Literature
Display
← All papers

TACTIC: Understanding Tactile Encoders and Conditioning for Contact-rich Robot Manipulation Policies

Seongjin Bien · Débora Oliveira Makowski · Carlo Kneissl · Reihaneh Mirjalili · Pankhuri Vanjani · Rudolf Lioutikov · Gitta Kutyniok · Florian Walter · Wolfram Burgard

Preprint posted · arXiv (preprint) ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

Overview

TACTIC compares five tactile encoders and five fusion strategies within a common ACT manipulation pipeline. Four real tasks impose different demands on pressure, slip, geometry, and termination detection. The study reports 2,180 rollouts including vision-only and visual-distribution-shift tests. Its central finding is task dependence: stronger average performance can hide severe failures on individual tasks, and more elaborate fusion is not reliably better.

Refer to caption
Fig. 1: An overview of TACTIC. We study the effect of five tactile encoders paired with five multimodal fusion methods, tested on four different challenging contact-rich manipulation tasks.

At a glance

The study evaluates how visual-tactile representation and fusion choices affect real robot manipulation under shared task and policy conditions. No tested encoder–fusion pair wins universally. Tactile pretraining and temporal signals can help, but their benefit depends on the task and how the policy uses them. Results are evidence for this sensor and ACT setup, rather than a universal ranking of tactile models.

I Related Works

Related work motivates a controlled comparison of tactile representations and multimodal fusion.

I-A Tactile Sensing for Robot Manipulation

Tactile observations help when vision misses contact state. Previous policies also vary temporal processing, hierarchy, and language-action architecture. TACTIC fixes the action-policy family to isolate a narrower set of representation and fusion choices.

I-B Encoding Tactile Information

Encoders differ in backbone, sensor diversity, self-supervised objectives, temporal inputs, and quantized latent structure. Common fusion choices include concatenation, feature modulation, cross-attention, and contrastive alignment. The experiment tests representative combinations rather than all current tactile models.

I-C Benchmarking Tactile Manipulation

Existing tactile benchmarks often vary the environment, policy, or sensing assumptions. Here a common real-world ACT setup supports direct task-level comparison. Hierarchical and vision-language-action systems are excluded because their extra scheduling and alignment choices would complicate interpretation.

II Method

The method specifies bilateral teleoperation, four tasks, tactile encoders, fusion, and the action-chunking policy.

II-A Data Collection and Inference Setup

Two Franka FR3 arms provide leader–follower teleoperation with external-torque feedback to the leader. The follower carries two DIGIT fingertips, a wrist camera, and a side-view camera. Policies receive both visual streams, both tactile streams, and joint/gripper state. The preserved control equation specifies position tracking and haptic feedback; gains vary by task.

Equations 1–3
τf​o​l\displaystyle\tau_{fol} =Kp​(qlead−qfol)−Kd​q˙fol\displaystyle=K_{p}(q_{\text{lead}}-q_{\text{fol}})-K_{d}\dot{q}_{\text{fol}} (1)
τl​e​a​d\displaystyle\tau_{lead} =−Gh⋅τext, fol\displaystyle=-G_{h}\cdot\tau_{\text{ext, fol}} (2)
𝒜t∼πθ​(IDIGITR,IDIGITL,Iwrist,Iside,q)\mathcal{A}_{t}\sim\pi_{\theta}(I_{\text{DIGIT}_{R}},I_{\text{DIGIT}_{L}},I_{\text{wrist}},I_{\text{side}},q) (3)

II-B Tasks

Each task has approximately 100 demonstrations collected at 30 Hz with four DIGIT cartridge pairs. Evaluation distinguishes initialization, grasp, task attempt, completion, and termination. Completing the contact objective and finishing the full episode are separate outcomes, so reported termination rates should not be interchanged with completion rates.

Refer to caption
Fig. 2: The 4 tasks tested in TACTIC. Each task has 5 stages, allowing for a more fine-grained analysis of the failure points for each individual condition. The success criterion for the wiping tasks is to leave no residue on the board or vase surface. For USB-C insertion, success is counted when the USB is inserted in the correct position, but not necessarily forced in. For screwing, success is defined as fully tightening the nut.

II-B 1 Board wiping

Board wiping varies chalk-line pose and board height. The robot must grasp an eraser, remove the line, and return the eraser. Tactile information principally helps maintain pressure, while camera information locates the line.

II-B 2 Screwing

Screwing tightens a pre-positioned nut and then retreats. The number of turns varies, and tactile shear or slip is intended to reveal when tightening is complete despite weak visual evidence.

II-B 3 USB-C insertion

USB-C insertion tests small grasp-induced alignment errors in an otherwise fixed scene. The figure’s criterion counts correct plug placement without requiring forceful full seating. This bounded setup does not cover arbitrary port locations or all practical insertion tolerances.

II-B 4 Vase wiping

Vase wiping requires pressure control and contact geometry on a curved, bisected printed vase. The marked line varies around its circumference, and the eraser must be returned after wiping.

II-C Encoder Backbone

The backbones are an ImageNet ResNet-18 baseline, SARL, UniT, Sparsh-DINO, and T³. Adaptation differs: SARL trains on collected tactile data, UniT and Sparsh adapt released weights, and T³ uses released weights directly. Sparsh consumes frames four steps apart; T³ uses raw images while others subtract a background estimated from 50 blank frames. Tactile adaptation samples 95% closed-gripper frames. During ACT training, pretrained tactile encoders freeze; the plain ResNet baseline trains end to end. These deliberate configurations compare practical recipes, not identical pretraining budgets.

II-D Multimodal Feature Fusion

Concatenation directly combines features. FiLM modulates tactile features using binary gripper state, while gated cross-attention lets tactile tokens query visual tokens. Two contrastive variants align visual and tactile features: CLIP-R samples random trajectories with batch size 256, while CLIP-T uses temporally spaced samples with batch size 32 and at least one second between samples. The variants therefore change both sampling structure and batch size. For pretrained backbones only a projection is learned; contrastive ResNet also updates its encoder. Learned representation changes are frozen during ACT training.

II-E Action Expert

ACT predicts 120 future joint-delta steps and temporally ensembles overlapping predictions. The ensembling coefficient is 0.01 except 0.1 for insertion. Policies specialize to a task dataset. They condition on current sensor observations, but this implementation lacks an explicit recurrent task-history state; this distinction matters when visually similar grasp and release phases are confused.

III Experiments

The main comparison runs 20 real rollouts per condition, with vision-only controls and smaller out-of-domain tests. Stage-level plots expose failure modes hidden by a single average. With only 20 trials per condition, rates change in five-point increments and close rankings warrant caution.

Original figure
Fig. 3: Success rates of initial grasping, task attempt, task completion, and task termination for the five ablated backbones and fusion schemata for the ACT policy and each proposed dataset. The first line shows the closed-loop metrics for the vision-only (no tactile) case. The numbers on the plot represent the task-termination success rate. If no number is shown, the success rate of task termination is zero. The average column also includes the standard deviation of the termination success rate across the different tasks.

III-A In-domain Experiments

Sparsh contrastive variants have the best mean termination rate, approximately 48.8%, but can fail badly on board wiping. UniT with concatenation is stronger on average completion. The ranking depends both on task and on whether completion or final termination is scored.

III-A 1 Board Wiping

Contrastive variants often drop the eraser or execute a limited wiping region. Non-contrastive variants better adapt to line placement; T³ and UniT concatenation reach 65% termination in the plotted comparison. The authors hypothesize that aligning similar pressure signals with diverse visual locations can suppress useful spatial distinctions. This is a proposed explanation of observed behavior, not an isolated causal experiment.

III-A 2 Vase Wiping

Grasping and returning the eraser can look similar, leading some policies to release it immediately. Concatenation and gated attention avoid this for several encoders. Sparsh with temporal contrastive alignment reaches 80% termination, while UniT-CLIP-T and UniT gated attention reach 55%. Temporal information may disambiguate phases, but the result is representation dependent.

III-A 3 USB-C Insertion

ResNet18 with simple concatenation achieves the best 80% insertion termination rate, compared with 55% for vision only. More complex fusion harms that baseline. Other encoders respond differently: T³ with FiLM reaches 55% termination and 75% completion. Contrastive policies sometimes move jerkily or overshoot. Tactile sensing is therefore useful only when integrated appropriately in this setup.

III-A 4 Screwing

Temporal contrastive alignment is strongest for tightening, except with SARL. Sparsh-CLIP-T and T³-CLIP-T both reach 90% termination. A linear probe better distinguishes the final two turns with Sparsh and T³ than SARL and UniT, consistent with useful dynamic contact cues. The probe scores are still modest, and do not alone explain all policy behavior.

Original figure
Fig. 4: Cosine similarity between the Sparsh embeddings obtained in the regular way, i.e., by feeding tactiles at time tt and t−4t-4 into the encoder and those without temporal information, i.e. feeding both times the tt tactile into the model.
TABLE I: Linear probing [36] of tactile encoders to discriminate between the last and second-to-last turn on the screw task.
Method Sparsh T3 SARL UniT
R2R^{2} 0.138 0.174 0.055 0.036

III-A 5 Effect of CLIP Fusion

A contrastive linear projection can redistribute existing representation variance but cannot create new input information. Participation-ratio analysis shows expansion for some encoders and compression for others. T³ demonstrates why this diagnostic is not a performance predictor: its temporal contrastive representation has lower participation ratio while its first principal component correlates more strongly with contact magnitude, from 0.16 to 0.75. Task-relevant signal ordering can matter more than global dimensionality.

TABLE II: Effect of the contrastive adapter on PR⁡(ΣCLIP-{R,T})PR⁡(Σpre)\frac{\operatorname{PR}\left(\Sigma_{\text{CLIP-\lx@text@lbrace R,T\lx@text@rbrace}}\right)}{\operatorname{PR}(\Sigma_{\text{pre}})} of the pooled tactile representation.
Encoder Screw (CLIP-R / CLIP-T) Vase (CLIP-R / CLIP-T)
SPARSH 5.16 / 4.25 3.40 / 1.60
T3 1.10 / 0.65 0.80 / 0.50
SARL 1.22 / 1.05 0.71 / 0.40
UniT 1.02 / 1.30 1.03 / 1.27
Equation 4
PR⁡(Σ)=(∑iλi)2∑iλi2\operatorname{PR}(\Sigma)=\frac{\bigl(\sum_{i}\lambda_{i}\bigr)^{2}}{\sum_{i}\lambda_{i}^{2}} (4)

III-B Out-of-Domain Analysis

Best selected variants are tested ten times under distractor objects or dark moving disco lights. Non-contrastive board and insertion policies degrade severely; Sparsh-CLIP-T preserves stronger vase and screwing behavior, including 50% vase termination under distractors. Additional contrastive insertion trials achieve three completions out of ten under disco lights, without successful episode termination. Attention plots suggest reduced reliance on visual camera features, but attention mass is correlational evidence, not proof of the cause. These are selected-policy tests of visual shifts, not all-encoder robustness across unseen tactile materials or sensors.

Refer to caption
Fig. 5: The two visual out-of-distribution conditions best models were tested under. Left: Setup with distractor objects. Right: Setup with darkened lights and moving disco lights.
Original figure
Fig. 6: Stage-level success rates of best-in-task model variants under out-of-distribution conditions. Top: Distractor condition. Middle: Disco light condition. Bottom: Best-in-task CLIP variants under disco light condition.
Original figure
Fig. 7: Attention score plots of the approach phase of the USB-C under normal lighting and disco lighting conditions for the Sparsh-CLIP-R (top row) and ResNet-Concat (bottom row) configuration.

IV Conclusion

Encoder and fusion selection should follow the task’s contact cues and termination demands. Temporal tactile features help on dynamic shear tasks; simple concatenation remains competitive. Contrastive alignment can produce specialists and some visual-shift robustness, but also severe task failures. Generalization beyond four tasks, DIGIT sensors, ACT, and the tested adaptation recipes remains open.

Acknowledgments

The authors acknowledge German research funding and computing support, and disclose using ChatGPT and Claude Code for portions of code that they reviewed and validated. They state that study design, evaluation, and interpretation were their own.

Concept demonstration

Open image ↗