Overview
TACTIC compares five tactile encoders and five fusion strategies within a common ACT manipulation pipeline. Four real tasks impose different demands on pressure, slip, geometry, and termination detection. The study reports 2,180 rollouts including vision-only and visual-distribution-shift tests. Its central finding is task dependence: stronger average performance can hide severe failures on individual tasks, and more elaborate fusion is not reliably better.
At a glance
The study evaluates how visual-tactile representation and fusion choices affect real robot manipulation under shared task and policy conditions. No tested encoder–fusion pair wins universally. Tactile pretraining and temporal signals can help, but their benefit depends on the task and how the policy uses them. Results are evidence for this sensor and ACT setup, rather than a universal ranking of tactile models.
I Related Works
Related work motivates a controlled comparison of tactile representations and multimodal fusion.
I-A Tactile Sensing for Robot Manipulation
Tactile observations help when vision misses contact state. Previous policies also vary temporal processing, hierarchy, and language-action architecture. TACTIC fixes the action-policy family to isolate a narrower set of representation and fusion choices.
I-B Encoding Tactile Information
Encoders differ in backbone, sensor diversity, self-supervised objectives, temporal inputs, and quantized latent structure. Common fusion choices include concatenation, feature modulation, cross-attention, and contrastive alignment. The experiment tests representative combinations rather than all current tactile models.
I-C Benchmarking Tactile Manipulation
Existing tactile benchmarks often vary the environment, policy, or sensing assumptions. Here a common real-world ACT setup supports direct task-level comparison. Hierarchical and vision-language-action systems are excluded because their extra scheduling and alignment choices would complicate interpretation.
II Method
The method specifies bilateral teleoperation, four tasks, tactile encoders, fusion, and the action-chunking policy.
II-A Data Collection and Inference Setup
Two Franka FR3 arms provide leader–follower teleoperation with external-torque feedback to the leader. The follower carries two DIGIT fingertips, a wrist camera, and a side-view camera. Policies receive both visual streams, both tactile streams, and joint/gripper state. The preserved control equation specifies position tracking and haptic feedback; gains vary by task.
Equations 1–3
| (1) | ||||
| (2) |
| (3) |
II-B Tasks
Each task has approximately 100 demonstrations collected at 30 Hz with four DIGIT cartridge pairs. Evaluation distinguishes initialization, grasp, task attempt, completion, and termination. Completing the contact objective and finishing the full episode are separate outcomes, so reported termination rates should not be interchanged with completion rates.
II-B 1 Board wiping
Board wiping varies chalk-line pose and board height. The robot must grasp an eraser, remove the line, and return the eraser. Tactile information principally helps maintain pressure, while camera information locates the line.
II-B 2 Screwing
Screwing tightens a pre-positioned nut and then retreats. The number of turns varies, and tactile shear or slip is intended to reveal when tightening is complete despite weak visual evidence.
II-B 3 USB-C insertion
USB-C insertion tests small grasp-induced alignment errors in an otherwise fixed scene. The figure’s criterion counts correct plug placement without requiring forceful full seating. This bounded setup does not cover arbitrary port locations or all practical insertion tolerances.
II-B 4 Vase wiping
Vase wiping requires pressure control and contact geometry on a curved, bisected printed vase. The marked line varies around its circumference, and the eraser must be returned after wiping.
II-C Encoder Backbone
The backbones are an ImageNet ResNet-18 baseline, SARL, UniT, Sparsh-DINO, and T³. Adaptation differs: SARL trains on collected tactile data, UniT and Sparsh adapt released weights, and T³ uses released weights directly. Sparsh consumes frames four steps apart; T³ uses raw images while others subtract a background estimated from 50 blank frames. Tactile adaptation samples 95% closed-gripper frames. During ACT training, pretrained tactile encoders freeze; the plain ResNet baseline trains end to end. These deliberate configurations compare practical recipes, not identical pretraining budgets.
II-D Multimodal Feature Fusion
Concatenation directly combines features. FiLM modulates tactile features using binary gripper state, while gated cross-attention lets tactile tokens query visual tokens. Two contrastive variants align visual and tactile features: CLIP-R samples random trajectories with batch size 256, while CLIP-T uses temporally spaced samples with batch size 32 and at least one second between samples. The variants therefore change both sampling structure and batch size. For pretrained backbones only a projection is learned; contrastive ResNet also updates its encoder. Learned representation changes are frozen during ACT training.
II-E Action Expert
ACT predicts 120 future joint-delta steps and temporally ensembles overlapping predictions. The ensembling coefficient is 0.01 except 0.1 for insertion. Policies specialize to a task dataset. They condition on current sensor observations, but this implementation lacks an explicit recurrent task-history state; this distinction matters when visually similar grasp and release phases are confused.
III Experiments
The main comparison runs 20 real rollouts per condition, with vision-only controls and smaller out-of-domain tests. Stage-level plots expose failure modes hidden by a single average. With only 20 trials per condition, rates change in five-point increments and close rankings warrant caution.
III-A In-domain Experiments
Sparsh contrastive variants have the best mean termination rate, approximately 48.8%, but can fail badly on board wiping. UniT with concatenation is stronger on average completion. The ranking depends both on task and on whether completion or final termination is scored.
III-A 1 Board Wiping
Contrastive variants often drop the eraser or execute a limited wiping region. Non-contrastive variants better adapt to line placement; T³ and UniT concatenation reach 65% termination in the plotted comparison. The authors hypothesize that aligning similar pressure signals with diverse visual locations can suppress useful spatial distinctions. This is a proposed explanation of observed behavior, not an isolated causal experiment.
III-A 2 Vase Wiping
Grasping and returning the eraser can look similar, leading some policies to release it immediately. Concatenation and gated attention avoid this for several encoders. Sparsh with temporal contrastive alignment reaches 80% termination, while UniT-CLIP-T and UniT gated attention reach 55%. Temporal information may disambiguate phases, but the result is representation dependent.
III-A 3 USB-C Insertion
ResNet18 with simple concatenation achieves the best 80% insertion termination rate, compared with 55% for vision only. More complex fusion harms that baseline. Other encoders respond differently: T³ with FiLM reaches 55% termination and 75% completion. Contrastive policies sometimes move jerkily or overshoot. Tactile sensing is therefore useful only when integrated appropriately in this setup.
III-A 4 Screwing
Temporal contrastive alignment is strongest for tightening, except with SARL. Sparsh-CLIP-T and T³-CLIP-T both reach 90% termination. A linear probe better distinguishes the final two turns with Sparsh and T³ than SARL and UniT, consistent with useful dynamic contact cues. The probe scores are still modest, and do not alone explain all policy behavior.
III-A 5 Effect of CLIP Fusion
A contrastive linear projection can redistribute existing representation variance but cannot create new input information. Participation-ratio analysis shows expansion for some encoders and compression for others. T³ demonstrates why this diagnostic is not a performance predictor: its temporal contrastive representation has lower participation ratio while its first principal component correlates more strongly with contact magnitude, from 0.16 to 0.75. Task-relevant signal ordering can matter more than global dimensionality.
| Encoder | Screw (CLIP-R / CLIP-T) | Vase (CLIP-R / CLIP-T) |
|---|---|---|
| SPARSH | 5.16 / 4.25 | 3.40 / 1.60 |
| T3 | 1.10 / 0.65 | 0.80 / 0.50 |
| SARL | 1.22 / 1.05 | 0.71 / 0.40 |
| UniT | 1.02 / 1.30 | 1.03 / 1.27 |
Equation 4
| (4) |
III-B Out-of-Domain Analysis
Best selected variants are tested ten times under distractor objects or dark moving disco lights. Non-contrastive board and insertion policies degrade severely; Sparsh-CLIP-T preserves stronger vase and screwing behavior, including 50% vase termination under distractors. Additional contrastive insertion trials achieve three completions out of ten under disco lights, without successful episode termination. Attention plots suggest reduced reliance on visual camera features, but attention mass is correlational evidence, not proof of the cause. These are selected-policy tests of visual shifts, not all-encoder robustness across unseen tactile materials or sensors.
IV Conclusion
Encoder and fusion selection should follow the task’s contact cues and termination demands. Temporal tactile features help on dynamic shear tasks; simple concatenation remains competitive. Contrastive alignment can produce specialists and some visual-shift robustness, but also severe task failures. Generalization beyond four tasks, DIGIT sensors, ACT, and the tested adaptation recipes remains open.
Acknowledgments
The authors acknowledge German research funding and computing support, and disclose using ChatGPT and Claude Code for portions of code that they reviewed and validated. They state that study design, evaluation, and interpretation were their own.