Literature
Display
← All papers

T-Rex: Tactile-Reactive Dexterous Manipulation

Dantong Niu · Zhuoyang Liu · Zekai Wang · Boning Shao · Zhao-Heng Yin · Anirudh Pai · Yuvan Sharma · Stefano Saravalle · Ruijie Zheng · Jing Wang · Ryan Punamiya · Mengda Xu · Yuqi Xie · Yunfan Jiang · Letian Fu · Konstantinos Kallidromitis · Matteo Gioia · Junyi Zhang · Jiaxin Ge · Haiwen Feng · Fabio Galasso · Wei Zhan · David M. Chan · Yutong Bai · Roei Herzig · Jiahui Lei · Li Fei-Fei · Ken Goldberg · Jitendra Malik · Pieter Abbeel · Yuke Zhu · Danfei Xu · Jim(Linxi) Fan · Trevor Darrell

Preprint posted · arXiv (preprint) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete 30-page v2 dated 18 June 2026. It retains all experimental appendices and the original instructions and scoring criteria for every benchmark task. Figures 1 and 3 label the slow and fast experts at approximately 5 and 20 Hz; the separate 300 Hz low-level controller is not the neural policy frequency.

Complete original figure with every panel, label and legend.
Figure 1: Overview. T-Rex is a tactile-reactive dexterous manipulation framework that combines large-scale human egocentric pre-training with tactile-grounded robot mid-training. Built on a Mixture-of-Transformer-Experts (MoT) architecture, the T-Rex Model integrates low-frequency visuomotor planning with high-frequency tactile refinement and employs a spatial-temporal tactile encoder. The T-Rex Dataset introduces 100 hours of real robot data containing diverse motor primitives and object interactions with synchronized tactile signals as well as 12 manipulation tasks.

At a glance

T-Rex adds touch to a large manipulation policy through 100 hours of synchronized robot mid-training, a temporal force encoder and a fast tactile expert that refines a slower visual action plan. Human-video pretraining supplies broad visuomotor priors before contact-rich robot adaptation.

Across twelve tasks, its reported average score is 65 versus 35 for the strongest baseline. These are percentages of partial-credit task scores from sixteen trials per task, not binary full-task completion rates. The difference is about 30 percentage points; the paper’s broad “30%” wording does not consistently distinguish this from relative improvement.

1 Introduction

Touch can reveal contact, deformation and slip before vision supports a useful reaction. A large visual-language model is expensive to rerun at every tactile update, and diverse synchronized dexterous data is scarce.

The proposed recipe combines EgoScale-style human-video pretraining, motor-primitive robot mid-training and task-specific post-training. A cached visual context supports faster tactile refinement without rebuilding the entire visual plan each tick. The evaluation covers fragile objects, deformable tools, insertion, handovers and multi-finger coordination.

2 Related Work

Prior tactile policies use force projections, learned contact representations, multimodal alignment or fast corrective controllers. Reactive Diffusion Policy separates slow and fast control, while ViTacFormer fuses touch and vision with future tactile prediction. T-Rex combines temporal touch encoding with large-scale dexterous pretraining.

Unified robot models increasingly combine continuous action experts, latent future prediction and reasoning. Human egocentric video adds broad hand-motion priors; synchronized robot mid-training supplies the contact dynamics absent from that video. The comparison tests a complete data-and-architecture recipe, rather than isolating touch through every baseline.

3 The T-Rex Dataset

The mid-training set contains 7,755 episodes and 100 hours of bimanual demonstrations covering 207 objects, 22 motor primitives and 502 feasible object–primitive pairs. Each episode records three RGB views, robot states/actions, ten-fingertip tactile measurements and an instruction.

Organizing around reusable interactions aims for broader coverage than collecting only downstream task demonstrations. Appendix G specifies feasibility filtering, scene randomization, cleaning, annotation and the planned dataset release.

Complete original figure with every panel, label and legend.
Figure 2: Statistics of the T-Rex Dataset. Top-left: distribution of object categories. Top-right: distribution of motor primitives. Bottom: long-tail distribution of individual objects. T-Rex dataset contains 100 hours of tactile-synchronized bimanual manipulation mid-training data spanning 200+ daily objects and 22 motor primitives, designed for broad coverage of contact-rich interactions.

4 Tactile-Reactive Dexterous Manipulation

The policy conditions a flow-matching action generator on images, language, recent fingertip wrench histories and current deformation maps. The clean demonstrated chunk and Gaussian noise define a linear interpolation, with noise minus action as the target velocity.

Equations 1
ℒFM​(θ)=𝔼⁡[‖vθ​(xτ,τ∣ct)−(x1−x0)‖2]\mathcal{L}_{\mathrm{FM}}(\theta)=\mathbb{E}\left[\left\|v_{\theta}(x_{\tau},\tau\mid c_{t})-(x_{1}-x_{0})\right\|^{2}\right] (1)
𝐜t={𝐨t,ℓ,𝐟t−Hf:t,𝐝t}\mathbf{c}_{t}=\{\mathbf{o}_{t},\ell,\mathbf{f}_{t-H_{f}:t},\mathbf{d}_{t}\}

4.1 Model Architecture

Three transformer experts share multimodal context: a latent expert predicts future visual representations, an action expert performs the early part of denoising, and a tactile expert completes it from the intermediate boundary state.

For each finger, a VQ-VAE compresses sixteen wrench frames into a temporal token. A direct projection preserves the current wrench, and a convolutional encoder captures the current deformation map. Their concatenation supplies dynamics, instantaneous contact and spatial geometry.

Complete original figure with every panel, label and legend.
Figure 3: T-Rex Model Architecture. T-Rex uses a Mixture-of-Transformer-Experts (MoT) backbone with three experts: a latent expert for future visual prediction, an action expert for low-frequency action denoising, and a tactile expert for high-frequency tactile refinement. During inference, the tactile expert reuses cached visual-language context to asynchronously refine intermediate actions using spatial-temporal tactile features, enabling fast tactile-reactive closed-loop control.
Equation 2
𝐳tτ=[Embvq(Ef(𝐟t−15:t));Projf(𝐟t);Projd(Ed(𝐝t))].\mathbf{z}^{\tau}_{t}=\bigl[\mathrm{Emb}_{\mathrm{vq}}\!\bigl(E_{f}(\mathbf{f}_{t-15:t})\bigr);\mathrm{Proj}_{f}(\mathbf{f}_{t});\mathrm{Proj}_{d}\!\bigl(E_{d}(\mathbf{d}_{t})\bigr)\bigr]. (2)

4.2 Asynchronous Tactile-Reactive Cascaded Flow Matching

Ten Euler steps move from noise time one to action time zero. The default slow expert performs six steps to .4 once per sixteen-action chunk. At offsets 0, 4, 8 and 12, the tactile expert reuses the boundary state and visual/action key-value cache, then performs the remaining four steps with fresh touch.

Both experts regress the same velocity target. Despite the earlier disjoint-interval description, the actual training rule samples the action expert across the full (0,1] interval to retain standalone generation; tactile training uses the lower interval scaled by .4. The cache comes from a detached slow pass.

The joint objective weights tactile loss by one and future-visual prediction by .5. Appendix B adds delayed tactile-context augmentation and specifies cache refresh and runtime synchronization.

Equations 3–7
𝐱τ=(1−τ)​𝐀demo+τ​ϵ,v⋆=ϵ−𝐀demo.\mathbf{x}_{\tau}=(1-\tau)\,\mathbf{A}^{\mathrm{demo}}+\tau\,\boldsymbol{\epsilon},\qquad v^{\star}=\boldsymbol{\epsilon}-\mathbf{A}^{\mathrm{demo}}. (3)
𝐱^τsplit=Euler(fθact;𝐱1,τ: 1→0.4,Kslow=6).\hat{\mathbf{x}}_{\tau_{\mathrm{split}}}=\mathrm{Euler}\bigl(f_{\theta}^{\mathrm{act}};\,\mathbf{x}_{1},\,\tau{:}\,1{\to}0.4,\,K_{\mathrm{slow}}=6\bigr). (4)
𝐀t:t+Ta=Euler(fθtac;𝐱^τsplit,τ: 0.4→0,Kfast=4).\mathbf{A}_{t:t+T_{a}}=\mathrm{Euler}\bigl(f_{\theta}^{\mathrm{tac}};\,\hat{\mathbf{x}}_{\tau_{\mathrm{split}}},\,\tau{:}\,0.4{\to}0,\,K_{\mathrm{fast}}=4\bigr). (5)
ℒact=∥fθact(𝐱τact,τact)−v⋆∥2,ℒtac=∥fθtac(𝐱τtac,τtac;KVτsplit)−v⋆∥2,\mathcal{L}_{\mathrm{act}}=\bigl\lVert f_{\theta}^{\mathrm{act}}(\mathbf{x}_{\tau_{\mathrm{act}}},\tau_{\mathrm{act}})-v^{\star}\bigr\rVert^{2},\quad\mathcal{L}_{\mathrm{tac}}=\bigl\lVert f_{\theta}^{\mathrm{tac}}(\mathbf{x}_{\tau_{\mathrm{tac}}},\tau_{\mathrm{tac}};\,\mathrm{KV}_{\tau_{\mathrm{split}}})-v^{\star}\bigr\rVert^{2}, (6)
ℒ=ℒact+λtac​ℒtac+λfuture​ℒfuture\mathcal{L}=\mathcal{L}_{\mathrm{act}}+\lambda_{\mathrm{tac}}\,\mathcal{L}_{\mathrm{tac}}+\lambda_{\mathrm{future}}\,\mathcal{L}_{\mathrm{future}} (7)
τtac=τsplit⋅τ~\tau_{\mathrm{tac}}=\tau_{\mathrm{split}}\cdot\tilde{\tau}

4.3 Training Recipe

Stage one trains latent/action experts on 22,889 hours of human egocentric video with retargeted arm and hand motions, without touch. Stage two adapts to robot cameras, executable actions and tactile feedback using the 100-hour T-Rex dataset. Stage three uses approximately 100 demonstrations per target task.

Zero-shot evaluation is limited to four easier primitive tasks after mid-training; the twelve-task main benchmark uses task-specific post-training. These two settings should not be conflated.

5 Experiments

Experiments evaluate the complete model, tactile and architecture ablations, data efficiency and the contribution of each training stage on the same physical robot platform.

5.1 Experiment Setup

A fixed-base Dexmate Vega-1 uses two seven-joint arms and two 22-joint Sharpa Wave hands. Head and wrist RGB views accompany six-axis fingertip wrenches and deformation maps. T-Rex predicts relative end-effector arm actions and absolute hand joints.

Each of twelve tasks receives sixteen randomized trials, then scores are averaged equally across tasks. Appendix F lists additive partial-credit rubrics; a score of 65 does not mean 65% of entire tasks finished. Baselines differ in pretraining, tactile adaptation and sometimes arm-control representation, as documented in Appendix E.

5.2 Main Results

T-Rex averages 65, versus 35 for EgoScale, 17 for π0.5, 15 for Tactile-VLA, 6 for RDP and tactile-augmented π0.5, and 3 for ViTacFormer. Its task scores range from 96 for page turning to 35 for screwing a bulb. The original per-task values remain in Table 1.

Naively concatenating touch into π0.5 lowers the average in this implementation. Small models trained only on task demonstrations also struggle. These comparisons support the combined recipe, but do not isolate touch from pretraining and mid-training advantages.

Table 1: Comparison of T-Rex and Baseline Methods Across 12 Tactile-reactive Manipulation Tasks. Success rates (%) are computed over 16 evaluation rollouts per task then across tasks.
Method
Flip
Page
Transfer
Egg
Wipe
Plate
Apply
Paste
Split
Cup
Sort
Mahjong
Open
Lock
Refill
Tablet
Acid-Base
Neut.
Extract
Card
Deal
Poker
Screw
Bulb
Avg.
ViTacFormer [24] 9 0 4 1 4 7 0 0 0 2 2 1 3
RDP [61] 12 8 18 2 6 9 2 0 0 1 2 7 6
Tactile-VLA [28] 38 14 24 0 21 27 8 0 9 4 11 18 15
EgoScale [70] 68 44 34 38 33 36 19 12 43 41 28 18 35
π0.5\pi_{0.5} [32] 36 17 28 13 18 32 5 1 24 8 9 11 17
π0.5\pi_{0.5} + tactile 8 9 27 2 4 14 2 0 7 3 0 0 6
Ours 96 75 69 66 78 65 47 41 76 70 57 35 65
Table 2: Ablation Studies on Tactile Modality and Architectural Design. Results are reported on six representative tactile-reactive manipulation tasks, with the final column showing the average.
Configuration
Flip
Page
Apply
Toothpaste
Split
Cup
Open
Lock
Extract
Card
Screw
Lightbulb
Average
Full Model (Ours) 96 66 78 47 70 35 65
Tactile Modality Ablation
w/o Tactile 76 39 58 23 34 20 42 (-23%)
MLP Force + Deform 89 58 72 44 58 29 58 (-7%)
Deform 82 57 71 36 55 25 54 (-11%)
MLP Force + VQVAE Force 92 63 65 38 67 28 59 (-6%)
Architecture Design
w/o Async 92 61 73 45 59 30 60 (-5%)

5.3 Ablation Studies

On six tasks, removing touch lowers the rounded average from 65 to 42. Static force plus deformation scores 58, deformation alone 54, and temporal/current force without deformation 59. Synchronous refinement scores 60; the full configuration uses both temporal and spatial touch with asynchronous updates.

Figure 4 sweeps slow-step count, with six steps best among the plotted settings. The prose describes the extremes using τ_split in the reverse direction from τ_split=1−K_slow/10; the plotted axis is K_slow, and the original curve is retained.

At equal 100-hour mid-training budgets, the motor-primitive dataset improves six post-trained tasks and four zero-shot primitive tasks over an eleven-task dataset. Figure 5 shows gains with 10–200 post-training demonstrations. These are selected-task studies, not evaluations on every possible skill.

Table 3 gives averages 18 with neither pretraining nor mid-training, 34 with mid-training only, 45 with pretraining only, and 65 with both. The colored inclusion/exclusion markers are retained with the exact source table.

Complete original figure with every panel, label and legend.
Figure 4: Ablation Studies on Cascaded Denoising Split Steps KslowK_{\mathrm{slow}}. We show the success rate curve of different split steps.
Complete original figure with every panel, label and legend.
Figure 5: Data Efficiency of T-Rex. We show the success rate curve of different numbers of demonstrations. Blue: with our tactile-grounded T-Rex mid-training data; Green: without mid-training.
Complete original figure with every panel, label and legend.
Figure 6: Ablation Studies on Mid-training Datasets. We select 6 representative tasks for post training evaluation and 4 easier tasks for zero-shot evaluation, including motor primitives of pick, slide, press and wipe in T-Rex dataset.
Complete original figure with every panel, label and legend.
Table 3: Effectiveness of the Training Recipe of T-Rex. We selected six representative tasks and report the success rates and compare the success rate (%) on different training recipes.
Selectable original table
Pre-training Mid-training
Flip
Page
Apply
Toothpaste
Split
Cup
Open
Lock
Extract
Card
Screw
Lightbulb
Average
✗ ✗ 46 16 20 6 14 5 18
✓ 75 34 45 10 32 9 34
✓ ✗ 88 40 52 22 46 20 45
✓ 96 66 78 47 70 35 65

6 Conclusion

The work combines a reusable tactile dataset, temporal touch tokens and cached-context action refinement. Its gains are demonstrated on one bimanual hardware setup; the benchmark’s score definition and task-specific post-training remain part of the claim.

7 Limitation and Future Work

Long tasks with tight contact tolerances remain hard to demonstrate and execute; reinforcement learning or online refinement is suggested. Hardware limits include sensor distortion, inter-device calibration drift and missing dense palm sensing. Heterogeneous-sensor transfer and whole-hand touch remain future work.

Acknowledgments

The authors acknowledge Sharpa maintenance, Panasonic assistance and support, Berkeley’s Humanoid Intelligence Center, Sapienza funding and research discussions.

Appendix A Model and Training Details

Table 4 retains all model/training settings. The Qwen3VL-2B-based latent and action experts each list 1.41B parameters; the tactile expert lists .62B, with a 1,536-wide feed-forward layer. Action dimension is 62 and chunk length sixteen.

Supervised training uses AdamW at peak learning rate 10⁻⁴, cosine decay, zero weight decay/warmup, gradient clipping one, bf16 and 24 H100 GPUs with batch sixteen each. The table specifies six slow and four fast inference steps.

Table 4: Model and Training Configurations for T-Rex.
   Latent Expert   
   VLM Backbone       Qwen3VL-2B   
   Hidden Feature Dimension       2048   
   Transformer Layers       28   
   Max Sequence Length       2048   
   Parameter size       1.41B   
   Attention Implementation       Flash Attention 2   
   Action Expert (Flow Matching)   
   VLM Backbone       Qwen3VL-2B   
   Action Dimension       62   
   Action Chunk       16   
   Training Timestep Sampling       Beta⁡(1.5,1.0)\mathrm{Beta}(1.5,1.0)   
   Num Inference Timesteps       6   
   Parameter size       1.41B   
   Tactile Expert (Flow Matching)   
   Action Dimension       62   
   Action Chunk       16   
   FFN Intermediate Size       1536   
   Training Timestep Sampling       Beta⁡(1.5,1.0)\mathrm{Beta}(1.5,1.0)   
   Num Inference Timesteps       4   
   Parameter size       0.62B   
   Training Configurations (SFT)   
   Optimizer       AdamW   
   Peak Learning Rate       1×10−41\times 10^{-4}   
   Min Learning Rate       0   
   LR Scheduler       Cosine with min\min LR   
   Weight Decay       0   
   Warmup Ratio       0   
   Gradient Clipping       1.0   
   GPU Type       NVIDIA H100   
   Number of GPUs       24   
   Deepspeed Zero Stage       1   
   Per Device Batch Size       16   
   Gradient Accumulation Steps       1   
   Mixed Precision Training       bf16   

Appendix B Additional Details for Asynchronous Cascaded Denoising

The slow expert sees visual-language and future-prediction context. A detached pass supplies both latent keys/values and action positions re-encoded at the .4 boundary. The tactile expert combines this cache with fresh touch, without rerunning raw-image processing.

Delay augmentation shifts tactile frame indices by a uniform choice from 0, 4, 8 or 12, matching deployment offsets. The visual tower and slow experts are amortized across fast ticks.

Runtime prose describes a request socket and execution lock that serialize expert execution. The printed pseudocode explicitly locks cache publication and cloning, leaving integration loops outside those marked critical sections. It should therefore be read with the implementation description, not as a standalone concurrency or real-time proof. The original two-column algorithm and exact selectable instructions remain below.

Equations 8–10
ℒact=𝔼∥fθact(𝐱τact,τact;𝐜vl)−v⋆∥22\mathcal{L}_{\mathrm{act}}=\mathbb{E}\bigl\lVert f_{\theta}^{\mathrm{act}}(\mathbf{x}_{\tau_{\mathrm{act}}},\tau_{\mathrm{act}};\mathbf{c}^{\mathrm{vl}})-v^{\star}\bigr\rVert_{2}^{2} (8)
ℒtac=𝔼∥fθtac(𝐱τtac,τtac;𝐜tac,KVτsplit)−v⋆∥22\mathcal{L}_{\mathrm{tac}}=\mathbb{E}\bigl\lVert f_{\theta}^{\mathrm{tac}}(\mathbf{x}_{\tau_{\mathrm{tac}}},\tau_{\mathrm{tac}};\mathbf{c}^{\mathrm{tac}},\mathrm{KV}_{\tau_{\mathrm{split}}})-v^{\star}\bigr\rVert_{2}^{2} (9)
ℒ=ℒact+λtac​ℒtac+λfuture​ℒfuture,where ​λtac=1.0,λfuture=0.5.\mathcal{L}=\mathcal{L}_{\mathrm{act}}+\lambda_{\mathrm{tac}}\mathcal{L}_{\mathrm{tac}}+\lambda_{\mathrm{future}}\mathcal{L}_{\mathrm{future}},\qquad\text{where }\lambda_{\mathrm{tac}}=1.0,\;\;\lambda_{\mathrm{future}}=0.5. (10)
Complete original pseudocode with exact line numbers, comments and indentation.
Algorithm 1 Asynchronous Tactile-Reactive Cascaded Flow Matching Inference
Selectable original pseudocode
Require: Pre-trained experts fθactf_{\theta}^{\mathrm{act}} and fθtacf_{\theta}^{\mathrm{tac}}; Total flow steps NN, slow segment steps KslowK_{\mathrm{slow}} (Kfast=N−KslowK_{\mathrm{fast}}=N-K_{\mathrm{slow}}); Step size Δτ=−1/N\Delta\tau=-1/N; Boundary threshold τsplit=1−Kslow/N\tau_{\mathrm{split}}=1-K_{\mathrm{slow}}/N.
Ensure: Executed actions 𝐀t:t+Ta\mathbf{A}_{t:t+T_{a}} at corresponding execution offsets.
1: Shared Memory: Intermediate state 𝐱^τsplit\hat{\mathbf{x}}_{\tau_{\mathrm{split}}}, KV Cache KVτsplit\mathrm{KV}_{\tau_{\mathrm{split}}}, Execution Lock lock
2: procedure Slow-Stream Loop (LowFreq)
3: for each action chunk window TaT_{a} do
4: Get vision-language context 𝐜vl\mathbf{c}^{\mathrm{vl}}
5: Sample initial noise 𝐱1∼𝒩⁡(𝟎,𝐈)\mathbf{x}_{1}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
⊳\triangleright Upper Segment Integration
6: for k=1k=1 to KslowK_{\mathrm{slow}} do
7: τ←1−(k−1)/N\tau\leftarrow 1-(k-1)/N
8: v←fθact​(𝐱τ,τ,𝐜vl)v\leftarrow f_{\theta}^{\mathrm{act}}(\mathbf{x}_{\tau},\tau;\,\mathbf{c}^{\mathrm{vl}})
9: 𝐱τ+Δ​τ←𝐱τ+Δ​τ⋅v\mathbf{x}_{\tau+\Delta\tau}\leftarrow\mathbf{x}_{\tau}+\Delta\tau\cdot v
10: end for
11: acquire lock
12: 𝐱^τsplit←𝐱τsplit\hat{\mathbf{x}}_{\tau_{\mathrm{split}}}\leftarrow\mathbf{x}_{\tau_{\mathrm{split}}}
13: Refresh and re-encode position cache:
KVτsplit←[KVlat|KVτsplitact]\mathrm{KV}_{\tau_{\mathrm{split}}}\leftarrow\bigl[\,\mathrm{KV}^{\mathrm{lat}}\;\big|\;\mathrm{KV}^{\mathrm{act}}_{\tau_{\mathrm{split}}}\,\bigr]
14: release lock
15: end for
16: end procedure
17: procedure Fast-Stream Loop (HighFreq)
18: for offsets δ∈{0,4,8,12}\delta\in\{0,4,8,12\} inside window do
19: Sample real-time tactile stream 𝐜tac\mathbf{c}^{\mathrm{tac}}
20: acquire lock
21: Clone context: 𝐤𝐯←clone​(KVτsplit)\mathbf{kv}\leftarrow\text{clone}(\mathrm{KV}_{\tau_{\mathrm{split}}})
22: 𝐱←𝐱^τsplit\mathbf{x}\leftarrow\hat{\mathbf{x}}_{\tau_{\mathrm{split}}}
23: release lock
⊳\triangleright Terminal Segment Integration
24: for k=1k=1 to KfastK_{\mathrm{fast}} do
25: τ←τsplit−(k−1)/N\tau\leftarrow\tau_{\mathrm{split}}-(k-1)/N
26: v←fθtac​(𝐱,τ,𝐜tac,𝐤𝐯)v\leftarrow f_{\theta}^{\mathrm{tac}}(\mathbf{x},\tau;\,\mathbf{c}^{\mathrm{tac}},\mathbf{kv})
27: 𝐱←𝐱+Δ​τ⋅v\mathbf{x}\leftarrow\mathbf{x}+\Delta\tau\cdot v
28: end for
29: 𝐀^t+δ:t+δ+Ta←𝐱\hat{\mathbf{A}}_{t+\delta:t+\delta+T_{a}}\leftarrow\mathbf{x}
30: Execute updated action chunk
31: end for
32: end procedure
Equation
KVτsplit=[KVlat|KVτsplitact]\mathrm{KV}_{\tau_{\mathrm{split}}}=\bigl[\mathrm{KV}^{\mathrm{lat}}\big|\mathrm{KV}^{\mathrm{act}}_{\tau_{\mathrm{split}}}\bigr]

Appendix C Implementation Details for Spacial-Temporal Tactile Encoder

A shared per-finger temporal convolutional encoder downsamples sixteen six-axis wrench frames twice, mean-pools to 256 dimensions and quantizes into a 64-entry codebook. EMA updates and reseeding discourage collapse; magnitude-weighted reconstruction MSE emphasizes contacts over dominant no-contact states. Finger embeddings identify digits.

The spatial encoder adapts ResNet-18 to one-channel deformation maps, keeps three residual stages, projects each to 128 channels and maps flattened features into policy tokens. It is pretrained as a convolutional autoencoder and frozen for policy learning. Temporal tokens, current-force projections and spatial features feed the tactile expert.

Appendix D Real-World Setup and Teleoperation Stack

Wheels, torso and head are fixed. Relative wrist targets pass through Pink differential inverse kinematics, low-pass filtering and the manufacturer’s cascaded PID controller. An asynchronous policy thread updates targets tracked by the separate 300 Hz control thread.

A head ZED X Mini left image and two wrist ZED X One S wide views supply 640×360 RGB. Every fingertip supplies deformation depth and a six-axis net wrench. Manus glove fingertip targets and VIVE wrist poses drive teleoperation; hand retargeting uses Pinocchio and CasADi. High-level collection and synchronized recording run at 30 Hz.

Complete original figure with every panel, label and legend.
Figure 7: Robot system setup on the Dexmate Vega-1 bimanual robot and the Sharpa Wave dexterous hands. Two ZED X One S (wide view) cameras are mounted at the wrists, and one ZED X Mini camera is mounted on the head. For teleoperation we use Manus gloves to retrieve hand target gesture and VIVE trackers for wrist target pose.

Appendix E Implementation Details of Baselines

ViTacFormer trains separate ACT policies for 100 epochs on 100 demonstrations per task, with chunk length 100 and all 22 hand joints predicted. RDP separately trains its tactile-conditioned action tokenizer for 100 epochs and latent diffusion policy for 200, using ten-finger wrenches.

Tactile-VLA is adapted from GelSight images to six-axis fingertip wrenches through a simple MLP, then trained per task for 100 epochs. This representation change matters when interpreting its comparison with the original method.

EgoScale uses the released GR00T-N1.7-3B checkpoint and 200-epoch per-task finetuning. π0.5 uses OpenPI and its released checkpoint; the tactile variant concatenates single-step wrenches to state. The appendix specifies joint-space arm control for these π0.5 variants, unlike T-Rex’s relative end-effector control, despite the general statement that action spaces are unified.

The reported results compare these reproduced implementations and training recipes, with different model sizes, initial checkpoints and tactile representations.

Appendix F Evaluation Tasks

The tasks span force control, deformation-sensitive contact and longer bimanual sequences. Every original task instruction, description and additive scoring rubric is retained beside its figure. Partial credit is awarded for individual substeps; all twelve listed rubrics are additive even though the introduction also describes hierarchical progress scoring.

Tasks 1–4 test single-page turning, intact egg transfer, bimanual wiping and toothpaste dispensing. Tasks 5–8 test separating one nested cup, tactile mahjong identification and sorting, key insertion/unlocking, and opening/filling/closing a compartment. Tasks 9–12 test dropper manipulation, separating one card from a sleeve, dealing one card, and handover/alignment/rotation of a lightbulb.

The acid-base task instruction says the solution should turn green or yellow, whereas its description and .15-credit rubric say colorless. Both source statements remain unchanged. The compartment task also differs on which right-hand digit closes the lid; the exact description and rubric are preserved.

Complete original figure with every panel, label and legend.
Figure 8: Key stages of Task I: Flip Page.
Original task instruction and scoring rubric

Task I: Flip Page. Text Instruction: “Turn a page of the book from right to left using your right index finger.” The robot must lift a single sheet from the right side of an open book, sweep it across the spine, and smooth it down flat on the left side.

Grading rubric (additive):

  • •

    +0.3+0.3: (a) Successfully touched the book page with a single finger.

  • •

    +0.3+0.3: (b) Using the index finger turn the page up.

  • •

    +0.4+0.4: (c) Successfully flips exactly one page from right to left and smooths it flat.

Complete original figure with every panel, label and legend.
Figure 9: Key stages of Task II: Transfer Egg.
Original task instruction and scoring rubric

Task II: Transfer Egg. Text Instruction: “Using the right thumb and index finger, pick up the egg from the green egg tray and place it into the yellow egg tray.” The robot must grasp a fragile egg without cracking the shell from the green container, lift it off the surface, transport it above the yellow container, and gently release it inside.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Approaches the egg and makes contact without knocking it off the table.

  • •

    +0.3+0.3: (b) Lifts the egg off the table without cracking it.

  • •

    +0.2+0.2: (c) Transports the egg above the yellow container.

  • •

    +0.3+0.3: (d) Releases the egg inside the container intact.

Complete original figure with every panel, label and legend.
Figure 10: Key stages of Task III: Wipe Plate.
Original task instruction and scoring rubric

Task III: Wipe Plate. Text Instruction: “There is a white plate and a white cloth on the table; the white plate has colored stains on it. Use your right hand to pick up the cloth, hold the plate steady with your left hand, and then use the cloth to wipe away the stains.” The robot must grasp the cloth with the right hand, press down on the plate with the left hand to hold it steady, bring the cloth into contact with the plate surface, wipe the plate until the colored stains are fully removed, and place the cloth back on the table while releasing the plate.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Right hand grasps the rag.

  • •

    +0.1+0.1: (b) Left hand presses down on the plate to hold it steady.

  • •

    +0.2+0.2: (c) Brings the rag into contact with the plate surface.

  • •

    +0.4+0.4: (d) Wipes the plate until the design is fully removed (no visible ink remaining).

  • •

    +0.1+0.1: (e) Places the rag back on the table and releases the plate.

Complete original figure with every panel, label and legend.
Figure 11: Key stages of Task IV: Apply Toothpaste.
Original task instruction and scoring rubric

Task IV: Apply Toothpaste. Text Instruction: “On the left side of the countertop sits a cup holding a toothbrush, while an open tube of toothpaste rests on the right. Pick up the toothbrush with your left hand and the toothpaste with your right, squeeze some toothpaste onto the brush, and then set the tube back down.” The robot must grasp a toothbrush in one hand and a toothpaste tube in the other, align the tube nozzle above the bristles and squeeze out a bead of toothpaste, and return the toothbrush upright into its holder and the toothpaste back onto the table.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Grasps the toothbrush.

  • •

    +0.1+0.1: (b) Grasps the toothpaste tube.

  • •

    +0.4+0.4: (c) dispenses a bead of toothpaste onto the bristles.

  • •

    +0.2+0.2: (d) Returns the toothbrush upright into its holder.

  • •

    +0.1+0.1: (e) Places the toothpaste tube back on the table.

Complete original figure with every panel, label and legend.
Figure 12: Key stages of Task V: Split Cup.
Original task instruction and scoring rubric

Task V: Split Cup. Text Instruction: “A stack of red plastic cups sits on the desktop; use the right hand to slide out the topmost one, exerting effort to separate it from the rest of the stack.” Given a stack of nested cups on the table, the robot must grasp the stack with the left hand to stabilize it, and use the right hand to twist and rub exactly one cup off the top of the stack.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Left hand grasps and stabilizes the cup stack.

  • •

    +0.3+0.3: (b) Right hand grasps the topmost cup of the stack.

  • •

    +0.3+0.3: (c) Right hand twists and separates exactly one cup from the stack.

  • •

    +0.2+0.2: (d) Right hand holds the single separated cup intact.

Complete original figure with every panel, label and legend.
Figure 13: Key stages of Task VI: Sort Mahjong.
Original task instruction and scoring rubric

Task VI: Sort Mahjong. Text Instruction: “Three boxes are placed on the table, representing the Mahjong tiles ’Red Zhong’, ’Green Fa’, and ’White Blank’, respectively. In the center of the table lies a single Mahjong tile, placed face-down. Now, using your right hand, grasp the tile and discern its pattern; then, use your left hand to open the box corresponding to that pattern and place the tile inside.” The robot must pick up a face-down mahjong tile with the right hand and feel its surface via tactile sensing to identify its category, then use the left hand to slide open the lid of the matching compartment in the organizer box, place the tile into the compartment with the right hand, and close the lid with the right thumb.

Grading rubric (additive):

  • •

    +0.1+0.1: (a) Right hand picks up the face-back mahjong tile.

  • •

    +0.5+0.5: (b) Left hand slides open the lid of the correct compartment.

  • •

    +0.2+0.2: (c) Right hand places the tile into the correct compartment.

  • •

    +0.2+0.2: (d) Right thumb closes the compartment lid.

Complete original figure with every panel, label and legend.
Figure 14: Key stages of Task VII: Open Lock.
Original task instruction and scoring rubric

Task VII: Open Lock. Text Instruction: “On the left side of the desk lies a red book, atop which rests a gray key; on the right side is a lock. Using your left thumb and index finger, slide the key free; then, pick up the lock with your right hand and use the key to unlock it.” The robot must first grasp the key with one hand and the padlock with the other, align and insert the key into the keyhole, and rotate it to release the shackle.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Grasps the key.

  • •

    +0.1+0.1: (b) Grasps the padlock.

  • •

    +0.4+0.4: (c) Aligns and inserts the key into the keyhole.

  • •

    +0.3+0.3: (d) Rotates the key and successfully opens the lock.

Complete original figure with every panel, label and legend.
Figure 15: Key stages of Task VIII: Refill Tablet.
Original task instruction and scoring rubric

Task VIII: Refill Tablet. Text Instruction: “Use your left hand to open one of the compartments in the small box, use your right hand to grasp the small ball on the table, place the ball into the box, and then close the box.” The robot must use the left index finger to press the button on a compartment lid to unlock it, flip the lid open with the left thumb, pick up the ball with the right hand, place the ball into the open compartment, and press the lid closed with the right index finger.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Left index finger presses the compartment button to unlock the lid.

  • •

    +0.2+0.2: (b) Left thumb flips the lid open.

  • •

    +0.2+0.2: (c) Right hand picks up the ball.

  • •

    +0.2+0.2: (d) Right hand places the ball into the open compartment.

  • •

    +0.2+0.2: (e) Right thumb presses the lid closed.

Complete original figure with every panel, label and legend.
Figure 16: Key stages of Task IX: Acid-Base Neutralization.
Original task instruction and scoring rubric

Task IX: Acid-Base Neutralization. Text Instruction: “On the right side of the desktop stands an Erlenmeyer flask containing 200 mL of citric acid solution; on the left is a beaker holding 20 mL of NaOH solution, which includes bromothymol blue indicator—appearing blue due to its alkaline nature. Using your right hand, pick up the dropper and draw up approximately 5 mL of the acid solution; then, using your left hand to hold the beaker, perform an acid-base titration until the liquid in the beaker turns green or yellow.” The robot uses a dropper held in the right hand to aspirate liquid from a conical flask, dispenses it into a beaker held in the left hand, and swirls the beaker until the blue indicator solution fully turns colorless, and then returns the dropper to the conical flask and places the beaker back on the table.

Grading rubric (additive):

  • •

    +0.1+0.1: (a) Right hand grasps the dropper from the conical flask.

  • •

    +0.15+0.15: (b) Right hand aspirates liquid from the conical flask.

  • •

    +0.1+0.1: (c) Left hand picks up the beaker.

  • •

    +0.15+0.15: (d) Right hand dispenses liquid from the dropper into the beaker.

  • •

    +0.15+0.15: (e) Left hand swirls the beaker to mix the contents.

  • •

    +0.15+0.15: (f) The solution in the beaker fully transitions from blue to colorless.

  • •

    +0.1+0.1: (g) Right hand returns the dropper to the conical flask.

  • •

    +0.1+0.1: (h) Left hand places the beaker back on the table.

Complete original figure with every panel, label and legend.
Figure 17: Key stages of Task X: Extract Card.
Original task instruction and scoring rubric

Task X: Extract Card. Text Instruction: “Next to the cube on the table lies a card case containing two cards. Pick up the case with the left hand, then use the right thumb to slide the cards out through the central opening; subsequently, use the right thumb and index finger to slide out the first card, taking care not to pull out the second one.” The robot must pick up the card sleeve (containing two cards) with the left hand, use the right thumb to rub the cards partially out, then use the right thumb and index finger to push the bottom card back in so that only the top card remains exposed, and extract that single card.

Grading rubric (additive):

  • •

    +0.2+0.2: (a) Left hand picks up and holds the card sleeve.

  • •

    +0.3+0.3: (b) Right thumb rubs the cards partially out of the sleeve.

  • •

    +0.3+0.3: (c) Right thumb and index finger push the bottom card in.

  • •

    +0.2+0.2: (d) Right hand extracts the single top card from the sleeve.

Complete original figure with every panel, label and legend.
Figure 18: Key stages of Task XI: Deal Poker.
Original task instruction and scoring rubric

Task XI: Deal Poker. Text Instruction: “Pick up a stack of playing cards with your right hand, then transfer it to your left; hold the stack aloft with your left hand, use your right thumb to slide out the top card, grasp it, and place it into the card holder.” The robot must grasp the full card stack from above with the right hand, transfer it to the left hand, use the right thumb to flick the top card partially out, adjust with the right thumb and index finger until exactly one card protrudes, grasp that single card, and insert it vertically into the dedicated card slot.

Grading rubric (additive):

  • •

    +0.1+0.1: (a) Right hand grasps the card stack from above.

  • •

    +0.2+0.2: (b) Right hand transfers the stack to the left hand (handover).

  • •

    +0.3+0.3: (c) Right thumb flicks and adjusts until exactly one card protrudes from the top.

  • •

    +0.3+0.3: (d) Right hand successfully grasps the single protruding card.

  • •

    +0.1+0.1: (e) Right hand inserts the card vertically into the card slot.

Complete original figure with every panel, label and legend.
Figure 19: Key stages of Task XII: Screw Lightbulb.
Original task instruction and scoring rubric

Task XII: Screw Lightbulb. Text Instruction: “There is a lightbulb and a base on the desktop. Use your left hand to pick up the lightbulb and transfer it to your right hand; then, use your left hand to hold down the base while using your right hand to screw the lightbulb into the base until it lights up.” The robot must pick up the lightbulb with the left hand, transfer it to the right hand (handover), stabilize the lamp socket with the left hand, and use the right hand to rotate the bulb through multiple turns into the socket until it is fully seated and illuminates.

Grading rubric (additive):

  • •

    +0.1+0.1: (a) Left hand picks up the lightbulb.

  • •

    +0.2+0.2: (b) Left hand transfers the bulb to the right hand (handover).

  • •

    +0.1+0.1: (c) Left hand stabilizes the lamp socket.

  • •

    +0.4+0.4: (d) Right hand aligns the bulb and rotates continuously to engage the threads.

  • •

    +0.2+0.2: (e) Bulb is fully seated and the lamp illuminates.

Appendix G T-Rex Dataset

Episodes synchronize three 640×360, 30 Hz RGB streams, arm positions/velocities, 44 hand joints, both wrist poses, ten deformation maps/wrenches and language. All streams are recorded at the 30 Hz high-level cadence, separate from the faster low-level controller.

Manual feasibility filtering reduces 207×22=4,554 potential pairs to 502. The set contains 7,755 episodes over ten weeks, approximately sixteen demonstrations per pair, with median length 29.8 seconds and interquartile range 21.0–41.1 seconds. Six tabletop backgrounds, randomized object poses and zero to five distractors from a pool exceeding 210 objects vary scenes.

Cleaning rejects unstable tactile measurements, corrupted streams, teleoperation failures and extreme joint velocities. A commercial VLM receives four to six sampled head-camera frames plus object/primitive labels to draft one imperative instruction; humans check hallucinations and imprecision.

The appendix says release of sensor streams, derived touch, annotations, loaders and preprocessing under MIT is planned. The main text’s “open-source” description is therefore not treated here as verification that release already occurred. Collection uses consenting operators in a controlled laboratory; incidental operator/reset frames are clipped.

Appendix H Failure Case Analysis

Six failure categories remain visible in the complete figure: bulb/base collision from poor alignment, a slipping key, incorrect egg placement, unintended neighboring-compartment contact, excessive toothpaste force and misaligned card extraction.

Together they expose limits in visual alignment, small-object grip, behavior-cloning distribution shift, individual-finger coordination and temporal force regulation. Fast tactile refinement reduces some errors but does not remove these failure modes.

Complete original figure with every panel, label and legend.
Figure 20: Failure Case Analysis. The order from left to right indicates the execution progress of the tasks, while the final column illustrates the specific failure scenarios.
Concept demonstration

Open image ↗