Overview
This edition covers the complete 30-page v2 dated 18 June 2026. It retains all experimental appendices and the original instructions and scoring criteria for every benchmark task. Figures 1 and 3 label the slow and fast experts at approximately 5 and 20 Hz; the separate 300 Hz low-level controller is not the neural policy frequency.
At a glance
T-Rex adds touch to a large manipulation policy through 100 hours of synchronized robot mid-training, a temporal force encoder and a fast tactile expert that refines a slower visual action plan. Human-video pretraining supplies broad visuomotor priors before contact-rich robot adaptation.
Across twelve tasks, its reported average score is 65 versus 35 for the strongest baseline. These are percentages of partial-credit task scores from sixteen trials per task, not binary full-task completion rates. The difference is about 30 percentage points; the paper’s broad “30%” wording does not consistently distinguish this from relative improvement.
1 Introduction
Touch can reveal contact, deformation and slip before vision supports a useful reaction. A large visual-language model is expensive to rerun at every tactile update, and diverse synchronized dexterous data is scarce.
The proposed recipe combines EgoScale-style human-video pretraining, motor-primitive robot mid-training and task-specific post-training. A cached visual context supports faster tactile refinement without rebuilding the entire visual plan each tick. The evaluation covers fragile objects, deformable tools, insertion, handovers and multi-finger coordination.
2 Related Work
Prior tactile policies use force projections, learned contact representations, multimodal alignment or fast corrective controllers. Reactive Diffusion Policy separates slow and fast control, while ViTacFormer fuses touch and vision with future tactile prediction. T-Rex combines temporal touch encoding with large-scale dexterous pretraining.
Unified robot models increasingly combine continuous action experts, latent future prediction and reasoning. Human egocentric video adds broad hand-motion priors; synchronized robot mid-training supplies the contact dynamics absent from that video. The comparison tests a complete data-and-architecture recipe, rather than isolating touch through every baseline.
3 The T-Rex Dataset
The mid-training set contains 7,755 episodes and 100 hours of bimanual demonstrations covering 207 objects, 22 motor primitives and 502 feasible object–primitive pairs. Each episode records three RGB views, robot states/actions, ten-fingertip tactile measurements and an instruction.
Organizing around reusable interactions aims for broader coverage than collecting only downstream task demonstrations. Appendix G specifies feasibility filtering, scene randomization, cleaning, annotation and the planned dataset release.
4 Tactile-Reactive Dexterous Manipulation
The policy conditions a flow-matching action generator on images, language, recent fingertip wrench histories and current deformation maps. The clean demonstrated chunk and Gaussian noise define a linear interpolation, with noise minus action as the target velocity.
Equations 1
| (1) |
4.1 Model Architecture
Three transformer experts share multimodal context: a latent expert predicts future visual representations, an action expert performs the early part of denoising, and a tactile expert completes it from the intermediate boundary state.
For each finger, a VQ-VAE compresses sixteen wrench frames into a temporal token. A direct projection preserves the current wrench, and a convolutional encoder captures the current deformation map. Their concatenation supplies dynamics, instantaneous contact and spatial geometry.
Equation 2
| (2) |
4.2 Asynchronous Tactile-Reactive Cascaded Flow Matching
Ten Euler steps move from noise time one to action time zero. The default slow expert performs six steps to .4 once per sixteen-action chunk. At offsets 0, 4, 8 and 12, the tactile expert reuses the boundary state and visual/action key-value cache, then performs the remaining four steps with fresh touch.
Both experts regress the same velocity target. Despite the earlier disjoint-interval description, the actual training rule samples the action expert across the full (0,1] interval to retain standalone generation; tactile training uses the lower interval scaled by .4. The cache comes from a detached slow pass.
The joint objective weights tactile loss by one and future-visual prediction by .5. Appendix B adds delayed tactile-context augmentation and specifies cache refresh and runtime synchronization.
Equations 3–7
| (3) |
| (4) |
| (5) |
| (6) |
| (7) |
4.3 Training Recipe
Stage one trains latent/action experts on 22,889 hours of human egocentric video with retargeted arm and hand motions, without touch. Stage two adapts to robot cameras, executable actions and tactile feedback using the 100-hour T-Rex dataset. Stage three uses approximately 100 demonstrations per target task.
Zero-shot evaluation is limited to four easier primitive tasks after mid-training; the twelve-task main benchmark uses task-specific post-training. These two settings should not be conflated.
5 Experiments
Experiments evaluate the complete model, tactile and architecture ablations, data efficiency and the contribution of each training stage on the same physical robot platform.
5.1 Experiment Setup
A fixed-base Dexmate Vega-1 uses two seven-joint arms and two 22-joint Sharpa Wave hands. Head and wrist RGB views accompany six-axis fingertip wrenches and deformation maps. T-Rex predicts relative end-effector arm actions and absolute hand joints.
Each of twelve tasks receives sixteen randomized trials, then scores are averaged equally across tasks. Appendix F lists additive partial-credit rubrics; a score of 65 does not mean 65% of entire tasks finished. Baselines differ in pretraining, tactile adaptation and sometimes arm-control representation, as documented in Appendix E.
5.2 Main Results
T-Rex averages 65, versus 35 for EgoScale, 17 for π0.5, 15 for Tactile-VLA, 6 for RDP and tactile-augmented π0.5, and 3 for ViTacFormer. Its task scores range from 96 for page turning to 35 for screwing a bulb. The original per-task values remain in Table 1.
Naively concatenating touch into π0.5 lowers the average in this implementation. Small models trained only on task demonstrations also struggle. These comparisons support the combined recipe, but do not isolate touch from pretraining and mid-training advantages.
| Method |
Flip
Page
|
Transfer
Egg
|
Wipe
Plate
|
Apply
Paste
|
Split
Cup
|
Sort
Mahjong
|
Open
Lock
|
Refill
Tablet
|
Acid-Base
Neut.
|
Extract
Card
|
Deal
Poker
|
Screw
Bulb
|
Avg. |
| ViTacFormer [24] | 9 | 0 | 4 | 1 | 4 | 7 | 0 | 0 | 0 | 2 | 2 | 1 | 3 |
| RDP [61] | 12 | 8 | 18 | 2 | 6 | 9 | 2 | 0 | 0 | 1 | 2 | 7 | 6 |
| Tactile-VLA [28] | 38 | 14 | 24 | 0 | 21 | 27 | 8 | 0 | 9 | 4 | 11 | 18 | 15 |
| EgoScale [70] | 68 | 44 | 34 | 38 | 33 | 36 | 19 | 12 | 43 | 41 | 28 | 18 | 35 |
| [32] | 36 | 17 | 28 | 13 | 18 | 32 | 5 | 1 | 24 | 8 | 9 | 11 | 17 |
| + tactile | 8 | 9 | 27 | 2 | 4 | 14 | 2 | 0 | 7 | 3 | 0 | 0 | 6 |
| Ours | 96 | 75 | 69 | 66 | 78 | 65 | 47 | 41 | 76 | 70 | 57 | 35 | 65 |
| Configuration |
Flip
Page
|
Apply
Toothpaste
|
Split
Cup
|
Open
Lock
|
Extract
Card
|
Screw
Lightbulb
|
Average |
| Full Model (Ours) | 96 | 66 | 78 | 47 | 70 | 35 | 65 |
| Tactile Modality Ablation | |||||||
| w/o Tactile | 76 | 39 | 58 | 23 | 34 | 20 | 42 (-23%) |
| MLP Force + Deform | 89 | 58 | 72 | 44 | 58 | 29 | 58 (-7%) |
| Deform | 82 | 57 | 71 | 36 | 55 | 25 | 54 (-11%) |
| MLP Force + VQVAE Force | 92 | 63 | 65 | 38 | 67 | 28 | 59 (-6%) |
| Architecture Design | |||||||
| w/o Async | 92 | 61 | 73 | 45 | 59 | 30 | 60 (-5%) |
5.3 Ablation Studies
On six tasks, removing touch lowers the rounded average from 65 to 42. Static force plus deformation scores 58, deformation alone 54, and temporal/current force without deformation 59. Synchronous refinement scores 60; the full configuration uses both temporal and spatial touch with asynchronous updates.
Figure 4 sweeps slow-step count, with six steps best among the plotted settings. The prose describes the extremes using τ_split in the reverse direction from τ_split=1−K_slow/10; the plotted axis is K_slow, and the original curve is retained.
At equal 100-hour mid-training budgets, the motor-primitive dataset improves six post-trained tasks and four zero-shot primitive tasks over an eleven-task dataset. Figure 5 shows gains with 10–200 post-training demonstrations. These are selected-task studies, not evaluations on every possible skill.
Table 3 gives averages 18 with neither pretraining nor mid-training, 34 with mid-training only, 45 with pretraining only, and 65 with both. The colored inclusion/exclusion markers are retained with the exact source table.
Selectable original table
| Pre-training | Mid-training |
Flip
Page
|
Apply
Toothpaste
|
Split
Cup
|
Open
Lock
|
Extract
Card
|
Screw
Lightbulb
|
Average |
| ✗ | ✗ | 46 | 16 | 20 | 6 | 14 | 5 | 18 |
| ✓ | 75 | 34 | 45 | 10 | 32 | 9 | 34 | |
| ✓ | ✗ | 88 | 40 | 52 | 22 | 46 | 20 | 45 |
| ✓ | 96 | 66 | 78 | 47 | 70 | 35 | 65 |
6 Conclusion
The work combines a reusable tactile dataset, temporal touch tokens and cached-context action refinement. Its gains are demonstrated on one bimanual hardware setup; the benchmark’s score definition and task-specific post-training remain part of the claim.
7 Limitation and Future Work
Long tasks with tight contact tolerances remain hard to demonstrate and execute; reinforcement learning or online refinement is suggested. Hardware limits include sensor distortion, inter-device calibration drift and missing dense palm sensing. Heterogeneous-sensor transfer and whole-hand touch remain future work.
Acknowledgments
The authors acknowledge Sharpa maintenance, Panasonic assistance and support, Berkeley’s Humanoid Intelligence Center, Sapienza funding and research discussions.
Appendix A Model and Training Details
Table 4 retains all model/training settings. The Qwen3VL-2B-based latent and action experts each list 1.41B parameters; the tactile expert lists .62B, with a 1,536-wide feed-forward layer. Action dimension is 62 and chunk length sixteen.
Supervised training uses AdamW at peak learning rate 10⁻⁴, cosine decay, zero weight decay/warmup, gradient clipping one, bf16 and 24 H100 GPUs with batch sixteen each. The table specifies six slow and four fast inference steps.
| Latent Expert | |
| VLM Backbone | Qwen3VL-2B |
| Hidden Feature Dimension | 2048 |
| Transformer Layers | 28 |
| Max Sequence Length | 2048 |
| Parameter size | 1.41B |
| Attention Implementation | Flash Attention 2 |
| Action Expert (Flow Matching) | |
| VLM Backbone | Qwen3VL-2B |
| Action Dimension | 62 |
| Action Chunk | 16 |
| Training Timestep Sampling | |
| Num Inference Timesteps | 6 |
| Parameter size | 1.41B |
| Tactile Expert (Flow Matching) | |
| Action Dimension | 62 |
| Action Chunk | 16 |
| FFN Intermediate Size | 1536 |
| Training Timestep Sampling | |
| Num Inference Timesteps | 4 |
| Parameter size | 0.62B |
| Training Configurations (SFT) | |
| Optimizer | AdamW |
| Peak Learning Rate | |
| Min Learning Rate | 0 |
| LR Scheduler | Cosine with LR |
| Weight Decay | 0 |
| Warmup Ratio | 0 |
| Gradient Clipping | 1.0 |
| GPU Type | NVIDIA H100 |
| Number of GPUs | 24 |
| Deepspeed Zero Stage | 1 |
| Per Device Batch Size | 16 |
| Gradient Accumulation Steps | 1 |
| Mixed Precision Training | bf16 |
Appendix B Additional Details for Asynchronous Cascaded Denoising
The slow expert sees visual-language and future-prediction context. A detached pass supplies both latent keys/values and action positions re-encoded at the .4 boundary. The tactile expert combines this cache with fresh touch, without rerunning raw-image processing.
Delay augmentation shifts tactile frame indices by a uniform choice from 0, 4, 8 or 12, matching deployment offsets. The visual tower and slow experts are amortized across fast ticks.
Runtime prose describes a request socket and execution lock that serialize expert execution. The printed pseudocode explicitly locks cache publication and cloning, leaving integration loops outside those marked critical sections. It should therefore be read with the implementation description, not as a standalone concurrency or real-time proof. The original two-column algorithm and exact selectable instructions remain below.
Equations 8–10
| (8) |
| (9) |
| (10) |
Selectable original pseudocode
Equation
Appendix C Implementation Details for Spacial-Temporal Tactile Encoder
A shared per-finger temporal convolutional encoder downsamples sixteen six-axis wrench frames twice, mean-pools to 256 dimensions and quantizes into a 64-entry codebook. EMA updates and reseeding discourage collapse; magnitude-weighted reconstruction MSE emphasizes contacts over dominant no-contact states. Finger embeddings identify digits.
The spatial encoder adapts ResNet-18 to one-channel deformation maps, keeps three residual stages, projects each to 128 channels and maps flattened features into policy tokens. It is pretrained as a convolutional autoencoder and frozen for policy learning. Temporal tokens, current-force projections and spatial features feed the tactile expert.
Appendix D Real-World Setup and Teleoperation Stack
Wheels, torso and head are fixed. Relative wrist targets pass through Pink differential inverse kinematics, low-pass filtering and the manufacturer’s cascaded PID controller. An asynchronous policy thread updates targets tracked by the separate 300 Hz control thread.
A head ZED X Mini left image and two wrist ZED X One S wide views supply 640×360 RGB. Every fingertip supplies deformation depth and a six-axis net wrench. Manus glove fingertip targets and VIVE wrist poses drive teleoperation; hand retargeting uses Pinocchio and CasADi. High-level collection and synchronized recording run at 30 Hz.
Appendix E Implementation Details of Baselines
ViTacFormer trains separate ACT policies for 100 epochs on 100 demonstrations per task, with chunk length 100 and all 22 hand joints predicted. RDP separately trains its tactile-conditioned action tokenizer for 100 epochs and latent diffusion policy for 200, using ten-finger wrenches.
Tactile-VLA is adapted from GelSight images to six-axis fingertip wrenches through a simple MLP, then trained per task for 100 epochs. This representation change matters when interpreting its comparison with the original method.
EgoScale uses the released GR00T-N1.7-3B checkpoint and 200-epoch per-task finetuning. π0.5 uses OpenPI and its released checkpoint; the tactile variant concatenates single-step wrenches to state. The appendix specifies joint-space arm control for these π0.5 variants, unlike T-Rex’s relative end-effector control, despite the general statement that action spaces are unified.
The reported results compare these reproduced implementations and training recipes, with different model sizes, initial checkpoints and tactile representations.
Appendix F Evaluation Tasks
The tasks span force control, deformation-sensitive contact and longer bimanual sequences. Every original task instruction, description and additive scoring rubric is retained beside its figure. Partial credit is awarded for individual substeps; all twelve listed rubrics are additive even though the introduction also describes hierarchical progress scoring.
Tasks 1–4 test single-page turning, intact egg transfer, bimanual wiping and toothpaste dispensing. Tasks 5–8 test separating one nested cup, tactile mahjong identification and sorting, key insertion/unlocking, and opening/filling/closing a compartment. Tasks 9–12 test dropper manipulation, separating one card from a sleeve, dealing one card, and handover/alignment/rotation of a lightbulb.
The acid-base task instruction says the solution should turn green or yellow, whereas its description and .15-credit rubric say colorless. Both source statements remain unchanged. The compartment task also differs on which right-hand digit closes the lid; the exact description and rubric are preserved.

Original task instruction and scoring rubric
Task I: Flip Page. Text Instruction: “Turn a page of the book from right to left using your right index finger.” The robot must lift a single sheet from the right side of an open book, sweep it across the spine, and smooth it down flat on the left side.
Grading rubric (additive):
- •
: (a) Successfully touched the book page with a single finger.
- •
: (b) Using the index finger turn the page up.
- •
: (c) Successfully flips exactly one page from right to left and smooths it flat.

Original task instruction and scoring rubric
Task II: Transfer Egg. Text Instruction: “Using the right thumb and index finger, pick up the egg from the green egg tray and place it into the yellow egg tray.” The robot must grasp a fragile egg without cracking the shell from the green container, lift it off the surface, transport it above the yellow container, and gently release it inside.
Grading rubric (additive):
- •
: (a) Approaches the egg and makes contact without knocking it off the table.
- •
: (b) Lifts the egg off the table without cracking it.
- •
: (c) Transports the egg above the yellow container.
- •
: (d) Releases the egg inside the container intact.

Original task instruction and scoring rubric
Task III: Wipe Plate. Text Instruction: “There is a white plate and a white cloth on the table; the white plate has colored stains on it. Use your right hand to pick up the cloth, hold the plate steady with your left hand, and then use the cloth to wipe away the stains.” The robot must grasp the cloth with the right hand, press down on the plate with the left hand to hold it steady, bring the cloth into contact with the plate surface, wipe the plate until the colored stains are fully removed, and place the cloth back on the table while releasing the plate.
Grading rubric (additive):
- •
: (a) Right hand grasps the rag.
- •
: (b) Left hand presses down on the plate to hold it steady.
- •
: (c) Brings the rag into contact with the plate surface.
- •
: (d) Wipes the plate until the design is fully removed (no visible ink remaining).
- •
: (e) Places the rag back on the table and releases the plate.

Original task instruction and scoring rubric
Task IV: Apply Toothpaste. Text Instruction: “On the left side of the countertop sits a cup holding a toothbrush, while an open tube of toothpaste rests on the right. Pick up the toothbrush with your left hand and the toothpaste with your right, squeeze some toothpaste onto the brush, and then set the tube back down.” The robot must grasp a toothbrush in one hand and a toothpaste tube in the other, align the tube nozzle above the bristles and squeeze out a bead of toothpaste, and return the toothbrush upright into its holder and the toothpaste back onto the table.
Grading rubric (additive):
- •
: (a) Grasps the toothbrush.
- •
: (b) Grasps the toothpaste tube.
- •
: (c) dispenses a bead of toothpaste onto the bristles.
- •
: (d) Returns the toothbrush upright into its holder.
- •
: (e) Places the toothpaste tube back on the table.

Original task instruction and scoring rubric
Task V: Split Cup. Text Instruction: “A stack of red plastic cups sits on the desktop; use the right hand to slide out the topmost one, exerting effort to separate it from the rest of the stack.” Given a stack of nested cups on the table, the robot must grasp the stack with the left hand to stabilize it, and use the right hand to twist and rub exactly one cup off the top of the stack.
Grading rubric (additive):
- •
: (a) Left hand grasps and stabilizes the cup stack.
- •
: (b) Right hand grasps the topmost cup of the stack.
- •
: (c) Right hand twists and separates exactly one cup from the stack.
- •
: (d) Right hand holds the single separated cup intact.

Original task instruction and scoring rubric
Task VI: Sort Mahjong. Text Instruction: “Three boxes are placed on the table, representing the Mahjong tiles ’Red Zhong’, ’Green Fa’, and ’White Blank’, respectively. In the center of the table lies a single Mahjong tile, placed face-down. Now, using your right hand, grasp the tile and discern its pattern; then, use your left hand to open the box corresponding to that pattern and place the tile inside.” The robot must pick up a face-down mahjong tile with the right hand and feel its surface via tactile sensing to identify its category, then use the left hand to slide open the lid of the matching compartment in the organizer box, place the tile into the compartment with the right hand, and close the lid with the right thumb.
Grading rubric (additive):
- •
: (a) Right hand picks up the face-back mahjong tile.
- •
: (b) Left hand slides open the lid of the correct compartment.
- •
: (c) Right hand places the tile into the correct compartment.
- •
: (d) Right thumb closes the compartment lid.

Original task instruction and scoring rubric
Task VII: Open Lock. Text Instruction: “On the left side of the desk lies a red book, atop which rests a gray key; on the right side is a lock. Using your left thumb and index finger, slide the key free; then, pick up the lock with your right hand and use the key to unlock it.” The robot must first grasp the key with one hand and the padlock with the other, align and insert the key into the keyhole, and rotate it to release the shackle.
Grading rubric (additive):
- •
: (a) Grasps the key.
- •
: (b) Grasps the padlock.
- •
: (c) Aligns and inserts the key into the keyhole.
- •
: (d) Rotates the key and successfully opens the lock.

Original task instruction and scoring rubric
Task VIII: Refill Tablet. Text Instruction: “Use your left hand to open one of the compartments in the small box, use your right hand to grasp the small ball on the table, place the ball into the box, and then close the box.” The robot must use the left index finger to press the button on a compartment lid to unlock it, flip the lid open with the left thumb, pick up the ball with the right hand, place the ball into the open compartment, and press the lid closed with the right index finger.
Grading rubric (additive):
- •
: (a) Left index finger presses the compartment button to unlock the lid.
- •
: (b) Left thumb flips the lid open.
- •
: (c) Right hand picks up the ball.
- •
: (d) Right hand places the ball into the open compartment.
- •
: (e) Right thumb presses the lid closed.

Original task instruction and scoring rubric
Task IX: Acid-Base Neutralization. Text Instruction: “On the right side of the desktop stands an Erlenmeyer flask containing 200 mL of citric acid solution; on the left is a beaker holding 20 mL of NaOH solution, which includes bromothymol blue indicator—appearing blue due to its alkaline nature. Using your right hand, pick up the dropper and draw up approximately 5 mL of the acid solution; then, using your left hand to hold the beaker, perform an acid-base titration until the liquid in the beaker turns green or yellow.” The robot uses a dropper held in the right hand to aspirate liquid from a conical flask, dispenses it into a beaker held in the left hand, and swirls the beaker until the blue indicator solution fully turns colorless, and then returns the dropper to the conical flask and places the beaker back on the table.
Grading rubric (additive):
- •
: (a) Right hand grasps the dropper from the conical flask.
- •
: (b) Right hand aspirates liquid from the conical flask.
- •
: (c) Left hand picks up the beaker.
- •
: (d) Right hand dispenses liquid from the dropper into the beaker.
- •
: (e) Left hand swirls the beaker to mix the contents.
- •
: (f) The solution in the beaker fully transitions from blue to colorless.
- •
: (g) Right hand returns the dropper to the conical flask.
- •
: (h) Left hand places the beaker back on the table.

Original task instruction and scoring rubric
Task X: Extract Card. Text Instruction: “Next to the cube on the table lies a card case containing two cards. Pick up the case with the left hand, then use the right thumb to slide the cards out through the central opening; subsequently, use the right thumb and index finger to slide out the first card, taking care not to pull out the second one.” The robot must pick up the card sleeve (containing two cards) with the left hand, use the right thumb to rub the cards partially out, then use the right thumb and index finger to push the bottom card back in so that only the top card remains exposed, and extract that single card.
Grading rubric (additive):
- •
: (a) Left hand picks up and holds the card sleeve.
- •
: (b) Right thumb rubs the cards partially out of the sleeve.
- •
: (c) Right thumb and index finger push the bottom card in.
- •
: (d) Right hand extracts the single top card from the sleeve.

Original task instruction and scoring rubric
Task XI: Deal Poker. Text Instruction: “Pick up a stack of playing cards with your right hand, then transfer it to your left; hold the stack aloft with your left hand, use your right thumb to slide out the top card, grasp it, and place it into the card holder.” The robot must grasp the full card stack from above with the right hand, transfer it to the left hand, use the right thumb to flick the top card partially out, adjust with the right thumb and index finger until exactly one card protrudes, grasp that single card, and insert it vertically into the dedicated card slot.
Grading rubric (additive):
- •
: (a) Right hand grasps the card stack from above.
- •
: (b) Right hand transfers the stack to the left hand (handover).
- •
: (c) Right thumb flicks and adjusts until exactly one card protrudes from the top.
- •
: (d) Right hand successfully grasps the single protruding card.
- •
: (e) Right hand inserts the card vertically into the card slot.

Original task instruction and scoring rubric
Task XII: Screw Lightbulb. Text Instruction: “There is a lightbulb and a base on the desktop. Use your left hand to pick up the lightbulb and transfer it to your right hand; then, use your left hand to hold down the base while using your right hand to screw the lightbulb into the base until it lights up.” The robot must pick up the lightbulb with the left hand, transfer it to the right hand (handover), stabilize the lamp socket with the left hand, and use the right hand to rotate the bulb through multiple turns into the socket until it is fully seated and illuminates.
Grading rubric (additive):
- •
: (a) Left hand picks up the lightbulb.
- •
: (b) Left hand transfers the bulb to the right hand (handover).
- •
: (c) Left hand stabilizes the lamp socket.
- •
: (d) Right hand aligns the bulb and rotates continuously to engage the threads.
- •
: (e) Bulb is fully seated and the lamp illuminates.
Appendix G T-Rex Dataset
Episodes synchronize three 640×360, 30 Hz RGB streams, arm positions/velocities, 44 hand joints, both wrist poses, ten deformation maps/wrenches and language. All streams are recorded at the 30 Hz high-level cadence, separate from the faster low-level controller.
Manual feasibility filtering reduces 207×22=4,554 potential pairs to 502. The set contains 7,755 episodes over ten weeks, approximately sixteen demonstrations per pair, with median length 29.8 seconds and interquartile range 21.0–41.1 seconds. Six tabletop backgrounds, randomized object poses and zero to five distractors from a pool exceeding 210 objects vary scenes.
Cleaning rejects unstable tactile measurements, corrupted streams, teleoperation failures and extreme joint velocities. A commercial VLM receives four to six sampled head-camera frames plus object/primitive labels to draft one imperative instruction; humans check hallucinations and imprecision.
The appendix says release of sensor streams, derived touch, annotations, loaders and preprocessing under MIT is planned. The main text’s “open-source” description is therefore not treated here as verification that release already occurred. Collection uses consenting operators in a controlled laboratory; incidental operator/reset frames are clipped.
Appendix H Failure Case Analysis
Six failure categories remain visible in the complete figure: bulb/base collision from poor alignment, a slipping key, incorrect egg placement, unintended neighboring-compartment contact, excessive toothpaste force and misaligned card extraction.
Together they expose limits in visual alignment, small-object grip, behavior-cloning distribution shift, individual-finger coordination and temporal force regulation. Fast tactile refinement reduces some errors but does not remove these failure modes.