Literature
Display
← All papers

FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

Zhi Cao · Yining Hong · Jiaolong Yang · Yejin Choi · Li Fei-Fei · Jiajun Wu · Huang Huang

Preprint posted · arXiv (preprint) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the 16-page arXiv v1 and retains all 20 figures, four tables, five numbered equations and five appendices. The original HTML omitted Figure 1; it is restored from the PDF. Read the results as task-specific adaptation on one physical station, using ten robot and 90 newly collected human demonstrations per task. Binary success, clean musical notes and pipette-dial engagement/conditional error are different metrics.

Original S0.F1 native PDF artwork with complete original panels, labels and layout.
Fig. 1: FACT combines large-scale force-rich human experience with robot demonstrations for pretraining. Its coarse-to-fine architecture uses visual tracks and video prediction to shape coarse action, followed by joint force–action refinement for fine-grained execution. The pretrained model adapts to diverse tasks, from tool use, musical instruments, to precision laboratory procedures.

At a glance

FACT learns from 2,000 hours of tactile human demonstrations plus 100 hours of robot data. A coarse model jointly predicts video, point tracks and actions; a second model co-denoises remaining actions and future fingertip wrenches while incorporating fresh touch. Ten-task hardware tests favor the full system over the two baselines, but binary success averages 41.4% and shoelace success is 20%. Gains reach 60 percentage points on test-tube opening. The data-ratio studies support useful human transfer in these tasks, not interchangeable human and robot data in general.

I Introduction

Human demonstrations scale more easily than robot teleoperation but usually lack contact measurements. EgoForce adds calibrated tactile observations; FACT combines prospective visual motion with force-conditioned refinement. Shared future video/tracks support transfer across embodiments, while continuous force prediction couples anticipated contact with action. The evaluated transfer includes substantial human–robot pretraining and task-specific fine-tuning, rather than zero-shot deployment from human video alone.

II Related Work

Related work uses egocentric video, tactile representations, action hierarchies and predicted visual motion. T-Rex refines coarse actions from quantized force history; FACT also retains quantized history but adds continuous current-force input and explicitly co-predicted future force. Its coarse visual/action/track prediction follows ideas in TrAct, with a Wan-based video transformer. This combination is tested empirically rather than proven embodiment invariant.

Original S2.F2 native PDF artwork with complete original panels, labels and layout.
Fig. 2: EgoForce task and skill distribution. Left: episode share by task category, with over two thirds of the corpus covering contact-rich fine manipulation across diverse task categories. Right: atomic-skill annotations on a log scale, showing a broad distribution of dexterous, force-dependent skills beyond simple transport.

III The EgoForce Dataset

EgoForce pairs synchronized egocentric views, estimated hand motion and calibrated pressure-derived contact labels for pretraining.

Scalable multimodal collection platform

A head camera, two wrist cameras and bimanual tactile gloves record aligned vision, hand movement and touch. The hardware aims to allow natural manipulation across diverse tasks and environments.

Automatic force calibration

A pneumatic bladder sweeps approximately uniform pressure from 0 to 50 psi while a reference sensor and each taxel record synchronously. Pressure times effective sensing area gives normal force; per-taxel A/D mappings calibrate gloves and sessions. This does not directly measure shear or a complete six-axis wrench.

The EgoForce dataset

The reported dataset contains about 2,000 hours, 40,000 episodes, 1,000 tasks, 120 scenes and 1,000 demonstrators. Appendix C explains inferred human action labels and the pressure-to-wrench approximation.

Task distribution: dexterous by construction

Deformable manipulation, assembly/construction and storage/organization account for 21.0%, 17.3% and 14.8%. Other fine manipulation raises the contact-rich share above two thirds. Frequent fold, regrasp and thread annotations motivate tactile supervision, but the distribution is intentionally dexterity-focused rather than a representative sample of all household activity.

Original S3.F3 native PDF artwork with complete original panels, labels and layout.
Fig. 3: FACT architecture. The Track-Guided Coarse Planner jointly denoises future video, visual tracks, and a coarse action chunk with a block-causal video DiT; per-token noise conditioning lets clean context and noisy targets coexist. At the τ=0.4\tau{=}0.4 handoff, video and tracks are read out and the coarse hidden states are frozen. The Force–Action Joint Refiner reads them through cached cross-attention and co-denoises the remaining action (τa:0.4→0\tau_{a}\!:0.4{\to}0) with continuous future fingertip force (τf:1→0\tau_{f}\!:1{\to}0), using observed tactile signals as clean context.

IV The FACT Framework

Pretraining mixes EgoForce with 100 hours of T-Rex robot teleoperation. At inference, visual/state/language/track/tactile context produces 16 actions at 20 Hz and four 5 Hz future-force slots for ten fingertips. The 62-dimensional action contains each arm’s relative end-effector pose and 22 absolute hand joints. Predicted force shapes the action representation; it is not sent as a force-control reference.

IV-A Track-Guided Coarse Planner

The block-causal coarse planner jointly generates future video, image-space tracks and a coarse action chunk. Its visual targets supply a common prediction interface for different embodiments.

Multimodal representation

Text and anchor-image embeddings enter through cross-attention. State, past tracks and conditioning video are clean context; future video, action and tracks are noisy prediction targets in the temporal self-attention sequence. Text/CLIP features remain outside that sequence.

Per-token flow matching

Flow matching interpolates clean targets with Gaussian noise; each token carries its own noise level, with zero clean and one noise. Generated temporal blocks draw Beta(1.5,1) levels; conditioning tokens stay clean. Per-token adaptive normalization permits the action pathway to stop partially denoised and continue in the refiner.

Equation 1
xτ=(1−τ)​c+τ​ε,v∗=ε−c,x_{\tau}=(1-\tau)c+\tau\varepsilon,\qquad v^{*}=\varepsilon-c, (1)

Attention topology

Within a causal temporal window, observation/action tokens attend to current and earlier visual/action/track tokens plus the current state. State remains local, while track tokens attend only to tracks. Separate heads decode video, actions and track motion; text/image embeddings supply cross-attention conditioning.

Training objective

The preserved objective sums video and action flow losses with a track objective weighted by 20. Track terms combine flow, reconstructed position and smoothness, with visibility masking and stronger weighting for moving points. The one-step reconstruction provides the position/smoothness target estimate.

Equations 2–3
ℒcoarse=ℒdyn+ℒact+λt​(ℒtrkflow+5​ℒtrkpos+2​ℒtrksmooth),\mathcal{L}_{\mathrm{coarse}}=\mathcal{L}_{\mathrm{dyn}}+\mathcal{L}_{\mathrm{act}}+\lambda_{t}\left(\mathcal{L}^{\mathrm{flow}}_{\mathrm{trk}}+5\,\mathcal{L}^{\mathrm{pos}}_{\mathrm{trk}}+2\,\mathcal{L}^{\mathrm{smooth}}_{\mathrm{trk}}\right), (2)
x^0=xτ−τ​v^.\hat{x}_{0}=x_{\tau}-\tau\hat{v}. (3)

Coarse handoff to the Force–Action Joint Refiner

Six Euler steps reduce noise from 1 to 0.4, followed by an additional forward pass at 0.4. Video and tracks are read out, while partially denoised actions pass to the refiner. Observation/action hidden states are detached and projected once into cached cross-attention memory.

IV-B Force–Action Joint Refiner

The second stage jointly predicts the remaining action and continuous future force, conditioned on current touch, quantized history and the fixed coarse plan.

Original S4.F4 native PDF artwork with complete original panels, labels and layout.
Fig. 4: Evaluation randomization. Overlaid initial frames from six evaluation tasks illustrate rollout-level variation in object positions and poses.

Force–action joint refinement

Current measured wrenches are clean observations; future-force tokens begin as noise. Bidirectional attention lets evolving actions and predicted contact influence one another throughout denoising. This is learned coupled prediction, not an analytic dynamics model or enforcement of contact-force constraints.

Coarse memory through cached cross-attention

Only observation/action hidden states are retained from the coarse handoff; track/state/conditioning tokens are discarded from memory. Cached projections are reused across refiner steps and tactile refreshes, avoiding a new expensive coarse pass each time.

Cascaded sampling

Four Euler steps take action noise from 0.4 to zero and force noise from one to zero. Only the unexecuted action suffix and unobserved force slots change. New 5 Hz touch measurements clamp observed slots before another suffix refinement.

Training objective

The refiner trains action-suffix flow error plus future-force error weighted 0.5 and vector-quantization loss. Its detached coarse memory is evaluated at the exact 0.4 handoff using the same action-noise draw, aligning training with the cascade rather than treating independently sampled plans as equivalent.

Equation 4
ℒrefine=‖v^a−va∗‖[o:16)2⏟action suffix+λf‖v^F−vF∗‖s>o/42⏟future force+ℒVQ.\mathcal{L}_{\mathrm{refine}}=\underbrace{\big\|\hat{v}_{a}-v_{a}^{*}\big\|^{2}_{[o:16)}}_{\text{action suffix}}+\lambda_{f}\underbrace{\big\|\hat{v}_{F}-v_{F}^{*}\big\|^{2}_{s>o/4}}_{\text{future force}}+\mathcal{L}_{\mathrm{VQ}}. (4)

IV-C Implementation Details

The planner has a 30-block, width-3072 Wan2.2 backbone; the refiner has six width-768 blocks. A 0.8-second temporal block carries 617 tokens, with four teacher-forced blocks during training. Explicit appendix details clarify that Wan is pretrained and fine-tuned, while new projections/refiner/codecs learn from scratch and VAE/text/CLIP encoders stay frozen. The main-text “all components from scratch” wording should not be read as replacing that initialization recipe.

TABLE I: Real-robot evaluation on the ten-task suite (scoring in Section V-A). For pipette volume, the indented second row gives the mean absolute error in μ\muL over the engaged trials. w/o force pred and w/o track pred remove future-force prediction and visual track prediction during training, respectively.
FACT (ours)
# Task π0.5\pi_{0.5} T-Rex w/o force pred w/o track pred full
Binary tasks: success rate (%) ↑\uparrow
1 Lab pipetting 0 0 30 20 40
2 Open test-tube cap 10 20 40 80 80
3 Open microcentrifuge flip cap 0 20 20 40 50
4 Tie shoelaces 0 0 0 20 20
5 Shuffle cards 0 10 0 20 30
6 Multi-object grasp 0 0 10 40 40
7 Use screwdriver 0 10 0 30 30
Mean 1.4 8.6 14.3 35.7 41.4
Continuous tasks: clean notes of 10 ↑\uparrow
8 Play harp 0.5 2.1 0 6.5 6.7
9 Play xylophone 0 3.2 1.9 3.9 5.4
Volume adjustment: engaged trials of 10 ↑\uparrow
10 Pipette volume 1 2 2 4 5
mean abs. error (μ\muL) ↓\downarrow 17.0 14.0 11.0 12.8 6.4

IV-D Training Recipe

The joint objective trains coarse and refinement pathways on an hours-proportional 2,000:100 human/robot mixture using 240 T-Head PPUs. Human tactile labels supervise the model from the beginning. Table IV retains optimizer, schedule, batch size and precision; the paper does not turn this compute description into a demonstrated low-cost pretraining recipe.

Equation 5
ℒFACT=ℒcoarse+λrefine​ℒrefine,\mathcal{L}_{\mathrm{FACT}}=\mathcal{L}_{\mathrm{coarse}}+\lambda_{\mathrm{refine}}\mathcal{L}_{\mathrm{refine}}, (5)

V Experiments

Experiments cover ten tasks on a single bimanual station, with architecture, pretraining and human-data-ratio ablations.

Refer to caption
Fig. 5: Qualitative examples. FACT executes contact-rich behaviors requiring dexterous multi-finger coordination, precision fingertip micro-manipulation, and in-hand multi-object grasping.
TABLE II: Pretraining recipe ablation on all ten tasks.
# Task human w/ force+robot (full) human w/o force+robot robot only
Binary tasks: success rate (%) ↑\uparrow
1 Lab pipetting 40 20 0
2 Open test-tube cap 80 0 20
3 Open microcentrifuge flip cap 50 10 0
4 Tie shoelaces 20 0 0
5 Shuffle cards 30 10 0
6 Multi-object grasp 40 40 30
7 Use screwdriver 30 10 0
Continuous tasks: clean notes of 10 ↑\uparrow
8 Play harp 6.7 2.0 1.2
9 Play xylophone 5.4 2.2 1.0
Volume adjustment: engaged trials of 10 ↑\uparrow
10 Pipette volume 5 2 0
mean abs. error (μ\muL) ↓\downarrow 6.4 20.5 –

V-A Experimental Setup

Every task gets its own 100-trajectory fine-tuning mixture and ten randomized evaluation rollouts.

Robot platform and data collection

Two YAM arms carry Sharpa Wave hands with 22 joints and five six-axis fingertips each. Policies observe head and wrist cameras plus state; tactile-aware models also use fingertip wrenches. Ten successful robot demonstrations and 90 human demonstrations are collected separately for each evaluation task.

Skill-specific fine-tuning

The pretrained model is fine-tuned separately for each task on eight GPUs. Human fine-tuning demonstrations are collected on the target tasks with Wuji gloves and wrist trackers, not drawn from EgoForce.

Baselines and ablations

π0.5 and T-Rex start from their released checkpoints and use the same task demonstration mixture. They differ in tactile access, action encoding and training recipe, so this is a complete-system comparison. FACT variants remove force-prediction or track pathways and associated losses.

Evaluation protocol and metrics

Seven tasks use binary success over ten rollouts. Harp and xylophone score clean notes out of ten per rollout, not whole-task success percentages. Pipette adjustment reports dial engagements out of ten and absolute volume error only among engagements. Scenes and objects are fixed across methods; placements vary. Never-engaged trials are failures with no conditional reading.

Original figure
Fig. 6: Robot-to-human mixture in fine-tuning. Performance versus the robot-to-human ratio of the 100-trajectory per-task fine-tuning mixture (10:90, 30:70, 50:50).
TABLE III: Contribution of human fine-tuning data on three dexterous tasks: ten robot trajectories alone (10:0) versus the 10:90 robot-to-human mixture, scored as in Table I.
robot : human data
# Task 10 : 0 10 : 90
Binary tasks: success rate (%) ↑\uparrow
1 Tie shoelaces 10 20
2 Shuffle cards 10 30
Volume adjustment: engaged trials of 10 ↑\uparrow
3 Pipette volume 4 5
mean abs. error (μ\muL) ↓\downarrow 11.2 6.4

V-B Main Results

FACT reaches 40% pipetting, 80% test-tube opening and 40% multi-object grasping, compared with T-Rex’s 0%, 20% and 0%. The seven binary tasks average 41.4% versus 8.6% for T-Rex; remaining failures are substantial. Harp scores 6.7 clean notes versus 2.1. Dial engagement succeeds in five trials with 6.4 μL conditional mean error; four of those five are within 4 μL. These small-sample results distinguish reaching useful contact from precise control after contact.

Original figure
Fig. 7: Scaling with human data. Validation loss (MSE) versus hours of human demonstration data (100, 500, 1k, 1.5k, and 2k hours on a linear axis), mixed with robot teleoperation data, with and without tactile supervision. Tactile supervision lowers the loss at every scale and the gap persists as data grows.

V-C Ablation Studies

The ablations examine predictive pathways, tactile human pretraining and the amount/composition of fine-tuning data.

Force and track prediction

Removing force prediction cuts test-tube success from 80% to 40% and harp performance from 6.7 to zero notes. Removing tracks lowers pipetting from 40% to 20% and xylophone from 5.4 to 3.9 notes, but leaves some tasks unchanged. The interventions remove tokens/heads/losses together, so they do not isolate an auxiliary loss from all architectural effects.

Force supervision in human pretraining data

Without human force supervision, test-tube opening falls to zero, pipetting to 20%, and dial engagement to two trials with 20.5 μL error; multi-object grasping remains 40%. That variant also masks human tactile inputs, so it combines removing observations with removing force targets. Robot-only pretraining loses the much larger human dataset and performs poorly. These comparisons support the full data recipe without equalizing all pretraining information or volume.

Scaling with pretraining human data

Across 100, 500, 1,000, 1,500 and 2,000 human hours, Figure 7 reports lower validation MSE with tactile supervision: 0.042 versus 0.047 at 100 hours and 0.031 versus 0.036 at 2,000. The 10–14% relative loss advantage persists. The figure uses a linear hour axis; this validation-loss trend is not a measured robot-success scaling law or proof of asymptotic behavior.

Human data in fine-tuning

Adding 90 human trajectories to ten robot trajectories improves shoelaces from 10% to 20%, cards from 10% to 30%, and dial engagement from four to five trials while conditional error falls from 11.2 to 6.4 μL. This comparison increases total fine-tuning examples from ten to 100; it demonstrates useful additional human data rather than equal sample efficiency by itself.

Human-demonstration efficiency in fine-tuning

At fixed 100-trajectory totals, changing robot:human ratios from 10:90 to 50:50 gives modest gains on the three plotted tasks: flip caps 50→60%, multi-object grasping 40→50%, dial engagement 5→6. Conditional volume errors are 6.4, 5.2 and 6.1 μL. Ten-rollout estimates suggest useful substitution in these settings, with limited evidence for a general human/robot equivalence claim.

VI Conclusion

The study demonstrates beneficial tactile human pretraining and coupled visual/force prediction for dexterous adaptation. Limits evident from the protocol include one physical setup, ten rollouts per task, substantial residual failures, task-specific human demonstrations and incomplete human wrench observability. It does not establish safety constraints, broad object/scene generalization or force-reference tracking.

Acknowledgements

The authors attribute the 2,000-hour collection system to TouchTronix FusionX and acknowledge hardware donations from Sharpa and Xdof. These collection/hardware credits are retained as provenance.

Appendix A Model and Training Details

Wan2.2-TI2V-5B is fully fine-tuned; new modality projections and the refiner train from scratch. VAE, umT5-XXL and CLIP remain frozen. Table IV records the 5.05B planner, 65M refiner and training configuration.

TABLE IV: Model and training configuration of FACT.
Track-Guided Coarse Planner (Flow Matching)
Backbone Wan2.2-TI2V-5B video DiT
Hidden dimension 3072
Number of layers 30
Max sequence length 2,518 tokens
Action dimension 62
Action chunk 16
Timestep sampling Beta⁡(1.5,1.0)\mathrm{Beta}(1.5,1.0)
Num inference timesteps 6 (τ:1→0.4\tau\!:1{\to}0.4)
Parameters 5.05B
Force–Action Joint Refiner (Flow Matching)
Hidden dimension 768
Number of layers 6
FFN intermediate size 3072
Action dimension 62
Action chunk 16
Num inference timesteps 4
Parameters 65M
Training configuration
Optimizer AdamW
Peak learning rate 1×10−51{\times}10^{-5}
Min learning rate 0
LR scheduler cosine
Warmup ratio 0.05
Weight decay 10−510^{-5}
Gradient clipping 1.0
GPU type Alibaba T-Head PPU (Zhenwu 810E, 96 GB HBM2e)
Number of GPUs 240
o DeepSpeed ZeRO stage 2
Per device batch size 16
Gradient accumulation steps 1
Mixed precision training bf16
Tactile dropout 0.1

Visual tracks

Offline SAM2/CoTracker processing yields 250 points per block: 100 hand points in the head view and 75 points per wrist view, including background grids. A 16-frame history encodes position, visibility and displacement; future heads predict 16 position velocities. Visibility masks and path-length weight 1+4ℓ emphasize observable motion. The text states offline extraction but does not detail a complete online tracker latency pipeline.

Data pipeline

LeRobot-v2 recordings at 30 fps are timestamp-resampled to actions/tracks at 20 Hz, video at 10 Hz and force at 5 Hz. Each arm uses three relative position coordinates, six relative rotation coordinates and 22 absolute hand joints. Percentile normalization maps values to [−1,1]. Three 160×320 camera views tile into a 2×2 image before VAE encoding.

Inference

Inference uses six coarse Euler updates plus an extra handoff forward pass, then four refinement updates, without classifier-free guidance. Actions execute at 20 Hz. The paper provides these algorithmic rates but no measured end-to-end deployment latency guarantee.

Appendix B Force–Action Joint Refiner

The refiner preserves a coarse plan while repeatedly updating its unexecuted suffix from new contact information.

Tactile input

Ten fingertips supply 60 wrench channels. Every 5 Hz refresh uses 16 history samples spanning 3.2 seconds; unavailable early samples are zeroed and masked. Current force is the history window’s last sample.

Force-history quantization

A shared temporal convolution encoder compresses each fingertip history into a 256-dimensional code. The 64-entry codebook uses EMA decay 0.99 and periodically reseeds dead codes. Reconstruction plus commitment loss trains the quantizer; hand/finger embeddings identify ten history tokens. Thus history remains quantized despite the continuous future-force pathway.

Token layout

The 76-token refiner combines 16 actions, ten current-force tokens, ten history tokens and 40 future-force tokens. Bidirectional attention and cached coarse cross-attention predict action/force flow velocities. The appendix gives 64-channel action projections/heads while the physical action and table are 62-dimensional; it does not explicitly explain that padding or adaptation, so the edition preserves the distinction.

Coarse memory

Observation/action hidden states at noise 0.4 are detached and projected to width 768. The appendix describes current plus up to four preceding blocks, whereas the main text describes a four-block causal window. These context-window descriptions are retained as a source ambiguity rather than silently reconciled.

Training

Four training blocks times offsets {0,4,8,12} create 16 refiner instances per sample. Executed prefixes and observed force slots remain clean; only remaining actions/unobserved force receive noise and loss. Action noise is restricted to the last 0.4 of the schedule, with a small floor. Current/history tactile dropout is 0.1 per sample.

Sampling and execution

At offsets 4, 8 and 12, new measured contact is clamped and the remaining chunk is refined again against cached coarse states. The 20 Hz action sequence is therefore refreshed using 5 Hz tactile updates. Predicted future wrenches are logged for analysis and explicitly are not control references.

Appendix C Hardware and Data Collection

Pretraining human data, target-task human data, robot teleoperation and robot pretraining use different capture systems whose labels are mapped into a shared representation.

Original A3.F8 native PDF artwork with complete original panels, labels and layout.
Fig. 8: EgoForce collection system and data. Left: collection hardware (egocentric stereo RGB camera, wrist RGB cameras, and piezoresistive tactile gloves) and front and side views of an operator wearing the system. Middle: the FusionX multimodal collection system (TouchTronix Robotics, NY), illustrated with synchronized head and wrist views and tactile readings from a shirt-folding recording; the head view includes estimated hand poses and image-space hand trajectories, which are not motion-capture ground truth. Right: twelve examples from the collection environments, each showing the egocentric view and the tactile maps of the left and right hand (finger labels T, I, M, R, L for thumb, index, middle, ring, and little finger). Heatmaps show per-taxel forces in newtons on a shared 0–8 N scale with square-root color mapping.

EgoForce collection (pretraining data)

FusionX combines a calibrated 30 fps stereo head pair, two wrist RGB cameras and TouchTronix gloves sampled near 200 Hz. Each glove has 12 taxels per fingertip plus 60 palm taxels, bend sensors and an orientation IMU. Per-user rest offsets/bend ranges and pneumatic pressure calibration accompany common-clock episode recordings.

Human action and force labels

Human hand motion is estimated from ViTPose/HaMeR camera fits, smoothed and scaled with stereo depth; glove bend/orientation cues assist occlusions. Retargeting maps hands to Sharpa’s 22 joints. Summed normal taxel forces and center of pressure yield normal force and two moments; unobserved tangential forces and twisting moment are set to zero. These approximate six-channel labels are not directly measured full human wrenches.

Refer to caption
Fig. 9: Real-world setup and data-collection interfaces. Top: the bimanual station, with two I2RT YAM Ultra arms driven by a YAM Box controller, a Sharpa Wave 1 hand on each arm, a RealSense D435 head camera, and a ZED Mini camera on each wrist. Bottom left: the teleoperation interface, Manus gloves with a VIVE Tracker 3.0 on each wrist. Bottom right: the human demonstration interface used for the fine-tuning data, Wuji tactile gloves with a RealSense D435 wrist camera and a VIVE Tracker 3.0 on each wrist.

Robot platform

The evaluation station uses two six-DoF YAM Ultra arms and Sharpa Wave 1 hands. End-effector targets pass through inverse kinematics to built-in joint controllers; hand joints are absolute targets. Fingertip wrenches arrive near 30 Hz and are resampled for the model. Three RGB views and state complete the observations.

Teleoperation

Robot teleoperation uses Manus gloves and VIVE wrist trackers; retargeted fingers and inverse-kinematics arm commands are logged alongside measured state/touch and synchronized cameras at 30 Hz. The dataset retains ten successful trajectories per target task.

Human demonstrations for fine-tuning

Target-task human demonstrations use Wuji gloves, wrist D435 cameras and wrist trackers. Ninety trajectories per task are retargeted into the robot’s pose/joint/wrench representation. They are new task-specific data rather than a sample of the large pretraining corpus; this interface description specifies two wrist views, unlike the robot’s three-view input.

Robot pretraining data

The 100-hour T-Rex robot corpus comes from seven-DoF Dexmate Vega-1 arms with the same hand/sensor family. End-effector actions permit the same 62-dimensional representation despite different arm kinematics; this is narrower than unrestricted embodiment transfer.

Appendix D Implementation Details of Baselines

Baseline implementations preserve their public-checkpoint conventions; tactile inputs, action encoding and optimizer schedules differ.

π 0.5

π0.5 has no tactile input and a fixed 32-dimensional action head. Each hand’s 22 joints are compressed to ten PCA coordinates (96–97% variance), while arm poses use six delta coordinates. Full fine-tuning uses 20k steps, batch 32 and EMA 0.999. Its representation and sensing differ materially from FACT’s full 62-dimensional tactile setup.

T-Rex

T-Rex uses its recommended mid-trained tactile checkpoint, the same 62-dimensional action and 16-step chunk as FACT, and ten-fingertip wrench histories. Its deformation-image branch is disabled because those images are unavailable; FLARE stays enabled. Ten denoising steps split at six, with tactile dropout 0.1 and 30-epoch fine-tuning.

FACT ablations

The force variant removes future-force tokens and loss but retains observed touch; the track variant removes track tokens, heads and losses. Both receive pretraining and fine-tuning. In the human-without-force recipe, human tactile inputs are masked and force supervision disabled; robot-only pretraining also removes the 2,000-hour human corpus.

Appendix E Evaluation Tasks and Additional Results

The original task figures retain five-frame sequences, exact instructions and success criteria. Pipetting requires transferring liquid, ejecting the tip and putting the tool down without drops; cap tasks require opened caps and upright returned tubes; shoelaces require a released two-loop bow; cards require a merged deck with no drops; multi-object grasp requires holding all three objects simultaneously before depositing them; screwdriver success requires screw removal and both objects returned. The screwdriver sequence is teleoperated, not an autonomous success demonstration. Musical tasks count clean individual notes rather than a melody; volume adjustment separates engagement from conditional error.

Original A5.F10 native PDF artwork with complete original panels, labels and layout.
Fig. 10: Full input modalities of FACT on Task 10: Adjust Pipette Volume. Text Instruction: “A pipette set to 420 μ\muL is on the table. Hold the pipette body with your left hand and turn the volume dial with your right thumb and index finger until the display reads 400 μ\muL.” Shown at one instant of an autonomous rollout while the dial is being turned: the head-camera view, the left and right ZED Mini wrist views, and the ten fingertip wrenches (rows R and L, one panel per finger; the dot gives the tangential force and the color the normal force), where the right thumb and index contacts on the dial appear while the left hand holds the pipette body. Together with the robot state, these are all the inputs the policy receives.

Failure modes

Observed failures include missed lace contact followed by useless open-loop motions, incorrect plunger/cap/card force, overly weak or damaging musical contact, screwdriver cam-out, and loss of earlier objects during regrasping. Half the volume trials never engage the dial, despite good precision in most engaged trials. These qualitative examples help interpret the ablations but are not independent causal tests of every failure mechanism.

Original A5.F11 native PDF artwork with complete original panels, labels and layout.
Fig. 11: Task 1: Lab Pipetting.

Task 1: Lab Pipetting. Text Instruction: “An orange rack with a large tube and a small tube stands on the table, with a pipette, a box of tips, and a beaker beside it. Pick up the pipette with your right hand and press it into a tip. Lift the large tube from the rack with your left hand, draw liquid from it, and put the tube back. Then dispense the liquid into the small tube, eject the tip into the beaker, and put the pipette back down.” The robot must press the pipette into a tip with the right hand, lift the large tube with the left hand and hold it steady, aspirate by pressing and releasing the plunger with the right index finger, return the tube to the rack, dispense into the small tube, eject the tip into the beaker, and put the pipette back down. A rollout succeeds only if liquid reaches the small tube, the tip is ejected, and the pipette is set back down, without dropping the pipette or the tube.

Original A5.F12 native PDF artwork with complete original panels, labels and layout.
Fig. 12: Task 2: Open Test-Tube Cap.

Task 2: Open Test-Tube Cap. Text Instruction: “An orange rack holding capped test tubes stands on the table in front of you. Pick up one tube with your left hand, unscrew its cap with your right hand, and place the tube back in the rack with your left hand.” The robot must lift a capped tube from the rack with the left hand, unscrew the cap with the right thumb and index finger while the left hand holds the tube against the twist, and return the tube to the rack. A rollout succeeds only if the cap is fully separated and the tube stands in the rack at the end.

Original A5.F13 native PDF artwork with complete original panels, labels and layout.
Fig. 13: Task 3: Open Microcentrifuge Flip Cap.

Task 3: Open Microcentrifuge Flip Cap. Text Instruction: “A blue rack holding microcentrifuge tubes sits at the front edge of the table. Pick up one tube with your left hand, pass it to your right hand, and flip its hinged cap open with your left thumb; then return the tube to the rack.” The robot must lift a tube from the rack with the left hand, pass it to the right hand, hook the left thumb under the lip of the cap and flip it open, and return the tube to the rack. A rollout succeeds only if the cap is fully open and the tube stands upright in the rack.

Original A5.F14 native PDF artwork with complete original panels, labels and layout.
Fig. 14: Task 4: Tie Shoelaces.

Task 4: Tie Shoelaces. Text Instruction: “A shoe with its laces untied sits on a board before you. Using both hands, pick up the two laces, cross and tighten them, form a loop with one lace, wrap the other lace around it, pull it through to make a second loop, and tighten the bow.” The robot must cross and tighten the laces, form a loop, wrap the other lace around it, and pull both loops tight. A rollout succeeds only if a two-loop bow stays tied after the hands release.

Original A5.F15 native PDF artwork with complete original panels, labels and layout.
Fig. 15: Task 5: Shuffle Cards.

Task 5: Shuffle Cards. Text Instruction: “Two halves of a deck of playing cards lie face down on a board in front of you. Pick up one half in each hand, bend the halves with your thumbs and riffle them together, then push the deck square.” The robot must lift one half in each hand, bend both halves with the thumbs, release the cards so that they interleave, and push the deck square. A rollout succeeds only if the two halves merge into a single deck with no card dropped.

Original A5.F16 native PDF artwork with complete original panels, labels and layout.
Fig. 16: Task 6: Multi-Object Grasp.

Task 6: Multi-Object Grasp. Text Instruction: “A basket, a plate, a toy shovel, a toy hammer, and a ping-pong ball are on the table. With your right hand, pick up the toy shovel, then the toy hammer, then the ping-pong ball without releasing the objects already in hand, and place them into the basket.” The robot must pick up the shovel, then the hammer and the ball with the same hand without releasing what it already holds, and place them into the basket. A rollout succeeds only if all three objects are held at the same time and end up in the basket.

Original A5.F17 native PDF artwork with complete original panels, labels and layout.
Fig. 17: Task 7: Use Screwdriver.

Task 7: Use Screwdriver. Text Instruction: “A Wuji Hand lies on the left of the table and a screwdriver on the right. Hold the Wuji Hand up with your left hand, base facing you, pick up the screwdriver with your right hand, seat its tip in the central screw, and turn it counterclockwise until the screw is out; then put both back on the table.” The robot must hold the Wuji Hand steady, seat the screwdriver tip in the screw with light axial pressure, and turn it counterclockwise with repeated regrasps while the left hand resists the torque. A rollout succeeds only if the screw is removed and both objects are back on the table.

Original A5.F18 native PDF artwork with complete original panels, labels and layout.
Fig. 18: Task 8: Play Harp.

Task 8: Play Harp. Text Instruction: “A harp stands in front of you. With your right hand, hook a fingertip behind each of the ten indicated strings in turn and pluck it so that it sounds.” The robot must bring the right hand to the strings and, for each of the ten indicated strings, hook a fingertip behind it and release it with enough force to make the string ring cleanly, neither muting it nor snapping it. The score is the number of the ten notes that sound cleanly.

Original A5.F19 native PDF artwork with complete original panels, labels and layout.
Fig. 19: Task 9: Play Xylophone.

Task 9: Play Xylophone. Text Instruction: “A xylophone lies open on the table with a mallet beside it. Pick up the mallet with your right hand and strike each of the ten indicated bars in turn so that it rings.” The robot must pick up the mallet with the right hand, hold it in a stable grip, and strike each of the ten indicated bars with enough force to ring cleanly, neither a muted tap nor a bounced double hit. The score is the number of the ten notes that sound cleanly.

Original A5.F20 native PDF artwork with complete original panels, labels and layout.
Fig. 20: Task 10: Adjust Pipette Volume.

Task 10: Adjust Pipette Volume. Text Instruction: “A pipette set to 420 μ\muL is on the table. Hold the pipette body with your left hand and turn the volume dial with your right thumb and index finger until the display reads 400 μ\muL.” The robot must hold the pipette body with the left hand, roll the volume dial with the right thumb and index fingertip in small increments from 420 toward 400 μ\muL, and stop when the display reads 400. A trial counts as engaged if the dial is turned; we report the engaged trials of ten and, over them, the mean absolute error of the final reading from 400 μ\muL.

Concept demonstration

Open image ↗