Overview
This edition follows the 16-page arXiv v1 and retains all 20 figures, four tables, five numbered equations and five appendices. The original HTML omitted Figure 1; it is restored from the PDF. Read the results as task-specific adaptation on one physical station, using ten robot and 90 newly collected human demonstrations per task. Binary success, clean musical notes and pipette-dial engagement/conditional error are different metrics.
At a glance
FACT learns from 2,000 hours of tactile human demonstrations plus 100 hours of robot data. A coarse model jointly predicts video, point tracks and actions; a second model co-denoises remaining actions and future fingertip wrenches while incorporating fresh touch. Ten-task hardware tests favor the full system over the two baselines, but binary success averages 41.4% and shoelace success is 20%. Gains reach 60 percentage points on test-tube opening. The data-ratio studies support useful human transfer in these tasks, not interchangeable human and robot data in general.
I Introduction
Human demonstrations scale more easily than robot teleoperation but usually lack contact measurements. EgoForce adds calibrated tactile observations; FACT combines prospective visual motion with force-conditioned refinement. Shared future video/tracks support transfer across embodiments, while continuous force prediction couples anticipated contact with action. The evaluated transfer includes substantial human–robot pretraining and task-specific fine-tuning, rather than zero-shot deployment from human video alone.
II Related Work
Related work uses egocentric video, tactile representations, action hierarchies and predicted visual motion. T-Rex refines coarse actions from quantized force history; FACT also retains quantized history but adds continuous current-force input and explicitly co-predicted future force. Its coarse visual/action/track prediction follows ideas in TrAct, with a Wan-based video transformer. This combination is tested empirically rather than proven embodiment invariant.
III The EgoForce Dataset
EgoForce pairs synchronized egocentric views, estimated hand motion and calibrated pressure-derived contact labels for pretraining.
Scalable multimodal collection platform
A head camera, two wrist cameras and bimanual tactile gloves record aligned vision, hand movement and touch. The hardware aims to allow natural manipulation across diverse tasks and environments.
Automatic force calibration
A pneumatic bladder sweeps approximately uniform pressure from 0 to 50 psi while a reference sensor and each taxel record synchronously. Pressure times effective sensing area gives normal force; per-taxel A/D mappings calibrate gloves and sessions. This does not directly measure shear or a complete six-axis wrench.
The EgoForce dataset
The reported dataset contains about 2,000 hours, 40,000 episodes, 1,000 tasks, 120 scenes and 1,000 demonstrators. Appendix C explains inferred human action labels and the pressure-to-wrench approximation.
Task distribution: dexterous by construction
Deformable manipulation, assembly/construction and storage/organization account for 21.0%, 17.3% and 14.8%. Other fine manipulation raises the contact-rich share above two thirds. Frequent fold, regrasp and thread annotations motivate tactile supervision, but the distribution is intentionally dexterity-focused rather than a representative sample of all household activity.
IV The FACT Framework
Pretraining mixes EgoForce with 100 hours of T-Rex robot teleoperation. At inference, visual/state/language/track/tactile context produces 16 actions at 20 Hz and four 5 Hz future-force slots for ten fingertips. The 62-dimensional action contains each arm’s relative end-effector pose and 22 absolute hand joints. Predicted force shapes the action representation; it is not sent as a force-control reference.
IV-A Track-Guided Coarse Planner
The block-causal coarse planner jointly generates future video, image-space tracks and a coarse action chunk. Its visual targets supply a common prediction interface for different embodiments.
Multimodal representation
Text and anchor-image embeddings enter through cross-attention. State, past tracks and conditioning video are clean context; future video, action and tracks are noisy prediction targets in the temporal self-attention sequence. Text/CLIP features remain outside that sequence.
Per-token flow matching
Flow matching interpolates clean targets with Gaussian noise; each token carries its own noise level, with zero clean and one noise. Generated temporal blocks draw Beta(1.5,1) levels; conditioning tokens stay clean. Per-token adaptive normalization permits the action pathway to stop partially denoised and continue in the refiner.
Equation 1
| (1) |
Attention topology
Within a causal temporal window, observation/action tokens attend to current and earlier visual/action/track tokens plus the current state. State remains local, while track tokens attend only to tracks. Separate heads decode video, actions and track motion; text/image embeddings supply cross-attention conditioning.
Training objective
The preserved objective sums video and action flow losses with a track objective weighted by 20. Track terms combine flow, reconstructed position and smoothness, with visibility masking and stronger weighting for moving points. The one-step reconstruction provides the position/smoothness target estimate.
Equations 2–3
| (2) |
| (3) |
Coarse handoff to the Force–Action Joint Refiner
Six Euler steps reduce noise from 1 to 0.4, followed by an additional forward pass at 0.4. Video and tracks are read out, while partially denoised actions pass to the refiner. Observation/action hidden states are detached and projected once into cached cross-attention memory.
IV-B Force–Action Joint Refiner
The second stage jointly predicts the remaining action and continuous future force, conditioned on current touch, quantized history and the fixed coarse plan.
Force–action joint refinement
Current measured wrenches are clean observations; future-force tokens begin as noise. Bidirectional attention lets evolving actions and predicted contact influence one another throughout denoising. This is learned coupled prediction, not an analytic dynamics model or enforcement of contact-force constraints.
Coarse memory through cached cross-attention
Only observation/action hidden states are retained from the coarse handoff; track/state/conditioning tokens are discarded from memory. Cached projections are reused across refiner steps and tactile refreshes, avoiding a new expensive coarse pass each time.
Cascaded sampling
Four Euler steps take action noise from 0.4 to zero and force noise from one to zero. Only the unexecuted action suffix and unobserved force slots change. New 5 Hz touch measurements clamp observed slots before another suffix refinement.
Training objective
The refiner trains action-suffix flow error plus future-force error weighted 0.5 and vector-quantization loss. Its detached coarse memory is evaluated at the exact 0.4 handoff using the same action-noise draw, aligning training with the cascade rather than treating independently sampled plans as equivalent.
Equation 4
| (4) |
IV-C Implementation Details
The planner has a 30-block, width-3072 Wan2.2 backbone; the refiner has six width-768 blocks. A 0.8-second temporal block carries 617 tokens, with four teacher-forced blocks during training. Explicit appendix details clarify that Wan is pretrained and fine-tuned, while new projections/refiner/codecs learn from scratch and VAE/text/CLIP encoders stay frozen. The main-text “all components from scratch” wording should not be read as replacing that initialization recipe.
| FACT (ours) | ||||||
| # | Task | T-Rex | w/o force pred | w/o track pred | full | |
| Binary tasks: success rate (%) | ||||||
| 1 | Lab pipetting | 0 | 0 | 30 | 20 | 40 |
| 2 | Open test-tube cap | 10 | 20 | 40 | 80 | 80 |
| 3 | Open microcentrifuge flip cap | 0 | 20 | 20 | 40 | 50 |
| 4 | Tie shoelaces | 0 | 0 | 0 | 20 | 20 |
| 5 | Shuffle cards | 0 | 10 | 0 | 20 | 30 |
| 6 | Multi-object grasp | 0 | 0 | 10 | 40 | 40 |
| 7 | Use screwdriver | 0 | 10 | 0 | 30 | 30 |
| Mean | 1.4 | 8.6 | 14.3 | 35.7 | 41.4 | |
| Continuous tasks: clean notes of 10 | ||||||
| 8 | Play harp | 0.5 | 2.1 | 0 | 6.5 | 6.7 |
| 9 | Play xylophone | 0 | 3.2 | 1.9 | 3.9 | 5.4 |
| Volume adjustment: engaged trials of 10 | ||||||
| 10 | Pipette volume | 1 | 2 | 2 | 4 | 5 |
| mean abs. error (L) | 17.0 | 14.0 | 11.0 | 12.8 | 6.4 | |
IV-D Training Recipe
The joint objective trains coarse and refinement pathways on an hours-proportional 2,000:100 human/robot mixture using 240 T-Head PPUs. Human tactile labels supervise the model from the beginning. Table IV retains optimizer, schedule, batch size and precision; the paper does not turn this compute description into a demonstrated low-cost pretraining recipe.
Equation 5
| (5) |
V Experiments
Experiments cover ten tasks on a single bimanual station, with architecture, pretraining and human-data-ratio ablations.
| # | Task | human w/ force+robot (full) | human w/o force+robot | robot only |
| Binary tasks: success rate (%) | ||||
| 1 | Lab pipetting | 40 | 20 | 0 |
| 2 | Open test-tube cap | 80 | 0 | 20 |
| 3 | Open microcentrifuge flip cap | 50 | 10 | 0 |
| 4 | Tie shoelaces | 20 | 0 | 0 |
| 5 | Shuffle cards | 30 | 10 | 0 |
| 6 | Multi-object grasp | 40 | 40 | 30 |
| 7 | Use screwdriver | 30 | 10 | 0 |
| Continuous tasks: clean notes of 10 | ||||
| 8 | Play harp | 6.7 | 2.0 | 1.2 |
| 9 | Play xylophone | 5.4 | 2.2 | 1.0 |
| Volume adjustment: engaged trials of 10 | ||||
| 10 | Pipette volume | 5 | 2 | 0 |
| mean abs. error (L) | 6.4 | 20.5 | – | |
V-A Experimental Setup
Every task gets its own 100-trajectory fine-tuning mixture and ten randomized evaluation rollouts.
Robot platform and data collection
Two YAM arms carry Sharpa Wave hands with 22 joints and five six-axis fingertips each. Policies observe head and wrist cameras plus state; tactile-aware models also use fingertip wrenches. Ten successful robot demonstrations and 90 human demonstrations are collected separately for each evaluation task.
Skill-specific fine-tuning
The pretrained model is fine-tuned separately for each task on eight GPUs. Human fine-tuning demonstrations are collected on the target tasks with Wuji gloves and wrist trackers, not drawn from EgoForce.
Baselines and ablations
π0.5 and T-Rex start from their released checkpoints and use the same task demonstration mixture. They differ in tactile access, action encoding and training recipe, so this is a complete-system comparison. FACT variants remove force-prediction or track pathways and associated losses.
Evaluation protocol and metrics
Seven tasks use binary success over ten rollouts. Harp and xylophone score clean notes out of ten per rollout, not whole-task success percentages. Pipette adjustment reports dial engagements out of ten and absolute volume error only among engagements. Scenes and objects are fixed across methods; placements vary. Never-engaged trials are failures with no conditional reading.
| robot : human data | |||
| # | Task | 10 : 0 | 10 : 90 |
| Binary tasks: success rate (%) | |||
| 1 | Tie shoelaces | 10 | 20 |
| 2 | Shuffle cards | 10 | 30 |
| Volume adjustment: engaged trials of 10 | |||
| 3 | Pipette volume | 4 | 5 |
| mean abs. error (L) | 11.2 | 6.4 | |
V-B Main Results
FACT reaches 40% pipetting, 80% test-tube opening and 40% multi-object grasping, compared with T-Rex’s 0%, 20% and 0%. The seven binary tasks average 41.4% versus 8.6% for T-Rex; remaining failures are substantial. Harp scores 6.7 clean notes versus 2.1. Dial engagement succeeds in five trials with 6.4 μL conditional mean error; four of those five are within 4 μL. These small-sample results distinguish reaching useful contact from precise control after contact.
V-C Ablation Studies
The ablations examine predictive pathways, tactile human pretraining and the amount/composition of fine-tuning data.
Force and track prediction
Removing force prediction cuts test-tube success from 80% to 40% and harp performance from 6.7 to zero notes. Removing tracks lowers pipetting from 40% to 20% and xylophone from 5.4 to 3.9 notes, but leaves some tasks unchanged. The interventions remove tokens/heads/losses together, so they do not isolate an auxiliary loss from all architectural effects.
Force supervision in human pretraining data
Without human force supervision, test-tube opening falls to zero, pipetting to 20%, and dial engagement to two trials with 20.5 μL error; multi-object grasping remains 40%. That variant also masks human tactile inputs, so it combines removing observations with removing force targets. Robot-only pretraining loses the much larger human dataset and performs poorly. These comparisons support the full data recipe without equalizing all pretraining information or volume.
Scaling with pretraining human data
Across 100, 500, 1,000, 1,500 and 2,000 human hours, Figure 7 reports lower validation MSE with tactile supervision: 0.042 versus 0.047 at 100 hours and 0.031 versus 0.036 at 2,000. The 10–14% relative loss advantage persists. The figure uses a linear hour axis; this validation-loss trend is not a measured robot-success scaling law or proof of asymptotic behavior.
Human data in fine-tuning
Adding 90 human trajectories to ten robot trajectories improves shoelaces from 10% to 20%, cards from 10% to 30%, and dial engagement from four to five trials while conditional error falls from 11.2 to 6.4 μL. This comparison increases total fine-tuning examples from ten to 100; it demonstrates useful additional human data rather than equal sample efficiency by itself.
Human-demonstration efficiency in fine-tuning
At fixed 100-trajectory totals, changing robot:human ratios from 10:90 to 50:50 gives modest gains on the three plotted tasks: flip caps 50→60%, multi-object grasping 40→50%, dial engagement 5→6. Conditional volume errors are 6.4, 5.2 and 6.1 μL. Ten-rollout estimates suggest useful substitution in these settings, with limited evidence for a general human/robot equivalence claim.
VI Conclusion
The study demonstrates beneficial tactile human pretraining and coupled visual/force prediction for dexterous adaptation. Limits evident from the protocol include one physical setup, ten rollouts per task, substantial residual failures, task-specific human demonstrations and incomplete human wrench observability. It does not establish safety constraints, broad object/scene generalization or force-reference tracking.
Acknowledgements
The authors attribute the 2,000-hour collection system to TouchTronix FusionX and acknowledge hardware donations from Sharpa and Xdof. These collection/hardware credits are retained as provenance.
Appendix A Model and Training Details
Wan2.2-TI2V-5B is fully fine-tuned; new modality projections and the refiner train from scratch. VAE, umT5-XXL and CLIP remain frozen. Table IV records the 5.05B planner, 65M refiner and training configuration.
| Track-Guided Coarse Planner (Flow Matching) | |
| Backbone | Wan2.2-TI2V-5B video DiT |
| Hidden dimension | 3072 |
| Number of layers | 30 |
| Max sequence length | 2,518 tokens |
| Action dimension | 62 |
| Action chunk | 16 |
| Timestep sampling | |
| Num inference timesteps | 6 () |
| Parameters | 5.05B |
| Force–Action Joint Refiner (Flow Matching) | |
| Hidden dimension | 768 |
| Number of layers | 6 |
| FFN intermediate size | 3072 |
| Action dimension | 62 |
| Action chunk | 16 |
| Num inference timesteps | 4 |
| Parameters | 65M |
| Training configuration | |
| Optimizer | AdamW |
| Peak learning rate | |
| Min learning rate | 0 |
| LR scheduler | cosine |
| Warmup ratio | 0.05 |
| Weight decay | |
| Gradient clipping | 1.0 |
| GPU type | Alibaba T-Head PPU (Zhenwu 810E, 96 GB HBM2e) |
| Number of GPUs | 240 |
| o DeepSpeed ZeRO stage | 2 |
| Per device batch size | 16 |
| Gradient accumulation steps | 1 |
| Mixed precision training | bf16 |
| Tactile dropout | 0.1 |
Visual tracks
Offline SAM2/CoTracker processing yields 250 points per block: 100 hand points in the head view and 75 points per wrist view, including background grids. A 16-frame history encodes position, visibility and displacement; future heads predict 16 position velocities. Visibility masks and path-length weight 1+4ℓ emphasize observable motion. The text states offline extraction but does not detail a complete online tracker latency pipeline.
Data pipeline
LeRobot-v2 recordings at 30 fps are timestamp-resampled to actions/tracks at 20 Hz, video at 10 Hz and force at 5 Hz. Each arm uses three relative position coordinates, six relative rotation coordinates and 22 absolute hand joints. Percentile normalization maps values to [−1,1]. Three 160×320 camera views tile into a 2×2 image before VAE encoding.
Inference
Inference uses six coarse Euler updates plus an extra handoff forward pass, then four refinement updates, without classifier-free guidance. Actions execute at 20 Hz. The paper provides these algorithmic rates but no measured end-to-end deployment latency guarantee.
Appendix B Force–Action Joint Refiner
The refiner preserves a coarse plan while repeatedly updating its unexecuted suffix from new contact information.
Tactile input
Ten fingertips supply 60 wrench channels. Every 5 Hz refresh uses 16 history samples spanning 3.2 seconds; unavailable early samples are zeroed and masked. Current force is the history window’s last sample.
Force-history quantization
A shared temporal convolution encoder compresses each fingertip history into a 256-dimensional code. The 64-entry codebook uses EMA decay 0.99 and periodically reseeds dead codes. Reconstruction plus commitment loss trains the quantizer; hand/finger embeddings identify ten history tokens. Thus history remains quantized despite the continuous future-force pathway.
Token layout
The 76-token refiner combines 16 actions, ten current-force tokens, ten history tokens and 40 future-force tokens. Bidirectional attention and cached coarse cross-attention predict action/force flow velocities. The appendix gives 64-channel action projections/heads while the physical action and table are 62-dimensional; it does not explicitly explain that padding or adaptation, so the edition preserves the distinction.
Coarse memory
Observation/action hidden states at noise 0.4 are detached and projected to width 768. The appendix describes current plus up to four preceding blocks, whereas the main text describes a four-block causal window. These context-window descriptions are retained as a source ambiguity rather than silently reconciled.
Training
Four training blocks times offsets {0,4,8,12} create 16 refiner instances per sample. Executed prefixes and observed force slots remain clean; only remaining actions/unobserved force receive noise and loss. Action noise is restricted to the last 0.4 of the schedule, with a small floor. Current/history tactile dropout is 0.1 per sample.
Sampling and execution
At offsets 4, 8 and 12, new measured contact is clamped and the remaining chunk is refined again against cached coarse states. The 20 Hz action sequence is therefore refreshed using 5 Hz tactile updates. Predicted future wrenches are logged for analysis and explicitly are not control references.
Appendix C Hardware and Data Collection
Pretraining human data, target-task human data, robot teleoperation and robot pretraining use different capture systems whose labels are mapped into a shared representation.
EgoForce collection (pretraining data)
FusionX combines a calibrated 30 fps stereo head pair, two wrist RGB cameras and TouchTronix gloves sampled near 200 Hz. Each glove has 12 taxels per fingertip plus 60 palm taxels, bend sensors and an orientation IMU. Per-user rest offsets/bend ranges and pneumatic pressure calibration accompany common-clock episode recordings.
Human action and force labels
Human hand motion is estimated from ViTPose/HaMeR camera fits, smoothed and scaled with stereo depth; glove bend/orientation cues assist occlusions. Retargeting maps hands to Sharpa’s 22 joints. Summed normal taxel forces and center of pressure yield normal force and two moments; unobserved tangential forces and twisting moment are set to zero. These approximate six-channel labels are not directly measured full human wrenches.
Robot platform
The evaluation station uses two six-DoF YAM Ultra arms and Sharpa Wave 1 hands. End-effector targets pass through inverse kinematics to built-in joint controllers; hand joints are absolute targets. Fingertip wrenches arrive near 30 Hz and are resampled for the model. Three RGB views and state complete the observations.
Teleoperation
Robot teleoperation uses Manus gloves and VIVE wrist trackers; retargeted fingers and inverse-kinematics arm commands are logged alongside measured state/touch and synchronized cameras at 30 Hz. The dataset retains ten successful trajectories per target task.
Human demonstrations for fine-tuning
Target-task human demonstrations use Wuji gloves, wrist D435 cameras and wrist trackers. Ninety trajectories per task are retargeted into the robot’s pose/joint/wrench representation. They are new task-specific data rather than a sample of the large pretraining corpus; this interface description specifies two wrist views, unlike the robot’s three-view input.
Robot pretraining data
The 100-hour T-Rex robot corpus comes from seven-DoF Dexmate Vega-1 arms with the same hand/sensor family. End-effector actions permit the same 62-dimensional representation despite different arm kinematics; this is narrower than unrestricted embodiment transfer.
Appendix D Implementation Details of Baselines
Baseline implementations preserve their public-checkpoint conventions; tactile inputs, action encoding and optimizer schedules differ.
π 0.5
π0.5 has no tactile input and a fixed 32-dimensional action head. Each hand’s 22 joints are compressed to ten PCA coordinates (96–97% variance), while arm poses use six delta coordinates. Full fine-tuning uses 20k steps, batch 32 and EMA 0.999. Its representation and sensing differ materially from FACT’s full 62-dimensional tactile setup.
T-Rex
T-Rex uses its recommended mid-trained tactile checkpoint, the same 62-dimensional action and 16-step chunk as FACT, and ten-fingertip wrench histories. Its deformation-image branch is disabled because those images are unavailable; FLARE stays enabled. Ten denoising steps split at six, with tactile dropout 0.1 and 30-epoch fine-tuning.
FACT ablations
The force variant removes future-force tokens and loss but retains observed touch; the track variant removes track tokens, heads and losses. Both receive pretraining and fine-tuning. In the human-without-force recipe, human tactile inputs are masked and force supervision disabled; robot-only pretraining also removes the 2,000-hour human corpus.
Appendix E Evaluation Tasks and Additional Results
The original task figures retain five-frame sequences, exact instructions and success criteria. Pipetting requires transferring liquid, ejecting the tip and putting the tool down without drops; cap tasks require opened caps and upright returned tubes; shoelaces require a released two-loop bow; cards require a merged deck with no drops; multi-object grasp requires holding all three objects simultaneously before depositing them; screwdriver success requires screw removal and both objects returned. The screwdriver sequence is teleoperated, not an autonomous success demonstration. Musical tasks count clean individual notes rather than a melody; volume adjustment separates engagement from conditional error.
Failure modes
Observed failures include missed lace contact followed by useless open-loop motions, incorrect plunger/cap/card force, overly weak or damaging musical contact, screwdriver cam-out, and loss of earlier objects during regrasping. Half the volume trials never engage the dial, despite good precision in most engaged trials. These qualitative examples help interpret the ablations but are not independent causal tests of every failure mechanism.
Task 1: Lab Pipetting. Text Instruction: “An orange rack with a large tube and a small tube stands on the table, with a pipette, a box of tips, and a beaker beside it. Pick up the pipette with your right hand and press it into a tip. Lift the large tube from the rack with your left hand, draw liquid from it, and put the tube back. Then dispense the liquid into the small tube, eject the tip into the beaker, and put the pipette back down.” The robot must press the pipette into a tip with the right hand, lift the large tube with the left hand and hold it steady, aspirate by pressing and releasing the plunger with the right index finger, return the tube to the rack, dispense into the small tube, eject the tip into the beaker, and put the pipette back down. A rollout succeeds only if liquid reaches the small tube, the tip is ejected, and the pipette is set back down, without dropping the pipette or the tube.
Task 2: Open Test-Tube Cap. Text Instruction: “An orange rack holding capped test tubes stands on the table in front of you. Pick up one tube with your left hand, unscrew its cap with your right hand, and place the tube back in the rack with your left hand.” The robot must lift a capped tube from the rack with the left hand, unscrew the cap with the right thumb and index finger while the left hand holds the tube against the twist, and return the tube to the rack. A rollout succeeds only if the cap is fully separated and the tube stands in the rack at the end.
Task 3: Open Microcentrifuge Flip Cap. Text Instruction: “A blue rack holding microcentrifuge tubes sits at the front edge of the table. Pick up one tube with your left hand, pass it to your right hand, and flip its hinged cap open with your left thumb; then return the tube to the rack.” The robot must lift a tube from the rack with the left hand, pass it to the right hand, hook the left thumb under the lip of the cap and flip it open, and return the tube to the rack. A rollout succeeds only if the cap is fully open and the tube stands upright in the rack.
Task 4: Tie Shoelaces. Text Instruction: “A shoe with its laces untied sits on a board before you. Using both hands, pick up the two laces, cross and tighten them, form a loop with one lace, wrap the other lace around it, pull it through to make a second loop, and tighten the bow.” The robot must cross and tighten the laces, form a loop, wrap the other lace around it, and pull both loops tight. A rollout succeeds only if a two-loop bow stays tied after the hands release.
Task 5: Shuffle Cards. Text Instruction: “Two halves of a deck of playing cards lie face down on a board in front of you. Pick up one half in each hand, bend the halves with your thumbs and riffle them together, then push the deck square.” The robot must lift one half in each hand, bend both halves with the thumbs, release the cards so that they interleave, and push the deck square. A rollout succeeds only if the two halves merge into a single deck with no card dropped.
Task 6: Multi-Object Grasp. Text Instruction: “A basket, a plate, a toy shovel, a toy hammer, and a ping-pong ball are on the table. With your right hand, pick up the toy shovel, then the toy hammer, then the ping-pong ball without releasing the objects already in hand, and place them into the basket.” The robot must pick up the shovel, then the hammer and the ball with the same hand without releasing what it already holds, and place them into the basket. A rollout succeeds only if all three objects are held at the same time and end up in the basket.
Task 7: Use Screwdriver. Text Instruction: “A Wuji Hand lies on the left of the table and a screwdriver on the right. Hold the Wuji Hand up with your left hand, base facing you, pick up the screwdriver with your right hand, seat its tip in the central screw, and turn it counterclockwise until the screw is out; then put both back on the table.” The robot must hold the Wuji Hand steady, seat the screwdriver tip in the screw with light axial pressure, and turn it counterclockwise with repeated regrasps while the left hand resists the torque. A rollout succeeds only if the screw is removed and both objects are back on the table.
Task 8: Play Harp. Text Instruction: “A harp stands in front of you. With your right hand, hook a fingertip behind each of the ten indicated strings in turn and pluck it so that it sounds.” The robot must bring the right hand to the strings and, for each of the ten indicated strings, hook a fingertip behind it and release it with enough force to make the string ring cleanly, neither muting it nor snapping it. The score is the number of the ten notes that sound cleanly.
Task 9: Play Xylophone. Text Instruction: “A xylophone lies open on the table with a mallet beside it. Pick up the mallet with your right hand and strike each of the ten indicated bars in turn so that it rings.” The robot must pick up the mallet with the right hand, hold it in a stable grip, and strike each of the ten indicated bars with enough force to ring cleanly, neither a muted tap nor a bounced double hit. The score is the number of the ten notes that sound cleanly.
Task 10: Adjust Pipette Volume. Text Instruction: “A pipette set to 420 L is on the table. Hold the pipette body with your left hand and turn the volume dial with your right thumb and index finger until the display reads 400 L.” The robot must hold the pipette body with the left hand, roll the volume dial with the right thumb and index fingertip in small increments from 420 toward 400 L, and stop when the display reads 400. A trial counts as engaged if the dial is turned; we report the engaged trials of ten and, over them, the mean absolute error of the final reading from 400 L.