Literature
Display
← All papers

I3L: Iterative Interactive Imitation Learning for Whole-Body Mobile Manipulation

Wensi Ai · Christopher Luey · Karthik Dharmarajan · Evans Han · Yunfan Jiang · Stef Ren · Jianwen Xie · Jiajun Wu · Li Fei-Fei · Ruohan Zhang

Preprint posted · arXiv (preprint) ↗

Robotics18 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows arXiv v2 (29 October 2026), retaining all eight figures, 17 tables, 14 displayed equations and six implementation appendices. Read the headline gains as results from eight specified tasks under equal additional trajectory counts. Active takeover time is measured, but total supervision time and independent training-seed uncertainty are not.

Refer to caption
Fig. 1: I3L combines interactive correction with iterative policy refinement. Left: 1) the policy approaches a failure, prompting the operator to take over with JoyLo+ and provide targeted corrections, 2) the operator uses JoyLo+ to take over during autonomous execution. Center: corrections are aggregated with prior demonstrations for policy retraining and redeployment. Right: eight evaluation tasks across three embodiments.

At a glance

I3L combines motorized, touch-triggered whole-body teleoperation with repeated corrective imitation learning. Four rounds of human-gated corrections raise mean autonomous success from 25.6% to 93.9% across eight tasks, 35.6 percentage points above adding the same number of demonstrations. Every final task rate is at least 90%. Mean active takeover duration falls 72% across collection rounds. These results use human-triggered scripted corrections in simulation and physical JoyLo+ leaders on real robots; the three-operator study excludes failed collection attempts.

I Introduction

Long manipulation sequences amplify distribution shift: a small error in the arm, torso or base can invalidate an otherwise competent policy. Corrections target such policy-visited failures instead of replaying already-mastered behavior. I3L contributes an integrated interface and repeated collection/retraining protocol, then examines reliability and intervention burden on four real mobile tasks plus two fixed-arm and two simulated tasks. It builds on DAgger and human-gated DAgger; its evidence concerns an implemented whole-body system rather than a new guarantee against compounding error.

Refer to caption
Fig. 2: JoyLo+ correction interface. Motorized arm leaders track policy targets during autonomous execution and switch to compliant current control during operator takeover. Capacitive touch sensors trigger takeover, while JoyCon inputs control the torso, mobile base, and grippers. The interface supports both single-arm and dual-arm configurations.

II Related Work

Motorized leader interfaces keep the operator aligned with autonomous commands; touch sensing makes takeover practical. Prior interactive-learning methods differ in intervention triggers and treatment of autonomous, corrective and pre-intervention data. I3L combines these interface ideas with comparisons of collection schedules and learning rules. Its mobile experiments extend earlier corrective manipulation systems, while algorithm comparisons are concentrated in simulation rather than performed across all physical tasks.

III Method

Deploy a policy, label operator takeovers, merge useful corrective segments with demonstrations, retrain, and deploy the new policy. JoyLo+ makes coordinated physical corrections possible; the learning study determines which collected segments are used.

TABLE I: Teleoperation and correction interfaces. ✓\checkmark: demonstrated; –: not demonstrated in the cited work.
System Mobile Bimanual Multi-embodiment Actuated Correction
ALOHA/2 [2, 51] – ✓\checkmark – – –
GELLO [20] – ✓\checkmark ✓\checkmark – –
Mobile ALOHA [31] ✓\checkmark ✓\checkmark – – –
TeleMoMa [32] ✓\checkmark ✓\checkmark ✓\checkmark – –
JoyLo [21] ✓\checkmark ✓\checkmark – ✓\checkmark –
UME [52] ✓\checkmark ✓\checkmark ✓\checkmark ✓\checkmark –
CR-DAgger [18] – – – – ✓\checkmark
HACTS [34] – – – ✓\checkmark ✓\checkmark
RoboCopilot [17] ✓\checkmark ✓\checkmark – ✓\checkmark ✓\checkmark
JoyLo+ (ours) ✓\checkmark ✓\checkmark ✓\checkmark ✓\checkmark ✓\checkmark

Mobile: coordinated arm and mobile-base control; torso where available. Multi-embodiment: distinct arm kinematics on real hardware. Actuated: leader motors respond to robot state, torque, or policy targets. Correction: takeover or kinesthetic guidance during autonomous rollouts. JoyLo+: R1 Pro (mobile, bimanual); R1 Pro and Franka (multi-embodiment).

III-A JoyLo+: A Hardware Interface for Whole-Body Policy Intervention

JoyLo+ has one seven-joint leader for Franka or two for R1 Pro. It follows policy targets during autonomy and permits gravity-compensated hand guidance during takeover. Table I compares demonstrated capabilities of related interfaces; a dash means a capability was not demonstrated in the cited work.

Actuated leaders for tracking and correction

Printed links and calibrated joint maps match each robot. Leader motors track commanded policy poses, then support hand-guided correction. The quoted $1,257.49 is one R1 Pro leader and listed accessories; the two-leader estimate is about $2,515 and excludes the robot and several assembly costs.

Refer to caption
Fig. 3: Evaluation tasks and initialization randomization. Representative images show initial configurations and successful task outcomes. The eight tasks span 29 annotated subtasks (2–6 per task). Across the four real-world R1 Pro tasks, task executions average approximately 83 seconds, with book storage and breakfast preparation averaging close to two minutes each. Yellow mats indicate the R1 Pro base initialization regions as well as where some task-relevant objects are randomized at the start of each episode.

Grip-integrated sensing for direct takeover

Copper grip electrodes connect an MPR121 sensor to an ESP32-S3. Separate touch/release thresholds and delayed release prevent brief contact loss from returning authority prematurely. Touch detection, serial publication and authority switching have distinct timing; takeover is not a measured zero-latency process.

Coordinated takeover and policy resumption

Either grip or active Joy-Con input transfers the entire R1 Pro command to the human: both arms, torso, base and grippers. Leaders specify arm targets and Joy-Cons control the rest. Release triggers replanning from the corrected state. The appendix additionally specifies a two-second command blend at R1 takeover entry.

III-B Intervention Detection and Recording

Each frame stores observation, executed command, counterfactual policy proposal and supervision category: demonstration, intervention, the preceding 15 frames, or autonomy. Executed commands follow the human/oracle during correction. These labels support filtering and reweighting; they are separate from manually annotated task stages.

III-C Iterative Interactive Imitation Learning

Each round changes both the dataset and the policy that generates the next collection distribution.

Data aggregation

HG-DAgger retains original demonstrations and active correction starts from every round. The preserved aggregation equation formalizes this union. For diffusion-policy chunks, filtering selects start frames; contiguous action targets can cross a takeover boundary, so an eligible chunk need not contain exclusively corrective actions.

Equation 1
𝒟r=𝒟r−1∪{(𝐨t,𝐚texec,𝐚tπ)∈ℛr:st=2}.\mathcal{D}_{r}=\mathcal{D}_{r-1}\cup\{(\mathbf{o}_{t},\mathbf{a}^{\mathrm{exec}}_{t},\mathbf{a}^{\mathrm{\pi}}_{t})\in\mathcal{R}_{r}:s_{t}=2\}. (1)

Policy retraining and redeployment

Every round restarts from the same initial weights or pretrained checkpoint and trains on the accumulated dataset. It is repeated retraining, not an incremental update to the previous trained policy. The resulting policy collects corrections against its remaining errors.

IV Experiments

The experiments separate practical whole-body deployment, precise fixed-arm manipulation and controlled simulation comparisons. They examine additional demonstrations versus corrections, single versus iterative collection, body-part involvement, learning-rule choice and variation across operators.

Original figure
Fig. 4: Intervention burden and task completion across correction rounds. In (a–c), the dark line shows the equal-weight mean across eight tasks; (b) measures the fraction of execution under intervention, and (c) measures time under intervention per episode. In (d), rounds 1–4 show takeover-free collection episodes. The final bar and dashed demo-only baseline (58.3%) compare autonomous evaluation success under the same trajectory budgets.

IV-A Experimental Setup

Because multi-round physical collection is expensive, alternative algorithms and schedules are compared on two simulated tasks. Real experiments validate the selected four-round HG-DAgger protocol rather than establish a hardware-wide ranking of every method.

Tasks and platforms

Eight tasks span 29 stages and three embodiments: simulated A1 strawberry placement and R1 fridge/radio retrieval; real Franka mug hanging and peg insertion; and real R1 breakfast preparation, book storage, microwave loading and radio activation. Simulated corrections use privileged-state cuRobo oracles selected by a human. Real mobile tasks average about 83 seconds, with breakfast and books near two minutes; initialization regions appear in Figure 3.

Evaluation protocol

Each real evaluation condition uses 25 autonomous rollouts. Simulation uses 50 rollouts for each of five evaluation seeds. Those seeds randomize evaluation of one trained checkpoint, not five independent trainings. During supervised collection, takeover-free completion is a separate outcome from the final autonomous evaluation rate.

Training

A1 and Franka use task-trained Diffusion Policy with ImageNet encoders; R1 uses pretrained π0.5. Every round trains DP for 200,000 steps or the VLA for 40,000, then evaluates the final checkpoint. Initialization is held common across rounds; dataset size and policy-induced state coverage change.

IV-B Q1: Targeted Corrections Improve Task Performance

Adding 100 correction trajectories in four rounds improves mean success by 68.3 points, versus 32.7 points from 100 extra demonstrations, a 35.6-point difference. The real mobile subset gains 81 points over base policies and 30 over extra demonstrations. Individual outcomes remain visible: fridge simulation actually falls from 39.6% to 30.4% with extra demonstrations, while correction reaches 90%. Equal trajectory counts do not establish equal frame counts, compute or total operator time.

TABLE II: Autonomous full-task success rates before and after additional data collection. Base policies receive 100 additional demonstrations or 100 corrective trajectories through four HG-DAgger rounds. Simulation rates are means across five evaluation seeds.
Task Base 100 Demos 100 Corr. (Ours)
Simulation
Pick-and-Place Strawberries 53.2% 55.6% 92.8%
Open Fridge and Pick Up Radio 39.6% 30.4% 90.0%
Real world (Franka, Diffusion Policy)
Mug Hanging 6/25 (24%) 17/25 (68%) 23/25 (092%)
Peg Insertion 7/25 (28%) 12/25 (48%) 23/25 (092%)
Real world (R1 Pro, VLA)
Prepare Make-Ahead Breakfast Bowls 1/25 (04%) 19/25 (76%) 24/25 (096%)
Boxing Books up for Storage 3/25 (12%) 13/25 (52%) 23/25 (092%)
Make Microwave Popcorn 5/25 (20%) 20/25 (80%) 25/25 (100%)
Turning on Radio 6/25 (24%) 14/25 (56%) 24/25 (096%)

IV-C Q2: Iteration Improves Performance and Reduces Intervention Burden

Iteration lets later data target the updated policy, while the burden analysis tracks active takeover during collection.

Refer to caption
Fig. 5: Task-stage intervention patterns across correction rounds. Orange curves show intervention rate over normalized episode progress, where the color and line style distinguish Rounds 1–4. Each y-axis tick interval represents 10 percentage points. Red annotations indicate failures and green annotations indicate successful behaviors.
TABLE III: Correction-learning comparison in simulation. Entries report mean ±\pm sample standard deviation of success rates (%) across five evaluation seeds. Parentheses indicate cumulative corrective trajectories. The entry with the best mean for each task is bold.
Setting Residual BC (all data) SIRIUS HG-DAgger
Open Fridge and Pick Up Radio (VLA)
Single round (100) 21.6±17.521.6\pm 17.5 79.2±13.279.2\pm 13.2 73.2±12.573.2\pm 12.5 80.0±14.880.0\pm 14.8
Round 1 (25) 16.0±2.016.0\pm 2.0 70.8±4.670.8\pm 4.6 54.8±20.954.8\pm 20.9 51.6±12.351.6\pm 12.3
Round 2 (50) 24.0±5.124.0\pm 5.1 86.4±3.886.4\pm 3.8 68.4±13.168.4\pm 13.1 72.4±6.272.4\pm 6.2
Round 3 (75) 30.0±12.230.0\pm 12.2 87.2±6.687.2\pm 6.6 70.4±16.970.4\pm 16.9 82.0±13.682.0\pm 13.6
Round 4 (100) 19.6±6.219.6\pm 6.2 88.0±3.788.0\pm 3.7 71.6±7.971.6\pm 7.9 90.0±2.4\boldsymbol{90.0\pm 2.4}
Pick-and-Place Strawberries (Diffusion Policy)
Single round (100) 45.2±2.745.2\pm 2.7 14.8±2.314.8\pm 2.3 76.0±2.076.0\pm 2.0 89.2±3.089.2\pm 3.0
Round 1 (25) 49.2±4.149.2\pm 4.1 0.4±0.90.4\pm 0.9 70.8±8.970.8\pm 8.9 75.6±6.575.6\pm 6.5
Round 2 (50) 45.6±1.745.6\pm 1.7 1.2±1.11.2\pm 1.1 78.8±4.478.8\pm 4.4 88.4±3.888.4\pm 3.8
Round 3 (75) 40.0±7.640.0\pm 7.6 4.4±2.64.4\pm 2.6 37.6±6.837.6\pm 6.8 88.4±4.888.4\pm 4.8
Round 4 (100) 46.8±9.746.8\pm 9.7 10.8±5.010.8\pm 5.0 78.8±5.878.8\pm 5.8 92.8±2.7\boldsymbol{92.8\pm 2.7}

Iteration improves success under the same collection budgets

Four rounds of 25 correction episodes beat one 100-episode batch against the fixed base policy by 3.6 points on strawberries and 10 points on fridge/radio under HG-DAgger. This isolates collection scheduling within the reported protocol; other learning rules do not show the same consistent benefit.

Active intervention burden decreases across rounds

From round 1 to round 4, equal-weight task averages decrease 63.4% for takeover count, 69.5% for intervention-frame share and 72.0% for duration per episode. The final bar is autonomous evaluation, whereas earlier Figure 4 success bars count takeover-free collection episodes. Operators still supervise the whole execution.

Remaining bottlenecks are task dependent

Corrections become concentrated at task-specific bottlenecks: coordinated basket-handle pickup persists after book grasping improves, and the smaller second bread item remains difficult after drawer opening improves. Figure 5 aligns whole-episode progress, with smoothed intervention rates; peaks are descriptive rather than causal tests of these explanations.

IV-D Q3: Corrections Span the Whole Body

The base contributes 34.2% of normalized body-part votes on average; torso contributes 8.2% across the three tasks where it is active. A takeover can earn votes for several parts, with grippers assigned to arms. These shares measure qualifying segment activations, not time shares or fractions of episodes, and show why arm-only correction can be insufficient.

Original figure
Fig. 6: Body-part activation shares during JoyLo+ corrections. Bars pool four 25-episode rounds; each body part receives one vote per takeover segment after three consecutive active frames. Thresholds are 0.003 m/s for base translation and 0.003 rad/s for base rotation commands, 0.01 rad for torso joint differences, 0.02 rad for arm joint differences, and 0.75 for normalized gripper differences. Grippers are assigned to their respective arms.

IV-E Q4: Correction-Learning Algorithm Comparison

Each method collects its own policy-induced corrections and trains on its accumulated data. Table III reports mean and sample standard deviation across evaluation seeds; the datasets are not a shared fixed batch across algorithms.

IV-E 1 Residual Policy

A ResNet/MLP residual adds action-chunk corrections to a frozen base policy, with zero targets outside intervention and half of training draws from corrections. The residual model is reinitialized each round; large required corrections and the frozen base may limit improvement. Full target clipping and sampling rules appear in Appendix F.

IV-E 2 BC (all data)

All-data behavioral cloning samples every collected category, including potentially unsuccessful autonomous behavior. It reaches 88% on fridge/radio but only 10.8% on strawberries after four rounds, illustrating strong task dependence.

IV-E 3 SIRIUS

SIRIUS assigns 50% sampling mass to intervention, up to 50% to demonstrations, and the remainder to autonomy. Four-round HG-DAgger has the best final means: 92.8% on strawberries and 90.0% on fridge/radio. SIRIUS fluctuates and its fridge result is below single-round collection; residual learning gains little from iteration. These observations support the selected protocol on these two tasks, not universal algorithm superiority.

IV-F Q5: Operator Study

Three operators separately collect three rounds of 25 retained mug-hanging episodes. Cumulative intervention shares fall from 16.1 to 11.8%, 34.6 to 19.9%, and 16.4 to 14.7%; cumulative takeover-free episodes reach roughly 40–55%. These pooled retained-episode measures are not final held-out policy success or round-only intervention rates. Participants receive familiarization, and failed attempts are excluded.

Original figure
Fig. 7: Operator study on Mug Hanging. (a) Cumulative intervention share. (b) Cumulative takeover-free episode count. Both metrics aggregate episodes through the indicated round for each operator.

V Conclusion and Limitations

The system combines practical coordinated takeover with targeted iterative data collection and substantially improves the tested policies. Limits include only eight tasks, simulation-only algorithm comparisons, three operators on one task, repeated full retraining and active-intervention rather than total human-time accounting. Broader tasks/users, cheaper incremental updates and failure-aware assistance remain open.

Appendix A Robot Hardware, Observations, and Control

The appendix specifies robot-space commands before normalization. Real and simulated R1 action vectors have the same dimensions but different base semantics and gripper signs; they must not be treated as interchangeable.

A-A Action Representations

The retained equations define each platform’s action layout, while the notes distinguish policy state from the larger recorded state.

Franka (Real)

Real Franka commands seven absolute arm angles and one gripper value: negative opens, positive closes. Policy state contains measured angles and the previous gripper command; recording also includes velocities. The configured full opening is 80 mm.

Equation A1
𝐚tF=(q1:7,tcmd,gtcmd)∈ℝ8.\mathbf{a}^{F}_{t}=(q^{\mathrm{cmd}}_{1:7,t},g^{\mathrm{cmd}}_{t})\in\mathbb{R}^{8}. (A1)

R1 Pro (Real)

Real R1 commands 23 values: three base velocities, four torso angles, two seven-joint arms and two grippers. Base velocities are local-frame m/s and rad/s; joints are absolute radians; −1 opens grippers. The adapter applies Table A1 limits and deadbands. The VLA selects 26 of 46 recorded state/velocity coordinates, including base pose and velocity, then pads to 32.

Equation A2
𝐚tR=(𝐛3,t,𝐪4,tT,𝐪7,tL,gtL,𝐪7,tR,gtR).\mathbf{a}^{R}_{t}=(\mathbf{b}_{3,t},\mathbf{q}^{T}_{4,t},\mathbf{q}^{L}_{7,t},g^{L}_{t},\mathbf{q}^{R}_{7,t},g^{R}_{t}). (A2)
TABLE A1: Real-R1 adapter limits. Base bounds apply independently to each component; torso rows follow the robot’s joint ordering.
Command Unit Lower Upper
Base translation xx m/s −0.3-0.3 0.30.3
Base translation yy m/s −0.3-0.3 0.30.3
Base yaw rad/s −0.4-0.4 0.40.4
Torso joint 1 rad −1.1345-1.1345 1.83261.8326
Torso joint 2 rad −2.7925-2.7925 2.53072.5307
Torso joint 3 rad −1.8326-1.8326 1.57081.5708
Torso joint 4 rad −3.0543-3.0543 3.05433.0543

A1 (Simulation)

Simulated A1 commands six absolute joint angles and a gripper; +1 opens and −1 closes, opposite the real convention. It uses position control and physical grasping. Policy state derives a binary gripper state from the 0.024 m finger-position threshold.

Equation A3
𝐚tA=(q1:6,tcmd,gtcmd)∈ℝ7.\mathbf{a}^{A}_{t}=(q^{\mathrm{cmd}}_{1:6,t},g^{\mathrm{cmd}}_{t})\in\mathbb{R}^{7}. (A3)

R1 Pro (Simulation)

Simulated R1 base actions are local translation/yaw increments per step, converted to position targets; +1 opens grippers. It uses position control with assisted grasping. The 23 selected policy-state entries include base velocity but exclude base pose, then pad to 32.

A-B Home Poses and Kinematic Calibration

Home postures and encoder maps are platform-specific calibration settings, not learned universal transforms.

Franka (Real)

Table A2 retains Franka signs, offsets and home angles. Encoder conversion uses zero 2048 and π/2048 radians per count. Peg insertion changes joint 7 home to −3π/4 and adds a −π/2 leader offset to preserve the physical leader posture.

TABLE A2: Franka mapping and nominal home pose. Offsets and home angles are in radians; the insertion adjustment is applied separately.
Joint Sign Leader offset Home angle
1 +1+1 −1.3962-1.3962 00
2 +1+1 1.60461.6046 00
3 +1+1 3.14163.1416 00
4 −1-1 −2.3000-2.3000 −π/2-\pi/2
5 +1+1 −3.2681-3.2681 00
6 −1-1 1.64021.6402 π/2\pi/2
7 +1+1 −0.7000-0.7000 −π/4-\pi/4

R1 Pro (Real)

Real R1 starts with zero torso angles and zero base velocity, while base pose varies within the task’s marked region. Mirrored leaders use distinct sign vectors and zero additive offsets; encoder scale matches Franka.

A1 (Simulation)

A1 uses the specified six-joint home pose and 0.03 m fingers. Its base and bowl stay fixed while strawberry positions vary.

R1 Pro (Simulation)

Simulated R1 uses torso (−0.63, 1.65, 1.05, 0), elbows at −1.57 radians and 0.05 m fingers. Base translation varies ±0.15 m and heading ±18°.

A-C Camera Layout and Image Formation

All policies use RGB rather than depth. Cropping and state/prompt preparation differ by platform.

Franka (Real)

Franka uses external and wrist ZED cameras, selects the left stereo eye, applies Table A3 crops and resizes to 240². DP takes random 224² crops for training and centered crops for evaluation. Camera firmware, exposure settings and viewpoints stay fixed across rounds.

TABLE A3: Recorded Franka camera profile. Crop widths and offsets are in pixels, before resizing to 240×240240\times 240.
View Camera Crop width Offset (x,y)(x,y)
External ZED 2i 600 (−30,−60)(-30,-60)
Wrist ZED Mini 720 (0,0)(0,0)

R1 Pro (Real)

Real R1 uses head and both wrist views. The interface supplies 240² images, datasets store 224², and VLA resizing preserves aspect ratio with padding. It also receives a task prompt.

A1 (Simulation)

A1 renders external and wrist views at 240² and applies training-random/evaluation-centered 224² crops. Table A4 preserves camera translations and quaternion conventions.

TABLE A4: A1 simulation camera poses. Translation is in metres and quaternions use (x,y,z,w)(x,y,z,w) order. The external pose is relative to its scene parent and the wrist pose to the end-effector link.
View Translation Quaternion
External (0.7,0,0.6)(0.7,0,0.6) (0.255,0.255,0.66,0.66)(0.255,0.255,0.66,0.66)
Wrist (0.086,0,−0.066)(0.086,0,-0.066) (0.652,0.652,0.274,0.274)(0.652,0.652,0.274,0.274)

R1 Pro (Simulation)

Simulated R1 uses head and two wrist cameras from the robot asset, processed to 224² alongside the task prompt.

A-D Frequencies, Timing, and Input Devices

Table A5 separates 30 Hz action/rendering loops, 120 Hz simulation physics, 100 Hz leader control and 200 Hz touch-state publication. Publication frequency does not equal touch-sensor detection latency.

TABLE A5: Configured operating rates for the real and simulated setups.
Process Configured rate
Real Franka/R1 Pro environment loop 30 Hz
Franka camera capture 30 Hz
Simulated action / rendering 30 / 30 Hz
Simulated physics 120 Hz
JoyLo+ leader control 100 Hz
Touch-state serial publication 200 Hz

Franka (Real)

Franka uses Deoxys with a 100 Hz client and joint gains 200/14, while environment actions run at 30 Hz. Two observations condition DP; eight actions execute between plans. Table A6 retains recording and gripper controls.

TABLE A6: Franka Joy-Con controls for teleoperation and trial recording.
Control Function
R Close gripper
ZR Open gripper
X Start recording
A Accept and save trial
Y Discard trial
Home Home robot and leader
++ End collection block

R1 Pro (Real)

Real R1 targets a 30 Hz environment loop with 100 Hz joint interpolation. Leaders control arms; Joy-Cons control other parts. Inputs use a 20% dead zone and smoothing coefficient 0.9; Table A7 gives all mappings.

TABLE A7: R1 Pro Joy-Con controls for teleoperation and trial recording. Torso angles are in radians.
Control Function
Left stick, unpressed Base translation (up to 0.3 m/s per axis)
Left stick, pressed Base yaw via horizontal deflection (up to 0.4 rad/s)
Right stick, vertical Torso pitch (joint 3)
Right stick, horizontal Torso yaw (joint 4)
Left D-pad, up / down Move torso joints 1–2 toward stand (0,0)(0,0) / squat (1.74,−2.70)(1.74,-2.70)
ZL / ZR Toggle left / right gripper
X / A / Y Start recording / save / discard trial
Home Home robot and leaders
++ End collection block

A1 and R1 Pro (Simulation)

Both simulations apply actions/render at 30 Hz with physics at 120 Hz. Keyboard input gates privileged scripted corrections.

Appendix B JoyLo+ Mechanical and Takeover Design

The physical interfaces combine existing kinematic leader designs with policy tracking and capacitive takeover. Simulation uses a separate keyboard/oracle protocol rather than the hardware.

B-A Mechanical Structure and Actuation

At 100 Hz, autonomy sends joint targets to onboard position controllers. Takeover switches leaders to current mode with gravity compensation and current limits. This is leader hand-guidance control, not a claimed robot contact-force controller.

Franka (Real)

Franka uses three XM430 and four XC330 servos on separate 12 V/5 V buses. Its configuration adds no autonomous tracking-current cap; hand-guidance proportional, integral and damping gains are zero, with empirical gravity multipliers plus velocity assistance and a deadband.

R1 Pro (Real)

Each R1 leader has four XM430 and three XC330 motors; both leaders share one bus per family. Autonomous tracking caps are 1.076 A for larger motors and 0.150 A for smaller ones. Hand guidance uses reduced proportional/damping gains and mirrored empirical gravity compensation.

A1 and R1 Pro (Simulation)

Simulated corrections need no physical leaders, motor buses or hand-guidance tuning.

B-B Touch Hardware, Wiring, and Firmware

The original wiring diagram and Table A8 retain the electrical connections and sensor settings.

Shared electronics: Franka and R1 Pro (Real)

One MPR121 and ESP32-S3 read the grip electrode at input 11 using address 0x5A and GPIO 43/44. Sampling is every 16 ms; host publication is 200 Hz. Calibration collects untouched/held readings, sets touch between their robust percentiles and release at 60% of touch, then reflashes firmware. Disconnected or >0.5 s stale sensing is treated as inactive and follows release unless another input is active; this is not a fail-safe hold guarantee.

Original Figure A1, complete R1 Pro wiring and command flow with all electrical and control labels.
Fig. A1: R1 Pro touch wiring and command flow. Both grip electrodes connect to input 11 of one MPR121 and share one ESP32-S3; touching either grip activates takeover.
TABLE A8: MPR121 touch-sensor settings. Thresholds are in sensor register units.
Setting Value
Enabled electrodes All 12 channels
Thresholds
(touch / release)
195 / 117
Charge current 63 μ\muA
Sample interval 16 ms
Debounce
(touch / release)
Three consecutive qualifying detections
Baseline calibration Automatic configuration and tracking

Franka (Real)

Franka keeps human authority for 0.5 seconds after contact ends, unless another input renews it.

R1 Pro (Real)

Both R1 grip electrodes share one sensor input, so either grip triggers global authority. Release waits 0.3 seconds after both are clear, subject to other active inputs.

A1 and R1 Pro (Simulation)

Simulation has no touch thresholds or grip-release delay; keyboard input selects the oracle.

B-C Authority Switching and Policy Resumption

The resumption logic prevents stale autonomous plans from simply continuing after correction; platform implementations differ.

Franka (Real)

Franka selects human commands on touch/Joy-Con input, clears pending plans at takeover and release, and holds while awaiting a fresh policy prediction. At release, the leader target is first set to its measured pose before tracking resumes. Predictions during correction remain enabled, but temporary hold commands are not valid counterfactual policy actions.

R1 Pro (Real)

R1 transfers all body parts together, even if the human moves only one. Entry linearly blends policy and human commands over two seconds; the release step retains human commands before fresh policy execution. This qualification matters when interpreting “instant” touch-triggered takeover.

A1 (Simulation)

For strawberries, the human chooses intervention and resumption times; a simulator-state/cuRobo oracle generates arm and gripper motion. The learned policy receives only cameras/proprioception and replans afterward.

R1 Pro (Simulation)

For fridge/radio, the oracle corrects base, torso, arms and grippers from current task state. Human keyboard gating and recorded intervention labels remain the same.

B-D Bill of Materials and Cost Scope

Table A9 preserves component quantities and prices. Estimates exclude the robot, tax, shipping, cabling, fasteners, bearings and mounting hardware; project CAD is linked in the source.

Franka (Real)

One Franka leader and listed accessories total $1,087.49, including three larger and four smaller servos.

R1 Pro (Real)

One R1 leader and accessories total $1,257.49. The paper estimates the dual-leader arrangement at roughly twice that, about $2,515.

TABLE A9: Estimated bills of materials for one Franka or R1 Pro leader and the listed accessories. Costs are in USD. Robot, tax, shipping, cabling, fasteners, bearings, and mounting hardware are excluded.
Quantity
Component Unit cost Franka R1 Pro
XM430-W350-T servo $220 3 4
XC330-M288-T servo $50 4 3
U2D2 USB adapter $30 2 2
Power supply (12 V/5 V) $25 2 2
Printed links (set) $15 1 1
Joy-Con pair $80 1 1
Adafruit MPR121 $7.95 1 1
LILYGO T-Display S3 $9.54 1 1
Copper tape $5 1 1
Estimated total $1,087.49 $1,257.49

A1 and R1 Pro (Simulation)

These physical-interface costs do not apply to the keyboard-based simulation protocol.

Appendix C Simulation Setup and Data Collection

Simulation supplies controlled initialization and privileged motion oracles, allowing collection and learning comparisons at scale.

C-A Simulator Configuration

OmniGibson runs A1 in an empty scene and R1 in Rs_int, using BEHAVIOR assets. Actions/rendering run at 30 Hz and physics at 120 Hz. A1 uses physical grasping; R1 uses assisted grasping. Goals must hold for 20 steps; timeouts are 800 and 900 action steps respectively.

C-B Task Initialization

Randomization is bounded and task-specific; the study does not evaluate arbitrary new environments.

Strawberries

Strawberries are randomized with at least 0.15 m separation from each other and the fixed bowl. Robot initialization stays fixed.

Fridge and radio

The fridge begins closed and the radio on a separate table. Base pose varies ±0.15 m and ±18°; fridge/table offsets are shared up to ±0.10 m on one axis and independently ±0.05 m on the other.

C-C Demonstration and Correction Collection

The same privileged-state planners generate demonstrations and human-gated corrections. The oracle selects its next motion from current task state; these low-level sequences are grouped into the manually annotated stages used for analysis.

Strawberries (14 motion substeps)

Fourteen motions perform the same seven-step sequence for red then purple strawberry: open, approach above, lower, grasp, lift, move over bowl and release.

Fridge and radio (14 motion substeps)

Fourteen motions open the gripper, approach/reach/grasp/pull/release the fridge handle, reset torso/retract arms/open, navigate to the separate table, then approach/reach/grasp/lift the radio.

Appendix D Data Recording and Processing

Recordings retain both executed and proposed commands plus supervision labels; analysis annotations are produced separately.

D-A Recorded Data and Intervention Labels

A correction trajectory includes autonomous and intervention segments. It is an entire recorded rollout, not a trajectory composed only of human actions.

Franka

Franka records two 240² views, eight action values and 16 state/velocity values. Gripper velocity is a zero placeholder; policy input uses a smaller state subset.

Real R1 Pro

Real R1 records three views, 23 actions and 46 state/velocity values in matching body-part order. Gripper openings are mapped to command range and their velocity entries are zero placeholders.

Intervention labels

Demonstrations, active correction, preceding 15 frames and autonomy receive labels 1–4. Active intervention wins when pre-intervention windows overlap; the retained piecewise equation specifies precedence.

Equation A4
st={1,original demonstration,2,ht=1,3,ht=0,∃j∈{t+1,…,t+15}:hj=1,4,otherwise.s_{t}=\begin{cases}1,&\text{original demonstration},\\ 2,&h_{t}=1,\\ 3,&h_{t}=0,\ \exists j\in\{t+1,\ldots,t+15\}:h_{j}=1,\\ 4,&\text{otherwise}.\end{cases} (A4)

D-B Training-Data Preparation

Normalization and eligible-start sampling define which parts of recorded trajectories train the policy.

Normalization

DP scales joint and other non-gripper action groups to [−1,1] using configured limits. Grippers retain platform-specific encodings. The VLA uses its separate state/action adaptation.

Equation A5
q~=2​q−qminqmax−qmin−1.\widetilde{q}=2\frac{q-q_{\min}}{q_{\max}-q_{\min}}-1. (A5)

Sampling by supervision source

HG-DAgger samples uniformly over eligible demonstration/intervention starts, so correction mass depends on their counts. All-data BC preserves observed proportions of all four labels. SIRIUS uses intervention mass 0.5, demonstration mass min(fD,0.5), zero pre-intervention mass and residual autonomous mass, renormalizing when categories are absent. Action chunks can cross label transitions.

Equations A6–A8
pD=NDND+NI,pI=NIND+NI,pP=pA=0,p_{D}=\frac{N_{D}}{N_{D}+N_{I}},\qquad p_{I}=\frac{N_{I}}{N_{D}+N_{I}},\qquad p_{P}=p_{A}=0, (A6)
pc=NcND+NI+NP+NA,c∈{D,I,P,A},p_{c}=\frac{N_{c}}{N_{D}+N_{I}+N_{P}+N_{A}},\qquad c\in\{D,I,P,A\}, (A7)
pD\displaystyle p_{D} =min⁡(fD,0.5),\displaystyle=\min(f_{D},0.5), pI\displaystyle p_{I} =0.5,\displaystyle=0.5, (A8)
pP\displaystyle p_{P} =0,\displaystyle=0, pA\displaystyle p_{A} =0.5−pD,\displaystyle=0.5-p_{D},

D-C Subtask Annotation

Humans mark named subtask start/end frames from episode videos. These intervals support analysis only; takeover labels select training data.

Intervention-rate curves

Figure 5 bins normalized whole-episode progress into 50 bins and pools frame counts. Takeovers shorter than 15 frames are excluded from intervention counts while their frames remain in the denominator; Gaussian smoothing uses σ=3 bins with reflected boundaries.

Displayed subtask boundaries

Displayed boundaries are mean normalized subtask ends from a reference round: round 2 for books/popcorn and round 4 for other tasks. Curves are not separately time-aligned within each subtask.

Appendix E Policy Learning and Deployment

Every round restarts DP/VLA training and uses the final checkpoint. Multiple accelerator types support training, while all real deployments use RTX 4090. Tables A10–A12 preserve hyperparameters and prompts.

E-A Diffusion Policy

DP begins with 75 strawberry demonstrations or 100 per Franka task. Separate ResNet-18 encoders and proprioceptive features yield 1024-dimensional two-observation conditioning for a small 1D U-Net. Training predicts noise with padding-masked squared loss over 16 actions and 100 diffusion steps. It stops at 200k steps on a 300k cosine schedule. Inference uses 16 DDIM steps, discards the first predicted action for alignment and executes eight actions (~0.27 s) before replanning; takeover/reset clears plan/history.

TABLE A10: Diffusion policy settings for A1 and Franka.
Setting       Value
Initialization       ImageNet ResNet-18
Trainable params.       ≈\approx27.8M
Image input       2 views
Image resolution       224×224224\times 224
Optimizer       AdamW, wd 0
Batch size       128
Peak / min. LR       7×10−47\times 10^{-4} / 5×10−65\times 10^{-6}
Warmup / cosine       1k / 300k steps
Training steps       200k
Grad. clipping       1.0
Precision / EMA       float32 / none
Prediction horizon       16
Execution horizon       8
Seed       42
Equation A9
ℒDP=𝔼⁡[∑umu​‖ϵθ​(𝐀(k),k,z)u−ϵu‖22da​∑umu].\mathcal{L}_{\mathrm{DP}}=\mathbb{E}\!\left[\frac{\sum_{u}m_{u}\|\epsilon_{\theta}(\mathbf{A}^{(k)},k,z)_{u}-\epsilon_{u}\|_{2}^{2}}{d_{a}\sum_{u}m_{u}}\right]. (A9)

E-B Vision–Language–Action Policy

π0.5 fine-tunes all approximately 3.35B parameters from 100 base demonstrations per task for 40k steps (~7 hours on eight H100s). Three views, current state and prompt condition 30 predicted actions; 15 execute per plan. Arm/torso targets become state-relative differences while base/gripper commands retain their semantics; values use checkpoint normalization and padding to 32. EMA decay is 0.99. Table A12’s fridge prompt additionally says to turn on the radio, while the stated evaluation criterion requires opening and lifting it; the edition preserves that source distinction.

TABLE A11: π0.5\pi_{0.5} settings for R1 Pro.
Setting       Value
Initialization       π0.5\pi_{0.5} base checkpoint
Trainable params.       ≈\approx3.35B (full model)
Image input       3 views
Observation horizon       1 (current observation)
Image resolution       224×224224\times 224
Optimizer       AdamW, wd 10−1010^{-10}
Batch size       64
Peak / min. LR       2.5×10−52.5\times 10^{-5} / 2.5×10−62.5\times 10^{-6}
Warmup / cosine       1k / 30k steps
Training steps       40k
Grad. clipping       1.0
Precision / EMA       bfloat16 / 0.99
Prediction horizon       30
Execution horizon       15
Seed       42
TABLE A12: Language instructions used for the VLA tasks.
Task Language instruction
Open Fridge and Pick Up Radio Open the fridge, then pick up and turn on the radio.
Boxing Books up for Storage Place the basket onto the table, and put the two books from the shelf into the basket.
Make Microwave Popcorn Pick up the red popcorn bag from the shelf and place it into the microwave.
Prepare Make-Ahead Breakfast Bowls Retrieve the bowl from the drawer, put the two bread in from the plate, and place the bowl on the right side of the countertop.
Turning on Radio Pick up the radio on the table and poke the orange button.

Appendix F Evaluation Metrics and Experimental Protocols

The final appendix defines success, intervention aggregation, body-part votes, baselines and the retained-episode operator study.

F-A Autonomous Evaluation and Collection Outcomes

Full-task completion, takeover-free collection and annotated-stage intervention rates are distinct measurements.

Simulation

Simulation success requires both strawberries in the bowl, or the fridge open and the radio lifted from its separate table.

Fixed-base manipulation

Franka success requires the mug supported by its handle on the rack or the peg inserted into its matching fixture.

Whole-body mobile manipulation

Real mobile success requires the bowl containing both breads on the far countertop with the drawer closed; both books in the basket on the table; popcorn inside a closed microwave; or holding the radio and pressing its activation button. These criteria do not require cooking popcorn.

Evaluation metrics

Real rates are counts out of 25. Simulation means and sample standard deviations come from five sets of 50 randomized evaluation rollouts on the same checkpoint. Tables A13/A14 retain every seed and baseline; they do not quantify retraining-seed variation.

TABLE A13: Open Fridge and Pick Up Radio simulation success rates (%) across five evaluation seeds. For correction-learning methods, parentheses indicate cumulative corrective trajectories.
Method Setting Seed 0 Seed 1 Seed 2 Seed 3 Seed 4 Mean
Demo-only Base (100 demos) 36.0 48.0 38.0 36.0 40.0 39.6
Additional demos (200 total) 30.0 28.0 42.0 26.0 26.0 30.4
Residual Single round (100) 10.0 12.0 46.0 6.0 34.0 21.6
Round 1 (25) 16.0 14.0 18.0 14.0 18.0 16.0
Round 2 (50) 18.0 30.0 28.0 24.0 20.0 24.0
Round 3 (75) 20.0 20.0 40.0 24.0 46.0 30.0
Round 4 (100) 16.0 14.0 20.0 18.0 30.0 19.6
BC (all data) Single round (100) 86.0 84.0 88.0 56.0 82.0 79.2
Round 1 (25) 72.0 78.0 66.0 68.0 70.0 70.8
Round 2 (50) 84.0 88.0 92.0 86.0 82.0 86.4
Round 3 (75) 82.0 94.0 94.0 86.0 80.0 87.2
Round 4 (100) 84.0 88.0 94.0 86.0 88.0 88.0
SIRIUS Single round (100) 70.0 86.0 82.0 54.0 74.0 73.2
Round 1 (25) 76.0 36.0 48.0 78.0 36.0 54.8
Round 2 (50) 58.0 66.0 56.0 74.0 88.0 68.4
Round 3 (75) 82.0 50.0 84.0 54.0 82.0 70.4
Round 4 (100) 78.0 66.0 64.0 68.0 82.0 71.6
HG-DAgger Single round (100) 86.0 90.0 82.0 54.0 88.0 80.0
Round 1 (25) 70.0 58.0 40.0 44.0 46.0 51.6
Round 2 (50) 70.0 80.0 66.0 68.0 78.0 72.4
Round 3 (75) 84.0 90.0 88.0 90.0 58.0 82.0
Round 4 (100) 88.0 88.0 90.0 90.0 94.0 90.0
TABLE A14: Pick-and-Place Strawberries simulation success rates (%) across five evaluation seeds. For correction-learning methods, parentheses indicate cumulative corrective trajectories.
Method Setting Seed 0 Seed 1 Seed 2 Seed 3 Seed 4 Mean
Demo-only Base (75 demos) 50.0 42.0 58.0 48.0 68.0 53.2
Additional demos (175 total) 52.0 54.0 62.0 54.0 56.0 55.6
Residual Single round (100) 48.0 44.0 48.0 42.0 44.0 45.2
Round 1 (25) 56.0 46.0 50.0 46.0 48.0 49.2
Round 2 (50) 48.0 46.0 46.0 44.0 44.0 45.6
Round 3 (75) 50.0 36.0 44.0 30.0 40.0 40.0
Round 4 (100) 56.0 42.0 42.0 36.0 58.0 46.8
BC (all data) Single round (100) 12.0 14.0 16.0 14.0 18.0 14.8
Round 1 (25) 0.0 0.0 0.0 2.0 0.0 0.4
Round 2 (50) 0.0 2.0 2.0 0.0 2.0 1.2
Round 3 (75) 8.0 2.0 2.0 6.0 4.0 4.4
Round 4 (100) 10.0 6.0 16.0 6.0 16.0 10.8
SIRIUS Single round (100) 76.0 78.0 74.0 74.0 78.0 76.0
Round 1 (25) 84.0 64.0 64.0 76.0 66.0 70.8
Round 2 (50) 76.0 86.0 76.0 76.0 80.0 78.8
Round 3 (75) 46.0 28.0 40.0 34.0 40.0 37.6
Round 4 (100) 80.0 74.0 74.0 78.0 88.0 78.8
HG-DAgger Single round (100) 90.0 90.0 92.0 90.0 84.0 89.2
Round 1 (25) 78.0 84.0 74.0 76.0 66.0 75.6
Round 2 (50) 86.0 90.0 84.0 94.0 88.0 88.4
Round 3 (75) 92.0 94.0 82.0 86.0 88.0 88.4
Round 4 (100) 96.0 90.0 94.0 94.0 90.0 92.8

F-B Intervention Burden and Aggregation

Figure 4 counts retained takeover runs within annotated intervals, divides intervention frames by annotated coverage, and converts duration using recorded frame rate. It averages episodes within tasks, then weights tasks equally. The operator plot instead pools all recorded timesteps cumulatively through each round and counts cumulative takeover-free episodes. These differing denominators explain why the plots should not be read interchangeably.

Equations A10–A11
Ke=h~e,0+∑t=1Te−1(1−h~e,t−1)​h~e,t,Fe=1Me​∑t=0Te−1h~e,t.\begin{split}K_{e}&=\widetilde{h}_{e,0}+\sum_{t=1}^{T_{e}-1}(1-\widetilde{h}_{e,t-1})\widetilde{h}_{e,t},\\ F_{e}&=\frac{1}{M_{e}}\sum_{t=0}^{T_{e}-1}\widetilde{h}_{e,t}.\end{split} (A10)
Frcum=∑j=1r∑e∑thj,e,t∑j=1r∑eTj,e.F^{\mathrm{cum}}_{r}=\frac{\sum_{j=1}^{r}\sum_{e}\sum_{t}h_{j,e,t}}{\sum_{j=1}^{r}\sum_{e}T_{j,e}}. (A11)

F-C Whole-Body Activation Analysis

Pool four 25-episode rounds. A part earns at most one vote per takeover after three consecutive active frames; multiple parts may vote. Thresholds are 0.003 m/s or rad/s base commands, 0.01 rad torso error, 0.02 rad arm error and 0.75 gripper-command error. Grippers count toward arms, and shares normalize by all body-part votes rather than intervention duration.

Equation A12
Vb=∑kvk​b∑b′∑kvk​b′.V_{b}=\frac{\sum_{k}v_{kb}}{\sum_{b^{\prime}}\sum_{k}v_{kb^{\prime}}}. (A12)

F-D Baseline Configurations

Both augmentation conditions add 100 trajectories. Scheduling compares one fixed-policy batch with four updated-policy rounds. Each algorithm collects its own evolving data, and only the residual baseline keeps its base policy permanently frozen.

Residual learning

The residual network uses image/state/base-plan inputs and a zero-initialized output; it always adds an 8-step A1 or 15-step R1 chunk. Targets subtract the frozen base action, clip non-gripper differences to [−2,2], then halve; deployment doubles the output and clips composed commands to [−1,1]. Grippers are absolute commands. Demonstration/autonomous targets are zero and pre-intervention starts are excluded. Sampling masses are 0.3/0.5/0.2, with one fifth of correction draws from each episode’s first 20 correction frames. A padding-masked squared loss plus equally weighted zero-residual penalty trains for 200k steps. Each round reinitializes the residual on accumulated data; stored base predictions keep target meaning fixed.

Equation A13
Δ​𝐚t=𝐚texec−𝐚tπ0,\Delta\mathbf{a}_{t}=\mathbf{a}^{\mathrm{exec}}_{t}-\mathbf{a}^{\pi_{0}}_{t}, (A13)

F-E Operator Study Protocol

Three lab participants—two experienced and one novice—complete about 25 familiarization episodes, then separately train from the same base checkpoint over three rounds of 25 retained mug-hanging episodes. Failed attempts are excluded. Intervention and takeover-free metrics therefore describe the 225 retained episodes, not all attempts or total practice/supervision time; this small, single-task study cannot establish broad user independence.

Concept demonstration

Open image ↗