Literature
Display
← All papers

Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Ye Wang · Pei Lin · Xiong-Hui Chen · Haoqi Yuan · Zhixuan Liang · Yiyang Huang · Anzhe Chen · Zixing Lei · Jie Zhang · Tao Zhang · Haoyang Li · Tong Zhang · Chenxi Xiao · Ziyuan Jiao · Qin Jin

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

Keywords: Robot Data Synthesis, Egocentric Data, Generalization Evaluation

Ego2Robot Appendix

At a glance

This edition follows the complete twenty-three-page v1, retaining all nine figures, five numbered tables, twelve displayed equations and the complete segmentation, curation and model-input prompts. Ego2Robot converts human hand trajectories and images into parallel-gripper robot demonstrations, then mixes these with robot data for policy pretraining.

Approximately 1,940 source-video hours produce 18,561 synthesized hours across fifteen morphologies. Those are rendered variants of human footage, not independent hours of robot-collected interaction. Equal-budget pretraining improves several tested generalization settings, with important exceptions for Franka and some tasks.

Real-robot results use substep partial scores over twenty trials per task, despite the chart’s “success rate” label. Camera geometry, hand reconstruction, kinematic feasibility and compositing assumptions remain dependencies; synthesis does not physically re-execute or validate each human interaction.

1 Introduction

Robot demonstrations are expensive to diversify. Egocentric video offers varied scenes and interactions, but anatomy, kinematics and appearance differ from robot data. The proposed pipeline aligns actions and images, filters errors, and tests whether the resulting data improves a pretrained action model after downstream robot fine-tuning.

The contribution is scaling retarget-and-render data plus a decomposed evaluation, rather than eliminating robot demonstrations. The largest-dataset claim is the authors’ comparison at publication, not an independently exhaustive census.

Complete original PDF block S1.F1
Figure 1: Ego2Robot pipeline. Converting egocentric video into 18,561h robot training data across 15 morphologies through action alignment, visual alignment, and quality curation.

2 Related Work

Robot Data Scaling and Learning from Human Data

Human video can supply visual features, rewards, tracks or paired training. PHANTOM retargets and renders human demonstrations; EgoMimic mixes human and robot learning. Ego2Robot tests large-scale robot-format pretraining and separates its effects across distribution shifts.

Data Augmentation for Robot Learning

MimicGen and RoboCasa generate simulated data, while Rovi-Aug and OXE-AugE alter robot embodiments/views. Here the source motion comes from humans, adding hand-estimation, retargeting and occlusion challenges.

Benchmarks for Robot Generalization

Many benchmarks bundle shifts. The paper extends RoboTwin 2.0 with independent visual, layout, embodiment and semantic settings, and separately evaluates EBench. Isolating benchmark factors helps describe where performance changes; it does not alone identify which training component causally produces a gain.

3 Ego2Robot Pipeline

Path A accepts annotated hand poses; Path B estimates them from video with WiLoR and DynHaMR, then uses Qwen3.5 to segment long recordings. Both paths retarget, remove hands, render robot arms and curate outputs. Object motion remains that of the original human interaction, so a plausible overlay is not a newly simulated contact trajectory.

1. Action Alignment

A virtual fingertip blends index and middle tips with weights 0.7/0.3. Its midpoint with the thumb gives the tool center; their separation gives gripper width. Equation 3 constructs an orthonormal frame from the jaw line and wrist direction, with a handedness sign. Degenerate widths or nearly parallel vectors require the appendix fallback.

Positions/widths use Savitzky–Golay filtering and rotations use Gaussian-weighted SLERP. The source reports per-source frame-rate resampling to 60%, 45% or 25% as slowing hand motion. The timestamp/replay convention is not fully specified; simply dropping frames at an unchanged execution rate would not itself establish this slowdown.

Equations 1–3
𝐩vf=0.7⋅𝐩index+0.3⋅𝐩middle\mathbf{p}_{\text{vf}}=0.7\cdot\mathbf{p}_{\text{index}}+0.3\cdot\mathbf{p}_{\text{middle}} (1)
𝐩tcp=𝐩thumb+𝐩vf2,w=‖𝐩thumb−𝐩vf‖\mathbf{p}_{\text{tcp}}=\frac{\mathbf{p}_{\text{thumb}}+\mathbf{p}_{\text{vf}}}{2},\quad w=\|\mathbf{p}_{\text{thumb}}-\mathbf{p}_{\text{vf}}\| (2)
𝐳=s⁡(𝐩thumb−𝐩vf)w,𝐲=𝐳×𝐝‖𝐳×𝐝‖,𝐱=𝐲×𝐳\mathbf{z}=\frac{s\,(\mathbf{p}_{\text{thumb}}-\mathbf{p}_{\text{vf}})}{w},\quad\mathbf{y}=\frac{\mathbf{z}\times\mathbf{d}}{\|\mathbf{z}\times\mathbf{d}\|},\quad\mathbf{x}=\mathbf{y}\times\mathbf{z} (3)

2. Visual Alignment

SAM3 masks arms, ProPainter fills their background, and a base-placement search fits robot kinematics to selected trajectory extremes. Framewise IK and camera-view rendering produce replacement arms. Keyframe feasibility and self-collision filtering do not establish object contact, force, torque or full continuous-motion feasibility.

Equation 5 gives a unified depth rule, but the appendix implements a different practical split: arm bodies are always overlaid; only grippers undergo depth occlusion testing outside a dilated original-hand mask. This depends on the assumption that scene objects never occlude the arm body.

Equations 4–5
𝐓base∗=arg⁡max𝐓base​1|𝒦|​∑k∈𝒦𝟙​[IK​(𝐓base−1​𝐓kee)​ is feasible]\mathbf{T}_{\text{base}}^{*}=\arg\max_{\mathbf{T}_{\text{base}}}\frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}}\mathbb{1}\left[\text{IK}(\mathbf{T}_{\text{base}}^{-1}\mathbf{T}_{k}^{\text{ee}})\text{ is feasible}\right] (4)
Ifinal​(u,v)={Irobot​(u,v)if ​Drobot​(u,v)<Dscene​(u,v)∧Mrobot​(u,v)=1Iinpaint​(u,v)otherwiseI_{\text{final}}(u,v)=\begin{cases}I_{\text{robot}}(u,v)&\text{if }D_{\text{robot}}(u,v)<D_{\text{scene}}(u,v)\land M_{\text{robot}}(u,v)=1\\ I_{\text{inpaint}}(u,v)&\text{otherwise}\end{cases} (5)

3. Quality Curation

Pipeline validity, dataset-level motion statistics and a VLM semantic audit reject poor frames/episodes. ANT 7h, EgoDex 732h, ViTRA 249h and EgoVerse 954h yield roughly 1,940 source hours and 18,561 rendered hours after fifteen-morphology processing. Filtering and repeated rendering explain why synthesized hours are not independent scene/task diversity.

Each sample includes robot-composited imagery, camera-frame action, camera parameters and instruction. Camera-relative output helps share a representation across embodiments, but converting poses and rendering still requires known or estimated camera/robot geometry.

4 Evaluation Framework

Eleven RoboTwin settings isolate background, lighting, robot color, table height, clutter, camera offset, three alternate robots, unseen objects and paraphrased instructions. EBench adds a separate seven-task precision suite. The decomposition does not test every interaction among shifts.

Cross-embodiment evaluation replaces Aloha with paired ARX-X5, UR5-WSG or Franka arms and aligns initial end-effectors and cameras. Zero-shot here means no target-embodiment downstream fine-tuning; related morphologies, including Panda, are present during synthesized pretraining.

Complete original PDF block S4.F2
Figure 2: Evaluation framework. Four generalization dimensions with 12 evaluation settings. Dashed borders: settings decoupled from bundled randomization for independent testing. Solid borders: newly introduced evaluation axes. Gray border: external benchmark (EBench).

5 Experiments

5.1 Experimental Setup

Qwen3.5-4B provides visual-language context to a Diffusion Transformer action head. Pretraining uses 200,000 steps, batch 96 and eightA 100 GPUs for about 19.2million sampled frames in every configuration. Equal exposure controls optimization budget, not equal unique data or equal exposure to each source.

Robot-only mixes DROID, AgibotWorld and synthetic InternData, totaling about 6,565h. Ego2R:Robot ratios 1:3,3:1 and 1:1 change sampling weights. All models then fine-tune on 2,500 clean RoboTwin demonstrations; EBench uses a separate 2,800-demonstration fine-tune. Main-text eight diffusion steps are qualified by the appendix’s four Euler steps at inference.

5.2 Main Results

The 1:1 mix improves Clean 62.2 → 68.1 and Randomized 50.9 → 53.5, a5.9 and 2.6 percentage-point gain. It leads five of seven aggregate columns;3:1 leads embodiment 28.2 and EBench 51.7. The 3:1 mix actually reduces Randomized 50.9 → 49.2, so improvement is not uniform across ratios.

At 1:1, background/lighting/color improve 3.7/7.6/6.4 points; camera offset improves 5.9. ARX improves 44.1 → 51.2 and UR5 peaks 31.4 at 3:1. Franka decreases from 7.0 to 5.3 at 1:1 and remains weak; exposure to multiple rendered morphologies does not solve every transfer.

Unseen-object performance rises 29.3 → 39.6 at 1:1, with paraphrases 63.1 → 68.5. EBench’s higher camera viewpoint plausibly helps transfer, but no controlled camera-only experiment isolates that explanation from its different tasks and simulator.

Complete original PDF block S5.T1
Table 1: Main results. Success rates (%) on RoboTwin2.0 and EBench. Green = gain >>5% vs. Robot-only.
Original selectable table
Pretraining RoboTwin Per-Dimension (RoboTwin) EBench
Clean Rand Visual Scene Embody Task Avg
Robot-only 62.2 50.9 61.4 52.9 23.8 46.2 39.6
Ego2R+Robot (1:3)
61.4
–0.8
51.0
+0.1
61.2
–0.2
52.5
–0.4
21.9
–1.9
49.5
+3.3
47.4
+7.8
Ego2R+Robot (3:1)
64.1
+1.9
49.2
–1.7
62.7
+1.3
54.3
+1.4
28.2
+4.4
51.6
+5.4
51.7
+12.1
Ego2R+Robot (1:1)
68.1
+5.9
53.5
+2.6
67.3
+5.9
56.9
+4.0
27.2
+3.4
54.1
+7.9
49.8
+10.2
Complete original PDF block S5.T2
Table 2: Per-perturbation breakdown. Change vs. Robot-only. Green = gain >>5%.
Original selectable table
Pretraining Visual Scene Embodiment Task
BG Light Color Height Clutter Camera ARX UR5 Franka Obj Lang
Robot-only 66.6 58.2 59.4 60.1 48.3 50.4 44.1 20.2 7.0 29.3 63.1
Ego2R+Robot (1:3)
65.0
–1.6
58.3
+0.1
60.3
+0.9
58.6
–1.5
49.3
+1.0
49.6
–0.8
43.7
–0.4
17.6
–2.6
4.5
–2.5
36.8
+7.5
62.2
–0.9
Ego2R+Robot (3:1)
65.5
–1.1
60.9
+2.7
61.8
+2.4
62.0
+1.9
49.2
+0.9
51.6
+1.2
47.6
+3.5
31.4
+11.2
5.6
–1.4
40.0
+10.7
63.1
+0.0
Ego2R+Robot (1:1)
70.3
+3.7
65.8
+7.6
65.8
+6.4
62.4
+2.3
52.0
+3.7
56.3
+5.9
51.2
+7.1
25.0
+4.8
5.3
–1.7
39.6
+10.3
68.5
+5.4

5.3 Ablation Studies

Ego-only pretraining followed by the same robot fine-tune gives 28.1% Randomized for filtered raw ego,31.7% for one rendered morphology and 33.5% for fifteen. Adding raw ego to fifteen morphologies reaches 37.3%. Raw ego has already passed quality filtering, so the first comparison is not unfiltered footage versus the entire pipeline.

The ablation changes image/action alignment and available rendered diversity together. It supports their combined value without separately measuring every stage. Raw human footage functions as an additional data domain; calling it a sixteenth morphology is an analogy, not a new robot body.

Complete original PDF block S5.F3
Figure 3: Pipeline value and embodiment scaling. Success rate on RoboTwin Randomized.

5.4 Real Robot Experiments

Five ARX ACone tasks use twenty teleoperated demonstrations each. About 35 minutes of scene-specific human play become 675 synthetic episodes, mixed 1:1 with teleoperation during the added-data fine-tune. Figure 4 includes Scratch, Robot-only, Mix and Mix+Ego2R Play, although the prose describes three pretrained configurations.

The best displayed partial scores are 31 fruits,49 blocks,30 towel,14 trash and 15 screw. The largest stated gains over Robot-only are 14 points for blocks and 13for screws. These are normalized subtask-progress scores under AppendixE’s rubric, not binary complete-task success percentages; no uncertainty intervals are supplied.

Complete original PDF block S5.F4
Figure 4: Real robot results. Success rates (%) on five tasks on the ARX ACone platform.
Complete original PDF block S5.F5
Figure 5: Real robot rollouts. Key frames from five evaluation tasks on the ARX ACone platform.

6 Limitations

Parallel-jaw retargeting discards finger articulation, and inpainting/depth errors remain under occlusion and difficult lighting. Kinematic overlays preserve human object motion without validating robot contact mechanics. The tested task families, limited real trials and missing uncertainty estimates limit broad deployment claims.

7 Conclusion

At fixed pretraining exposure, synthesized ego data complements robot-format data in several generalization settings after task fine-tuning. Benefits vary by ratio, task and embodiment. The study supports a useful data source rather than replacing robot collection or guaranteeing executable demonstrations.

Appendix A Ego2Robot Pipeline Details

A.1 Hand Pose Estimation (Path B).

WiLoR predicts MANO hands and 21 keypoints. Frames with over 80% of projected keypoints outside the SAM hand mask are removed. The strongest two-hand frame seeds nearest-wrist association in both directions; jumps above max(4 × median velocity,0.003m/frame) are rejected. Identity swaps or occlusion errors can survive these heuristics.

DynHaMR jointly smooths pose/shape with data and biomechanical terms, constraining hand depth to 0.05–0.4m. Gaps of at most ten frames interpolate positions and rotations; longer gaps use robot home pose with 5–90 frame blended transitions. Those filled motions are synthetic assumptions, not recovered observations.

Equation 6
ℒdyn=ℒdata+λsmooth​ℒsmooth+λbio​ℒbio\mathcal{L}_{\text{dyn}}=\mathcal{L}_{\text{data}}+\lambda_{\text{smooth}}\mathcal{L}_{\text{smooth}}+\lambda_{\text{bio}}\mathcal{L}_{\text{bio}} (6)

A.2 Subtask Segmentation (Path B).

Videos split at 60 seconds before Qwen subtask segmentation. The complete prompt asks for independent complete goals,5–12 word English instructions and JSON time boundaries. A fixed clip boundary can still cut a longer activity; no segmentation-accuracy or boundary-consistency audit is reported.

Complete original PDF block source-segment-prompt.part1Complete original PDF block source-segment-prompt.part2
Original subtask-segmentation prompt
Original selectable source block

You are watching a first-person manipulation video ({duration}s).
Task: "{task_desc}"
Segment this video by complete task goals. Each subtask should be an independent, complete objective.
Write a SHORT English action instruction (5--12 words) for each subtask.
Output ONLY a JSON array: [{"description":..., "start_time":..., "end_time":...}, ...]

A.3 Action Alignment Details.

Widths below 1cm or nearly parallel orientation vectors freeze the last valid rotation. Position jumps use max(5 × median,0.9/fps), with rotational floor 10/fps radians per frame, followed by two interpolation passes. These are outlier heuristics rather than contact-preserving physical constraints.

Savitzky–Golay uses at most 21 frames and polynomial order 3; quaternion smoothing uses hemisphere correction, Gaussianσ10 and a21 frame kernel. Filtering can reduce jitter while also changing timing and contact alignment.

Equation 7
τt=max⁡(5⋅median​({vs}s), 0.9/fps)\tau_{t}=\max\!\bigl(5\cdot\text{median}(\{v_{s}\}_{s}),\;0.9/\text{fps}\bigr) (7)

A.4 Visual Alignment Details.

SAM uses “person,” a middle-frame anchor,400 frame chunks and 50 frame overlap. Mask postprocessing fills short gaps, replaces abnormally small masks and closes holes; ProPainter uses the stated half-precision temporal settings. These operations can reconstruct plausible unseen pixels without recovering true occluded object geometry.

The exact base-search grid is retained below. Up to 20keyframes determine IK feasibility, while the practical score adds a5 × penalty for deviation from 65% normalized reach. The best five candidates per arm produce 25joint combinations. This refines the main feasibility-only objective and remains a finite search.

The arm/gripper composite split, hand-mask dilation and Depth Anything V3 estimate are preserved in equation 9. Always placing arm bodies in front is a scene assumption; the VLM consistency filter does not supply a geometric proof that it holds.

Complete original PDF block source-base-grid.part1
Exact base-placement search grid
Original selectable source block
  • •

    Lateral (left-right): r×{0.3,0.4,0.5,0.6,0.8,1.0,1.2}r\times\{0.3,0.4,0.5,0.6,0.8,1.0,1.2\} (7 levels, sign-flipped per arm)

  • •

    Forward-backward: r×{−0.1,0.0,0.1,0.3,0.5,0.7,0.9}r\times\{-0.1,0.0,0.1,0.3,0.5,0.7,0.9\} (7 levels)

  • •

    Vertical: r×{0.4,0.2,0.0,−0.2,−0.4}r\times\{0.4,0.2,0.0,-0.2,-0.4\} (5 levels)

  • •

    Orientation: pitch {30∘,45∘,60∘}\{30^{\circ},45^{\circ},60^{\circ}\} ×\times yaw {−45∘,−20∘,0∘,20∘,45∘}\{-45^{\circ},-20^{\circ},0^{\circ},20^{\circ},45^{\circ}\} ×\times roll {−15∘,0∘,15∘}\{-15^{\circ},0^{\circ},15^{\circ}\}

Equations 8–9
S=FR​(𝐓base)−5.0⋅|ρ¯−0.65|S=\text{FR}(\mathbf{T}_{\text{base}})-5.0\cdot|\bar{\rho}-0.65| (8)
Iout​(u,v)={Isim​(u,v)if ​(u,v)∈Marm∪Mgripper_visIinpaint​(u,v)otherwiseI_{\text{out}}(u,v)=\begin{cases}I_{\text{sim}}(u,v)&\text{if }(u,v)\in M_{\text{arm}}\cup M_{\text{gripper\_vis}}\\ I_{\text{inpaint}}(u,v)&\text{otherwise}\end{cases} (9)

A.5 Quality Curation Details.

A valid frame needs a hand, IK position error below 5cm, a visible robot and no self-collision. More than one cross-arm contact or a robot mask covering over 70% of the image invalidates it. Short valid runs under 0.3 seconds are eroded.

Q1/Q99-expanded bounds and distribution-based residual/acceleration/jerk thresholds flag motion outliers; episodes exceeding 60% invalid frames are dropped. Exact per-dimension sudden-change thresholds and final handling of every invalid frame are not fully enumerated.

The complete VLM prompt audits 4fps video, tolerates toy substitutes and small deviations, and rejects major semantic mismatches. No human-verified precision/recall for this curator is reported, so passing is not equivalent to physically correct grasping or successful execution.

Complete original PDF block source-curation-prompt.part1
Original semantic-curation prompt
Original selectable source block

You are evaluating whether a manipulation video matches its text description.
This is a ROBOT manipulation dataset. "Hand" refers to the robot’s gripper/end-effector.
Many tasks use FAKE or SIMULATED objects as stand-ins for real objects. This is EXPECTED.
Task Description: {description}
Determine if the robot’s actions match the described task.
Flag MAJOR MISMATCHES: wrong action type, wrong object category, wrong target location, or failed execution.
Be tolerant of: fake/toy objects, minor appearance variations, small spatial deviations, different grasping approaches.
Respond in JSON: {"is_consistent": true/false, "confidence": 0.0-1.0, "reasoning": "..."}

A.6 Color Randomization.

Each rendered robot receives a deterministic sampled HSV color shared across links: hue 0–1, saturation 0.3–1 and value 0.4–1. Robot-color robustness therefore has direct color-augmentation exposure as well as multi-morphology data; the main result does not isolate morphology count alone as its cause.

A.7 Supported Robot Morphologies.

Table 3 and Figure 6 preserve all fifteen supported kinematic configurations, gripper ranges and stated reach values. Rendering a morphology establishes supported synthesis, not successful downstream control on every listed robot.

Complete original PDF block A1.T3
Table 3: Supported robot morphologies.
Original selectable table
Robot DOF Gripper (mm) Reach (m)
Panda 7 0–80 1.272
Kinova Gen3 7 0–85 1.337
IIWA 7 0–85 1.411
Sawyer 7 14–79 1.420
FR3 7 0–80 1.272
xArm7 7 0–85 1.290
UR5e 6 0–85 1.236
UR10e 6 0–85 1.627
Jaco 6 0–125 1.200
ViperX 6 15–87 0.911
WidowX 6 11–55 0.787
ARX-L5 6 0–88 0.855
Piper 6 0–70 0.883
YAM 6 4–75 0.866
Aloha-Agilex 6 7–102 0.853
Complete original PDF block A1.F6
Figure 6: 15 supported robot morphologies. 3D models of all 15 robots.

Appendix B Evaluation Framework Details

B.1 Perturbation Parameters.

Appearance changes texture, illumination or link hue; layout changes table height±4cm, adds 3–5 distractors or shifts the head camera up to 5cm per axis. Semantic tests use 505 paraphrases and fifty new object-instance tasks across seven families. Their complete task names are retained.

Alternative dual-arm bases are spaced 0.60m ARX,0.59m UR5 and 0.65m Franka. Initial end-effector poses and all three camera extrinsics are aligned. This engineered alignment conditions the transfer claim; it is not arbitrary uncalibrated camera placement.

Complete original PDF block source-unseen-tasks.part1
Complete unseen-object task inventory
Original selectable source block
  • •

    Move-to-pad/pot (10): move bottle pad, move cup pad, move bowl pad, move cup pot, move bottle pot, move mug pot, move apple pot, move seal pot, move pillbottle pot, move bowl pot

  • •

    Place-in-basket (11): place soap basket, place mug basket, place perfume basket, place teabox basket, place toycar basket, place apple basket, place bottle basket, place cup basket, place coffeebox basket, place saucecan basket, place hamburg basket

  • •

    Place-on-stand (4): place cup stand, place mug stand, place bottle stand, place apple stand

  • •

    Place-on-scale (3): place bottle scale, place cup scale, place mug scale

  • •

    Put-in-cabinet (3): put apple cabinet, put seal cabinet, put bowl cabinet

  • •

    Bimanual place-A-to-B (12): place cup left bottle, place mug right apple, place bottle left cup, place seal right mug, place mug left cup, place apple left mug, place bottle right apple, place seal left cup, place mug left bottle, place apple right cup, place cup right seal, place bottle left apple

  • •

    Move-away (7): move cup away, move bottle away, move apple away, move mug away, move seal away, move bowl away, move pillbottle away

B.2 Task List.

The full fifty-task list and step limits remain below. Most tasks allow 400 steps, some complex tasks up to 1,700, and new object tasks default to 1,000. Task-dependent budgets affect opportunity to recover and should accompany score comparisons.

Complete original PDF block source-base-tasks.part1
Complete fifty-task inventory and step budgets
Original selectable source block

The 50 tasks: adjust bottle, beat block hammer, blocks ranking rgb, blocks ranking size, click alarmclock, click bell, dump bin bigbin, grab roller, handover block, handover mic, hanging mug, lift pot, move can pot, move pillbottle pad, move playingcard away, move stapler pad, open laptop, open microwave, pick diverse bottles, pick dual bottles, place a2b left, place a2b right, place bread basket, place bread skillet, place burger fries, place can basket, place cans plasticbox, place container plate, place dual shoes, place empty cup, place fan, place mouse pad, place object basket, place object scale, place object stand, place phone stand, place shoe, press stapler, put bottles dustbin, put object cabinet, rotate qrcode, scan object, shake bottle, shake bottle horizontally, stack blocks three, stack blocks two, stack bowls three, stack bowls two, stamp seal, turn switch.

Step limits per task range from 400 (e.g., click bell, adjust bottle) to 1700 (put bottles dustbin), with most tasks at 400. Complex multi-step tasks use higher limits: open microwave 1500, blocks ranking 1200, stack blocks three 1200, stack bowls 900–1200, hanging mug 900, stack blocks two / handover block 800, place cans plasticbox 800. New/unseen tasks default to 1000 steps.

B.3 Protocol.

The protocol states fifty binary episodes per task. Visual and Scene each average three perturbations, Embodiment averages three robots, and Task averages object/paraphrase settings. These macro averages are defined by the chosen factor grouping.

Table 5 includes odd integer percentages such as 77% despite the stated fifty-episode binary protocol, which alone gives two-point increments. Additional repeats or pooling are not specified here; exact reported values are preserved without inventing trial counts.

B.4 EBench Table Top.

EBench’s seven tasks cover coffee beans, cup flipping/cookies, frame placement, gear installation, peg insertion, glass placement and nut tightening in Isaac Sim. Its higher camera is closer to an ego perspective, but it also differs in tasks, data and simulator from RoboTwin.

Appendix C Model Architecture Details

The backbone takes two or three RGB views and a structured instruction. Intrinsics/extrinsics enter visual features through mRoPE; eight learned query tokens compress context for the action head. The appendix states 32 step chunks, eight training steps and four inference steps, with downstream chunk changes detailed later.

C.1 Structured Prompt.

The exact two-line template specifies embodiment and instruction. Embodiment identifies human/robot/synthesized source and model, and is replaced with None 15% of the time; instruction is always kept. This supplies explicit domain information rather than requiring all morphology differences to be inferred from pixels.

Complete original PDF block source-model-prompt.part1
Original model input template
Original selectable source block
embodiment: {type}_{model}
instruction: {task description}

C.2 Camera-Frame Relative EEF.

Each arm action has translation delta, rotation-vector delta and gripper command. Equation 10 transforms deltas with R_ce, derived from the end-effector-to-camera transform T_wc⁻¹T_we. This requires appropriate estimated or known geometry even though the prose describes avoiding explicit extrinsic calibration.

The paragraph calls its inputs base-frame deltas, but the displayed Δp_e^e and R_ce convention correspond to end-effector-expressed deltas. Base/world-expressed deltas would require their corresponding rotation to camera coordinates. The exact equation is retained and the frame wording left qualified; this does not establish an implementation error.

Equation 10
Δ​𝐩c​c=𝐑c​e​Δ​𝐩e​e,Δ​𝐑c​c=𝐑c​e​Δ​𝐑e​e​𝐑c​e⊤\Delta\mathbf{p}_{cc}=\mathbf{R}_{ce}\,\Delta\mathbf{p}_{ee},\quad\Delta\mathbf{R}_{cc}=\mathbf{R}_{ce}\,\Delta\mathbf{R}_{ee}\,\mathbf{R}_{ce}^{\top} (10)

C.3 Flow Matching.

Flow matching linearly interpolates clean actions at t=0 and Gaussian noise at t=1, predicting velocity ε−a₀ with squared-error loss. Inference uses four Euler steps. The exact equations remain; the main model description’s eight-step wording should not be treated as an independently established inference count.

Equations 11–12
𝐚t=(1−t)​𝐚0+t​ϵ\mathbf{a}_{t}=(1-t)\,\mathbf{a}_{0}+t\,\boldsymbol{\epsilon} (11)
ℒ=𝔼t,𝐚0,ϵ​‖𝐯θ​(𝐚t,t,𝐜)−(ϵ−𝐚0)‖2\mathcal{L}=\mathbb{E}_{t,\mathbf{a}_{0},\boldsymbol{\epsilon}}\left\|\mathbf{v}_{\theta}(\mathbf{a}_{t},t,\mathbf{c})-(\boldsymbol{\epsilon}-\mathbf{a}_{0})\right\|^{2} (12)

Appendix D Training Details

D.1 Pretraining.

Pretraining uses eightA 100s,96 samples/step,200ksteps, AdamWβ=(0.9,0.95),5kwarmup and cosine decay. The action head has ten times the backbone learning rate; bfloat 16 and ZeRO-1 reduce memory use. No image augmentation is used at this stage beyond the synthesized-data pipeline’s own rendering/color variations.

D.2 Pretraining Data.

Robot-format data includes DROID 511h and AgibotWorld 2,404h of real demonstrations plus InternData 3,650h of simulation. “Robot-only” therefore does not mean real-only. Synthesized ego data repeats source activities across bodies; mixing ratios are sampling weights under the shared 19.2million-frame budget.

Figure 7’s wedges show training sampling weights, not hour shares: Robot DROID 5%/Agibot 45%/InternData 50%; Ego ANT 3%/EgoDex 40%/ViTRA 17%/EgoVerse 40%. Total-volume labels should not be used to reinterpret those wedges as measured collection proportions.

Complete original PDF block A4.F7
Figure 7: Pretraining data composition. Slices show per-source sampling weights (training mix); panel titles give the total data volume. (a) Robot (∼\sim6,565 h). (b) Ego2R (∼\sim18,561 h).

D.3 Finetuning.

Fine-tuning uses the same optimizer/rates with ZeRO-2 and ColorJitter for 50ksteps. RoboTwin uses 2,500 episodes, chunk 20 and replanning every 16 steps. EBench uses 2,800 episodes and chunk 32; physical control also uses chunk 32. These settings qualify the generic 32 step architecture description.

D.4 Ablation Configurations.

Filtered raw ego is compared with ARX-only, five-, ten- and fifteen-morphology render sets, then fifteen plus raw ego. Approximate volumes range 1,237h for one body to 18,561h for fifteen, or 20,501h with raw ego. All exclude robot data during pretraining but receive RoboTwin robot fine-tuning afterward. Equal training steps do not equalize unique-view diversity.

Complete original PDF block A4.F8
Figure 8: Ego play →\to Ego2R synthesis on real-world scenes. Top: original egocentric human manipulation. Bottom: Ego2R pipeline output with ACone robot overlay.

Appendix E Real Robot Experiment Details

E.1 Platform and Tasks.

ARX ACone has two 6DoF arms, parallel grippers and head/two-wrist RGB views, running at 15Hz with replanning every 32 steps. Tasks involve three fruits; drawer opening/two blocks/closure; two towel folds; broom pickup/sweep/dump/return; and bimanual handover/insertion of two screws. Long open-loop chunks and these task scopes define the tested setting.

E.2 Data.

One hundred teleoperated demonstrations and about 35 minutes of human play across five scenes provide the real setup. Path B converts play into 675ACone episodes. Adding them changes available scene-specific supervision as well as modality, so the final gain is not solely a pretraining effect.

E.3 Ego Play to Ego2R Visualization.

Figure 8 preserves representative original human frames and their ACone overlays. It demonstrates visual synthesis, while selected frames alone cannot establish that every generated action is contact-feasible or that semantic curation removes every artifact.

E.4 Scoring.

Each task totals 100partial-credit points: fruits divide across three placements, towel across two folds, and the other tasks across four stages. Twenty trials are scored per task. Figure 4’s labels are preserved, but these means are progress scores; a low nonzero score can occur with zero completed whole tasks.

Complete original PDF block A5.T4
Table 4: Real robot scoring.
Original selectable table
Task Step 1 Step 2 Step 3 Step 4
Put Fruits 33.3 33.3 33.3 —
Put Blocks 25 25 25 25
Fold Towel 50 50 — —
Sweep Trash 25 25 25 25
Insert Screw 25 25 25 25

Appendix F Additional Experimental Results

F.1 Comparison with Pi0.5.

Pi0.5 uses the same camera-relative RoboTwin interface through OpenPI but a different backbone and pretraining corpus. This is a contextual system comparison rather than a controlled data-source ablation. Figure 9 retains all eleven perturbation bars, including Franka where the mixed model remains weak.

Complete original PDF block A6.F9
Figure 9: Comparison with Pi0.5 across RoboTwin settings.

F.2 Per-Task Results.

Table 5 retains every task/model/Clean/Randomized cell. The mixed model’s average improves, but tasks such as open microwave and grab roller regress relative to Robot-only. Average gains do not imply taskwise dominance, and the table provides no repeated-training uncertainty.

Complete original PDF block A6.T5
Table 5: Per-task results on RoboTwin including Pi0.5 (success rate %).
Original selectable table
Task Pi0.5 Robot-only Ego2R (1:3) Ego2R (3:1) Ego2R (1:1)
Clean Rand Clean Rand Clean Rand Clean Rand Clean Rand
adjust_bottle 62.0 18.0 94.0 77.0 92.0 79.0 100.0 72.0 89.0 91.0
beat_block_hammer 48.0 16.0 63.0 27.0 64.0 56.0 57.0 18.0 65.0 71.0
blocks_ranking_rgb 74.0 16.0 56.0 60.0 64.0 63.0 49.0 43.0 78.0 60.0
blocks_ranking_size 32.0 0.0 30.0 22.0 46.0 21.0 43.0 7.0 52.0 16.0
click_alarmclock 44.0 56.0 96.0 93.0 100.0 87.0 100.0 85.0 100.0 100.0
click_bell 32.0 40.0 100.0 95.0 100.0 94.0 100.0 93.0 100.0 100.0
dump_bin_bigbin 72.0 70.0 54.0 73.0 72.0 69.0 81.0 76.0 86.0 72.0
grab_roller 94.0 46.0 98.0 71.0 68.0 48.0 91.0 64.0 89.0 63.0
handover_block 16.0 0.0 5.0 2.0 18.0 4.0 39.0 9.0 36.0 9.0
handover_mic 26.0 4.0 69.0 13.0 86.0 34.0 83.0 17.0 97.0 23.0
hanging_mug 10.0 4.0 14.0 16.0 14.0 9.0 12.0 9.0 10.0 10.0
lift_pot 8.0 2.0 93.0 28.0 84.0 14.0 95.0 44.0 93.0 44.0
move_can_pot 28.0 0.0 47.0 50.0 30.0 40.0 42.0 85.0 47.0 65.0
move_pillbottle_pad 60.0 44.0 63.0 60.0 68.0 69.0 41.0 67.0 74.0 82.0
move_playingcard_away 90.0 52.0 74.0 58.0 88.0 92.0 88.0 42.0 92.0 63.0
move_stapler_pad 22.0 6.0 23.0 19.0 34.0 21.0 30.0 13.0 43.0 31.0
open_laptop 68.0 20.0 77.0 61.0 74.0 56.0 64.0 46.0 75.0 58.0
open_microwave 26.0 8.0 77.0 59.0 42.0 29.0 37.0 32.0 50.0 25.0
pick_diverse_bottles 56.0 18.0 60.0 37.0 58.0 53.0 71.0 50.0 70.0 50.0
pick_dual_bottles 82.0 14.0 91.0 59.0 80.0 77.0 83.0 55.0 97.0 54.0
place_a2b_left 60.0 10.0 40.0 55.0 58.0 42.0 54.0 48.0 73.0 44.0
place_a2b_right 58.0 14.0 42.0 49.0 52.0 41.0 53.0 51.0 64.0 53.0
place_bread_basket 68.0 44.0 75.0 55.0 72.0 64.0 76.0 59.0 79.0 62.0
place_bread_skillet 86.0 46.0 76.0 41.0 76.0 50.0 80.0 61.0 87.0 53.0
place_burger_fries 90.0 78.0 96.0 83.0 98.0 81.0 97.0 88.0 96.0 88.0
place_can_basket 40.0 0.0 49.0 25.0 38.0 9.0 34.0 15.0 34.0 16.0
place_cans_plasticbox 94.0 74.0 90.0 68.0 54.0 68.0 97.0 59.0 70.0 59.0
place_container_plate 96.0 58.0 86.0 76.0 92.0 79.0 93.0 73.0 93.0 81.0
place_dual_shoes 54.0 12.0 50.0 30.0 34.0 27.0 35.0 17.0 35.0 19.0
place_empty_cup 74.0 56.0 73.0 74.0 86.0 83.0 88.0 78.0 88.0 66.0
place_fan 36.0 30.0 48.0 54.0 48.0 24.0 41.0 50.0 39.0 46.0
place_mouse_pad 18.0 0.0 41.0 34.0 22.0 36.0 30.0 25.0 29.0 33.0
place_object_basket 24.0 4.0 65.0 50.0 76.0 28.0 77.0 51.0 69.0 48.0
place_object_scale 58.0 44.0 38.0 46.0 40.0 33.0 52.0 45.0 40.0 47.0
place_object_stand 94.0 40.0 70.0 75.0 80.0 77.0 87.0 71.0 82.0 82.0
place_phone_stand 40.0 4.0 65.0 49.0 48.0 63.0 70.0 61.0 75.0 45.0
place_shoe 64.0 24.0 84.0 84.0 78.0 84.0 63.0 58.0 75.0 76.0
press_stapler 42.0 32.0 86.0 71.0 82.0 62.0 76.0 70.0 83.0 62.0
put_bottles_dustbin 14.0 2.0 25.0 20.0 42.0 32.0 41.0 19.0 50.0 39.0
put_object_cabinet 22.0 0.0 41.0 20.0 30.0 19.0 39.0 36.0 43.0 29.0
rotate_qrcode 50.0 2.0 31.0 17.0 54.0 16.0 54.0 36.0 69.0 37.0
scan_object 62.0 38.0 62.0 37.0 66.0 46.0 66.0 46.0 56.0 33.0
shake_bottle 96.0 72.0 91.0 75.0 88.0 89.0 99.0 75.0 96.0 87.0
shake_bottle_horizontally 98.0 74.0 98.0 81.0 84.0 87.0 99.0 84.0 97.0 92.0
stack_blocks_three 68.0 16.0 14.0 23.0 14.0 28.0 10.0 13.0 28.0 23.0
stack_blocks_two 80.0 56.0 75.0 70.0 82.0 75.0 77.0 56.0 95.0 61.0
stack_bowls_three 38.0 36.0 50.0 41.0 40.0 43.0 48.0 50.0 47.0 45.0
stack_bowls_two 82.0 44.0 76.0 78.0 66.0 73.0 73.0 67.0 72.0 72.0
stamp_seal 32.0 2.0 35.0 42.0 46.0 30.0 34.0 30.0 44.0 42.0
turn_switch 56.0 40.0 55.0 42.0 40.0 42.0 53.0 41.0 50.0 48.0
Average 54.9 27.7 62.2 50.9 61.4 51.0 64.1 49.2 68.1 53.5
Concept demonstration

Open image ↗