Literature
Display
← All papers

Large Video Planner Enables Generalizable Robot Control

Boyuan Chen · Tianyuan Zhang · Haoran Geng · Caiyi Zhang · Peihao Li · Kiwhan Song · William T. Freeman · Jitendra Malik · Pieter Abbeel · Russ Tedrake · Vincent Sitzmann · Yilun Du

Preprint posted · arXiv (preprint) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

Large Video Planner uses generated video as an intermediate motion plan. Its central distinction is between imagining a plausible interaction and successfully reconstructing, retargeting and executing that interaction on a physical robot.

At a glance

LVP adapts a 14-billion-parameter video generator to action-focused human and robot clips, then extracts robot trajectories from generated hand motion. History-conditioned diffusion and curated data improve judged video-plan quality on 100 third-party prompts. Selected plans also support open-loop execution on a Franka gripper and a G1 dexterous hand. The evidence supports useful transfer in these settings, while video realism, metric reconstruction and reliable physical execution remain separate challenges.

1 Introduction

Robot action datasets are smaller and narrower than web video collections. LVP explores video pretraining as a source of interaction structure: given a scene image and task instruction, generate a plausible future interaction, then convert its hand trajectory into robot commands. It builds on a pretrained video model rather than learning a foundation model from scratch.

This edition covers all 29 pages of arXiv v2, dated 8 May 2026; the catalog retains the original 2025 posting date. “Zero-shot” refers to deployment without task-specific policy fine-tuning, with substantial prior video training and a reconstruction/calibration/retargeting pipeline. Third-party task choice does not establish that every action was absent from all underlying pretraining data.

Complete original PDF figure S1.F1.
Figure 1: Autonomous Robot Execution with Large Video Planner. Our approach uses video generation as a visual motion planner in pixel space. From a single image and a task instruction, the model generates a video depicting how the task should be completed. The predicted human motion is then retargeted to a robot hand for real-world execution, enabling zero-shot visual planning in diverse scenes.

2 Related Work

Video generation: Diffusion Forcing allows different noise levels across sequence tokens, while History Guidance strengthens conditioning on previous frames. Robot models: VLAs output actions directly; language-planner/controller systems split reasoning from execution. Video-based robotics: visual policies generate desired futures, whereas dynamics models predict outcomes conditioned on actions.

LVP follows the visual-policy route. Its action extraction combines HaMeR hand reconstruction, MegaSaM scene/depth reconstruction and Dex-Retargeting, rather than training one inverse-dynamics network on the generated frames.

3 Method

The two stages are video planning and action extraction. A scene-conditioned video supplies an imagined hand interaction; reconstruction and morphology-specific control convert that motion into commands. A convincing generated door-opening video is therefore only the first stage of a successful robot door-opening attempt.

3.1 LVP: A Video Foundation Model for Generative Planning

LVP retains the pretrained latent-video architecture while changing temporal noise conditioning, guidance and training data to favor manipulation plans.

Latent Diffusion

A frozen causal 3D VAE compresses video into 16-channel latents, and a transformer learns a flow-matching denoiser. Noise follows zₖ = (1−k)z₀ + kε; the printed loss is ‖fθ(zₖ,c,k)−k(ε−z₀)‖₂. The prose calls the target ε−z₀, so its extra k factor is a source inconsistency rather than a detail silently corrected here.

The stated 8×8×4 compression and Appendix B’s 832×480→104×60 latent grid agree on eightfold spatial compression. The main text instead prints H/4 and W/4 in its shape formula. The appendix’s concrete dimensions are the clearer implementation description.

Complete original PDF figure S3.F2.
Figure 2: LVP Overview: (a) Overview of the latent video diffusion framework. We first use a temporally causal VAE to encode video clips into compressed 3D latent representations. Then we train a diffusion transformer in this latent space with flow matching objectives. (b) We jointly train image-to-video (I2V) and video-to-video (V2V) with a modified diffusion forcing training strategy. During training, a random context length between 00 and 66 frames is selected, dividing the video into history and future segments. Two independent noise levels are applied to these segments, and the history segment is set to zero noise with a 50% probability. We visualize four representative cases of this noisy training strategy: the top row shows that longer contexts enable V2V training; the second row shows clean first-frame contexts, which exactly aligns with standard I2V training; and the botton two rows show noisy context frames, which improve robustness to out-of-distribution conditioning.

Diffusion Forcing Transformer

Initialize from Wan 2.1 I2V 14B. Sample a history length of zero to six latent frames and apply independently chosen noise levels to history and future segments. Set history noise to zero half the time; other cases teach conditioning on partially corrupted context. Clean history at sampling supports either image-to-video or video extension.

Per-token noise embeddings replace a uniform timestep embedding. LVP removes Wan’s extra mask/guidance input channels while retaining first-frame CLIP and UMT5 text cross-attention. The sampled context length is in latent frames, despite the shorter wording in Figure 2’s caption.

Enhanced Temporal Coherence with History Guidance

History guidance compares predictions with clean context against predictions with that context fully noised. The latter still receives the task text: “unconditional” here means unconditional on history, not on every input. Equation 3 amplifies the difference, and Equation 4 adds a text-guidance term to reinforce both observation and instruction.

The printed combined score contains two copies of the jointly conditioned score when both guidance weights are zero. It is retained exactly as published, rather than replaced with a more conventional normalized guidance formula. Better-looking plans do not constitute a dynamics or contact-consistency guarantee.

Equations 1–4
∇log⁡p​(xk|xhist,ctext,k).\nabla\log{p(x_{k}|x_{\text{hist}},c_{\text{text}},k)}. (1)
∇log⁡p​(xk|ctext,k).\nabla\log{p(x_{k}|c_{\text{text}},k)}. (2)
shist\displaystyle s_{\text{hist}} =(1+whist)∇logp(xk|xhist,ctext)−whist∇logp(xk|ctext).\displaystyle=(1+w_{\text{hist}})\nabla\log{p(x_{k}|x_{\text{hist}},c_{\text{text}})}-w_{\text{hist}}\nabla\log{p(x_{k}|c_{\text{text}})}. (3)
sfinal=\displaystyle s_{\text{final}}= (1+whist)∇logp(xk|xhist,ctext)−whist∇logp(xk|ctext)+\displaystyle(1+w_{\text{hist}})\nabla\log{p(x_{k}|x_{\text{hist}},c_{\text{text}})}-w_{\text{hist}}\nabla\log{p(x_{k}|c_{\text{text}})}+ (4)
(1+wtext)∇logp(xk|xhist,ctext)−wtext∇logp(xk|xhist)\displaystyle(1+w_{\text{text}})\nabla\log{p(x_{k}|x_{\text{hist}},c_{\text{text}})}-w_{\text{text}}\nabla\log{p(x_{k}|x_{\text{hist}})}

Autoregressive Extension for Multi-Stage Planning

Condition each extension on up to six latent history frames, described as about 24 video frames. Repeated extension produces multistage visual plans; Figure 7 shows two three-stage examples. These qualitative rollouts do not establish closed-loop long-horizon robot success rates.

Training Details

Continue pretraining for 60,000 steps at batch 128, reported as 200 billion tokens, then fine-tune 10,000 steps on low-camera-motion human clips from Ego4D, Epic-Kitchens and Panda. The full run takes about 14 days on 128 H100 SXM5 GPUs. The second stage suppresses camera drift; it does not remove downstream reconstruction and control requirements.

Complete original PDF figure S3.F3.
Figure 3: (a) Visualization of our eight dataset sources. First row: four robotics datasets. Second row: four human-centric datasets. (b) Illustration of our video diffusion sampling strategy, where scores estimated with and without history are linearly combined. Text conditioning and the diffusion transformer are omitted for clarity.

3.2 LVP-1M : A video dataset of human and robot actions

LVP-1M is described as 1.4 million distinct action clips with multiple captions. The rounded source counts in Table 1 sum to 1.486 million, so the headline is approximate. The 4.1 million captioned examples count repeated clips with different captions, not 4.1 million unique videos.

Video Sources

Eight sources mix four robot datasets—Bridge, DROID, Language Table and AgiBot-World—with Ego4D, Epic-Kitchens, Something-Something and filtered Panda-70M human videos. Robot clips provide morphology examples, egocentric clips provide hand interactions, and web clips broaden tasks/scenes. The raw mix is heavily dominated by AgiBot, motivating weighted sampling.

Temporal Alignment

Target roughly three-second clips at 16 fps. Choose dataset-specific subsampling/speed factors by visual inspection so slower teleoperated motions resemble typical human action speed, and split long sequences into atomic actions. This is motion-time normalization, not simply converting every source to the same nominal frame rate; robot execution later uses separate control frequencies.

Quality Filtering

Filter low-quality or badly sized clips, excessive camera motion, missing visible hands/grippers and labeled robot failures. Panda filtering combines caption keywords, person detection and Gemini judgments about hand activity, meaningful actions, normal speed and shot changes. These automated filters select preferred examples; their output is not a verified physical-success label for every clip.

Action-Centric Re-Caption

Gemini Flash expands existing short instructions using the initial image, or describes the action from the complete video when no annotation exists. Each clip receives two to five captions of varying detail. Existing short action labels improve the model’s caption accuracy, illustrating that broad visual description alone can miss fine-grained dynamics.

Name # Filtered clips Robot? Ego-centric? Morphology In-the-wild Bimanual
Bridge 25k Yes Third-person Gripper No No
DROID 192k Yes Third-person Gripper No Yes
Language-Tables 71k Yes Third-person Gripper No No
AgiBot-World 863k Yes Third-person Gripper No Yes
Ego4D 39k No Egocentric Human Hand Yes Yes
Epic-Kitchens 7k No Egocentric Human Hand No Yes
Something-Something 93k No Third-person Human Hand Yes Yes
Panda-70M (filtered) 196k No Third-person Human Hand Yes Yes
Table 1: LVP-1M sources and properties after curation. The dataset targets action-centric clips with broad diversity across embodiment (robot/human), viewpoint (ego/third-person), scene type (in-the-wild/lab), morphology, and bimanuality, totaling 1.4 million clips.

3.3 Robot Actions from Video Plans

Generated human-hand videos are reconstructed and retargeted to either a dexterous hand or a parallel gripper. Most dexterous experiments follow the human-hand route; visual resemblance between a generated motion and robot motion is insufficient without spatial alignment and feasible controls.

Complete original PDF figure S3.F4.
Figure 4: Pipeline from Video to Action. Given a generated video depicting a human hand performing a task, we first reconstruct and track the hand in 3D (second column). The reconstructed hand motion is then retargeted to dexterous hands or grippers (third column). Finally, the retargeted trajectory is transformed into the robot’s control frame and executed in the real world (rightmost column).

Human Hand Motion Estimation

HaMeR estimates per-frame MANO articulation and wrist orientation. MegaSaM supplies depth, intrinsics and camera poses; back-projecting wrist pixels gives a more temporally consistent translation trajectory. Invalid positions are interpolated, quaternion gaps use SLERP with sign continuity, and Savitzky–Golay filtering smooths positions/quaternion components before quaternion renormalization.

Reconstruction remains learned and fallible. The paper’s metric-scale claims are illustrated qualitatively, without a ground-truth wrist-error benchmark demonstrating exact scale recovery for every generated scene.

Robot Finger Motion Retargeting

Dex-Retargeting optimizes robot finger configurations to match estimated human keypoints with a DexPilot-style objective. Export wrist poses, finger joint trajectories and joint metadata, and inspect rendered robot-hand motion in simulation. This is kinematic transfer, not a learned model of grasp force or contact mechanics.

Real-Robot Execution

Transform wrist poses from the first-camera frame into the robot control frame, solve arm inverse kinematics with cuRobo, and send synchronized finger commands directly. Equation 5 includes both a fixed rotation and translation offset, despite prose emphasizing rotation. Executing a predetermined trajectory makes the overall system open-loop with respect to new visual feedback.

Equation 5
𝐩𝒲,tℛ=𝐌​𝐩𝒲,t𝒞0+t,𝐑𝒲,tℛ=𝐌​𝐑𝒲,t𝒞0\mathbf{p}^{\mathcal{R}}_{\mathcal{W},t}=\mathbf{M}\,\mathbf{p}^{\mathcal{C}_{0}}_{\mathcal{W},t}+t,\qquad\mathbf{R}^{\mathcal{R}}_{\mathcal{W},t}=\mathbf{M}\,\mathbf{R}^{\mathcal{C}_{0}}_{\mathcal{W},t} (5)

4 Evaluating task-level generalization

The intended test goes beyond new object positions to varied task verbs and surroundings. Assess generated-video plans separately from physical execution, and interpret “unseen” within the reported task-selection protocol rather than as an audited absence from all video pretraining.

4.1 Third-party selection of novel tasks

Participants propose 3–5-second manipulation tasks, photograph the scene with a hand and target visible, and supply an instruction. A separate group filters roughly 200 submissions down to 100, removing poor images, protocol violations and basic tabletop pick/push tasks. Gemini rewrites the instructions before generation. This reduces author task selection but remains a curated benchmark with language-model assistance.

4.2 Evaluating video motion planning

Each model produces four videos for each of 100 prompts. Third-party raters judge four levels: correct object contact; correct final state ignoring motion quality; complete task with continuous plausible motion, allowing minor physics artifacts; and visually flawless completion. These are judgments of rendered video, not measured physical robot success.

LVP’s average/Best@4 scores are 87.3/100 for contact, 63.2/85 for end state, 59.3/82 for completion and 44/71 for perfect completion. Wan’s completion and perfect-completion averages are 39.3 and 20.5, so LVP improves those by 20.0 and 23.5 percentage points. Best@4 assumes selecting the successful sample among four and is not single-sample reliability.

All baselines use the same four-sample protocol. The prose labels Hunyuan 13B while Table 2 says 14B; original labels are retained. Rater counts, agreement statistics and confidence intervals are not clearly supplied. Example contact/physics differences and multistage extensions are qualitative evidence, not independent mechanistic ablations of every training change.

Complete original PDF figure S4.F5.
Figure 5: Baseline Comparison. LVP accurately generates videos of hand interactions in a zero-shot setting, such as pulling out a tissue (left) and opening a gate (right). Baseline models (Wan, Cosmos-Predict 2, Hunyuan) often produce spatial or semantic inconsistencies, highlighted by red circles. The first frame and task instruction shown under each column serve as the generation conditions.
Complete original PDF figure S4.F6.
Figure 6: Visualization of generated video plans. Eight examples in our in-the-wild test set with generated videos by LVP.
Complete original PDF figure S4.F7.
Figure 7: Multi-Stage Video Plans. Our LVP can generate long-horizon video plans by repeatedly extending videos conditioned on the last six latent frames. Each example illustrates a three-stage motion plan obtained through two iterative extensions.
Method Level 1: Correct contact Level 2: End state Level 3: Task complete Level 4: Perfect
Average (%) Best@4 Average (%) Best@4 Average (%) Best@4 Average (%) Best@4
Wan 2.1 I2V 14B 83.9 99.0 47.0 80.0 39.3 76.0 20.5 53.0
Cosmos-Predict 2 14B 45.3 81.0 11.9 35.0 7.5 24.0 2.5 9.0
Hunyuan I2V 14B 68.7 96.0 27.3 65.0 13.5 42.0 7.2 27.0
Ours 87.3 100.0 63.2 85.0 59.3 82.0 44.0 71.0
Table 2: Video Plan Evaluation. Evaluation on 100 in-the-wild manipulation prompts collected from third-party participants. We report the average success rate (Average) and Best@4 for each level. Our method achieves substantially higher success at Levels 3–4 than the baselines, indicating stronger generation of coherent, task-complete plans in in-the-wild settings.

4.3 Evaluating Real-World Robot Manipulation

Physical tests combine video generation, reconstruction, calibration, retargeting and execution. Figure 8 contains the actual task success counts; references to “Table 8” in the prose point to that figure’s embedded table.

Tasks

Franka groups A/C test picking, placement, drawer opening, button pressing and out-of-distribution objects/scenes. G1/Inspire group B tests eight tasks including elevator buttons, sweeping, opening boxes/doors, wiping, scooping and tape tearing. The prose additionally mentions in-hand rotation and pen/screwdriver tool use, but these are not listed in the quantitative task table and should not be treated as measured tasks.

Complete original PDF figure S4.F8.
Figure 8: Robot Execution Evaluation. Left: Comparison of Task Success Across Methods on (1) Franka Arm with Parallel-Jew Gripper and (2) G1 with Inspire Hands. 1 denotes tests on OOD objects; 2 denotes scenes that differ substantially from the training videos. Right: Visualization of the robot tasks and experiments.
Selectable task results
Task Set and Tasks Ours π𝟎\mathbf{\pi_{0}} OpenVLA
Task Group A: w/ Parallel Gripper
Pick Objects 5/10 3/10 0/10
Pick A into B 3/10 1/10 0/10
Open Drawer 2/10 1/10 0/10
Press Button 4/10 0/10 0/10
Task Group B: w/ Dexterous Hands
Pick Objects 4/10 N/A N/A
Press Elevator Button 4/5 N/A N/A
Sweep Tennis Ball into Bucket 5/5 N/A N/A
Open Box (a) 2/10 N/A N/A
Open Door (b) 6/10 N/A N/A
Wipe Table (c) 8/10 N/A N/A
Scoop Coffee Beans (b) 3/5 N/A N/A
Tear off Clear Tape (b) 2/5 N/A N/A
Task Group C: Out-of-distribution Set
Pick Objects (OOD Object1) 4/10 0/10 0/10
Pick A into B (OOD Object1) 2/10 0/10 0/10
Pick Objects (OOD Scene2) 6/10 1/10 0/10
Pick A into B (OOD Scene2) 1/10 0/10 0/10
Complete original PDF figure S4.F9.
Figure 9: Zero Shot Robot Manipulation with LVP. The model generates videos for diverse tasks, enabling zero-shot execution on both a dexterous hand and a parallel-jaw gripper.

Baselines

Run released π0 and OpenVLA checkpoints without further fine-tuning on the Franka tasks. They are incompatible with the multi-finger setup and have N/A entries for all dexterous tasks. Accordingly, dexterous performance is a feasibility demonstration, not a head-to-head win over these baselines.

Results

Summing Figure 8’s counts, LVP succeeds in 14/40 standard Franka trials and 13/40 out-of-distribution trials, versus π0’s 5/40 and 1/40; OpenVLA records zero in both groups. These aggregates are computed from the table, not separately reported metrics.

The dexterous tasks range from 2/10 for opening a box to 5/5 for sweeping; door opening is 6/10, wiping 8/10, scooping 3/5 and tape tearing 2/5. Across its eight listed tasks this is 34/60 successes. Five or ten attempts per task give limited precision. The study demonstrates varied successes but not uniformly reliable execution, and does not fully specify how robot video candidates are selected or how failures divide among generation, reconstruction and control.

5 Limitations

Generating one plan takes several minutes on one A100, precluding direct real-time replanning. Reconstruction and pose-estimation errors propagate into control; even a correctly reconstructed human motion may not suit a robot hand or low-DOF gripper. Open-loop execution cannot correct contact errors as they develop. Distillation and causal generation are proposed improvements, not capabilities demonstrated by the evaluated system.

6 Conclusion

LVP shows that curated video training can support useful visual plans and some cross-embodiment execution. The contribution is the video-planning/data/control combination; stronger autonomous manipulation still needs reliable geometry, feasible contact, feedback and lower latency. The authors report releasing model, data and code; release usability was not independently tested in this edition.

Appendix A Details of LVP-1M Dataset

The dataset appendix specifies the Panda filtering pipeline and optical-flow processing used for curation.

A.1 Extracting hand interaction videos from Pandas

Use 108 whitelist and 84 blacklist keywords to select about 692,000 source videos, split them into four-second pieces, and run person detection at 1 fps on 768×1024 frames. Gemini 2.0 Flash keeps clips judged to contain rich hand motion, meaningful activity and normal playback, with no shot change, leaving 196,000 clips.

The appendix says one to three valid people and defines a valid box as height ≤60 pixels and width ≥120; the main text says one to four people. These unusual/inconsistent criteria are recorded as written, not silently replaced with presumed intended thresholds.

A.2 Data preprocessing pipeline

Figures 10–11 retain the curation pipeline, source composition and reweighted training mixture. The figure orders filtering, captioning and temporal alignment, whereas the main prose introduces temporal alignment before quality filtering; the paper does not fully resolve this sequencing difference.

Filtering our rapid camera motions

Compute OpenCV optical flow at 4 fps on 256×256 images and discard the top 30% of clips by mean spatiotemporal flow magnitude. This screens rapid camera motion using total image motion, which may also include substantial foreground activity.

Complete original PDF figure A1.F10.
Figure 10: LVP-1M curation pipelines. Videos are collected from eight public sources, including four teleoperated robotics datasets and four human-centric activity datasets. We apply three processing stages: (1) aggressive filtering for quality and embodiment, (2) action-focused captioning using Gemini, and (3) temporal frequency alignment to match human motion speeds. The final dataset contains 1.4 million clips with diverse, action-centric text captions.
Complete original PDF figure A1.F11.
Figure 11: Composition of LVP-1M . Distribution of filtered LVP-1M clips across eight sources. Greenish tones indicate teleoperated robot datasets, while reddish tones represent human-centric activity datasets. The rightmost column shows the final sampling ratios after reweighting during first-stage training.

Appendix B Details of Video Generative Model Training

Train on 49 frames at 832×480, encoded into a 104×60×13 latent grid. Continue pretraining uses batch 128, 60,000 steps and learning rate 10⁻⁵ after 1,000 warmup steps. The 10,000-step fine-tuning stage keeps batch size and lowers the rate to 2.5×10⁻⁶.

Data reweighting

Per-source sampling weights are AgiBot 0.375, DROID 0.75, Ego4D 1.5, filtered Panda 0.5, Something-Something 0.5, Bridge 1.0, Epic-Kitchens 2.0 and Language Table 0.05. These are multipliers on source counts, not normalized probabilities. Figure 11 shows the resulting intended mixture.

The printed weights and Table 1 counts do not exactly reproduce Figure 11: for example, their product-normalized DROID share is about 17.7%, whereas the pie chart says 30.4%. The intended training mixture therefore cannot be reconstructed unambiguously from these descriptions alone.

Appendix C Method: Comparison between HaMeR and Our 4D Alignment Module

Figure 12 contrasts raw HaMeR and depth-aligned wrist trajectories for one interaction. The aligned path is presented as more temporally coherent and metrically suitable. No ground-truth path, error table or controlled robot-success ablation is supplied here, so the image supports a qualitative comparison rather than a quantified accuracy gain.

Complete original PDF figure A3.F12.
Figure 12: Comparison between HaMeR and Our 4D Alignment Module. The red curve shows our 4D Alignment Module; the blue one is from HaMeR.

Appendix D Method: Details about Our Video Processing Modules

The reconstruction appendix details coordinate frames, missing-frame completion, smoothing and gripper adaptation. Frame conventions matter because the printed main and appendix transforms are not identical.

D.1 Data Acquisition and Input

Input is monocular RGB video without supplied camera intrinsics/extrinsics. Recover per-frame hand joints and a smooth wrist trajectory relative to the first camera frame. Later camera-to-robot alignment still requires an estimated external transform.

D.2 HaMeR: Hand Mesh and Camera Estimation

HaMeR provides MANO mesh/articulation, wrist pose and weak-perspective camera estimates. Monocular scale ambiguity and independent per-frame estimates make translations noisy; MegaSaM supplies additional depth/sequence constraints rather than guaranteeing exact geometry.

D.3 MegaSaM: Depth-Assisted Structure Recovery

MegaSaM estimates depth and camera motion with sequence-wide refinement/bundle adjustment. Use the first camera as the world reference. Learned depth supports approximate metric alignment, with reconstruction errors acknowledged in the limitations.

D.4 Alignment of HaMeR and MegaSaM

Back-project the wrist pixel using estimated depth and intrinsics (Equation 6), then transform it with frame extrinsics (7). Equation 8 maps translation with M and orientation with MRMᵀ, while main Equation 5 adds a translation offset and uses MR. These may reflect different frame/basis conventions, but the text does not reconcile them; the formulas are preserved and should not be substituted for one another without defining frames.

Equations 6–8
Complete original PDF equation 6. \mathbf{p}^{\text{cam}}_{t,w}=z_{t}\,{\mathbf{K}}^{-1}\begin{bmatrix}u_{t}\\
v_{t}\\
1\end{bmatrix}.
Original selectable MathML𝐩t,wcam=zt​𝐊−1​[utvt1].\mathbf{p}^{\text{cam}}_{t,w}=z_{t}\,{\mathbf{K}}^{-1}\begin{bmatrix}u_{t}\\ v_{t}\\ 1\end{bmatrix}.
𝐩t,ww=𝐑t​𝐩t,wcam+𝐭t.\mathbf{p}^{\text{w}}_{t,w}=\mathbf{R}_{t}\,\mathbf{p}^{\text{cam}}_{t,w}+\mathbf{t}_{t}. (7)
𝐩𝒲,tℛ=𝐌​𝐩𝒲,t𝒞0,𝐑𝒲,tℛ=𝐌​𝐑𝒲,t𝒞0​𝐌⊤.\mathbf{p}^{\mathcal{R}}_{\mathcal{W},t}=\mathbf{M}\,\mathbf{p}^{\mathcal{C}_{0}}_{\mathcal{W},t},\qquad\mathbf{R}^{\mathcal{R}}_{\mathcal{W},t}=\mathbf{M}\,\mathbf{R}^{\mathcal{C}_{0}}_{\mathcal{W},t}\,\mathbf{M}^{\top}. (8)

D.5 Wrist Pose and Robot Frame Transformation

Estimate a full rigid transform between the first-camera world and G1 base using base/view orientation information, then apply rotation and translation (Equation 9). Fill missing positions, interpolate quaternions with sign continuity, apply Savitzky–Golay smoothing and renormalize quaternions (10). The numerical smoothing window/order and detailed calibration procedure are not specified here.

Equations 9–10
𝐩t,wG1=𝐑G1←w​𝐩t,ww+𝐭G1←w\mathbf{p}^{\text{G1}}_{t,w}=\mathbf{R}_{\text{G1}\leftarrow{w}}\,\mathbf{p}^{\text{w}}_{t,w}+\mathbf{t}_{\text{G1}\leftarrow{w}} (9)
Complete original PDF equation 10. \hat{\mathbf{p}}^{\mathcal{R}}_{\mathcal{W},t}=\mathrm{SG}\!\big(\tilde{\mathbf{p}}^{\mathcal{R}}_{\mathcal{W},:}\big)_{t},\qquad\hat{\mathbf{q}}^{\mathcal{R}}_{\mathcal{W},t}=\frac{\mathrm{SG}\!\big(\mathbf{q}^{\mathcal{R}}_{\mathcal{W},:}\big)_{t}}{\left\lVert\mathrm{SG}\!\big(\mathbf{q}^{\mathcal{R}}_{\mathcal{W},:}\big)_{t}\right\rVert_{2}}.
Original selectable MathML𝐩^𝒲,tℛ=SG(𝐩~𝒲,:ℛ)t,𝐪^𝒲,tℛ=SG(𝐪ℛ𝒲,:)t‖SG(𝐪ℛ𝒲,:)t‖2.\hat{\mathbf{p}}^{\mathcal{R}}_{\mathcal{W},t}=\mathrm{SG}\!\big(\tilde{\mathbf{p}}^{\mathcal{R}}_{\mathcal{W},:}\big)_{t},\qquad\hat{\mathbf{q}}^{\mathcal{R}}_{\mathcal{W},t}=\frac{\mathrm{SG}\!\big(\mathbf{q}^{\mathcal{R}}_{\mathcal{W},:}\big)_{t}}{\left\lVert\mathrm{SG}\!\big(\mathbf{q}^{\mathcal{R}}_{\mathcal{W},:}\big)_{t}\right\rVert_{2}}.

D.6 Other End-effectors Retargeting: Parallel-Jaw Grippers.

For parallel-jaw grippers, use GraspNet to propose geometry-compatible grasps and heuristics on human hand motion to trigger grasping. Reducing five-finger motion to two fingers is underconstrained; this additional grasp model and intent logic are part of the execution pipeline.

Appendix E Experiments: Task Set and Tasks Description

Table 3 preserves all 16 task descriptions and robot assignments: eight Franka conditions and eight G1-hand tasks. Success descriptions include reaching a target position for drawers/doors, placing objects into containers, lifting scooped beans, sweeping into a bucket and tearing a strip of tape.

Task Name Robot Setup Task Description
Press Button (1) Franka+Gripper A robot arm is required to locate and press a button initialized at any arbitrary position within its reachable workspace.
Pick Objects (1) Franka+Gripper A robot arm is required to locate, grasp, and lift objects placed at arbitrary positions within its reachable workspace for manipulation.
Pick A into B (1) Franka+Gripper A robot arm is required to grasp object A and accurately place it into container B located at arbitrary positions in workspace.
Open Drawer (1) Franka+Gripper A robot arm is required to grasp the handle and pull open a drawer to a designated position.
Pick Objects (OOD Object) (1) Franka+Gripper A robot arm is required to pick unfamiliar, out-of-distribution objects randomly placed within its workspace.
Pick A into B (OOD Object) (1) Franka+Gripper A robot arm is required to place novel, out-of-distribution objects into container B within the workspace.
Pick Objects (OOD Scene) (1) Franka+Gripper A robot arm is required to pick objects in unseen, out-of-distribution scenes with new layouts or obstacles.
Pick A into B (OOD Scene) (1) Franka+Gripper A robot arm is required to place objects into container B under unseen, out-of-distribution scene configurations.
Pick Objects (2) G1+DexHand A dexterous robot hand is required to locate, grasp, and lift objects from arbitrary positions within workspace.
Open Box (2) G1+DexHand A dexterous robot hand is required to grasp and open a box lid placed in the workspace.
Open Door (2) G1+DexHand A dexterous robot hand is required to grasp and pull open a door handle to a designated position.
Wipe Table (2) G1+DexHand A dexterous robot hand is required to wipe the table surface clean using an object or cloth.
Press Elevator Button (2) G1+DexHand A dexterous robot hand is required to press an elevator button at any arbitrary position in the workspace.
Scoop Coffee Beans (2) G1+DexHand A dexterous robot hand is required to use a scoop to collect coffee beans and lift them successfully.
Sweep Tennis Ball into Bucket (2) G1+DexHand A dexterous robot hand is required to sweep a tennis ball across the surface into a target bucket.
Tear off Clear Tape (2) G1+DexHand A dexterous robot hand is required to peel and tear off a strip of transparent adhesive tape.
Table 3: Task Set and Tasks Description

Appendix F Experiments: Real-World Robots Set Up

Franka executes at 15 Hz; G1 with the Inspire DH56DFX hand runs synchronized arm/hand commands at 5 Hz through a teleoperation-derived control framework. Hand joints are remapped to valid motor ranges. Some humanoid trials use tape to improve friction or compensate for limited grip torque; this is a reported hardware modification. Control frequency is distinct from the several-minute video-planning latency.

Tasks Robot Control Frequency
Task Set1 Franka Panda Arm + Parallel-Jaw Gripper 15 Hz
Task Set2 Unitree G1 Arm + Inspire Hand (DH56DFX) 5 Hz
Table 4: Real-world robot experimental settings.

Appendix G Video Results

Figures 13–14 show nine human-hand and ten mixed hand/gripper video sequences, with original instructions before Gemini rewriting. Figures 15–16 show eight paired generated-video/robot-execution examples, including cabinets, a fridge, beans, sweeping, a laptop, wiping, a first-aid case and an elevator button. All frames and task labels are retained; selected successful examples do not replace the quantitative trial counts.

Complete original PDF figure A7.F13.
Figure 13: Visualization of generated video plans with human hand. Each row shows eight uniformly sampled frames from a generated video plan. The first frame depicts the input condition image that defines the scene. The caption below each sequence indicates the task instruction used to generate the video.
Complete original PDF figure A7.F14.
Figure 14: Visualization of generated video plans with human hand and robot gripper. Each row shows eight uniformly sampled frames from a generated video plan. The first frame depicts the input condition image that defines the scene. The caption below each sequence indicates the task instruction used to generate the video.
Complete original PDF figure A7.F15.
Figure 15: Visualization of generated videos and corresponding robot executions – A. The first row presents five uniformly sampled frames from the generated video plan conditioned on the input scene image. The second row illustrates the robot executing the same task in real world.
Complete original PDF figure A7.F16.
Figure 16: Visualization of generated videos and corresponding robot executions – B. The first row presents five uniformly sampled frames from the generated video plan conditioned on the input scene image. The second row illustrates the robot executing the same task in real world.
Concept demonstration

Open image ↗