Literature
Display
← All papers

Masked Visual Actions for Unified World Modeling

Hadi Alzayer · Wenlong Huang · Haonan Chen · Christopher Luey · Lvmin Zhang · Maneesh Agrawala · Gordon Wetzstein · Li Fei-Fei · Yilun Du · Jiajun Wu · Jia-Bin Huang

Preprint posted · arXiv (preprint) ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

Complete original PDF block S0.F1
Figure 1: Masked Visual Actions. We finetune a video model to condition on masked trajectories of robots, representing robot actions as pixel-space masked motions. Efficiently finetuned on only 15 hours of data, a single checkpoint of the model can act as an action-conditioned forward model to simulate robotic interactions with diverse and unseen embodiments. By conditioning it on object motion, it can also act as an inverse model that synthesizes the robot motion needed to achieve the desired outcome. We showcase the efficacy of our model for policy evaluation, model-based-planning, and action extraction where the video model acts as a policy.

At a glance

This edition follows the complete twenty-one-page v1, retaining all eleven numbered figures, three tables, eight displayed equations and both evaluator prompts. Masked Visual Actions condition a video model on partially revealed entity trajectories: robot pixels specify a forward query, while desired object motion specifies an inverse query.

One video checkpoint is adapted with fifteen hours of robot data. Tests cover video reconstruction, candidate-plan selection, policy evaluation and action extraction. The inverse pipeline reaches 18/20 successes on one simulated task, using a separately trained inverse-dynamics model. The video model’s zero-shot inverse use should not be confused with a pipeline trained without task demonstrations.

The model learns conditional visual correlations, not a certified causal simulator. Generated outcomes tend to overestimate progress; table inconsistencies, evaluator assumptions and unequal baseline inputs qualify the reported results.

1 Introduction

Video priors contain useful motion and interaction patterns, but conventional robot action vectors may be poorly aligned with those representations. The paper encodes the proposed intervention directly as a masked pixel trajectory, allowing one completion model to infer different missing entities.

The same checkpoint is used for forward prediction and inverse video synthesis. Candidate ranking and physical action extraction require additional components: a stochastic policy and VLM evaluator for planning, and a learned inverse-dynamics mapper for execution. These downstream components are part of the system’s data and compute requirements.

2 Related Work

Complete original PDF block S2.F2
Figure 2: Comparing action representations for learning. Low-dimensional robot actions are compact, but embodiment-specific and not aligned with the image observations used by video models. End-effector poses or robot skeletons are more visual, but remain sparse and require the model to infer geometry, contact, and interaction effects. Our masked visual actions provide dense, image-aligned conditioning, making robot motion and action directly visible, yielding a more learnable representation across embodiments and object interaction.

Controllable video generation as a robotic interface.

Controllable video methods use forces, tracks, hand poses, flow and goals. Visual prompting by inpainting and probabilistic structure integration motivate completing missing visual regions under different conditioning. Masked Visual Actions applies that interface to robot interactions; the cited diversity should not be read as proof that every earlier control representation lacks cross-embodiment transfer.

Pixel-grounded action conditioning for robot world models.

Rendered robot masks, point maps, occupancy and action images already supply spatially grounded control. Other methods edit human videos for embodiment transfer. The distinction emphasized here is changing which entity is revealed at query time, reusing a robot-conditioned training checkpoint for object-conditioned completion.

Unified video-action models and downstream uses.

Unified video/action models combine prediction and control through modality masks or diffusion schedules. This work instead masks spatial entities. Forward prediction supports planning and evaluation; inverse video synthesis still needs an embodiment-specific conversion from generated frames to executable controls, supplied here by a separately trained model.

3 Masked Visual Actions

The formulation treats a video as a joint distribution over entity trajectories. A binary spatiotemporal mask reveals the selected entities, and the model completes the rest conditioned on that evidence and the initial frame. The active/passive partition describes how the interface is queried, not a learned agency variable.

Equations 1–5 express conditional distributions. Conditional completion from observational training does not, by itself, identify an interventional causal distribution or guarantee physical feasibility of a supplied trajectory.

Equations 1–3
p⁡(V)=p⁡(e1,e2,…,en),p(V)=p(e_{1},e_{2},\ldots,e_{n}), (1)
p⁡({ei}i∉𝒮|{ej}j∈𝒮,I0),p\bigl(\{e_{i}\}_{i\notin\mathcal{S}}\,\big|\,\{e_{j}\}_{j\in\mathcal{S}},\,I_{0}\bigr), (2)
M⁡(𝒮)=⋃i∈𝒮ei,M(\mathcal{S})=\bigcup_{i\in\mathcal{S}}e_{i}, (3)

Forward model.

Revealing the robot trajectory asks the model to predict the environment’s response. Pixel alignment exposes geometry and contact locations more directly than low-dimensional action vectors. To render novel robot actions, the implementation still needs a mesh, camera calibration and a kinematic trajectory.

Equation 4
p⁡({ei}i∈𝒫|{ej}j∈𝒜,I0).p\bigl(\{e_{i}\}_{i\in\mathcal{P}}\,\big|\,\{e_{j}\}_{j\in\mathcal{A}},\,I_{0}\bigr). (4)

Inverse model.

Revealing desired object motion asks the model to complete corresponding robot motion. The video checkpoint was trained on robot masks only, so this change of conditioning is zero-shot with respect to inverse video training. The demonstrated action-extraction task subsequently uses supervised inverse dynamics; neither the equations nor the examples ensure every desired object path is achievable.

Equation 5
p⁡({ei}i∈𝒜|{ej}j∈𝒫,I0).p\bigl(\{e_{i}\}_{i\in\mathcal{A}}\,\big|\,\{e_{j}\}_{j\in\mathcal{P}},\,I_{0}\bigr). (5)

4 Method

Complete original PDF block S4.F3
Figure 3: Applications. The masked visual actions allow using the video model as a forward model, conditioned on robot actions, or as an inverse model that predicts the robot motion that satisfies the object trajectory. The forward model can be used for planning and choosing the best trajectory sampled from a policy, or policy evaluation. On the other hand, the inverse modeling can be combined with an inverse dynamics model to estimate robot actions from the generated video.
Complete original PDF block S4.F4
Figure 4: Dataset construction. (a) to train our model, we need a reference frame of the initial scene, and the masked visual actions, and train it to reproduce a realistic video of the robot executing the input actions. (b) We use segmentation-based approach by segmenting the robot arm from robotics datasets as the masked visual actions. (c) However, to allow the user to provide arbitrary action trajectory at inference, the model needs to also accept simulated mesh visualization of those actions. As a result, we also include rendering-based dataset. Given that DROID also contains the robot state at each timestep, we render the robot URDF that matches the original video to construct masked visual actions.

4.1 Dataset construction

Training combines successes and failures from DROID and RoboCasa. Segmentation-derived conditioning teaches completion from real robot appearance, while rendering-derived conditioning supports candidate trajectories that were never physically executed.

Segmentation based dataset

SAM 3, prompted with “A robotic arm,” isolates the arm in recorded video without requiring camera calibration. Occlusion boundaries can leak information about the objects’ actual future, and obtaining the exact future segmentation at inference is difficult. These are reasons to include independently rendered trajectories rather than relying only on segmented demonstrations.

Rendering based dataset

Robot states, URDF geometry and refined camera calibration align rendered robot motion to DROID imagery. RoboCasa renders the robot separately from the scene. Translucent rendering and red gripper fingers expose motion despite self-occlusion. This route supports arbitrary proposed kinematic trajectories, while requiring known geometry and calibration; it is not a calibration-free interface for every new robot.

4.2 Model implementation and training

The backbone is Wan-Fun-Control 2.2 14B from the Wan family. Its autoencoder encodes masked videos with uniform gray missing regions, and concatenation injects aligned conditioning. Rank-256 LoRA adapts the model with batch size four on eight H200 GPUs for about ten thousand steps over four days.

Fifteen hours describes adaptation data, not training compute or all data used by the pretrained backbone. The source does not provide an end-to-end inference-time or planning-cost measurement supporting real-time deployment.

Complete original PDF block S4.F5
Figure 5: Generalization to unseen embodiment. While using the raw action state such as in Ctrl-world [25] can work well within the training domain, it collapses on unseen embodiments. However, our method can generalize well to unseen embodiments.

5 Experiments

Evaluation first compares reconstruction and visual control, then tests three robot applications. The same video checkpoint is reused, while proposal policies, the VLM judge and the inverse-dynamics model have distinct training and evaluation roles.

5.1 Controllable video generation

On DROID, Masked Visual Actions reports LPIPS/SSIM/PSNR 0.0945/0.887/23.74 versus Ctrl-World 0.362/0.708/18.15. On unseen BEHAVIOR embodiment, the corresponding main-table values are 0.123/0.843/22.90 versus 0.196/0.837/18.39. These are image/video reconstruction metrics, not direct guarantees of correct contact dynamics.

The held-out DROID labs are unseen by this model’s adaptation but were included in Ctrl-World’s training. BEHAVIOR transfers a single-arm baseline through an active-arm heuristic and adapted cameras; the comparison therefore includes interface adaptation differences.

Wan-Move receives ground-truth mesh tracks. Its main-table BEHAVIOR LPIPS/SSIM are 0.312/0.756, whereas Table C1 gives 0.454/0.528 with the same PSNR 13.17. Both original tables are retained, and no undocumented correction is chosen.

Complete original PDF block S5.F6
Figure 6: Comparing baselines on DROID. Using image-to-video [63] or even trajectory conditioned video generation [13] with GT tracks fails to execute the robot motion or preserve the input scene. On the other hand, our model can competitively match and outperform models that take the raw robot actions [25] while maintaining generalization.
Complete original PDF block S5.T1
Table 1: Baseline comparison on diverse embodiments. We evaluate our method against Ctrl-World [25] on DROID as a seen robotic embodiment, as well as BEHAVIOR, which uses a bimanual robotic embodiment that’s unseen for all methods. Our model outperform the baseline on both datasets, and we include image-to-video and trajectory conditioned video models as a reference.
Original selectable table
DROID BEHAVIOR
Method LPIPS ↓\downarrow SSIM ↑\uparrow PSNR ↑\uparrow LPIPS ↓\downarrow SSIM ↑\uparrow PSNR ↑\uparrow
Image-to-video [63] 0.521 0.548 12.42 0.602 0.457 10.22
Wan-move [13] 0.534 0.562 12.99 0.312 0.756 13.17
Ctrl-World [25] 0.362 0.708 18.15 0.196 0.837 18.39
Masked Visual Actions (Ours) 0.0945 0.887 23.74 0.123 0.843 22.90
Complete original PDF block S5.F7
Figure 7: Comparing action conditioning. Training a video model on different conditioning signals on DROID such as masked visual actions, end effector visualization, or skeleton all work well within the training domain. However, when going beyond the training distribution, such as using a custom end-effector, the models trained on skeleton and end effector position would hallucinate the robot seen in training or transform the robot to match the training. Furthermore, on unseen embodiments such as bimanual robots in BEHAVIOR, using masked actions generalizes gracefully, while other conditioning signals transform and disfigure the robot.
Complete original PDF block S5.T2
Table 2: Ablation on visual conditioning signal. When using sparse conditioning signal, the performance on held out data from the training distribution on DROID is similar to using masked visual actions. However, when using the same robot with unseen gripper (such as on our real world data), or on a robot from unseen embodiment in BEHAVIOR, the gap increases significantly between our masked actions and the other conditioning methods.
Original selectable table
DROID Real world BEHAVIOR
Method LPIPS ↓\downarrow SSIM ↑\uparrow PSNR ↑\uparrow LPIPS ↓\downarrow SSIM ↑\uparrow PSNR ↑\uparrow LPIPS ↓\downarrow SSIM ↑\uparrow PSNR ↑\uparrow
End-effector vis. 0.107 0.878 22.64 0.183 0.858 20.32 0.171 0.815 19.23
Skeleton vis. 0.106 0.878 22.74 0.169 0.866 21.02 0.162 0.824 19.58
Masked Visual Actions 0.0945 0.887 23.74 0.148 0.864 22.79 0.123 0.843 22.90

Comparing the choice of visual actions

The visual-conditioning ablation trains the same backbone with DROID data using end-effector pose, skeleton or dense robot pixels. Dense conditioning better preserves the custom gripper and unseen bimanual morphology in selected examples, while in-domain differences are smaller.

On thirteen custom-gripper real-world reconstruction samples, dense conditioning gives LPIPS 0.148 and PSNR 22.79 versus skeleton 0.169/21.02. Skeleton’s SSIM is slightly higher, 0.866 versus 0.864, within the reported SEMs. Thus dense masks do not lead every metric; broad claims of universal collapse exceed these sampled tests.

5.2 Robotics applications

RoboCasa experiments use the model to rank proposed futures, compare predicted outcomes with simulator outcomes, and synthesize actions from goal-object motion. These settings test different components and should not be collapsed into one general manipulation success rate.

Planning

A Diffusion Policy proposes ten action sequences for each of ten scenes per task. The video model simulates each candidate; Gemini judges success and realism, and the selected original action sequence executes in the simulator. The planner ranks existing proposals rather than optimizing actions through the video model.

Across six tasks, the Figure 8 bars report gains of 24, 26, 21, 11, 9 and 7 percentage points. Curves enumerate subsets of the same ten candidates per scene and average the selected candidate’s simulator success; their points are not independent newly collected runs. Benefits depend on proposal coverage and evaluator accuracy, and extra generation/judging compute is not timed.

Complete original PDF block S5.F8
Figure 8: Application on Planning. By rolling out multiple trajectories from a pretrained diffusion policy, we can evaluate each trajectory by simulating the actions with the video model, and then using a VLM judge to pick the best action trajectory. We observe consistent improvement in task success when using the video model to roll out and choose best action sequences, as well as the positive correlation with the number of samples evaluated at test time. This demonstrates the ability of the model to simulate counterfactuals given the same initial condition.

Policy evaluation

Across seven simulated tasks, predicted policy success correlates with simulator success at r=0.982, with a positive success bias. This aggregate correlation does not establish calibrated per-trial probabilities or eliminate hallucinated progress. The main text says rollouts are judged manually, while Appendix E describes fully automated Gemini policy evaluation; the source does not clearly reconcile the two protocols.

Real-world evaluation pairs twenty demonstrations per task with model rollouts for four tasks, scoring partial stage completion. The displayed distributions are similar and again optimistic. These eighty demonstration pairs evaluate modeled progress, rather than eighty autonomous deployments of the learned planner. Matching marginal distributions alone would not establish accurate trial-by-trial prediction.

Complete original PDF block S5.F9
Figure 9: Robocasa policy evaluation. Video-model rollouts consistently track ground-truth success rates across RoboCasa tasks.
Complete original PDF block S5.F10
Figure 10: Real world policy evaluation. Rolling out real-world demonstrations with our video model produces videos with success progress closely aligned with what is observed in the real world execution.
Complete original PDF block S5.F11
Figure 11: Action extraction. Even without task-specific video-model training, inverse modeling recovers competitive robot behavior.

Action extraction

On CoffeeServeMug, desired object motion conditions generated robot video, and a learned inverse-dynamics model extracts a twelve-dimensional action sequence. The video model has not seen this task according to the main text, but the inverse mapper and each baseline use one hundred task demonstrations.

With twenty trials each, success is 50% Diffusion Policy, 80% ACT, 85% SmolVLA and 90% the proposed pipeline: ten, sixteen, seventeen and eighteen successes. The proposed margin over SmolVLA is one trial. Appendix F specifies that baselines receive only the first generated frame, while the inverse mapper sees the full goal-conditioned video, so this is not a matched-information isolation of representation quality.

6 Discussion and Conclusions

Masked entity trajectories are a reusable control interface for a pretrained video generator, with promising sampled transfer and planning results. The authors explicitly acknowledge learning correlations rather than causal relationships and inheriting the backbone’s speed and expressiveness limits. Physical plausibility, evaluator reliability and deployment performance remain empirical concerns.

The broader-impact discussion suggests lower development cost and increased accessibility, while recognizing possible unsafe or unauthorized deployment. Neither proposed benefit nor risk is quantified in these experiments.

Appendix Overview

The supplement provides reconstruction uncertainty, data splits, exact VLM prompts, policy/inverse-model training, physical data collection, task rubrics and unseen-embodiment baseline adaptation. These details are essential to interpret zero-shot claims, planning curves and real-world progress scores.

Appendix A Project webpage

The project page is the source’s destination for video demonstrations. The complete static PDF panels remain in this edition; linked external demonstrations are not treated as additional measured evidence here.

Appendix B Reproducibility

The appendix states that code and model weights are released, while the main text uses future-release wording and also mentions data. This edition records the paper’s statement; it does not independently verify repository completeness or reproducibility of the trained checkpoint.

Appendix C Full quantitative results

Table C1 reports means ± standard error of the mean for fifty BEHAVIOR, fifty DROID and thirteen real-world reconstruction examples. Dense conditioning improves the listed LPIPS/PSNR results, while real-world skeleton SSIM 0.866±0.005 exceeds 0.864±0.008. SEM describes the reported sample mean, not a confidence guarantee for unseen tasks.

The Wan-Move BEHAVIOR LPIPS/SSIM discrepancy with Table 1 is substantial: 0.454/0.528 here versus 0.312/0.756 there. DROID Wan-Move PSNR/SSIM also differ slightly, 12.96/0.558 versus 12.99/0.562. Minor rounding changes occur for other entries. Exact tables preserve both versions.

Complete original PDF block A3.T1
Table C1: Reconstruction metrics across datasets (mean ±\pm SEM; best per column in bold).
Original selectable table
BEHAVIOR-1K (n=50n{=}50) DROID (n=50n{=}50) Real-World (n=13n{=}13)
Method PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow
Ctrl-World 18.39±0.5418.39{\pm}0.54 0.836±0.0080.836{\pm}0.008 0.196±0.0110.196{\pm}0.011 18.15±0.1618.15{\pm}0.16 0.708±0.0080.708{\pm}0.008 0.362±0.0060.362{\pm}0.006 — — —
Wan-Move 13.17±0.3013.17{\pm}0.30 0.528±0.0150.528{\pm}0.015 0.454±0.0160.454{\pm}0.016 12.96±0.3712.96{\pm}0.37 0.558±0.0170.558{\pm}0.017 0.534±0.0230.534{\pm}0.023 — — —
Wan2.2-I2V 10.22±0.3810.22{\pm}0.38 0.457±0.0230.457{\pm}0.023 0.602±0.0240.602{\pm}0.024 12.42±0.3412.42{\pm}0.34 0.548±0.0170.548{\pm}0.017 0.521±0.0230.521{\pm}0.023 — — —
EEF 19.23±0.3419.23{\pm}0.34 0.815±0.0060.815{\pm}0.006 0.171±0.0070.171{\pm}0.007 22.64±0.2622.64{\pm}0.26 0.878±0.0050.878{\pm}0.005 0.107±0.0050.107{\pm}0.005 20.32±0.2520.32{\pm}0.25 0.858±0.0050.858{\pm}0.005 0.183±0.0080.183{\pm}0.008
Skeleton 19.58±0.3419.58{\pm}0.34 0.824±0.0060.824{\pm}0.006 0.162±0.0070.162{\pm}0.007 22.74±0.2822.74{\pm}0.28 0.878±0.0050.878{\pm}0.005 0.106±0.0040.106{\pm}0.004 21.02±0.3821.02{\pm}0.38 0.866±0.005\mathbf{0.866{\pm}0.005} 0.169±0.0100.169{\pm}0.010
Ours 22.90±0.52\mathbf{22.90{\pm}0.52} 0.842±0.007\mathbf{0.842{\pm}0.007} 0.123±0.007\mathbf{0.123{\pm}0.007} 23.74±0.24\mathbf{23.74{\pm}0.24} 0.887±0.004\mathbf{0.887{\pm}0.004} 0.095±0.004\mathbf{0.095{\pm}0.004} 22.79±0.35\mathbf{22.79{\pm}0.35} 0.864±0.0080.864{\pm}0.008 0.148±0.010\mathbf{0.148{\pm}0.010}

Appendix D Additional training details

Approximately one thousand DROID demonstrations from two external cameras are processed through both segmentation and rendering pipelines. CLVR and RAD scenes are reserved for testing. Four thousand RoboCasa examples exclude navigation tasks and include generated failures; DROID failures are also retained.

A high-level text prompt accompanies the required video-model conditioning. Restricting that prompt encourages use of visual control but does not prove the model ignores language. The main inverse experiment separately states that CoffeeServeMug was unseen by the video model; the appendix does not enumerate every task exclusion.

Appendix E VLM Evaluation Protocol

Gemini 3.1 Pro Preview is queried at temperature zero with all eighty-one frames sampled at fifteen FPS. It assesses visible causal contact, task success/confidence, gripper contact and physical realism. Desired task outcomes are supplied in the user message, and structured field order asks the judge to explain the mechanism before declaring success.

Candidates are sorted lexicographically by success flag, realism, confidence, contact, then lower index. The selected rollout is evaluated against simulator ground truth; subset enumeration averages the judge’s selection over every size-N subset. At N=1 this reduces to random candidate performance. A structured explanation is an evaluation heuristic, not a verified causal test.

E.1 Gemini system prompt

The complete system instruction is preserved below, including all three PDF pages. It rejects ghost contact, object motion after disengagement and frame discontinuities. Its categorical “no momentum” assumption is a task-specific judging rule, not a universal physical principle: real contact-driven motion can continue after release.

The prompt can reduce obvious hallucination credit, but contact visibility, occlusion and the underlying assumptions can still affect judgments. The paper does not report an independent confusion matrix establishing evaluator accuracy.

Complete original PDF block source-system-prompt.part1Complete original PDF block source-system-prompt.part2Complete original PDF block source-system-prompt.part3
Original Gemini evaluator system prompt
Original selectable prompt
You are a strict evaluator of robot-manipulation videos.
The video was produced by a generative video model that is meant to
*simulate* a robot policy executing a task. The video is often
visually flawed and may HALLUCINATE the desired outcome -- e.g. the
target object teleports into its goal pose, vanishes and reappears in
place, or moves on its own without the robot causing it. Your job is
to judge whether the *robot* actually caused the outcome via
plausible physical contact, not whether the final frame happens to
look correct.
CRITICAL ANTI-PATTERNS -- these are common world-model artifacts that
look superficially like success but must NOT be credited as
robot_pushed / robot_grasped:
(a) Ghost / weak contact. The gripper must be in visible mechanical
coupling with the object on the frames the object moves: the
gripper geometry should be touching/compressing against the
object surface, and the object's motion should be co-located in
time with that contact. If the gripper is merely *near* the
object, in front of it, or has only momentary glancing contact
and the object still moves, treat the motion as autonomous or
passthrough -- NOT as robot_pushed. "Implied" or "magic" contact
does not count.
(b) Post-disengagement coasting / "momentum". The simulated
environment has NO momentum or follow-through: an object does
not continue moving toward its goal after the gripper retracts
or stops. If the gripper disengages (retracts, stops, or moves
away) and the object continues to move closed/into-goal on its
own afterwards, the post-disengagement frames are autonomous,
regardless of what came before. Only credit the extent of
progress that occurred WHILE the gripper was in active contact
AND moving the object. If the goal state is only reached after
the gripper has disengaged, robot_caused_outcome.caused is
FALSE and task_success.success is FALSE.
(c) Glitchy reach-around or frame jumps. If the goal state appears
in a frame following a visual discontinuity, scene reset, or
sudden teleport -- even if the robot's pose looks plausible
around it -- treat as teleported.
Score these fields:
1. robot_caused_outcome
caused True only if the robot, via its gripper, is the
visible physical cause of the goal-relevant motion,
AND the goal state is reached DURING the contact
window (not after it).
mechanism one of:
"robot_pushed" gripper pushed/pulled the object
in sustained contact
"robot_grasped" gripper grasped and moved it
"teleported" object snapped to goal state
"vanished" object disappeared from scene
"autonomous" object moved on its own
(incl. post-disengagement
coasting, ghost contact)
"passthrough" gripper went through the object
"no_attempt" robot never engaged the object
"other" describe in reasoning
2. task_success success=TRUE requires ALL THREE of: (a) the
end-state matches the success criterion, (b)
robot_caused_outcome.caused is true, and (c)
the goal state was reached during a frame
range where the gripper was actively in
mechanical contact with the object. Outcomes
reached after disengagement, via ghost contact,
teleport, or autonomous motion are FALSE.
3. gripper_contact Did the gripper visibly make physical contact
with the target object? "Hovering near",
"passing through", or single-frame glancing
contact does not count.
4. physics_realism 1-5 plausibility. Penalize teleporting,
morphing, gripper passing through solids,
vanishing or duplicated objects, frame-to-frame
jumps, ghost contact, post-disengagement
coasting.
Be terse but specific. Reference what you see in the video (frame
ranges, objects, motions, whether the gripper was in contact when
the goal state was reached). Do not speculate beyond what is
visible.

Per-rollout user message.

The exact user template supplies the intended task and success criterion, then asks for a one-to-five realism score and schema-conforming JSON. Changing the task criterion changes what the evaluator rewards; the reusable system instruction does not replace task-specific semantics.

Complete original PDF block source-user-prompt.part1
Original per-rollout user prompt
Original selectable prompt
Intended task: {task_prompt}
Success criterion: {expected_outcome}
Score physics_realism on a 1-5 integer scale: 1 = nonsensical,
3 = obvious artifacts but recognizable, 5 = indistinguishable from
real video. Return ONLY the JSON object matching the response schema.

Structured response schema.

The ordered schema contains robot-caused outcome, task success, gripper contact, physical realism and a summary. Mechanism categories distinguish pushing/grasping from teleportation, disappearance, autonomous motion, pass-through and no attempt. The earlier overview calls the contact field end-effector contact; the detailed schema and prompt use gripper_contact.

Appendix F Policy and Inverse-Dynamics Training Details

Proposal-policy training, inverse-dynamics training and physical policy datasets are separate settings. Only the video checkpoint is shared across the claimed forward/inverse applications; the downstream controls have their own task-specific supervision.

F.1 Simulation Policy Training for Planning and Policy Evaluation

Simulation Diffusion Policies use two fixed RGB views plus robot state and predict an entire twelve-dimensional action sequence. Coordinates cover position, rotation, gripper, mobile base, torso and control mode. Training mixes roughly one hundred human demonstrations with task-dependent MimicGen expansions, around nine thousand trajectories in the stated atomic-task setting.

The video model ranks these proposals against benchmark success conditions. More candidates can help only when sufficiently successful proposals exist and ranking identifies them; proposal generation and open-loop execution remain limitations.

F.2 Learned Inverse Dynamics for Action Extraction

The transformer inverse-dynamics model is trained on CoffeeServeMug with AdamW for three hundred epochs and predicts the native twelve-dimensional sequence. It observes the complete synthesized interaction. The DP/ACT/SmolVLA controls observe its first frame only, a material information difference alongside the shared one-hundred-demonstration count.

F.3 Real-World Policy Training

The appendix also trains DP and ACT on forty trajectories each for mug-on-rack and close-cabinet, sampled at ten Hz with four RGB views and eight-dimensional joint/end-effector actions. These two policy-training datasets differ from the four twenty-demonstration tasks used for real-world progress evaluation; their counts should not be merged.

Appendix G Robot Data Collection

A Franka Panda with a custom printed compliant end-effector is teleoperated through GELLO. Synchronized robot states and wrist/external video supply the paired real-world observations used for rendering and evaluation.

G.1 Hardware Setup

The stated sensing hardware comprises two ZED Mini 2 cameras, one wrist-mounted and one external, recording 720p RGB at thirty Hz. The separate policy-training subsection reports four synchronized RGB views and ten-Hz sampling; no unreported sensor arrangement is inferred.

G.2 Camera and Robot Frame Calibration

A fixed table AprilTag links robot and camera frames. Diverse wrist poses combine robot kinematics with tag detections to estimate hand–eye calibration and the tag’s base-frame pose. Observing the same tag calibrates the external camera via transform composition. Projected robot geometry is checked against images; no numerical calibration-error distribution is supplied.

Equation
TCextB=TAB​(TACext)−1.T^{B}_{C^{\mathrm{ext}}}=T^{B}_{A}\left(T^{C^{\mathrm{ext}}}_{A}\right)^{-1}.

G.3 Trajectory Representation

Each HDF5 rollout stores RGB frames, measured joints, timestamps and camera calibration metadata. Robot kinematics, the custom gripper model and these intrinsics/extrinsics render visual action masks. Accurate registration is therefore a dependency of the rendering route, not something the video model is shown to solve automatically.

Equations
τ={(Itext,Itwrist,qt)}t=1T,\tau=\{(I^{\mathrm{ext}}_{t},I^{\mathrm{wrist}}_{t},q_{t})\}_{t=1}^{T},
Kext,Kwrist,TAB,TCextB,TCwristE.K_{\mathrm{ext}},\quad K_{\mathrm{wrist}},\quad T^{B}_{A},\quad T^{B}_{C^{\mathrm{ext}}},\quad T^{E}_{C^{\mathrm{wrist}}}.

Appendix H Real World Tasks Rubrics

Real-world progress awards equal points to completed substages within each task. Bagging an orange and placing a towel each have two stages: pickup, then placement. Block stacking has four: pick and place the purple block on orange, then pick and place black on purple. Drawer closing has one full-closure stage.

Scores normalize by each task’s stage count. They distinguish partial progress from binary completion but do not equalize difficulty or make fractional gains physically interchangeable between tasks.

Appendix I Adapting Baselines to Unseen Embodiments

For BEHAVIOR-1K, the new robot is rendered from its URDF. Ctrl-World’s camera setup is adapted toward DROID, and a heuristic chooses the more active arm by relative end-effector motion to map bimanual data into its single-arm action interface. Sparse baselines display both end effectors or both arm skeletons.

These adaptations permit a comparison but do not expose all methods to identical control information. Results demonstrate transfer to the tested morphology/data, not unrestricted embodiment-independent executable control.

Concept demonstration

Open image ↗