Overview
At a glance
This edition follows the complete twenty-one-page v1, retaining all eleven numbered figures, three tables, eight displayed equations and both evaluator prompts. Masked Visual Actions condition a video model on partially revealed entity trajectories: robot pixels specify a forward query, while desired object motion specifies an inverse query.
One video checkpoint is adapted with fifteen hours of robot data. Tests cover video reconstruction, candidate-plan selection, policy evaluation and action extraction. The inverse pipeline reaches 18/20 successes on one simulated task, using a separately trained inverse-dynamics model. The video model’s zero-shot inverse use should not be confused with a pipeline trained without task demonstrations.
The model learns conditional visual correlations, not a certified causal simulator. Generated outcomes tend to overestimate progress; table inconsistencies, evaluator assumptions and unequal baseline inputs qualify the reported results.
1 Introduction
Video priors contain useful motion and interaction patterns, but conventional robot action vectors may be poorly aligned with those representations. The paper encodes the proposed intervention directly as a masked pixel trajectory, allowing one completion model to infer different missing entities.
The same checkpoint is used for forward prediction and inverse video synthesis. Candidate ranking and physical action extraction require additional components: a stochastic policy and VLM evaluator for planning, and a learned inverse-dynamics mapper for execution. These downstream components are part of the system’s data and compute requirements.
2 Related Work
Controllable video generation as a robotic interface.
Controllable video methods use forces, tracks, hand poses, flow and goals. Visual prompting by inpainting and probabilistic structure integration motivate completing missing visual regions under different conditioning. Masked Visual Actions applies that interface to robot interactions; the cited diversity should not be read as proof that every earlier control representation lacks cross-embodiment transfer.
Pixel-grounded action conditioning for robot world models.
Rendered robot masks, point maps, occupancy and action images already supply spatially grounded control. Other methods edit human videos for embodiment transfer. The distinction emphasized here is changing which entity is revealed at query time, reusing a robot-conditioned training checkpoint for object-conditioned completion.
Unified video-action models and downstream uses.
Unified video/action models combine prediction and control through modality masks or diffusion schedules. This work instead masks spatial entities. Forward prediction supports planning and evaluation; inverse video synthesis still needs an embodiment-specific conversion from generated frames to executable controls, supplied here by a separately trained model.
3 Masked Visual Actions
The formulation treats a video as a joint distribution over entity trajectories. A binary spatiotemporal mask reveals the selected entities, and the model completes the rest conditioned on that evidence and the initial frame. The active/passive partition describes how the interface is queried, not a learned agency variable.
Equations 1–5 express conditional distributions. Conditional completion from observational training does not, by itself, identify an interventional causal distribution or guarantee physical feasibility of a supplied trajectory.
Equations 1–3
| (1) |
| (2) |
| (3) |
Forward model.
Revealing the robot trajectory asks the model to predict the environment’s response. Pixel alignment exposes geometry and contact locations more directly than low-dimensional action vectors. To render novel robot actions, the implementation still needs a mesh, camera calibration and a kinematic trajectory.
Equation 4
| (4) |
Inverse model.
Revealing desired object motion asks the model to complete corresponding robot motion. The video checkpoint was trained on robot masks only, so this change of conditioning is zero-shot with respect to inverse video training. The demonstrated action-extraction task subsequently uses supervised inverse dynamics; neither the equations nor the examples ensure every desired object path is achievable.
Equation 5
| (5) |
4 Method
4.1 Dataset construction
Training combines successes and failures from DROID and RoboCasa. Segmentation-derived conditioning teaches completion from real robot appearance, while rendering-derived conditioning supports candidate trajectories that were never physically executed.
Segmentation based dataset
SAM 3, prompted with “A robotic arm,” isolates the arm in recorded video without requiring camera calibration. Occlusion boundaries can leak information about the objects’ actual future, and obtaining the exact future segmentation at inference is difficult. These are reasons to include independently rendered trajectories rather than relying only on segmented demonstrations.
Rendering based dataset
Robot states, URDF geometry and refined camera calibration align rendered robot motion to DROID imagery. RoboCasa renders the robot separately from the scene. Translucent rendering and red gripper fingers expose motion despite self-occlusion. This route supports arbitrary proposed kinematic trajectories, while requiring known geometry and calibration; it is not a calibration-free interface for every new robot.
4.2 Model implementation and training
The backbone is Wan-Fun-Control 2.2 14B from the Wan family. Its autoencoder encodes masked videos with uniform gray missing regions, and concatenation injects aligned conditioning. Rank-256 LoRA adapts the model with batch size four on eight H200 GPUs for about ten thousand steps over four days.
Fifteen hours describes adaptation data, not training compute or all data used by the pretrained backbone. The source does not provide an end-to-end inference-time or planning-cost measurement supporting real-time deployment.
5 Experiments
Evaluation first compares reconstruction and visual control, then tests three robot applications. The same video checkpoint is reused, while proposal policies, the VLM judge and the inverse-dynamics model have distinct training and evaluation roles.
5.1 Controllable video generation
On DROID, Masked Visual Actions reports LPIPS/SSIM/PSNR 0.0945/0.887/23.74 versus Ctrl-World 0.362/0.708/18.15. On unseen BEHAVIOR embodiment, the corresponding main-table values are 0.123/0.843/22.90 versus 0.196/0.837/18.39. These are image/video reconstruction metrics, not direct guarantees of correct contact dynamics.
The held-out DROID labs are unseen by this model’s adaptation but were included in Ctrl-World’s training. BEHAVIOR transfers a single-arm baseline through an active-arm heuristic and adapted cameras; the comparison therefore includes interface adaptation differences.
Wan-Move receives ground-truth mesh tracks. Its main-table BEHAVIOR LPIPS/SSIM are 0.312/0.756, whereas Table C1 gives 0.454/0.528 with the same PSNR 13.17. Both original tables are retained, and no undocumented correction is chosen.
Original selectable table
| DROID | BEHAVIOR | |||||
|---|---|---|---|---|---|---|
| Method | LPIPS | SSIM | PSNR | LPIPS | SSIM | PSNR |
| Image-to-video [63] | 0.521 | 0.548 | 12.42 | 0.602 | 0.457 | 10.22 |
| Wan-move [13] | 0.534 | 0.562 | 12.99 | 0.312 | 0.756 | 13.17 |
| Ctrl-World [25] | 0.362 | 0.708 | 18.15 | 0.196 | 0.837 | 18.39 |
| Masked Visual Actions (Ours) | 0.0945 | 0.887 | 23.74 | 0.123 | 0.843 | 22.90 |
Original selectable table
| DROID | Real world | BEHAVIOR | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | LPIPS | SSIM | PSNR | LPIPS | SSIM | PSNR | LPIPS | SSIM | PSNR |
| End-effector vis. | 0.107 | 0.878 | 22.64 | 0.183 | 0.858 | 20.32 | 0.171 | 0.815 | 19.23 |
| Skeleton vis. | 0.106 | 0.878 | 22.74 | 0.169 | 0.866 | 21.02 | 0.162 | 0.824 | 19.58 |
| Masked Visual Actions | 0.0945 | 0.887 | 23.74 | 0.148 | 0.864 | 22.79 | 0.123 | 0.843 | 22.90 |
Comparing the choice of visual actions
The visual-conditioning ablation trains the same backbone with DROID data using end-effector pose, skeleton or dense robot pixels. Dense conditioning better preserves the custom gripper and unseen bimanual morphology in selected examples, while in-domain differences are smaller.
On thirteen custom-gripper real-world reconstruction samples, dense conditioning gives LPIPS 0.148 and PSNR 22.79 versus skeleton 0.169/21.02. Skeleton’s SSIM is slightly higher, 0.866 versus 0.864, within the reported SEMs. Thus dense masks do not lead every metric; broad claims of universal collapse exceed these sampled tests.
5.2 Robotics applications
RoboCasa experiments use the model to rank proposed futures, compare predicted outcomes with simulator outcomes, and synthesize actions from goal-object motion. These settings test different components and should not be collapsed into one general manipulation success rate.
Planning
A Diffusion Policy proposes ten action sequences for each of ten scenes per task. The video model simulates each candidate; Gemini judges success and realism, and the selected original action sequence executes in the simulator. The planner ranks existing proposals rather than optimizing actions through the video model.
Across six tasks, the Figure 8 bars report gains of 24, 26, 21, 11, 9 and 7 percentage points. Curves enumerate subsets of the same ten candidates per scene and average the selected candidate’s simulator success; their points are not independent newly collected runs. Benefits depend on proposal coverage and evaluator accuracy, and extra generation/judging compute is not timed.
Policy evaluation
Across seven simulated tasks, predicted policy success correlates with simulator success at r=0.982, with a positive success bias. This aggregate correlation does not establish calibrated per-trial probabilities or eliminate hallucinated progress. The main text says rollouts are judged manually, while Appendix E describes fully automated Gemini policy evaluation; the source does not clearly reconcile the two protocols.
Real-world evaluation pairs twenty demonstrations per task with model rollouts for four tasks, scoring partial stage completion. The displayed distributions are similar and again optimistic. These eighty demonstration pairs evaluate modeled progress, rather than eighty autonomous deployments of the learned planner. Matching marginal distributions alone would not establish accurate trial-by-trial prediction.
Action extraction
On CoffeeServeMug, desired object motion conditions generated robot video, and a learned inverse-dynamics model extracts a twelve-dimensional action sequence. The video model has not seen this task according to the main text, but the inverse mapper and each baseline use one hundred task demonstrations.
With twenty trials each, success is 50% Diffusion Policy, 80% ACT, 85% SmolVLA and 90% the proposed pipeline: ten, sixteen, seventeen and eighteen successes. The proposed margin over SmolVLA is one trial. Appendix F specifies that baselines receive only the first generated frame, while the inverse mapper sees the full goal-conditioned video, so this is not a matched-information isolation of representation quality.
6 Discussion and Conclusions
Masked entity trajectories are a reusable control interface for a pretrained video generator, with promising sampled transfer and planning results. The authors explicitly acknowledge learning correlations rather than causal relationships and inheriting the backbone’s speed and expressiveness limits. Physical plausibility, evaluator reliability and deployment performance remain empirical concerns.
The broader-impact discussion suggests lower development cost and increased accessibility, while recognizing possible unsafe or unauthorized deployment. Neither proposed benefit nor risk is quantified in these experiments.
Appendix Overview
The supplement provides reconstruction uncertainty, data splits, exact VLM prompts, policy/inverse-model training, physical data collection, task rubrics and unseen-embodiment baseline adaptation. These details are essential to interpret zero-shot claims, planning curves and real-world progress scores.
Appendix A Project webpage
The project page is the source’s destination for video demonstrations. The complete static PDF panels remain in this edition; linked external demonstrations are not treated as additional measured evidence here.
Appendix B Reproducibility
The appendix states that code and model weights are released, while the main text uses future-release wording and also mentions data. This edition records the paper’s statement; it does not independently verify repository completeness or reproducibility of the trained checkpoint.
Appendix C Full quantitative results
Table C1 reports means ± standard error of the mean for fifty BEHAVIOR, fifty DROID and thirteen real-world reconstruction examples. Dense conditioning improves the listed LPIPS/PSNR results, while real-world skeleton SSIM 0.866±0.005 exceeds 0.864±0.008. SEM describes the reported sample mean, not a confidence guarantee for unseen tasks.
The Wan-Move BEHAVIOR LPIPS/SSIM discrepancy with Table 1 is substantial: 0.454/0.528 here versus 0.312/0.756 there. DROID Wan-Move PSNR/SSIM also differ slightly, 12.96/0.558 versus 12.99/0.562. Minor rounding changes occur for other entries. Exact tables preserve both versions.
Appendix D Additional training details
Approximately one thousand DROID demonstrations from two external cameras are processed through both segmentation and rendering pipelines. CLVR and RAD scenes are reserved for testing. Four thousand RoboCasa examples exclude navigation tasks and include generated failures; DROID failures are also retained.
A high-level text prompt accompanies the required video-model conditioning. Restricting that prompt encourages use of visual control but does not prove the model ignores language. The main inverse experiment separately states that CoffeeServeMug was unseen by the video model; the appendix does not enumerate every task exclusion.
Appendix E VLM Evaluation Protocol
Gemini 3.1 Pro Preview is queried at temperature zero with all eighty-one frames sampled at fifteen FPS. It assesses visible causal contact, task success/confidence, gripper contact and physical realism. Desired task outcomes are supplied in the user message, and structured field order asks the judge to explain the mechanism before declaring success.
Candidates are sorted lexicographically by success flag, realism, confidence, contact, then lower index. The selected rollout is evaluated against simulator ground truth; subset enumeration averages the judge’s selection over every size-N subset. At N=1 this reduces to random candidate performance. A structured explanation is an evaluation heuristic, not a verified causal test.
E.1 Gemini system prompt
The complete system instruction is preserved below, including all three PDF pages. It rejects ghost contact, object motion after disengagement and frame discontinuities. Its categorical “no momentum” assumption is a task-specific judging rule, not a universal physical principle: real contact-driven motion can continue after release.
The prompt can reduce obvious hallucination credit, but contact visibility, occlusion and the underlying assumptions can still affect judgments. The paper does not report an independent confusion matrix establishing evaluator accuracy.
Original selectable prompt
Per-rollout user message.
The exact user template supplies the intended task and success criterion, then asks for a one-to-five realism score and schema-conforming JSON. Changing the task criterion changes what the evaluator rewards; the reusable system instruction does not replace task-specific semantics.
Structured response schema.
The ordered schema contains robot-caused outcome, task success, gripper contact, physical realism and a summary. Mechanism categories distinguish pushing/grasping from teleportation, disappearance, autonomous motion, pass-through and no attempt. The earlier overview calls the contact field end-effector contact; the detailed schema and prompt use gripper_contact.
Appendix F Policy and Inverse-Dynamics Training Details
Proposal-policy training, inverse-dynamics training and physical policy datasets are separate settings. Only the video checkpoint is shared across the claimed forward/inverse applications; the downstream controls have their own task-specific supervision.
F.1 Simulation Policy Training for Planning and Policy Evaluation
Simulation Diffusion Policies use two fixed RGB views plus robot state and predict an entire twelve-dimensional action sequence. Coordinates cover position, rotation, gripper, mobile base, torso and control mode. Training mixes roughly one hundred human demonstrations with task-dependent MimicGen expansions, around nine thousand trajectories in the stated atomic-task setting.
The video model ranks these proposals against benchmark success conditions. More candidates can help only when sufficiently successful proposals exist and ranking identifies them; proposal generation and open-loop execution remain limitations.
F.2 Learned Inverse Dynamics for Action Extraction
The transformer inverse-dynamics model is trained on CoffeeServeMug with AdamW for three hundred epochs and predicts the native twelve-dimensional sequence. It observes the complete synthesized interaction. The DP/ACT/SmolVLA controls observe its first frame only, a material information difference alongside the shared one-hundred-demonstration count.
F.3 Real-World Policy Training
The appendix also trains DP and ACT on forty trajectories each for mug-on-rack and close-cabinet, sampled at ten Hz with four RGB views and eight-dimensional joint/end-effector actions. These two policy-training datasets differ from the four twenty-demonstration tasks used for real-world progress evaluation; their counts should not be merged.
Appendix G Robot Data Collection
A Franka Panda with a custom printed compliant end-effector is teleoperated through GELLO. Synchronized robot states and wrist/external video supply the paired real-world observations used for rendering and evaluation.
G.1 Hardware Setup
The stated sensing hardware comprises two ZED Mini 2 cameras, one wrist-mounted and one external, recording 720p RGB at thirty Hz. The separate policy-training subsection reports four synchronized RGB views and ten-Hz sampling; no unreported sensor arrangement is inferred.
G.2 Camera and Robot Frame Calibration
A fixed table AprilTag links robot and camera frames. Diverse wrist poses combine robot kinematics with tag detections to estimate hand–eye calibration and the tag’s base-frame pose. Observing the same tag calibrates the external camera via transform composition. Projected robot geometry is checked against images; no numerical calibration-error distribution is supplied.
Equation
G.3 Trajectory Representation
Each HDF5 rollout stores RGB frames, measured joints, timestamps and camera calibration metadata. Robot kinematics, the custom gripper model and these intrinsics/extrinsics render visual action masks. Accurate registration is therefore a dependency of the rendering route, not something the video model is shown to solve automatically.
Equations
Appendix H Real World Tasks Rubrics
Real-world progress awards equal points to completed substages within each task. Bagging an orange and placing a towel each have two stages: pickup, then placement. Block stacking has four: pick and place the purple block on orange, then pick and place black on purple. Drawer closing has one full-closure stage.
Scores normalize by each task’s stage count. They distinguish partial progress from binary completion but do not equalize difficulty or make fractional gains physically interchangeable between tasks.
Appendix I Adapting Baselines to Unseen Embodiments
For BEHAVIOR-1K, the new robot is rendered from its URDF. Ctrl-World’s camera setup is adapted toward DROID, and a heuristic chooses the more active arm by relative end-effector motion to map bimanual data into its single-arm action interface. Sparse baselines display both end effectors or both arm skeletons.
These adaptations permit a comparison but do not expose all methods to identical control information. Results demonstrate transfer to the tested morphology/data, not unrestricted embodiment-independent executable control.