Overview
Heng Yu, David D. Yuan and Juze Zhang contributed equally. This Stanford framework’s project page and code repository accompany the complete paper and implementation appendices.
At a glance
Stanford’s OpenWAM compares joint, video-first, action-first and decoupled world–action generation under a common causal video backbone. Separate local inverse/forward dynamics models connect visual futures with actions.
Video-first policy success averages 98.6% over four LIBERO suites. A frozen counterfactual-trained local inverse model reaches 84.0% across four target tasks after adapting their video predictors. The latter is component reuse with target-task training, rather than zero-shot transfer of the complete system.
1 Introduction
World–action systems often change data, architecture and inference together, making design choices hard to compare. This framework shares a video initialization and downstream recipe while varying cross-modal attention and generation order.
A second study learns reusable dynamics from alternatives to successful demonstrations. Simulation branches broaden local transition coverage; independently trained dynamics components then receive explicit future-video or action trajectories.
2 Related Work
Related work spans direct visuomotor control, predictive policies and dynamics learning beyond successful demonstrations. The framework separates interaction design from component-level transition supervision.
2.1 Visuomotor Policies and Predictive Control
Behavior cloning maps current observations to actions, while predictive control uses imagined observations or latent dynamics. OpenWAM exposes both kinds of trajectory interface within one modeling framework [13] [16].
2.2 Video Foundations and World–Action Architectures
Existing WAMs use joint denoising, video-first prediction, specialized branches and multiple policy/dynamics modes. This paper holds several shared choices fixed to study interaction programs and independent component reuse.
A concurrent, separately authored project is also named OpenWAM. Its paper is the distinct modular-pretraining study already in this library; the two entries should remain separate [79].
2.3 Dynamics Learning Beyond Successful Demonstrations
Demonstrations show actions chosen by a successful policy, leaving many alternative outcomes uncovered. Prior recovery, failure and counterfactual methods broaden that experience.
Here the collection restores simulator states and executes modified action continuations. This directly observes alternative simulator outcomes; it does not infer them solely from observational demonstrations.
3 Methodology
The system combines a causal robot-video backbone, configurable dual-stream interaction and separately trained local dynamics. Composition within a model and composition between independently trained components are distinct capabilities.
3.1 World–Action Modeling Formulation
Context contains past/current observations, past actions, proprioception and task language. Future actions start at the current control step, while future observations follow the executed actions.
The two chain-rule factorizations identify video prediction/inverse dynamics and direct action/forward dynamics roles. They describe a reference joint distribution. Independently trained components need not be mutually consistent factors of one learned joint distribution; their composition is evaluated empirically.
Equations 1–3
| (1) | ||||
| (2) |
Original selectable mathematics
| (3) | ||||
3.2 An Open Causal Robot-Video Backbone
Wan2.2-5B is adapted on heterogeneous interaction video without action labels. Chunk-causal attention permits current/past observations and earlier generated chunks, while processing tokens inside each future chunk jointly.
Temporal causality is separate from future video/action communication. The same pretrained video checkpoint initializes all main downstream variants and remains usable for standalone video prediction.
3.3 Configurable Video–Action Interaction Programs
A shared Mixture-of-Transformers architecture supports four native policy programs. Each retains the same data/representations/objective family, while changing future cross-modal visibility and sampling order.
Shared architecture.
The video and action experts have separate normalization, projection and feed-forward parameters. Attention over packed modality tokens exchanges information, with a separate text cross-attention path.
The video expert is about 5B parameters and the action expert about 2B. Common initialization helps compare the four programs, while comparisons with a shared single-transformer architecture also change parameter allocation [49].
Interaction programs.
Joint denoises video and actions with bidirectional future attention. Video-then-action (VTA) finishes the visual future before predicting actions; action-then-video (ATV) reverses that order.
Decoupled removes attention between future modalities while retaining observed history and language. Sequential programs use two generation passes, so shared training settings do not imply identical inference computation or latency.
Accessible table and original citation links
| Task | Target behavior | Transfer challenge |
|---|---|---|
| 64 | Stack bowls; place stack in tray | Novel skill composition |
| 74 | Book into caddy’s left compartment | Retarget familiar manipulation |
| 21 | Turn on stove; place frying pan on it | Object and grasp change |
| 45 | Same objective as Task 21 | Additional scene and layout shift |
Shared flow-matching objective.
Latent flow matching interpolates from noise to clean targets and learns the corresponding velocity. Multiple active target modalities contribute summed losses.
For VTA/ATV training, the second modality receives the recorded first-modality trajectory. At inference it receives the generated trajectory instead. This teacher-forcing shift is part of the tested pipeline.
Equations 4–5
| (4) |
| (5) |
3.4 Counterfactual Learning of Local-Context Dynamics Interfaces
Local inverse dynamics maps a proposed visual trajectory into actions; local forward dynamics predicts a visual trajectory from supplied actions. The models train independently through explicit trajectory interfaces.
Local-context dynamics interfaces.
Both local components receive the current observation and proprioception, plus the supplied future video or action trajectory. They exclude language, past actions and earlier observations.
This restricts conditioning without proving that local inputs are sufficient under every contact state, occlusion or embodiment. The paper tests adequacy in its benchmark transitions and composed policies.
Equation 6
| (6) | ||||
Broader local-transition supervision.
Alternative simulator action sequences branch from the same starting state, providing recorded state/action/outcome transitions whether or not a task succeeds. Simulator state controls collection but is not a model input.
Training samples individual transitions; there is no explicit paired-branch contrastive loss. Mixed dynamics supervision draws 60%counterfactual and 40%demonstration transitions. Broader coverage also adds much more data, without a demonstration-only control matched to that full diversity/volume.
Equation 7
| (7) |
Local dynamics learning.
Inverse dynamics denoises actions conditioned on the realized future; forward dynamics denoises future video conditioned on executed actions. The supplied conditioning modality is not a simultaneous denoising target in these specialist objectives.
Equation 8
| (8) | ||||
Inference and component composition.
A video producer passes VAE latents to the receiving inverse model, which re-encodes them through its conditioning pathway. Transformer states/caches are not exchanged. An external action policy can similarly supply actions to a forward model.
Reuse requires compatible visual latent representation, time alignment, camera/action conventions and normalization. The “arbitrary model” diagram should be read within those interface constraints, rather than as direct compatibility with every pretrained model.
Equations 9–10
| (9) |
| (10) |
4 Experiments
Experiments distinguish native policy performance, frozen inverse-component transfer and short-horizon forward prediction. All benchmark/task tables and the complete implementation appendix are retained.
4.1 Experimental Setup
The common foundation is adapted to four LIBERO suites and three physical bimanual tasks. Counterfactual data train separate dynamics models, with same-start branches kept in the same split.
Robot-video pretraining and simulation.
The video corpus mixes robot demonstrations, synthetic manipulation and human interaction. Downstream video chunks contain four latent frames aligned with 16 action steps, plus per-chunk proprioception.
Component transfer starts from LIBERO-Long dynamics and uses four LIBERO-90 tasks outside that source task set. This is a task holdout within the benchmark, rather than evidence of arbitrary unseen assets, embodiments or environments.
Accessible table and original citation links
| Task | Objective | Main challenge |
|---|---|---|
| Toast Bread | Insert bread and activate toaster | Deformability and precise insertion |
| Rubik’s Cube | Restore final layer of cube | Contact-rich push and rotation |
| Sort Cups | Sort cups according to color | Variable ordering and sequencing |
Real-world tasks.
The real system uses two Franka FR3 arms, parallel-jaw grippers, two wrist cameras and a third-person camera. Tasks are toasting bread, restoring the final layer of a 2×2 cube and sorting cups by color.
The cube task is the stated final-layer procedure, rather than unrestricted solving from arbitrary cube configurations.
Dynamics supervision.
Counterfactual branches start from LIBERO-Long demonstration states and executed perturbations. Demonstration-only, counterfactual-only and 60/40 mixed data train local dynamics without changing their interfaces.
Keeping all alternatives from a shared initial state in one partition prevents direct same-context branch leakage. It does not by itself establish transfer to new tasks beyond the separately reported four-target experiment.
Evaluation.
Closed-loop task success evaluates policies and compositions. Forward-dynamics evaluation measures RGB error and whether a prediction is closest to its own realized outcome among two or 16 same-state alternatives.
Outcome identification measures action-dependent visual discrimination. It is not a physical success predicate or a demonstration that the model improves a downstream planner.
Accessible table and original citation links
| Method | Object | Goal | Spatial | Long | Mean |
|---|---|---|---|---|---|
| OpenVLA [38] | 88.4 | 79.2 | 84.7 | 53.7 | 76.5 |
| OpenVLA-OFT [36] | 98.4 | 97.9 | 97.6 | 94.5 | 97.1 |
| [6] | 98.8 | 95.8 | 96.8 | 85.2 | 94.1 |
| [33] | 98.2 | 98.0 | 98.8 | 92.4 | 96.9 |
| GR00T-N1 [5] | 97.6 | 93.0 | 94.4 | 90.6 | 93.9 |
| Motus [3] | 99.8 | 96.6 | 96.8 | 97.6 | 97.7 |
| Fast-WAM [94] | 100.0 | 97.0 | 98.2 | 95.2 | 97.6 |
| LingBot-VA [43] | 99.6 | 97.2 | 98.5 | 98.5 | 98.5 |
| OpenWAM-VTA | 98.6 | ||||
| OpenWAM-ATV | 96.8 | ||||
| OpenWAM-Joint | 97.6 | ||||
| OpenWAM-Decoupled | 98.0 |
Accessible table and original citation links
| Video / policy | Action component (frozen) | 64 Composition | 74 Retarget | 21 Object/grasp | 45 Scene shift | Transfer mean |
| Task-tuned VTA (finetuned policy reference) | 94% | 86% | 92% | 100% | 93.0% | |
| Task-tuned video | Full-context IDM, demo-only | 88% | 90% | 0% | 0% | 44.5% |
| Task-tuned video | Full-context IDM, mixed | 54% | 92% | 4% | 38% | 47.0% |
| Task-tuned video | Local-context IDM, demo-only | 36% | 50% | 0% | 0% | 21.5% |
| Task-tuned video | Local-context IDM, CF-only | 80% | 92% | 90% | 76% | 84.5% |
| Task-tuned video | Local-context IDM, mixed | 74% | 86% | 88% | 88% | 84.0% |
| Original VTA (unadapted policy reference) | 42% | 0% | 0% | 0% | 10.5% | |
4.2 Native Policy Performance
All four interaction programs produce strong LIBERO results from the common robot-video backbone. Real-robot results compare VTA and Joint only.
Broad LIBERO evaluation.
VTA averages 98.6%success across four suites, versus 97.6%Joint, 96.8%ATV and 98.0%Decoupled. Three independent training seeds are evaluated; task-level rollout conditions are shared across programs.
Published baseline results use their original papers’ protocols. The 0.1-point mean difference from reported LingBot-VA is not a controlled significance result. Decoupled’s competitive score also shows that future cross-modal attention is not necessary for strong control in this setting.
Backbone initialization ablation.
With VTA architecture and downstream training fixed, original Wan2.2 initialization scores 68.4%on LIBERO-Long and the robot-video-pretrained causal initialization scores 97.8%, a 29.4-percentage-point gain.
This isolates the combined new initialization, including both robot-video training and causal adaptation. It does not separate those two contributions.
Real-world native policies.
Physical mean success is 92.1%VTA and 91.9%Joint across the three tasks. Evaluation uses 50 toast, 50 cube and 36 cup trials per method, with no reported independent training-seed intervals.
The rounded table values correspond to 125 successes out of 136 for each model. The slight difference in equally weighted task means should not be read as an established physical reliability advantage.
4.3 Component Composability
The component study pairs the same target-video producer with different frozen source-trained inverse models. It separates conditioning scope from demonstration/counterfactual supervision.
Frozen action-component reuse.
Target video predictors come from task-tuned VTA models, which use target task training. Only the reused source inverse component remains frozen; the full composition is not zero-shot from an unadapted predictor.
Local inverse transfer rises from 21.5%demo-only to 84.0%mixed; full-context transfer rises 44.5%to 47.0%. Counterfactual-only local dynamics reaches 84.5%, slightly above mixed 84.0%, while mixed retains better source success, 94.4%versus 90.6%.
Per-task reversals remain: demo-only full context reaches 88/90%on the first two targets but 0%on the other two. Task-tuned native VTA averages 93.0%, setting a useful adapted-policy reference rather than an unsupervised transfer baseline.
4.4 Action-Conditioned Forward Dynamics
Forward evaluation uses 160 contexts across ten tasks with 16 action branches each, totaling 2,560 futures. Counterfactual-only training lowers RGB MSE from 14.35 to 9.40 in 10⁻³units, a 34.5%relative reduction.
It raises two-outcome identification from 68.3%to 93.6%and 16-outcome identification from 21.1%to 71.3%. Mixed data is slightly weaker on those held-out alternatives but stronger on demonstrations. These are local prediction results, not long-horizon real-world planning tests.
5 Discussion and Conclusion
The framework offers a controlled way to compare generation programs and demonstrates that broader simulated transition coverage can make local dynamics easier to reuse after visual task adaptation.
Limits include predominantly simulated composability, interface compatibility requirements, task-adapted target predictors and short-horizon dynamics metrics. A unified checkpoint is feasible but does not match all specialist policy/dynamics results.
Original robot demonstration
The Stanford project page provides the original 34.75-second robot demonstration, preserved unchanged with its original poster. It shows selected toasting and final-layer cube execution.
The public repository remains linked. These qualitative rollouts complement the complete task/trial tables.
Appendix A Implementation Details
The complete appendix preserves architectural, optimization, sampling, execution and robot-data specifications. Simulation and real hardware use different action/state parameterizations.
A.1 Model Architecture and Representation
The architecture uses 30 dual-stream transformer layers. Two visual streams and proprioception align with action chunks through a common temporal representation.
Dual-stream backbone.
The video expert has width 3072,24 heads of 128 channels and FFN width 14336. The action expert has hidden width 2048 and FFN 8192, projecting its queries/keys/values to 3072 for shared attention and returning to 2048 afterward.
Visual observations.
Agent and wrist 128×128 views are separately encoded to 8×8×48 latents, then concatenated to an 8×16×48 canvas. A 1×2×2 patch gives 32 video tokens per latent frame.
The VAE temporally compresses four environment steps per latent frame. Four future latent frames align with a 16-step action chunk.
Actions and proprioception.
Simulation actions have seven entries: end-effector position, axis-angle rotation and gripper command. The eight-entry state contains position, orientation and two gripper joint positions.
Proprioception is projected and added to each stream’s hidden representation, rather than appended as a separate token.
A.2 Downstream Training
The action expert starts from width-adapted video layers, with separately initialized input/output and conditioning layers. Both experts train jointly for 10,000 optimizer steps.
AdamW uses learning rate 10⁻⁵,β=(0.9,0.95), weight decay 0.1, ten warm-up steps, effective batch 10 through accumulation, gradient clipping 2.0 and bf16. The same recipe is used for the four programs.
Flow-matching objective.
The implementation uses noise parameterσ=1−τand a 1,000-level base grid. Video/action timestep shifts are 5/1, and normalized weighting emphasizes intermediate noise.
The printed weighting expression is zero at bothσ=0 andσ=1, although the prose mentions only the pure-noise endpoint. Future video/action losses have equal weight after their respective masks and normalizations.
Equations 11–13
| (11) |
| (12) |
| (13) |
A.3 Video–Action Interaction Programs
The appendix restates the four attention/generation programs and their common context. Sequential programs differ from concurrent programs in inference staging.
Joint.
Joint video/action streams exchange evolving noisy future tokens while denoising together. Neither trajectory must be completed before cross-modal information can flow.
Video-to-action (VTA).
VTA completes future video without future actions, then generates actions conditioned on that completed trajectory. Recorded video replaces generated video during second-stage training.
Action-to-video (ATV).
ATV completes actions first and then predicts video conditioned on them. Its second stage uses recorded actions during training and predicted actions at inference.
Decoupled.
Decoupled generates the two futures without future cross-attention. Both streams still see observed context and language, so decoupling does not mean removing every shared input or the common initialization.
A.4 Inference and Closed-Loop Evaluation
Closed-loop evaluation fixes denoising, history, startup and rollout termination across programs. It executes each complete 16-step action chunk before replanning.
Sampling.
Each modality uses 20 denoising steps, video guidance scale 5 and no action guidance. Key/value caching is enabled, with a history window of 15 chunks or 60 latent video frames.
The paper does not provide a matched latency/compute benchmark for all generation programs. Recorded 30 Hzhardware data does not imply 30 Hz large-model replanning.
Action execution.
After reset, five zero-action steps settle the simulator; the first chunk conditions on the resulting single observation. Training uses the same startup convention.
Episodes stop on success, 800 environment steps including startup, or 50 action chunks. The first successful simulator predicate ends the rollout.
Evaluation protocol.
Each program has three independently trained seeds and 50 episodes on each of ten tasks per suite, or 500 episodes per seed/suite. Programs share the same task-specific environment seeds.
Means and deviations summarize training-seed results. Published external baselines are not necessarily matched to that full protocol.
A.5 Real Robot Setup
GELLO collects bimanual demonstrations with two FR3 arms, Flexiv Grav grippers, D405 wrist cameras and an Orbbec Femto Mega scene camera. Images/state/control timestamps are recorded at 30 Hz.
Hardware actions contain 20 values: per arm, position 3, orientation 6 and gripper width 1. Poses are relative to the chunk-start observed pose; gripper widths remain absolute. This differs from the seven-entry simulation action.
Data comprise 200 toast and 200 cube demonstrations plus 180 cup demonstrations, each 30 seconds long. The full setup image and task definitions remain.
Accessible table and original citation links
| Dataset | Trajectories / takes | Hours | Source type |
|---|---|---|---|
| Open X-Embodiment (49 manipulation datasets) | 1,300,749 | 1,911.29 | Multi-embodiment robot data |
| AgiBot World Beta | 1,003,672 | 2,976.4 | Real robot manipulation |
| Ego-Exo4D v2 | 5,035 | 221.26 | Human interaction |
| InternData-A1 | 637,498 | 7,433.91 | Synthetic robot manipulation |
| RoboCOIN | 183,157 | 1,306.83 | Real bimanual manipulation |
| RoboMIND | 107,877 | 305.5 | Multi-embodiment robot data |
| FastUMI-100K | 92,823 | 461.77 | Human-guided manipulation |
| UMI family | 5,430 | 24.60 | Human-guided manipulation |
Appendix B Robot-Video Pretraining
Robot-video pretraining adapts a general video prior using robot, human-interaction and synthetic sequences without action labels or proprioception.
B.1 Pretraining Corpus
The corpus totals about 3.34 million trajectories/takes and 14.64k nominal source hours before window sampling. InternData-A1 contributes 7,433.91 synthetic hours, roughly half the total; the corpus is not exclusively physical-robot recordings.
The original table retains all eight source groups. The UMI-family total includes UMI, DexUMI, UMI on Legs and MV-UMI.
B.2 Causal Adaptation
Chunk-causal video pretraining excludes later chunks while permitting joint denoising within the current chunk. No robot action, state or policy target is supplied in this stage.
B.3 Training Compute
Pretraining uses 32 NVIDIA B200 accelerators for 14 days. That substantial shared initialization cost is separate from the downstream 10,000-step training comparisons.
Appendix C LIBERO-Long-CF: Counterfactual Dynamics Data
LIBERO-Long-CF executes alternative normalized controls from task-relevant simulator states, preserving the original ten tasks/assets/physics. It records actual simulated outcomes, including failures.
C.1 Scale and Intervention Coverage
The collection has 32,000 segments with 128 controls each, or 4,096,000 controls, versus 138,090 in 500 source demonstrations. At 20 Hzthose are 56.9 and 1.92 control-equivalent hours, about 29.7×more controls.
Intervention recipes vary magnitude, direction, timing, axis biases, noise and gripper state. Recipe fractions are rounded and describe generation categories, while resulting physical effects can overlap.
Accessible table and original citation links
| Intervention family | Fraction (%) |
|---|---|
| Stop / rescale arm motion | 9.4 |
| Reverse / redirect translation | 6.3 |
| Axis biases and pulses | 12.5 |
| Dedicated yaw perturbation | 3.1 |
| Noise / randomized arm controls | 12.5 |
| Dedicated gripper interventions | 31.3 |
| Random-duration arm / gripper interventions | 25.0 |
C.2 State and Interaction Coverage
Three quarters of segments start on demonstration prefixes; one quarter starts after an executed perturbation. Characterized perturbed starts average 3.34 cmfrom the demonstrated path, with median 1.90 cm.
Under the stated thresholds, 61.6%also change object configuration beyond the demonstrated configuration. Reported contact/grasp/object-effect rates are 90.4/50.0/72.3%, with overlapping categories and a characterization subset rather than an entirely new task distribution.
C.3 Representation for Local Dynamics Learning
Each segment has two 128×128 RGB streams, 128 seven-entry controls and 129 states. VAE encoding produces one boundary observation plus 32 future latent frames, aligned at four controls per frame.
The specialist models receive only local boundary/state inputs and the supplied future trajectory; task language and pre-start history remain excluded.
Appendix D Ablations
Two ablations compare backbone initialization and a shared-transformer versus separate-expert parameterization. All original result rows remain.
D.1 Effect of Video-Backbone Initialization
Random, original Wan and robot-video-causal initializations score 20.0/68.4/97.8%for VTA and 27.2/62.2/96.6%for Joint. The combined domain/causal initialization helps both programs.
Accessible table and original citation links
| Video initialization | VTA (%) | Joint (%) |
|---|---|---|
| Random initialization | 20.0 | 27.2 |
| Original Wan2.2 | 68.4 | 62.2 |
| Robot-video pretrained | 97.8 | 96.6 |
Accessible table and original citation links
| Policy | Real demonstrations | Counterfactual transitions | |||||||
| Model | Succ. (%) | FDM MSE | FDM SSIM | IDM pos. (cm) | IDM rot. (∘) | FDM MSE | FDM SSIM | IDM pos. (cm) | IDM rot. (∘) |
| OpenWAM, LC-IDM, CF-only | – | – | – | 1.76 | 2.39 | – | – | 1.17 | 3.60 |
| OpenWAM, LC-IDM, mixed | – | – | – | 0.44 | 1.21 | – | – | 1.22 | 3.68 |
| OpenWAM, LC-FDM, CF-only | – | 0.00734 | 0.9126 | – | – | 0.00940 | 0.8889 | – | – |
| OpenWAM, LC-FDM, mixed | – | 0.00186 | 0.9776 | – | – | 0.00962 | 0.8859 | – | – |
| OpenWAM, multi-objective | 92.8 | 0.00058 | 0.9934 | 0.43 | 1.10 | 0.01416 | 0.8398 | 1.73 | 3.55 |
| UVA | 88.0 | 0.00279 | 0.9645 | 2.03 | 1.60 | 0.01514 | 0.8134 | 3.57 | 12.99 |
D.2 Effect of the Mixture-of-Transformers Architecture
With the robot-video foundation fixed, the shared non-MoT transformer scores 92.8%VTA and 93.6%Joint; MoT scores 97.8/96.6%.
The MoT system adds a separate roughly 2B action expert to the video expert. This is not a total-parameter-matched test of attention organization alone.
Appendix E External Dynamics Comparison
The external comparison uses UVA, a released policy/inverse/forward model. RGB errors are measured in common agent-view image space; inverse predictions are compared through executed end-effector trajectories because action parameterizations differ [44].
These system comparisons do not match every backbone, pretraining corpus and compute resource.
E.1 Independent Dynamics Components
Counterfactual-only specialists have the best specialist prediction on held-out alternative transitions, while mixed training substantially improves fit on demonstrations.
The full table retains position/orientation errors separately. No single variant wins every metric; for example, the unified model’s counterfactual rotation error is slightly lower even though specialist position and video prediction are stronger.
E.2 Unified Policy and Dynamics Modeling
The unified checkpoint allocates 60%updates to joint policy data and 20%each to local inverse/forward objectives. Each dynamics portion is split equally between demonstration and counterfactual transitions.
It scores 92.8%policy success versus UVA 88.0%, but remains below policy-specialized variants and weaker than specialists on most counterfactual dynamics measures. Lowest demonstration-distribution errors can include training examples seen by the policy objective, so they are fit rather than held-out generalization evidence.