Literature
Display
← All papers

OpenWAM: An Open Framework for Composable World-Action Models

Heng Yu · David D. Yuan · Juze Zhang · Changan Chen · Yao Feng · Michelle Baldonado · Steve Cousins · Li Fei-Fei · Jiajun Wu · Ehsan Adeli

Preprint posted · arXiv (preprint) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

Heng Yu, David D. Yuan and Juze Zhang contributed equally. This Stanford framework’s project page and code repository accompany the complete paper and implementation appendices.

At a glance

Stanford’s OpenWAM compares joint, video-first, action-first and decoupled world–action generation under a common causal video backbone. Separate local inverse/forward dynamics models connect visual futures with actions.

Video-first policy success averages 98.6% over four LIBERO suites. A frozen counterfactual-trained local inverse model reaches 84.0% across four target tasks after adapting their video predictors. The latter is component reuse with target-task training, rather than zero-shot transfer of the complete system.

1 Introduction

World–action systems often change data, architecture and inference together, making design choices hard to compare. This framework shares a video initialization and downstream recipe while varying cross-modal attention and generation order.

A second study learns reusable dynamics from alternatives to successful demonstrations. Simulation branches broaden local transition coverage; independently trained dynamics components then receive explicit future-video or action trajectories.

Complete original Figure 1 : Overview of OpenWAM . Left: Starting from Wan2.2 [ 75 ] , causal pretraining on robot and
human-interaction video builds a predictive backbone without action labels,
followed by video–action training on simulated and real-robot demonstrations. Top right: A shared Mixture-of-Transformers couples temporally
aligned video and action streams through joint attention while retaining
modality-specific experts. Middle right: Configurable cross-modal attention and generation
order support joint, video-then-action, action-then-video, and decoupled
prediction within the same architecture. Bottom right: Local-context inverse and forward dynamics models
are trained independently using demonstrations and counterfactual
transitions. A frozen IDM translates predictions from compatible
task-adapted video models into actions, while an FDM predicts the visual
consequences of candidate actions.
Figure 1: Overview of OpenWAM. Left: Starting from Wan2.2 [75], causal pretraining on robot and human-interaction video builds a predictive backbone without action labels, followed by video–action training on simulated and real-robot demonstrations. Top right: A shared Mixture-of-Transformers couples temporally aligned video and action streams through joint attention while retaining modality-specific experts. Middle right: Configurable cross-modal attention and generation order support joint, video-then-action, action-then-video, and decoupled prediction within the same architecture. Bottom right: Local-context inverse and forward dynamics models are trained independently using demonstrations and counterfactual transitions. A frozen IDM translates predictions from compatible task-adapted video models into actions, while an FDM predicts the visual consequences of candidate actions.

2 Related Work

Related work spans direct visuomotor control, predictive policies and dynamics learning beyond successful demonstrations. The framework separates interaction design from component-level transition supervision.

2.1 Visuomotor Policies and Predictive Control

Behavior cloning maps current observations to actions, while predictive control uses imagined observations or latent dynamics. OpenWAM exposes both kinds of trajectory interface within one modeling framework [13] [16].

2.2 Video Foundations and World–Action Architectures

Existing WAMs use joint denoising, video-first prediction, specialized branches and multiple policy/dynamics modes. This paper holds several shared choices fixed to study interaction programs and independent component reuse.

A concurrent, separately authored project is also named OpenWAM. Its paper is the distinct modular-pretraining study already in this library; the two entries should remain separate [79].

2.3 Dynamics Learning Beyond Successful Demonstrations

Demonstrations show actions chosen by a successful policy, leaving many alternative outcomes uncovered. Prior recovery, failure and counterfactual methods broaden that experience.

Here the collection restores simulator states and executes modified action continuations. This directly observes alternative simulator outcomes; it does not infer them solely from observational demonstrations.

3 Methodology

The system combines a causal robot-video backbone, configurable dual-stream interaction and separately trained local dynamics. Composition within a model and composition between independently trained components are distinct capabilities.

3.1 World–Action Modeling Formulation

Context contains past/current observations, past actions, proprioception and task language. Future actions start at the current control step, while future observations follow the executed actions.

The two chain-rule factorizations identify video prediction/inverse dynamics and direct action/forward dynamics roles. They describe a reference joint distribution. Independently trained components need not be mutually consistent factors of one learned joint distribution; their composition is evaluated empirically.

Equations 1–3
Ot−\displaystyle O_{t}^{-} =(ot−Ko,…,ot−1),\displaystyle=(o_{t-K_{o}},\ldots,o_{t-1}), (1)
At−\displaystyle A_{t}^{-} =(at−Ka,…,at−1),\displaystyle=(a_{t-K_{a}},\ldots,a_{t-1}),
Ot0\displaystyle O_{t}^{0} =ot.\displaystyle=o_{t}.
At+=(at,…,at+H−1),Ot+=(ot+1,…,ot+H).A_{t}^{+}=(a_{t},\ldots,a_{t+H-1}),\quad O_{t}^{+}=(o_{t+1},\ldots,o_{t+H}). (2)
𝒞t=(Ot−,Ot0,At−,qt,ℓ).\mathcal{C}_{t}=(O_{t}^{-},O_{t}^{0},A_{t}^{-},q_{t},\ell).
Original Equation 3: joint-distribution factorizations with CVP, IDM, VLA and FDM underbrace labels.
Original selectable mathematics
pWAM​(Ot+,At+∣𝒞t)\displaystyle p_{\mathrm{WAM}}(O_{t}^{+},A_{t}^{+}\mid\mathcal{C}_{t}) =p⁡(Ot+∣𝒞t)⏟CVP​p⁡(At+∣Ot+,𝒞t)⏟IDM\displaystyle=\underbrace{p(O_{t}^{+}\mid\mathcal{C}_{t})}_{\mathrm{CVP}}\underbrace{p(A_{t}^{+}\mid O_{t}^{+},\mathcal{C}_{t})}_{\mathrm{IDM}} (3)
=p⁡(At+∣𝒞t)⏟VLA​p⁡(Ot+∣At+,𝒞t)⏟FDM.\displaystyle=\underbrace{p(A_{t}^{+}\mid\mathcal{C}_{t})}_{\mathrm{VLA}}\underbrace{p(O_{t}^{+}\mid A_{t}^{+},\mathcal{C}_{t})}_{\mathrm{FDM}}.

3.2 An Open Causal Robot-Video Backbone

Wan2.2-5B is adapted on heterogeneous interaction video without action labels. Chunk-causal attention permits current/past observations and earlier generated chunks, while processing tokens inside each future chunk jointly.

Temporal causality is separate from future video/action communication. The same pretrained video checkpoint initializes all main downstream variants and remains usable for standalone video prediction.

3.3 Configurable Video–Action Interaction Programs

A shared Mixture-of-Transformers architecture supports four native policy programs. Each retains the same data/representations/objective family, while changing future cross-modal visibility and sampling order.

Shared architecture.

The video and action experts have separate normalization, projection and feed-forward parameters. Attention over packed modality tokens exchanges information, with a separate text cross-attention path.

The video expert is about 5B parameters and the action expert about 2B. Common initialization helps compare the four programs, while comparisons with a shared single-transformer architecture also change parameter allocation [49].

Complete original Figure 2 : Shared MoT architecture. Two modality-specific branches keep their own
normalization, projection and feed-forward parameters and meet in a single
attention over the packed video and action tokens; a separate
cross-attention consumes the task text. Noisy future tokens enter at the
bottom and the preceding chunks sit behind as history context.
Figure 2: Shared MoT architecture. Two modality-specific branches keep their own normalization, projection and feed-forward parameters and meet in a single attention over the packed video and action tokens; a separate cross-attention consumes the task text. Noisy future tokens enter at the bottom and the preceding chunks sit behind as history context.

Interaction programs.

Joint denoises video and actions with bidirectional future attention. Video-then-action (VTA) finishes the visual future before predicting actions; action-then-video (ATV) reverses that order.

Decoupled removes attention between future modalities while retaining observed history and language. Sequential programs use two generation passes, so shared training settings do not imply identical inference computation or latency.

Complete original Figure 3 : Attention masks the shared architecture exposes: four interaction programs
(A–D) and the two local-context dynamics programs (E–F). Columns are the
streams a row may condition on, cell colour gives what it attends to there,
and the numeral marks the generation pass. E and F carry one row each
because each predicts a single stream: the inverse model produces actions, the forward model observations.
Figure 3: Attention masks the shared architecture exposes: four interaction programs (A–D) and the two local-context dynamics programs (E–F). Columns are the streams a row may condition on, cell colour gives what it attends to there, and the numeral marks the generation pass. E and F carry one row each because each predicts a single stream: the inverse model produces actions, the forward model observations.
Complete original Table 1 : LIBERO-90 targets used for component-transfer evaluation.
Table 1: LIBERO-90 targets used for component-transfer evaluation.
Accessible table and original citation links
Task Target behavior Transfer challenge
64 Stack bowls; place stack in tray Novel skill composition
74 Book into caddy’s left compartment Retarget familiar manipulation
21 Turn on stove; place frying pan on it Object and grasp change
45 Same objective as Task 21 Additional scene and layout shift
Complete original Figure 4 : Evaluation tasks. Left: the three real-world bimanual
tasks of Table 2 , each shown as frames sampled
along one successful rollout. Right: the four LIBERO-90 targets of Table 1 used for component transfer, which lie
outside the LIBERO-Long source-task set.
Figure 4: Evaluation tasks. Left: the three real-world bimanual tasks of Table 2, each shown as frames sampled along one successful rollout. Right: the four LIBERO-90 targets of Table 1 used for component transfer, which lie outside the LIBERO-Long source-task set.

Shared flow-matching objective.

Latent flow matching interpolates from noise to clean targets and learns the corresponding velocity. Multiple active target modalities contribute summed losses.

For VTA/ATV training, the second modality receives the recorded first-modality trajectory. At inference it receives the generated trajectory instead. This teacher-forcing shift is part of the tested pipeline.

Equations 4–5
ZP,τ=(1−τ)​ϵP+τ​ZP⋆,uP=ZP⋆−ϵP.Z_{P,\tau}=(1-\tau)\epsilon_{P}+\tau Z_{P}^{\star},\quad u_{P}=Z_{P}^{\star}-\epsilon_{P}. (4)
ℒP=𝔼⁡[‖vθ,P​(ZP,τ∣cP,τ)−uP‖22],\mathcal{L}_{P}=\mathbb{E}\!\left[\left\|v_{\theta,P}\!\left(Z_{P,\tau}\mid c_{P},\tau\right)-u_{P}\right\|_{2}^{2}\right], (5)

3.4 Counterfactual Learning of Local-Context Dynamics Interfaces

Local inverse dynamics maps a proposed visual trajectory into actions; local forward dynamics predicts a visual trajectory from supplied actions. The models train independently through explicit trajectory interfaces.

Local-context dynamics interfaces.

Both local components receive the current observation and proprioception, plus the supplied future video or action trajectory. They exclude language, past actions and earlier observations.

This restricts conditioning without proving that local inputs are sufficient under every contact state, occlusion or embodiment. The paper tests adequacy in its benchmark transitions and composed policies.

Equation 6
pIDMlocal\displaystyle p_{\mathrm{IDM}}^{\mathrm{local}} :=p⁡(At+∣Ot+,Ot0,qt),\displaystyle:=p(A_{t}^{+}\mid O_{t}^{+},O_{t}^{0},q_{t}), (6)
pFDMlocal\displaystyle p_{\mathrm{FDM}}^{\mathrm{local}} :=p⁡(Ot+∣At+,Ot0,qt).\displaystyle:=p(O_{t}^{+}\mid A_{t}^{+},O_{t}^{0},q_{t}).

Broader local-transition supervision.

Alternative simulator action sequences branch from the same starting state, providing recorded state/action/outcome transitions whether or not a task succeeds. Simulator state controls collection but is not a model input.

Training samples individual transitions; there is no explicit paired-branch contrastive loss. Mixed dynamics supervision draws 60%counterfactual and 40%demonstration transitions. Broader coverage also adds much more data, without a demonstration-only control matched to that full diversity/volume.

Equation 7
𝒟dyn=(1−η)​𝒟taskloc+η​𝒟cf,η∈[0,1].\mathcal{D}_{\mathrm{dyn}}=(1-\eta)\mathcal{D}_{\mathrm{task}}^{\mathrm{loc}}+\eta\mathcal{D}_{\mathrm{cf}},\qquad\eta\in[0,1]. (7)

Local dynamics learning.

Inverse dynamics denoises actions conditioned on the realized future; forward dynamics denoises future video conditioned on executed actions. The supplied conditioning modality is not a simultaneous denoising target in these specialist objectives.

Equation 8
ℒLC​-​IDM\displaystyle\mathcal{L}_{\mathrm{LC\text{-}IDM}} =𝔼⁡[‖vθID​(ZA,τ∣O0,qt,O+,τ)−uA‖22],\displaystyle=\mathbb{E}\!\left[\left\|v_{\theta_{\mathrm{ID}}}(Z_{\mathrm{A},\tau}\mid O^{0},q_{t},O^{+},\tau)-u_{\mathrm{A}}\right\|_{2}^{2}\right], (8)
ℒLC​-​FDM\displaystyle\mathcal{L}_{\mathrm{LC\text{-}FDM}} =𝔼⁡[‖vθFD​(ZO,τ∣O0,qt,A+,τ)−uO‖22].\displaystyle=\mathbb{E}\!\left[\left\|v_{\theta_{\mathrm{FD}}}(Z_{\mathrm{O},\tau}\mid O^{0},q_{t},A^{+},\tau)-u_{\mathrm{O}}\right\|_{2}^{2}\right].

Inference and component composition.

A video producer passes VAE latents to the receiving inverse model, which re-encodes them through its conditioning pathway. Transformer states/caches are not exchanged. An external action policy can similarly supply actions to a forward model.

Reuse requires compatible visual latent representation, time alignment, camera/action conventions and normalization. The “arbitrary model” diagram should be read within those interface constraints, rather than as direct compatibility with every pretrained model.

Equations 9–10
O^t+∼pϕCVP(⋅∣𝒞t),A^t+∼pθIDLC​-​IDM(⋅∣O^t+,Ot0,qt).\hat{O}_{t}^{+}\sim p_{\phi}^{\mathrm{CVP}}(\cdot\mid\mathcal{C}_{t}),\quad\hat{A}_{t}^{+}\sim p_{\theta_{\mathrm{ID}}}^{\mathrm{LC\text{-}IDM}}(\cdot\mid\hat{O}_{t}^{+},O_{t}^{0},q_{t}). (9)
A^t+∼pψVLA(⋅∣𝒞t),O^t+∼pθFDLC​-​FDM(⋅∣A^t+,Ot0,qt).\hat{A}_{t}^{+}\sim p_{\psi}^{\mathrm{VLA}}(\cdot\mid\mathcal{C}_{t}),\quad\hat{O}_{t}^{+}\sim p_{\theta_{\mathrm{FD}}}^{\mathrm{LC\text{-}FDM}}(\cdot\mid\hat{A}_{t}^{+},O_{t}^{0},q_{t}). (10)

4 Experiments

Experiments distinguish native policy performance, frozen inverse-component transfer and short-horizon forward prediction. All benchmark/task tables and the complete implementation appendix are retained.

4.1 Experimental Setup

The common foundation is adapted to four LIBERO suites and three physical bimanual tasks. Counterfactual data train separate dynamics models, with same-start branches kept in the same split.

Robot-video pretraining and simulation.

The video corpus mixes robot demonstrations, synthetic manipulation and human interaction. Downstream video chunks contain four latent frames aligned with 16 action steps, plus per-chunk proprioception.

Component transfer starts from LIBERO-Long dynamics and uses four LIBERO-90 tasks outside that source task set. This is a task holdout within the benchmark, rather than evidence of arbitrary unseen assets, embodiments or environments.

Complete original Table 2 : Real-world bimanual manipulation tasks.
Table 2: Real-world bimanual manipulation tasks.
Accessible table and original citation links
Task Objective Main challenge
Toast Bread Insert bread and activate toaster Deformability and precise insertion
Rubik’s Cube Restore final layer of 2×22\times 2 cube Contact-rich push and rotation
Sort Cups Sort cups according to color Variable ordering and sequencing

Real-world tasks.

The real system uses two Franka FR3 arms, parallel-jaw grippers, two wrist cameras and a third-person camera. Tasks are toasting bread, restoring the final layer of a 2×2 cube and sorting cups by color.

The cube task is the stated final-layer procedure, rather than unrestricted solving from arbitrary cube configurations.

Dynamics supervision.

Counterfactual branches start from LIBERO-Long demonstration states and executed perturbations. Demonstration-only, counterfactual-only and 60/40 mixed data train local dynamics without changing their interfaces.

Keeping all alternatives from a shared initial state in one partition prevents direct same-context branch leakage. It does not by itself establish transfer to new tasks beyond the separately reported four-target experiment.

Evaluation.

Closed-loop task success evaluates policies and compositions. Forward-dynamics evaluation measures RGB error and whether a prediction is closest to its own realized outcome among two or 16 same-state alternatives.

Outcome identification measures action-dependent visual discrimination. It is not a physical success predicate or a demonstration that the model improves a downstream planner.

Complete original Table 3 : Closed-loop success (%) on four LIBERO suites.
Baselines are previously reported results.
Table 3: Closed-loop success (%) on four LIBERO suites. Baselines are previously reported results.
Accessible table and original citation links
Method Object Goal Spatial Long Mean
OpenVLA [38] 88.4 79.2 84.7 53.7 76.5
OpenVLA-OFT [36] 98.4 97.9 97.6 94.5 97.1
π0\pi_{0} [6] 98.8 95.8 96.8 85.2 94.1
π0.5\pi_{0.5} [33] 98.2 98.0 98.8 92.4 96.9
GR00T-N1 [5] 97.6 93.0 94.4 90.6 93.9
Motus [3] 99.8 96.6 96.8 97.6 97.7
Fast-WAM [94] 100.0 97.0 98.2 95.2 97.6
LingBot-VA [43] 99.6 97.2 98.5 98.5 98.5
OpenWAM-VTA 99.4±0.3\text{99.4}_{\pm\text{0.3}} 98.4±0.5\text{98.4}_{\pm\text{0.5}} 98.6±0.2\text{98.6}_{\pm\text{0.2}} 97.8±0.4\text{97.8}_{\pm\text{0.4}} 98.6
OpenWAM-ATV 98.0±0.4\text{98.0}_{\pm\text{0.4}} 97.2±0.2\text{97.2}_{\pm\text{0.2}} 96.6±0.6\text{96.6}_{\pm\text{0.6}} 95.4±0.3\text{95.4}_{\pm\text{0.3}} 96.8
OpenWAM-Joint 98.2±0.2\text{98.2}_{\pm\text{0.2}} 97.8±0.4\text{97.8}_{\pm\text{0.4}} 97.6±0.3\text{97.6}_{\pm\text{0.3}} 96.6±0.5\text{96.6}_{\pm\text{0.5}} 97.6
OpenWAM-Decoupled 99.0±0.3\text{99.0}_{\pm\text{0.3}} 98.0±0.2\text{98.0}_{\pm\text{0.2}} 97.8±0.5\text{97.8}_{\pm\text{0.5}} 97.0±0.4\text{97.0}_{\pm\text{0.4}} 98.0
Complete original Table 4 : Closed-loop success (%) on real-world bimanual manipulation tasks.
Table 4: Closed-loop success (%) on real-world bimanual manipulation tasks.
Accessible table and original citation links
Method Toast Cube Cups Mean
OpenWAM-VTA 92.0 90.0 94.4 92.1
OpenWAM-Joint 90.0 94.0 91.7 91.9
Complete original Table 5 : Frozen inverse dynamics transfer to four LIBERO-90 targets.
Target rows use task-tuned video predictors while source-trained IDMs
remain frozen. Mean is averaged over the four target tasks.
Table 5: Frozen inverse dynamics transfer to four LIBERO-90 targets. Target rows use task-tuned video predictors while source-trained IDMs remain frozen. Mean is averaged over the four target tasks.
Accessible table and original citation links
Video / policy Action component (frozen) 64 Composition 74 Retarget 21 Object/grasp 45 Scene shift Transfer mean
Task-tuned VTA (finetuned policy reference) 94% 86% 92% 100% 93.0%
Task-tuned video Full-context IDM, demo-only 88% 90% 0% 0% 44.5%
Task-tuned video Full-context IDM, mixed 54% 92% 4% 38% 47.0%
Task-tuned video Local-context IDM, demo-only 36% 50% 0% 0% 21.5%
Task-tuned video Local-context IDM, CF-only 80% 92% 90% 76% 84.5%
Task-tuned video Local-context IDM, mixed 74% 86% 88% 88% 84.0%
Original VTA (unadapted policy reference) 42% 0% 0% 0% 10.5%
Complete original Table 6 : Source-task composition on LIBERO-Long, all action components evaluated with LIBERO-Long video
predictor.
Table 6: Source-task composition on LIBERO-Long, all action components evaluated with LIBERO-Long video predictor.
Accessible table and original citation links
Action component Supervision Success (%)
Full-context IDM Demo-only (Native VTA) 97.8
Full-context IDM Mixed 92.2
Local-context IDM Demo-only 25.8
Local-context IDM CF-only 90.6
Local-context IDM Mixed 94.4
Complete original Table 7 : Local-context FDM prediction on counterfactual transitions.
MSE values are reported in units of 10 − 3 10^{-3} .
Outcome accuracy measures identification of the matching realized future
by RGB MSE among K K same-state outcomes.
Table 7: Local-context FDM prediction on counterfactual transitions. MSE values are reported in units of 10−310^{-3}. Outcome accuracy measures identification of the matching realized future by RGB MSE among KK same-state outcomes.
Accessible table and original citation links
Supervision RGB MSE ↓\downarrow Acc. K=2K{=}2 ↑\uparrow Acc. K=16K{=}16 ↑\uparrow
Demo-only 14.35 68.3 21.1
CF-only 9.40 93.6 71.3
Mixed 9.62 91.8 67.7

4.2 Native Policy Performance

All four interaction programs produce strong LIBERO results from the common robot-video backbone. Real-robot results compare VTA and Joint only.

Broad LIBERO evaluation.

VTA averages 98.6%success across four suites, versus 97.6%Joint, 96.8%ATV and 98.0%Decoupled. Three independent training seeds are evaluated; task-level rollout conditions are shared across programs.

Published baseline results use their original papers’ protocols. The 0.1-point mean difference from reported LingBot-VA is not a controlled significance result. Decoupled’s competitive score also shows that future cross-modal attention is not necessary for strong control in this setting.

Backbone initialization ablation.

With VTA architecture and downstream training fixed, original Wan2.2 initialization scores 68.4%on LIBERO-Long and the robot-video-pretrained causal initialization scores 97.8%, a 29.4-percentage-point gain.

This isolates the combined new initialization, including both robot-video training and causal adaptation. It does not separate those two contributions.

Real-world native policies.

Physical mean success is 92.1%VTA and 91.9%Joint across the three tasks. Evaluation uses 50 toast, 50 cube and 36 cup trials per method, with no reported independent training-seed intervals.

The rounded table values correspond to 125 successes out of 136 for each model. The slight difference in equally weighted task means should not be read as an established physical reliability advantage.

4.3 Component Composability

The component study pairs the same target-video producer with different frozen source-trained inverse models. It separates conditioning scope from demonstration/counterfactual supervision.

Frozen action-component reuse.

Target video predictors come from task-tuned VTA models, which use target task training. Only the reused source inverse component remains frozen; the full composition is not zero-shot from an unadapted predictor.

Local inverse transfer rises from 21.5%demo-only to 84.0%mixed; full-context transfer rises 44.5%to 47.0%. Counterfactual-only local dynamics reaches 84.5%, slightly above mixed 84.0%, while mixed retains better source success, 94.4%versus 90.6%.

Per-task reversals remain: demo-only full context reaches 88/90%on the first two targets but 0%on the other two. Task-tuned native VTA averages 93.0%, setting a useful adapted-policy reference rather than an unsupervised transfer baseline.

4.4 Action-Conditioned Forward Dynamics

Forward evaluation uses 160 contexts across ten tasks with 16 action branches each, totaling 2,560 futures. Counterfactual-only training lowers RGB MSE from 14.35 to 9.40 in 10⁻³units, a 34.5%relative reduction.

It raises two-outcome identification from 68.3%to 93.6%and 16-outcome identification from 21.1%to 71.3%. Mixed data is slightly weaker on those held-out alternatives but stronger on demonstrations. These are local prediction results, not long-horizon real-world planning tests.

5 Discussion and Conclusion

The framework offers a controlled way to compare generation programs and demonstrates that broader simulated transition coverage can make local dynamics easier to reuse after visual task adaptation.

Limits include predominantly simulated composability, interface compatibility requirements, task-adapted target predictors and short-horizon dynamics metrics. A unified checkpoint is feasible but does not match all specialist policy/dynamics results.

Original robot demonstration

The Stanford project page provides the original 34.75-second robot demonstration, preserved unchanged with its original poster. It shows selected toasting and final-layer cube execution.

The public repository remains linked. These qualitative rollouts complement the complete task/trial tables.

Original Stanford OpenWAM robot demonstration: toasting bread and restoring the final layer of a 2×2 Rubik’s cube.

Appendix A Implementation Details

The complete appendix preserves architectural, optimization, sampling, execution and robot-data specifications. Simulation and real hardware use different action/state parameterizations.

A.1 Model Architecture and Representation

The architecture uses 30 dual-stream transformer layers. Two visual streams and proprioception align with action chunks through a common temporal representation.

Dual-stream backbone.

The video expert has width 3072,24 heads of 128 channels and FFN width 14336. The action expert has hidden width 2048 and FFN 8192, projecting its queries/keys/values to 3072 for shared attention and returning to 2048 afterward.

Visual observations.

Agent and wrist 128×128 views are separately encoded to 8×8×48 latents, then concatenated to an 8×16×48 canvas. A 1×2×2 patch gives 32 video tokens per latent frame.

The VAE temporally compresses four environment steps per latent frame. Four future latent frames align with a 16-step action chunk.

Actions and proprioception.

Simulation actions have seven entries: end-effector position, axis-angle rotation and gripper command. The eight-entry state contains position, orientation and two gripper joint positions.

Proprioception is projected and added to each stream’s hidden representation, rather than appended as a separate token.

A.2 Downstream Training

The action expert starts from width-adapted video layers, with separately initialized input/output and conditioning layers. Both experts train jointly for 10,000 optimizer steps.

AdamW uses learning rate 10⁻⁵,β=(0.9,0.95), weight decay 0.1, ten warm-up steps, effective batch 10 through accumulation, gradient clipping 2.0 and bf16. The same recipe is used for the four programs.

Flow-matching objective.

The implementation uses noise parameterσ=1−τand a 1,000-level base grid. Video/action timestep shifts are 5/1, and normalized weighting emphasizes intermediate noise.

The printed weighting expression is zero at bothσ=0 andσ=1, although the prose mentions only the pure-noise endpoint. Future video/action losses have equal weight after their respective masks and normalizations.

Equations 11–13
Ss​(σ)=s​σ1+(s−1)​σ,sV=5.0,sA=1.0,S_{s}(\sigma)=\frac{s\sigma}{1+(s-1)\sigma},\qquad s_{V}=5.0,\quad s_{A}=1.0, (11)
wP(σ)∝exp[−2(σ−12)2]−e−1/2,w_{P}(\sigma)\propto\exp\!\left[-2\left(\sigma-\tfrac{1}{2}\right)^{2}\right]-e^{-1/2}, (12)
ℒ=ℒV+ℒA.\mathcal{L}=\mathcal{L}_{V}+\mathcal{L}_{A}. (13)

A.3 Video–Action Interaction Programs

The appendix restates the four attention/generation programs and their common context. Sequential programs differ from concurrent programs in inference staging.

Joint.

Joint video/action streams exchange evolving noisy future tokens while denoising together. Neither trajectory must be completed before cross-modal information can flow.

Video-to-action (VTA).

VTA completes future video without future actions, then generates actions conditioned on that completed trajectory. Recorded video replaces generated video during second-stage training.

Action-to-video (ATV).

ATV completes actions first and then predicts video conditioned on them. Its second stage uses recorded actions during training and predicted actions at inference.

Decoupled.

Decoupled generates the two futures without future cross-attention. Both streams still see observed context and language, so decoupling does not mean removing every shared input or the common initialization.

A.4 Inference and Closed-Loop Evaluation

Closed-loop evaluation fixes denoising, history, startup and rollout termination across programs. It executes each complete 16-step action chunk before replanning.

Sampling.

Each modality uses 20 denoising steps, video guidance scale 5 and no action guidance. Key/value caching is enabled, with a history window of 15 chunks or 60 latent video frames.

The paper does not provide a matched latency/compute benchmark for all generation programs. Recorded 30 Hzhardware data does not imply 30 Hz large-model replanning.

Action execution.

After reset, five zero-action steps settle the simulator; the first chunk conditions on the resulting single observation. Training uses the same startup convention.

Episodes stop on success, 800 environment steps including startup, or 50 action chunks. The first successful simulator predicate ends the rollout.

Evaluation protocol.

Each program has three independently trained seeds and 50 episodes on each of ten tasks per suite, or 500 episodes per seed/suite. Programs share the same task-specific environment seeds.

Means and deviations summarize training-seed results. Published external baselines are not necessarily matched to that full protocol.

A.5 Real Robot Setup

GELLO collects bimanual demonstrations with two FR3 arms, Flexiv Grav grippers, D405 wrist cameras and an Orbbec Femto Mega scene camera. Images/state/control timestamps are recorded at 30 Hz.

Hardware actions contain 20 values: per arm, position 3, orientation 6 and gripper width 1. Poses are relative to the chunk-start observed pose; gripper widths remain absolute. This differs from the seven-entry simulation action.

Data comprise 200 toast and 200 cube demonstrations plus 180 cup demonstrations, each 30 seconds long. The full setup image and task definitions remain.

Refer to caption
Figure 5: Real-world bimanual setup with two Franka FR3 arms, Flexiv Grav grippers, bimanual GELLO teleoperation [83], two wrist-mounted RealSense D405 cameras, and a central Orbbec Femto Mega camera.
Complete original Table 8 : Data used for causal robot-video pretraining.
Table 8: Data used for causal robot-video pretraining.
Accessible table and original citation links
Dataset Trajectories / takes Hours Source type
Open X-Embodiment (49 manipulation datasets) 1,300,749 1,911.29 Multi-embodiment robot data
AgiBot World Beta 1,003,672 2,976.4 Real robot manipulation
Ego-Exo4D v2 5,035 221.26 Human interaction
InternData-A1 637,498 7,433.91 Synthetic robot manipulation
RoboCOIN 183,157 1,306.83 Real bimanual manipulation
RoboMIND 107,877 305.5 Multi-embodiment robot data
FastUMI-100K 92,823 461.77 Human-guided manipulation
UMI family 5,430 24.60 Human-guided manipulation

Appendix B Robot-Video Pretraining

Robot-video pretraining adapts a general video prior using robot, human-interaction and synthetic sequences without action labels or proprioception.

B.1 Pretraining Corpus

The corpus totals about 3.34 million trajectories/takes and 14.64k nominal source hours before window sampling. InternData-A1 contributes 7,433.91 synthetic hours, roughly half the total; the corpus is not exclusively physical-robot recordings.

The original table retains all eight source groups. The UMI-family total includes UMI, DexUMI, UMI on Legs and MV-UMI.

B.2 Causal Adaptation

Chunk-causal video pretraining excludes later chunks while permitting joint denoising within the current chunk. No robot action, state or policy target is supplied in this stage.

B.3 Training Compute

Pretraining uses 32 NVIDIA B200 accelerators for 14 days. That substantial shared initialization cost is separate from the downstream 10,000-step training comparisons.

Appendix C LIBERO-Long-CF: Counterfactual Dynamics Data

LIBERO-Long-CF executes alternative normalized controls from task-relevant simulator states, preserving the original ten tasks/assets/physics. It records actual simulated outcomes, including failures.

C.1 Scale and Intervention Coverage

The collection has 32,000 segments with 128 controls each, or 4,096,000 controls, versus 138,090 in 500 source demonstrations. At 20 Hzthose are 56.9 and 1.92 control-equivalent hours, about 29.7×more controls.

Intervention recipes vary magnitude, direction, timing, axis biases, noise and gripper state. Recipe fractions are rounded and describe generation categories, while resulting physical effects can overlap.

Complete original Table 9 : Scale of LIBERO-Long-CF and the source LIBERO-Long demonstration corpus.
Table 9: Scale of LIBERO-Long-CF and the source LIBERO-Long demonstration corpus.
Accessible table and original citation links
Quantity Demonstrations LIBERO-Long-CF
Tasks 10 10
Stored sequences 500 32,000
Sequences / task 50 3,200
Controls / sequence 276.2 mean 128
Total controls 138,090 4,096,000
Duration 1.92 h 56.9 h
Complete original Table 10 : Intervention composition of LIBERO-Long-CF. Categories denote generation
recipes and may induce overlapping physical effects.
Table 10: Intervention composition of LIBERO-Long-CF. Categories denote generation recipes and may induce overlapping physical effects.
Accessible table and original citation links
Intervention family Fraction (%)
Stop / rescale arm motion 9.4
Reverse / redirect translation 6.3
Axis biases and pulses 12.5
Dedicated yaw perturbation 3.1
Noise / randomized arm controls 12.5
Dedicated gripper interventions 31.3
Random-duration arm / gripper interventions 25.0

C.2 State and Interaction Coverage

Three quarters of segments start on demonstration prefixes; one quarter starts after an executed perturbation. Characterized perturbed starts average 3.34 cmfrom the demonstrated path, with median 1.90 cm.

Under the stated thresholds, 61.6%also change object configuration beyond the demonstrated configuration. Reported contact/grasp/object-effect rates are 90.4/50.0/72.3%, with overlapping categories and a characterization subset rather than an entirely new task distribution.

Complete original Table 11 : Physical interaction statistics for LIBERO-Long-CF. Categories overlap.
Table 11: Physical interaction statistics for LIBERO-Long-CF. Categories overlap.
Accessible table and original citation links
Measured interaction Rate (%)
Gripper–object / fixture contact 90.4
Detected grasp 50.0
Object-configuration effect 72.3

C.3 Representation for Local Dynamics Learning

Each segment has two 128×128 RGB streams, 128 seven-entry controls and 129 states. VAE encoding produces one boundary observation plus 32 future latent frames, aligned at four controls per frame.

The specialist models receive only local boundary/state inputs and the supplied future trajectory; task language and pre-start history remain excluded.

Appendix D Ablations

Two ablations compare backbone initialization and a shared-transformer versus separate-expert parameterization. All original result rows remain.

D.1 Effect of Video-Backbone Initialization

Random, original Wan and robot-video-causal initializations score 20.0/68.4/97.8%for VTA and 27.2/62.2/96.6%for Joint. The combined domain/causal initialization helps both programs.

Complete original Table 12 : Effect of video-backbone initialization on closed-loop LIBERO-Long success.
Downstream architecture and training are held fixed within each interaction
program.
Table 12: Effect of video-backbone initialization on closed-loop LIBERO-Long success. Downstream architecture and training are held fixed within each interaction program.
Accessible table and original citation links
Video initialization VTA (%) Joint (%)
Random initialization 20.0 27.2
Original Wan2.2 68.4 62.2
Robot-video pretrained 97.8 96.6
Complete original Table 13 : Effect of the Mixture-of-Transformers architecture on closed-loop
LIBERO-Long success. All variants use robot-video-pretrained
initialization.
Table 13: Effect of the Mixture-of-Transformers architecture on closed-loop LIBERO-Long success. All variants use robot-video-pretrained initialization.
Accessible table and original citation links
Architecture VTA (%) Joint (%)
Non-MoT 92.8 93.6
MoT 97.8 96.6
Complete original Table 14 : Policy and dynamics accuracy on LIBERO-Long.
LC-IDM and LC-FDM are independently trained specialist checkpoints;
the multi-objective model trains policy, IDM, and FDM within a single
checkpoint.
Lower is better for MSE and IDM error; higher is better for success and SSIM.
Table 14: Policy and dynamics accuracy on LIBERO-Long. LC-IDM and LC-FDM are independently trained specialist checkpoints; the multi-objective model trains policy, IDM, and FDM within a single checkpoint. Lower is better for MSE and IDM error; higher is better for success and SSIM.
Accessible table and original citation links
Policy Real demonstrations Counterfactual transitions
Model Succ. (%) FDM MSE ↓\downarrow FDM SSIM ↑\uparrow IDM pos. (cm) ↓\downarrow IDM rot. (∘) ↓\downarrow FDM MSE ↓\downarrow FDM SSIM ↑\uparrow IDM pos. (cm) ↓\downarrow IDM rot. (∘) ↓\downarrow
OpenWAM, LC-IDM, CF-only – – – 1.76 2.39 – – 1.17 3.60
OpenWAM, LC-IDM, mixed – – – 0.44 1.21 – – 1.22 3.68
OpenWAM, LC-FDM, CF-only – 0.00734 0.9126 – – 0.00940 0.8889 – –
OpenWAM, LC-FDM, mixed – 0.00186 0.9776 – – 0.00962 0.8859 – –
OpenWAM, multi-objective 92.8 0.00058 0.9934 0.43 1.10 0.01416 0.8398 1.73 3.55
UVA 88.0 0.00279 0.9645 2.03 1.60 0.01514 0.8134 3.57 12.99

D.2 Effect of the Mixture-of-Transformers Architecture

With the robot-video foundation fixed, the shared non-MoT transformer scores 92.8%VTA and 93.6%Joint; MoT scores 97.8/96.6%.

The MoT system adds a separate roughly 2B action expert to the video expert. This is not a total-parameter-matched test of attention organization alone.

Appendix E External Dynamics Comparison

The external comparison uses UVA, a released policy/inverse/forward model. RGB errors are measured in common agent-view image space; inverse predictions are compared through executed end-effector trajectories because action parameterizations differ [44].

These system comparisons do not match every backbone, pretraining corpus and compute resource.

E.1 Independent Dynamics Components

Counterfactual-only specialists have the best specialist prediction on held-out alternative transitions, while mixed training substantially improves fit on demonstrations.

The full table retains position/orientation errors separately. No single variant wins every metric; for example, the unified model’s counterfactual rotation error is slightly lower even though specialist position and video prediction are stronger.

E.2 Unified Policy and Dynamics Modeling

The unified checkpoint allocates 60%updates to joint policy data and 20%each to local inverse/forward objectives. Each dynamics portion is split equally between demonstration and counterfactual transitions.

It scores 92.8%policy success versus UVA 88.0%, but remains below policy-specialized variants and weaker than specialists on most counterfactual dynamics measures. Lowest demonstration-distribution errors can include training examples seen by the policy objective, so they are fit rather than held-out generalization evidence.

Concept demonstration

Open image ↗