Literature
Display
← All papers

LeWAM: End-to-End World and Action Modeling with JEPAs

Minghao Fu · Tavis Siebert · Eryk Halicki · Randall Balestriero

Preprint posted · arXiv (preprint) ↗

World models11 min text · CondensedOriginal paper ↗
Contents

Overview

Website Code
Refer to caption
 
Original figure
Figure 1: LeWAM Training. A shared encoder encodes observed frames with SIGReg (Balestriero and LeCun, 2025) regularization. The Mixture-of-Transformers (Liang et al., 2025) learns latent-state prediction from clean actions and action flow matching from noisy actions with goal information. Results Overview. The mean of (a) planning success on contact-rich tasks; (b) planning success on a 100-step goal offset; (c) plans/s versus CEM (Rubinstein, 1999); (d) action chunks per TFLOP versus π0.5\pi_{0.5} (Physical Intelligence et al., 2025). Full results are in Section 4.
Original figure
Figure 2: Positioning of LeWAM. The columns place LeWAM alongside related approaches to world-model training, visual representation learning, world action modeling, and action search. LeWAM jointly trains latent dynamics and a policy through a shared encoder, predicts in latent space without reconstruction, and refines learned action proposals through its dynamics. The final column summarizes its evaluation on contact-rich tasks and longer goal horizons.
Contents 
A Implementation Details p. A
LeWAM Planning p. A.1
Baseline Implementations p. A.2
B Benchmarks and Evaluation p. B
Tasks and Datasets p. B.1
Goals and Success Criteria p. B.2
Planning and Timing Protocols p. B.3
C Additional Results p. C
Visual Planning Results p. C.1
Component and Visual-Encoder Ablations p. C.2
Probing Latent Representations p. C.3
Visual Reconstructions and Open-Loop Rollouts p. C.4
Multimodal Action Distributions p. C.5
Simulation Task-Completion Results p. C.6
D Physical-Robot Setup and Evaluation p. D
Platform and Data Collection p. D.1
Model Implementation p. D.2
Physical-Robot Baseline Configurations p. D.3
Closed-Loop Evaluation p. D.4
Representation Diagnostics and Evaluation p. D.5
E Discussion of the Role of Joint Prediction p. E
Representation and Dynamics p. E.1
Planning with Learned Action Proposals p. E.2
F Author Contribution Statement p. F

At a glance

LeWAM jointly trains a visual encoder, action-conditioned latent dynamics, and an action flow-matching policy from reward-free offline trajectories. Predicting both future states and actions encourages control-relevant representations; SIGReg keeps the latent distribution diverse.

The same architecture supports direct action sampling or goal-image planning that refines learned action proposals through the dynamics model. Across ten simulated planning tasks it averages 93.5% success versus LeWM's 50.9%; a separate four-task timing comparison reports 32.3× higher mean planning throughput. Physical-robot experiments use direct policy execution, without online planning.

1 Introduction

Latent world models avoid expensive pixel prediction, but their representations can retain distracting visual information and their planners can waste samples on ineffective actions (Sobal et al., 2022). Adding a policy only after freezing the representation disconnects action supervision from perception.

LeWAM learns perception, consequences, and behavior together. Its roughly 17M-parameter simulated model uses a joint state/action objective plus one latent-regularization weight. A learned action distribution supplies useful proposals, while gradients through predicted consequences refine them toward a visual goal.

2 Related Work

Unlike pixel-generating world models, JEPAs predict representations (LeCun, 2022). DINO-WM uses pretrained visual features; LeWM learns them with SIGReg (Zhou et al., 2024) (Maes et al., 2026b). LeWAM adds joint policy learning so action supervision also shapes the encoder.

GC-IDM and PRISM learn action predictors on frozen world-model features (Nguyen et al., 2026) (Wang et al., 2026). LeWAM instead trains clean-action dynamics and noisy-action flow matching on the same sample and forward pass. Unlike video/action generative approaches, its training needs neither reward labels nor pixel reconstruction.

3 Methodology

Training data are offline image-and-action trajectories. The model predicts future image representations and a distribution of action chunks, supporting goal-conditioned planning or goal-free behavior cloning.

3.1 End-to-End Training with LeWAM

A shared ResNet-18 encoder with spatial softmax produces visual latents. A MoT predictor has separate state and action projections but shared masked attention. Its state query reads history and clean recorded actions; noisy action tokens read history and their own causal prefix. History cannot read clean actions, preventing indirect target leakage across layers.

The policy predicts a contiguous C-action chunk while dynamics targets are K environment steps apart. Optional goal context includes a goal-image latent and remaining horizon, visible only to noisy action tokens. Flow matching regresses the velocity from Gaussian noise to recorded actions; dynamics regresses the encoded future state.

The total loss adds the two coordinate-averaged MSE terms and SIGReg with weight λ. SIGReg uses one-dimensional normality tests on random latent projections to encourage a standard isotropic Gaussian distribution (Balestriero and LeCun, 2025).

Original figure
Figure 3: MoT Attention. Filled cells permit attention, while white cells are masked. The state query reads history and clean actions, and noisy action tokens read state history and their own causal prefix. Small grids show causal attention. Arrows mark the state branch and action velocity branch.
Equations 1–5
𝒛^t+K\displaystyle\hat{{\bm{z}}}_{t+K} =fz​(𝒒t+Kz,𝒛≤t,𝑨t,K),\displaystyle=f_{z}({\bm{q}}^{z}_{t+K};{\bm{z}}_{\leq t},{\bm{A}}_{t,K}), (1)
𝒗^t,C\displaystyle\hat{{\bm{v}}}_{t,C} =fa​(𝑨t,C(s),s,𝒛≤t,𝒄t)\displaystyle=f_{a}({\bm{A}}_{t,C}^{(s)};s,{\bm{z}}_{\leq t},{\bm{c}}_{t})
𝑨t,C(s)=(1−s)​ϵ+s​𝑨t,C,𝒗t,C=d​𝑨t,C(s)d​s=𝑨t,C−ϵ{\bm{A}}^{(s)}_{t,C}=(1-s){\bm{\epsilon}}+s{\bm{A}}_{t,C},\qquad{\bm{v}}_{t,C}=\frac{\mathrm{d}{\bm{A}}^{(s)}_{t,C}}{\mathrm{d}s}={\bm{A}}_{t,C}-{\bm{\epsilon}} (2)
ℒdyn\displaystyle{\mathcal{L}}_{\mathrm{dyn}} =𝔼𝒟​[MSE⁡(𝒛^t+K,𝒛t+K)],\displaystyle=\mathbb{E}_{\mathcal{D}}\!\left[\operatorname{MSE}\!\left(\hat{{\bm{z}}}_{t+K},{\bm{z}}_{t+K}\right)\right], (3)
ℒact\displaystyle{\mathcal{L}}_{\mathrm{act}} =𝔼𝒟,s,ϵ​[MSE⁡(𝒗^t,C,𝑨t,C−ϵ)].\displaystyle=\mathbb{E}_{\mathcal{D},s,{\bm{\epsilon}}}\!\left[\operatorname{MSE}\!\left(\hat{{\bm{v}}}_{t,C},{\bm{A}}_{t,C}-{\bm{\epsilon}}\right)\right].
ℒsig=1M​∑m=1MT⁡(𝒁​𝒖(m)).{\mathcal{L}}_{\mathrm{sig}}=\frac{1}{M}\sum_{m=1}^{M}T\!\left({\bm{Z}}{\bm{u}}^{(m)}\right). (4)
ℒLeWAM=ℒdyn+ℒact+λsig​ℒsig.{\mathcal{L}}_{\rm LeWAM}={\mathcal{L}}_{\mathrm{dyn}}+{\mathcal{L}}_{\mathrm{act}}+\lambda_{\mathrm{sig}}{\mathcal{L}}_{\mathrm{sig}}. (5)
Refer to caption
Figure 4: Planning with LeWAM. Each MoT predictor stage samples NN action chunks from Gaussian noise and uses the first KK actions in each chunk, denoted 𝑨t,K{\bm{A}}_{t,K}, to advance the latent dynamics from step tt to t+Kt+K. Five-point curves depict action chunks (we use K=5K=5), and stacked curves denote NN sampled plans of length LL. Step 1 refines all plans by gradient descent through the frozen LeWAM dynamics using the goal-state cost. Step 2 selects the lowest-cost refined iterate as the final plan, shown as 𝑨⋆{\bm{A}}^{\star}. Appendix A.1 gives the full procedure in a pseudo-algorithm.

3.2 Efficient Control with LeWAM

Direct control integrates the action flow from Gaussian noise, executes the first K actions, and samples again after observing the result.

Planning first generates N candidate sequences by alternating action sampling and predicted-state updates. It then differentiates terminal latent distance to the goal through the learned dynamics to refine each sequence. A small proximity penalty discourages large departures from the sampled plans. The lowest unpenalized goal-cost iterate is selected; only its first K actions execute before replanning.

Equations 6–8
d​𝑨t,C(s)d​s=fa​(𝑨t,C(s),s,𝒛≤t,𝒄t),𝑨t,C(0)∼𝒩⁡(𝟎,𝑰),𝑨^t,C=𝑨t,C(1)\frac{\mathrm{d}{\bm{A}}_{t,C}^{(s)}}{\mathrm{d}s}=f_{a}({\bm{A}}_{t,C}^{(s)};s,{\bm{z}}_{\leq t},{\bm{c}}_{t}),\qquad{\bm{A}}_{t,C}^{(0)}\sim\mathcal{N}({\bm{0}},{\bm{I}}),\qquad\hat{{\bm{A}}}_{t,C}={\bm{A}}_{t,C}^{(1)} (6)
𝒞⁡(𝑨)=MSE⁡(𝒛^t+L​(𝑨),𝒛g){\mathcal{C}}({\bm{A}})=\operatorname{MSE}\!\left(\hat{{\bm{z}}}_{t+L}({\bm{A}}),{\bm{z}}_{g}\right) (7)
𝒂t+k←𝒂t+k−η∂𝒞∂𝒂t+k,k=0,…,L−1,{\bm{a}}_{t+k}\leftarrow{\bm{a}}_{t+k}-\eta\frac{\partial{\mathcal{C}}}{\partial{\bm{a}}_{t+k}},\qquad k=0,\ldots,L-1, (8)

4 Experiments

Experiments separate visual-goal planning, diagnostic analysis of learned representations and dynamics, and goal-free behavior cloning in simulation and on physical robots.

4.1 Evaluation Protocol

Ten simulated tasks span navigation and contact-rich manipulation using expert, random, and mixed-quality trajectories. Visual planning receives a goal image and a fixed action budget. Task completion instead learns one demonstrated behavior without a goal image. Within each protocol, methods share observations, action limits, goals, budgets, and success criteria.

Refer to caption
Figure 5: Simulation Environments. Symbols beneath each frame denote overlapping task attributes, with their meanings shown in the legend to the left.

4.2 Visual Planning with LeWAM

Against LeWM with CEM, LeWAM improves mean success from 50.9% to 93.5% over ten tasks and wins every task. Contact-rich task averages rise from 28.6% to 89.7%, including ToolHang 35.4→94.7% and Transport 2.8→83.3%.

On four tasks with goals 100 steps after the start, the same checkpoints achieve 46.2% mean success versus 10.9%. These longer goals are distinct from the default short-horizon benchmark.

On an H100, mean per-task planning throughput is 52.50 versus 1.62 plans/s, a reported 32.3× ratio. This compares five gradient updates with LeWM's 300-candidate, 30-iteration CEM; it is planner-call timing, not an end-to-end robot-control speedup.

Original figure
Figure 6: Visual Planning across Ten Simulated Environments. Most panels report the success rate under a goal offset of L=25L=25 environment steps and permit 2​L=502L=50 actions across 3 seeds. For Two-Room, we use 100/150 action steps following LeWM setting; For LeWM (CEM), we report 10 seeds for stability. Details are present in Appendix B.2. Error bars denote one standard deviation; indicates models with end-to-end training. Linear axes start at 50% for the first five panels.
Original figure
Figure 7: Visual Planning Success across Different Horizons. The task panels show mean success rates across 3 seeds. Each sweep starts at the first trajectory frame but with different goal offsets.

4.3 Understanding LeWAM’s Dynamics, Representations, and Actions

Linear probes find more decodable proprioception and actions in jointly trained representations. Without SIGReg, latent-only LeWM collapses on the reported Push-T probes, while action-supervised variants retain information. Action supervision alone is insufficient for the strongest planning: SIGReg also improves object information and goal sensitivity.

Across four tasks, direct success averages 86.7% with a jointly trained encoder, versus 79.0% with frozen DINOv2 CLS features and 75.8% with frozen patch features. A stochastic flow head gives larger planning gains than a deterministic MSE head with added Gaussian noise. Distribution modeling can distinguish histories whose action means match but whose valid behaviors differ.

Table 1: Linear probing and the role of SIGReg. All linear probes report R2R^{2} score using ridge regression. Object denotes the average R2R^{2} over all object ground truth information and Proprio the ground truth proprioceptive state. Action predicts the action chunk from latent 𝒛t{\bm{z}}_{t}; parenthesized IDM scores use both 𝒛t,𝒛t+K{\bm{z}}_{t},{\bm{z}}_{t+K} where reported. Eff. rank denotes effective rank (Roy and Vetterli, 2007).
Environment Model SIGReg Eff. rank Object ↑\uparrow Proprio ↑\uparrow Action (IDM) ↑\uparrow
Push-T LeWAM Yes 42.9042.90 0.900.90 0.86\mathbf{0.86} 0.40​(0.75)\mathbf{0.40}(\mathbf{0.75})
No 10.9410.94 0.880.88 0.850.85 0.38​(0.73)0.38(0.73)
LeWAM (MSE) No 6.966.96 0.660.66 0.86\mathbf{0.86} 0.36​(0.74)0.36\,(0.74)
LeWM Yes 94.2094.20 0.95\mathbf{0.95} 0.800.80 0.29​(0.71)0.29(0.71)
No 16.2316.23 0.000.00 0.000.00 0.00​(0.00)0.00\,(0.00)
Original figure
Figure 8: (a) Planning gain (percentage points) from SIGReg. (b) Direct success and planning gain for visual encoders, averaged over four tasks. (c) Planning gain for flow (circles) and MSE (squares) action heads. (d) Task-completion success for LeWAM and Diffusion Policy.

Isolating Action-Conditioned Dynamics.

Removing the action head at evaluation and planning with CEM isolates learned dynamics. LeWAM remains competitive on Push-T and improves Cube success by 22.2 points over LeWM. Decoded recorded-action rollouts also show clearer control-relevant details on the three-dimensional tasks; these diagnostics use separate post-trained pixel decoders.

4.4 Behavior Cloning and Physical Robot Control

Goal-free LeWAM directly executes sampled action chunks. On DrawerCleanup, Transport, and ToolHang it is competitive with Diffusion Policy, improving Transport by 16 points (Chi et al., 2023).

Physical experiments compare LeWAM trained from scratch, a frozen V-JEPA 2.1 variant, Diffusion Policy, and π0.5 on three tasks. All train or adapt on the collected teleoperation demonstrations and execute half of each action chunk. LeWAM is competitive; its two encoder variants have similar average success. Each method receives 30 trials per task, and no physical experiment uses online planning.

Refer to caption
Figure 9: Paired scene and wrist view examples from three physical-robot tasks.
Table 2: Physical robot evaluation. Successful trials out of 30 on Pick and Place T, Push-T, and Tape Stack. GFLOPs count one inference for a full action chunk, including all denoising steps. One full LeWAM action chunk uses 270.2 GFLOPs versus 3603.6 GFLOPs for π0.5\pi_{0.5}. Figure 1(d) expresses the reciprocal as action chunks per TFLOP, a 13.3×13.3\times difference in operation-count efficiency.
Method Pick and Place T Push-T Tape Stack Average GFLOPs
Diffusion Policy (Chi et al., 2023) 19/3019/30 16/3016/30 0/30 38.9% 270.0270.0
π0.5\pi_{0.5} (Physical Intelligence et al., 2025) 14/3014/30 3/30 9/30 28.9% 3603.63603.6
LeWAM (Frozen V-JEPA 2.1 Encoder) 15/3015/30 12/3012/30 5/30 35.6% 270.2270.2
LeWAM 16/3016/30 8/308/30 6/30 33.3% 270.2270.2

5 Conclusion and Limitations

Joint state and action prediction supports useful latent dynamics, learned planning proposals, and direct control. Evidence remains limited to the tested data and tasks: models are trained or adapted separately for each task and embodiment.

Open directions include broader contact-rich play data, unfamiliar objects and viewpoints, multitask and language-conditioned training, hierarchical subgoals, and scaling models or test-time search. The reported results do not establish a universal cross-robot policy.

Reproducibility statement

Architecture, training, data, and evaluation settings are supplied in the appendices, with project and code links in the original paper.

Ethics statement

The authors report no new human-subject data. Reused resources retain their licenses, and physical deployment requires independent safety mechanisms.

AI use statement

Language-model assistants helped polish the manuscript. The authors retain responsibility for the research, analysis, and text.

Appendix A Implementation Details

Simulation uses two history latents, dynamics stride K=5, and ten-action chunks. History stride is five for goal reaching and one for task completion. The encoder maps spatial-softmax features to 192 dimensions; the predictor has four blocks and four heads, totaling 16.9M trainable parameters. Action sampling uses eight Euler steps.

Targets use the online encoder, without a momentum encoder or state noise. SIGReg uses 1,024 projections, 17 knots, and weight 0.04. Visual planning trains 50 epochs at batch 128; goal-free cloning trains 120 epochs at batch 64. Checkpoints minimize validation action flow loss, not task success.

Separate branch parameters share one masked attention operation. Goal and flow-time conditioning affect noisy action tokens. The explicit mask below blocks clean-action leakage into history and the action flow, while letting the future-state query read recorded actions.

Equations 9–10
[𝒛t−Δ,𝒛t,𝒒t+Kz,𝑨t,K,𝑨t,C(s)].\left[{\bm{z}}_{t-\Delta},{\bm{z}}_{t},{\bm{q}}^{z}_{t+K},{\bm{A}}_{t,K},{\bm{A}}_{t,C}^{(s)}\right]. (9)
𝒱⁡(𝒛i)\displaystyle\mathcal{V}({\bm{z}}_{i}) ={𝒛t−Δ,𝒛t},\displaystyle=\{{\bm{z}}_{t-\Delta},{\bm{z}}_{t}\}, i∈{t−Δ,t},\displaystyle i\in\{t-\Delta,t\}, (10)
𝒱⁡(𝒒t+Kz)\displaystyle\mathcal{V}({\bm{q}}^{z}_{t+K}) ={𝒛t−Δ,𝒛t,𝒒t+Kz,𝑨t,K},\displaystyle=\{{\bm{z}}_{t-\Delta},{\bm{z}}_{t},{\bm{q}}^{z}_{t+K},{\bm{A}}_{t,K}\},
𝒱⁡(𝒂j)\displaystyle\mathcal{V}({\bm{a}}_{j}) ={𝒛t−Δ,𝒛t,𝒂≤j},\displaystyle=\{{\bm{z}}_{t-\Delta},{\bm{z}}_{t},{\bm{a}}_{\leq j}\}, j∈{t,…,t+K−1},\displaystyle j\in\{t,\ldots,t+K-1\},
𝒱⁡(𝒂j(s))\displaystyle\mathcal{V}({\bm{a}}_{j}^{(s)}) ={𝒛t−Δ,𝒛t,𝒂≤j(s)},\displaystyle=\{{\bm{z}}_{t-\Delta},{\bm{z}}_{t},{\bm{a}}_{\leq j}^{(s)}\}, j∈{t,…,t+2​K−1}.\displaystyle j\in\{t,\ldots,t+2K-1\}.
Original figure
Figure 10: Token-Level MoT Attention. Expansion of Figure 3 for two history latents, one learned state query, five clean actions, and ten interpolated action tokens. Rows are queries; columns are keys. Filled cells permit attention; white cells are masked. Blue and brown mark latent state and action velocity prediction queries; gray marks context. Wider gaps separate token groups. Optional goal conditioning 𝒄t{\bm{c}}_{t} is omitted. Δ\Delta is the history stride, and action indices remain at the raw environment rate.

A.1 LeWAM Planning

Only normalized candidate actions receive gradients during planning; model weights are frozen. Adam uses learning rate 0.05 and gradient clipping at 10. Updates include a 0.01 quadratic proximity penalty and clip normalized actions to ±3, but candidate retention and final selection use the unpenalized goal cost.

Equation 11
𝒥prox​(𝑨,𝑨sample)=𝒞⁡(𝑨)+λprox​‖𝑨−𝑨sample‖F2.{\mathcal{J}}_{\mathrm{prox}}({\bm{A}};{\bm{A}}^{\mathrm{sample}})={\mathcal{C}}({\bm{A}})+\lambda_{\mathrm{prox}}\left\|{\bm{A}}-{\bm{A}}^{\mathrm{sample}}\right\|_{F}^{2}. (11)
Algorithm 1 Candidate refinement and selection for LeWAM planning
Require: 𝒛≤t,𝒄t,𝒛g,L,N,J,η,λprox,amax{\bm{z}}_{\leq t},{\bm{c}}_{t},{\bm{z}}_{g},L,N,J,\eta,\lambda_{\mathrm{prox}},a_{\max}
Sample candidates from the learned policy
1: 𝑨1:Nsample←N{\bm{A}}_{1:N}^{\mathrm{sample}}\leftarrow N policy rollouts from (𝒛≤t,𝒄t)({\bm{z}}_{\leq t},{\bm{c}}_{t}) for L/KL/K blocks
2: c1:N←𝒞(𝑨1:Nsample)c_{1:N}\leftarrow{\mathcal{C}}({\bm{A}}_{1:N}^{\mathrm{sample}})
3: 𝑨⋆1:N←𝑨1:Nsample{\bm{A}}^{\star}_{1:N}\leftarrow{\bm{A}}_{1:N}^{\mathrm{sample}},  𝑨1:N←𝑨1:Nsample{\bm{A}}_{1:N}\leftarrow{\bm{A}}_{1:N}^{\mathrm{sample}}
4: c⋆1:N←c1:Nc^{\star}_{1:N}\leftarrow c_{1:N}
Refine each candidate
5: for j=0,…,J−1j=0,\ldots,J-1 do
6:   𝑨1:N←AdamStep(𝑨1:N,∇𝑨1:N𝒥prox(𝑨1:N;𝑨1:Nsample),η){\bm{A}}_{1:N}\leftarrow\operatorname{AdamStep}\left({\bm{A}}_{1:N},\nabla_{{\bm{A}}_{1:N}}{\mathcal{J}}_{\mathrm{prox}}({\bm{A}}_{1:N};{\bm{A}}_{1:N}^{\mathrm{sample}}),\eta\right)
7:   𝑨1:N←clip(𝑨1:N,−amax,amax){\bm{A}}_{1:N}\leftarrow\operatorname{clip}({\bm{A}}_{1:N},-a_{\max},a_{\max})
8:   c1:N←𝒞(𝑨1:N)c_{1:N}\leftarrow{\mathcal{C}}({\bm{A}}_{1:N})
9:   𝑨n⋆←𝑨n,cn⋆←cn∀n​ such that ​cn<cn⋆{\bm{A}}^{\star}_{n}\leftarrow{\bm{A}}_{n},\quad c_{n}^{\star}\leftarrow c_{n}\quad\forall n\text{ such that }c_{n}<c_{n}^{\star}
10: end for
Select the lowest-cost iterate
11: n⋆←arg​minn=1,…,N⁡cn⋆n^{\star}\leftarrow\argmin_{n=1,\ldots,N}c_{n}^{\star}
12: return 𝑨n⋆⋆{\bm{A}}^{\star}_{n^{\star}}

A.2 Baseline Implementations

Baselines share environment wrappers and evaluation interfaces from stable-worldmodel (Maes et al., 2026a). LeWM-IDM freezes a separately trained world model; LeWM-CEM follows its released recipe. DINO-WM compares patch and CLS tokenization, using ten epochs on the six added tasks. GCBC uses frozen DINOv2 features and 20 training epochs.

The simulated Diffusion Policy baseline uses a ResNet-18 encoder per camera and a convolutional U-Net. It predicts 16 actions, executes eight, and samples with 100 DDPM steps. Training uses 120 epochs, a 90/10 window split, and validation denoising loss for selection. Each checkpoint is evaluated on 50 episodes for each of three seeds.

Appendix B Benchmarks and Evaluation

B.1 Tasks and Datasets

The suite retains Two-Room, Reacher, Push-T, and Cube, then adds PointMaze-Large, Scene, Puzzle, ToolHang, Transport, and DrawerCleanup. Data range from random or noisy navigation and interaction to scripted or human demonstrations. Scene and Puzzle use newly collected scripted-oracle trajectories; the complex manipulation tasks use robomimic or DexMimicGen demonstrations.

Task-focused demonstrations broaden contact complexity but constrain the diversity of visual goals. Table 3 retains action dimensions, views, and dataset scale; synchronized cameras count once toward duration.

Table 3: Simulation Benchmark Datasets. Ten tasks in Figure 5 order. Views gives camera-stream count; Dim. is the raw action dimension. Statistics cover complete source datasets before splits. Total hours are computed before rounding per-task values.
Task Collection regime Views Dim. Traj. Mean frames Total frames Hz Hours
Two-Room Noisy heuristic 1 2 10,000 92 920,809 10 25.58
Reacher Uniform random 1 2 10,000 201 2,010,000 25 22.33
Push-T Non-expert 1 2 18,685 125 2,336,736 10 64.91
Cube Scripted expert 1 5 10,000 201 2,010,000 20 27.92
PointMaze-Large Scripted navigation 1 2 1,000 1,000 1,000,000 10 27.78
Scene Scripted expert 1 5 10,024 200 2,004,800 20 27.84
Puzzle (3×\times3) Scripted expert 1 5 10,024 200 2,004,800 20 27.84
ToolHang Human demos 1 7 200 480 95,962 20 1.33
Transport Generated demos 3 14 1,029 408 419,712 20 5.83
DrawerCleanup Generated demos 3 24 1,026 291 298,235 20 4.14
Reported total (10) 71,988 13,101,054 235.51

B.2 Goals and Success Criteria

Default goal images come 25 recorded steps after the initial state, with 50 executable actions. Two-Room instead uses a 100-step offset and 150-action budget. Results generally average three evaluation seeds and 50 episodes per seed; first-hit success matches the baseline protocol.

Cube now requires both cube and end-effector positions within 0.04 m of their goal positions. The old object-only criterion rewarded doing nothing on 42.7% of short-horizon episodes; the corrected criterion reduces that floor to 2.7%. Blind zero-action and random-policy controls are retained in Table 4.

Refer to caption
Figure 11: A Degenerate Cube Success under Goal Reaching. Each row shows a policy rollout (left, frames over time) beside its goal frame (far right, after the divider). Throughout the execution, the gripper never reaches the goal end-effector pose, yet the cube stays within the 0.040.04 m object-only threshold of the goal, so object-position-only predicate scores the episode a spurious success. Our end-effector term (Table 4) rejects these degenerate successes by additionally requiring the gripper to reach the goal pose.
Environment Raw action dim. 25-step goal reaching Task Completion
Zero-action Random Zero-action Random
Two-Room 2 7.3±0.97.3\pm 0.9 25.3±5.025.3\pm 5.0 0.0±0.00.0\pm 0.0 2.7±2.52.7\pm 2.5
Reacher 2 3.3±1.93.3\pm 1.9 12.0±1.612.0\pm 1.6 0.0±0.00.0\pm 0.0 12.0±4.912.0\pm 4.9
Push-T 2 0.0±0.00.0\pm 0.0 1.3±0.91.3\pm 0.9 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0
Cube 5 42.7±6.642.7\pm 6.6 44.0±1.644.0\pm 1.6 4.7±2.54.7\pm 2.5 6.0±1.66.0\pm 1.6
Cube (++EEF) 5 2.7±0.92.7\pm 0.9 4.7±0.94.7\pm 0.9 4.7±2.54.7\pm 2.5 6.0±1.66.0\pm 1.6
PointMaze-Large 2 6.0±0.06.0\pm 0.0 9.3±1.99.3\pm 1.9 0.7±0.90.7\pm 0.9 6.0±3.36.0\pm 3.3
Scene 5 2.7±2.32.7\pm 2.3 2.7±2.32.7\pm 2.3 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0
Puzzle 5 0.7±1.20.7\pm 1.2 0.7±1.20.7\pm 1.2 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0
ToolHang 7 22.7±2.522.7\pm 2.5 26.0±1.626.0\pm 1.6 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0
Transport 14 3.3±0.93.3\pm 0.9 2.7±0.92.7\pm 0.9 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0
DrawerCleanup 24 0.0±0.00.0\pm 0.0 6.0±0.06.0\pm 0.0 0.0±0.00.0\pm 0.0 0.0±0.00.0\pm 0.0

Darker shading indicates a higher control success rate.

Table 4: Action Dimensions and Policy-Floor Controls. Rows cover all ten environments in Figure 5, plus a Cube (++EEF) row isolating the corrected end-effector predicate. Raw action dimensionality is reported beside zero-action and random-policy success rates (%) for short-horizon goal reaching and full-trajectory task completion, averaged over three seeds. These blind controls measure baseline success from zero actions and random actions. Cube has the same task-completion values in both rows because the end-effector correction applies only to short-horizon goal reaching. DrawerCleanup exposes 24 raw coordinates; the policy predicts 22 active dimensions after removing two constant joints.

B.3 Planning and Timing Protocols

Long-horizon sweeps use the first held-out trajectory frame and goals 25, 50, 75, or 100 steps later. Push-T sensitivity varies gradient updates at 32 candidates, or candidate count at five updates.

Timing measures one planner call on the same H100. Throughput is averaged after converting each task's milliseconds per plan to plans per second. The headline ratio compares LeWM's 300×30 CEM with five LeWAM gradient updates.

Appendix C Additional Results

C.1 Visual Planning Results

Tables 5–6 retain horizon-dependent success and per-call timing. In the tested Push-T sweep, success plateaus after five refinement updates while latency keeps rising; more than eight candidates gives no consistent gain.

Table 5: Goal-Reaching Success across Horizons. Success rate (%), mean ±\pm standard deviation. Measured values for Figure 7. The Cube horizon sweep is distinct from the sampled-start 25-step evaluation in Figure 6.
Task Method Goal horizon (steps)
25 50 75 100
Push-T LeWAM 94.7±3.194.7\pm 3.1 60.7±3.160.7\pm 3.1 44.0±3.544.0\pm 3.5 43.3±4.643.3\pm 4.6
LeWM (CEM) 68.7±3.168.7\pm 3.1 12.0±4.012.0\pm 4.0 3.3±2.33.3\pm 2.3 2.7±1.22.7\pm 1.2
Cube LeWAM 100.0±0.0100.0\pm 0.0 88.0±2.088.0\pm 2.0 80.7±2.380.7\pm 2.3 82.0±2.082.0\pm 2.0
LeWM (CEM) 100.0±0.0100.0\pm 0.0 0.0±0.00.0\pm 0.0 4.0±3.54.0\pm 3.5 4.0±3.54.0\pm 3.5
Scene LeWAM 82.7±3.182.7\pm 3.1 73.3±8.173.3\pm 8.1 65.3±3.165.3\pm 3.1 56.0±4.056.0\pm 4.0
LeWM (CEM) 44.7±13.044.7\pm 13.0 31.3±2.331.3\pm 2.3 37.3±5.837.3\pm 5.8 36.0±6.036.0\pm 6.0
Puzzle LeWAM 74.0±12.574.0\pm 12.5 14.0±3.514.0\pm 3.5 4.0±3.54.0\pm 3.5 3.3±1.23.3\pm 1.2
LeWM (CEM) 10.0±0.010.0\pm 0.0 4.7±2.34.7\pm 2.3 4.0±2.04.0\pm 2.0 0.7±1.20.7\pm 1.2
Table 6: Planning Speed across Four Visual-Goal Tasks. Each plan is one planner call. LeWM (CEM) (Maes et al., 2026b) uses 300×30300\times 30 CEM; LeWAM uses five gradient steps. Action dimensions are raw environment action dimensions. Throughput is the reciprocal of time per plan. The mean row averages per-task throughput, and its ratio compares those two means.
LeWM (CEM) LeWAM
Task Action dim. ms/plan plans/s ms/plan plans/s Speedup
Scene 55 705705 1.421.42 18.818.8 53.1953.19 37.5×37.5\times
Puzzle 55 731731 1.371.37 16.316.3 61.3561.35 44.8×44.8\times
Cube 55 771771 1.301.30 21.121.1 47.3947.39 36.5×36.5\times
Push-T 22 414414 2.422.42 20.820.8 48.0848.08 19.9×19.9\times
Task mean – – 1.621.62 – 52.5052.50 32.3×32.3\times
Original figure
Figure 12: Planner Sensitivity on Push-T. Goal-reaching success (mean and standard deviation across three seeds, 50 episodes each) and per-plan latency versus (left) Adam refinement updates with 32 candidate plans and (right) sampled candidate plans with five refinement updates.

C.2 Component and Visual-Encoder Ablations

Table 7 compares SIGReg and action heads using epoch-30 checkpoints. Table 8 separately compares encoders. Direct execution has no search; planning adds action refinement through latent dynamics. Encoder results average Push-T, Cube, Scene, and Puzzle equally.

Table 7: SIGReg and Action-Head Ablations. Direct and planning success rates (%) using epoch-30 checkpoints for all variants. Gain is planning minus direct success in percentage points. Standard deviations are reported where available; none is inferred for gains.
Environment Variant Direct Planning Gain (pp)
Push-T LeWAM 82.7±8.182.7\pm 8.1 97.3±3.197.3\pm 3.1 +14.7+14.7
w/o SIGReg 69.3±10.169.3\pm 10.1 72.0±3.572.0\pm 3.5 +2.7+2.7
MSE action head 90.0±4.090.0\pm 4.0 95.3±1.295.3\pm 1.2 +5.3+5.3
Scene LeWAM 92.0±5.392.0\pm 5.3 97.3±1.297.3\pm 1.2 +5.3+5.3
w/o SIGReg 91.3±2.391.3\pm 2.3 93.3±3.193.3\pm 3.1 +2.0+2.0
MSE action head 89.3±4.289.3\pm 4.2 90.0±2.090.0\pm 2.0 +0.7+0.7
Puzzle LeWAM 76.0±7.276.0\pm 7.2 80.7±11.080.7\pm 11.0 +4.7+4.7
w/o SIGReg 67.3±9.567.3\pm 9.5 75.3±9.075.3\pm 9.0 +8.0+8.0
MSE action head 76.7±6.176.7\pm 6.1 74.7±6.174.7\pm 6.1 −2.0-2.0
Cube LeWAM 99.3±1.299.3\pm 1.2 100.0100.0 +0.7+0.7
w/o SIGReg 99.3±1.299.3\pm 1.2 99.3±1.299.3\pm 1.2 0.00.0
MSE action head 98.0±2.098.0\pm 2.0 99.3±1.299.3\pm 1.2 +1.3+1.3
Table 8: Visual Encoders for Direct Control and Planning. Success rate (%) with the same goal-conditioned evaluation protocol for all encoder variants. Direct executes the policy without search; planning uses test-time action search. Gain is planning minus direct success in percentage points. Averages are equal-weight means over the four tasks; gains use unrounded means.
Visual encoder Mode Push-T Cube Scene Puzzle Average
LeWAM (end-to-end) Direct 85.3±5.085.3\pm 5.0 100.0±0.0100.0\pm 0.0 88.0±6.988.0\pm 6.9 73.3±5.873.3\pm 5.8 86.786.7
Planning 98.7±1.298.7\pm 1.2 100.0±0.0100.0\pm 0.0 93.3±3.193.3\pm 3.1 74.0±12.574.0\pm 12.5 91.591.5
Gain (pp) +13.4+13.4 0.00.0 +5.3+5.3 +0.7+0.7 +4.9+4.9
frozen DINOv2 CLS Direct 72.7±7.672.7\pm 7.6 98.7±1.298.7\pm 1.2 82.0±4.082.0\pm 4.0 62.7±7.062.7\pm 7.0 79.079.0
Planning 78.7±6.478.7\pm 6.4 99.3±1.299.3\pm 1.2 91.3±1.291.3\pm 1.2 61.3±13.061.3\pm 13.0 82.782.7
Gain (pp) +6.0+6.0 +0.6+0.6 +9.3+9.3 −1.4-1.4 +3.6+3.6
frozen DINOv2 patch mean Direct 66.0±12.566.0\pm 12.5 93.3±2.393.3\pm 2.3 84.0±2.084.0\pm 2.0 60.0±7.260.0\pm 7.2 75.875.8
Planning 67.3±8.167.3\pm 8.1 98.7±1.298.7\pm 1.2 86.0±4.086.0\pm 4.0 58.7±7.658.7\pm 7.6 77.777.7
Gain (pp) +1.3+1.3 +5.4+5.4 +2.0+2.0 −1.3-1.3 +1.9+1.9

C.3 Probing Latent Representations

Ridge-regression probes use a 4,000/1,000-frame train/test split. Object and proprioception scores average R² over their ground-truth variables. Action probes use the current latent; inverse-dynamics probes also use the future latent. Effective rank measures the dimensional spread of representations (Roy and Vetterli, 2007).

Table 9: Effective Rank and Linear Probing. Results on Push-T, Cube, Scene, and Puzzle, with the Object, Proprio, and Action columns reporting linear-probe R2R^{2}. Rows compare models with and without SIGReg; parenthesized values use both 𝒛t{\bm{z}}_{t} and 𝒛t+K{\bm{z}}_{t+K} for inverse-dynamics probing.
Environment Model SIGReg Eff. rank Object ↑\uparrow Proprio ↑\uparrow Action (IDM) ↑\uparrow
Push-T LeWAM Yes 42.9042.90 0.900.90 0.86\mathbf{0.86} 0.40​(0.75)\mathbf{0.40}\,(\mathbf{0.75})
No 10.9410.94 0.880.88 0.850.85 0.38​(0.73)0.38\,(0.73)
LeWAM (MSE) No 6.966.96 0.660.66 0.86\mathbf{0.86} 0.36​(0.74)0.36\,(0.74)
LeWM Yes 94.2094.20 0.95\mathbf{0.95} 0.800.80 0.29​(0.71)0.29\,(0.71)
No 16.2316.23 0.000.00 0.000.00 0.00​(0.00)0.00\,(0.00)
Cube LeWAM Yes 46.6046.60 0.99\mathbf{0.99} 0.670.67 0.37​(0.66)\mathbf{0.37}\,(0.66)
No 8.008.00 0.99\mathbf{0.99} 0.70\mathbf{0.70} 0.37​(0.68)\mathbf{0.37}\,(\mathbf{0.68})
LeWM Yes 108.10108.10 0.99\mathbf{0.99} 0.580.58 0.17​(0.55)0.17\,(0.55)
Scene LeWAM Yes 48.9048.90 0.48\mathbf{0.48} 0.78\mathbf{0.78} 0.39​(0.74)0.39\,(\mathbf{0.74})
No 12.5012.50 0.400.40 0.770.77 0.43​(0.74)\mathbf{0.43}\,(\mathbf{0.74})
LeWM Yes 111.10111.10 0.400.40 0.370.37 0.07​(0.35)0.07\,(0.35)
Puzzle LeWAM Yes 46.5046.50 0.370.37 0.70\mathbf{0.70} 0.33​(0.69)\mathbf{0.33}\,(\mathbf{0.69})
No 5.805.80 0.220.22 0.660.66 0.31​(0.68)0.31\,(0.68)
LeWM Yes 97.4097.40 0.60\mathbf{0.60} 0.420.42 0.16​(0.48)0.16\,(0.48)

C.4 Visual Reconstructions and Open-Loop Rollouts

Post-trained pixel decoders visualize frozen representations and open-loop latent rollouts under recorded actions. Displayed clips are excluded from decoder training; each rollout receives only its initial observation context and the fixed recorded action sequence. These are diagnostics, not generated-policy rollouts.

Both models reproduce Push-T motion; LeWAM preserves clearer robot and object details in Cube, Scene, and Puzzle. Encoders, training schedules, and decoders differ between models. Table 10 removes LeWAM's action head and uses CEM, supporting the usefulness of its learned dynamics independently of its policy.

Table 10: CEM Planning with Learned Dynamics. Goal-reaching success rates (%), reported as mean ±\pm standard deviation. LeWAM is evaluated with its action prediction head removed.
Model Push-T Cube
LeWM (CEM) 92.4±1.2\mathbf{92.4\pm 1.2} 63.8±2.263.8\pm 2.2
LeWAM (CEM) 84.0±1.384.0\pm 1.3 86.0±3.2\mathbf{86.0\pm 3.2}
Refer to caption
Figure 13: Aligned Visual Sequences on Push-T and Cube. Each task shows recorded ground truth, framewise reconstructions for LeWM and LeWAM, and decoded open-loop predictions for both models. Columns use the same relative frame indices; predictions follow the recorded actions.
Refer to caption
Figure 14: Aligned Visual Sequences on Scene and Puzzle. Rows show recorded ground truth, LeWM and LeWAM framewise reconstructions, and their decoded open-loop predictions. Columns use the same relative frame indices and the same recorded action sequence for both prediction rows.

C.5 Multimodal Action Distributions

A two-door navigation dataset contains equal upper- and lower-route demonstrations with fixed endpoints. Flow matching preserves both route choices in closed-loop execution; the deterministic MSE head concentrates on the lower route and sometimes hits walls. This illustrates useful multimodal proposals rather than proving coverage of every valid behavior.

(a) Data Trajectories

Original figure

(b) flow head

Original figure

(c) MSE head

Original figure
Figure 15: Two-Door Demonstrations and Action-Head Rollouts. Expert demonstrations (a), executed trajectories from the flow-matching action head (b), and executed trajectories from the deterministic MSE action head (c). Black segments mark walls, and the green cross marks the goal. Blue and orange indicate upper- and lower-door trajectories; dashed red paths end in collision.

C.6 Simulation Task-Completion Results

Table 11 retains per-task behavior-cloning success and deviations for the three goal-free simulated manipulation tasks.

Table 11: Task completion. Mean success rates (%) ±\pm standard deviation across three evaluation seeds, with 50 episodes per seed. Bold marks the better result.
Method DrawerCleanup Transport ToolHang Average
Diffusion Policy (Chi et al., 2023) 54.7±1.2\mathbf{54.7\pm 1.2} 45.3±9.045.3\pm 9.0 84.7±3.1\mathbf{84.7\pm 3.1} 61.6
LeWAM 52.7±5.052.7\pm 5.0 61.3±9.9\mathbf{61.3\pm 9.9} 80.0±3.580.0\pm 3.5 64.7\mathbf{64.7}

Appendix D Physical-Robot Setup and Evaluation

The physical study evaluates direct closed-loop policies trained on task-specific teleoperation data.

D.1 Platform and Data Collection

One proficient operator records successful demonstrations with a reBot Arm 102 leader and B601-DM follower through LeRobot (Cadene et al., 2026). Training uses scene-left and wrist camera crops; the recorded right stereo view is unused. At 30 Hz, recordings include seven joint-target actions and a 21-dimensional position/torque/velocity state. No intentional exploration is collected.

Table 12: Datasets for the physical-robot tasks in Section 4.4.
Task Episodes Frames Duration Task Description
Pick and Place T 200 52,297 29 min ”Pick up the T block and place it onto the outline.”
Push-T 100 61,996 34 min ”Push the T block onto the outline.”
Tape Stack 100 23,726 13 min ”Stack the green roll of tape on the white roll of tape.”

D.2 Model Implementation

Both physical LeWAM variants use a ViT-B visual backbone, width 512, eight predictor blocks, and state-MSE plus action-flow losses. One encoder trains from scratch; the other freezes V-JEPA 2.1 features. They take two video context frames and predict three future frames sampled at 2.5 Hz.

Compute in Table 2 counts a complete action chunk: 100 DDPM steps for Diffusion Policy and ten Euler steps for π0.5 and both LeWAM variants. This is an operation count, not measured end-to-end latency.

D.3 Physical-Robot Baseline Configurations

Every physical policy returns 36 actions and executes 18 at 30 Hz, corresponding to 1.2-second predictions and 0.6-second execution intervals. Baselines use the same cameras, seven joint-target actions, and demonstrations.

The stated Pick and Place T/Push-T recipes hold out 10% of episodes and train 20,000 updates at effective batch 256. Diffusion Policy uses per-camera ImageNet-initialized ResNet-18 encoders, a U-Net, 100 sampling steps, and moving-average weights. π0.5 freezes its vision/language backbone and trains its action expert; discretized joint torques enter the language prompt.

D.4 Closed-Loop Evaluation

Observations travel to a remote inference machine; the robot executes returned action chunks at 30 Hz. A human labels success using the demonstration criteria. Each task/method has 30 trials, limiting the precision of small observed differences.

D.5 Representation Diagnostics and Evaluation

A separately trained pixel decoder maps frozen latents to images. Figure 16 compares the two encoder variants on one held-out Pick and Place T episode: recorded frames, framewise reconstructions, and autoregressive latent predictions. Rollouts follow recorded actions without intermediate ground-truth images or newly generated actions; ten displayed frames span 10.4 seconds.

Equation 12
ℒdec​(ψ)=𝔼𝒐t​[MSE⁡(dψ​(Enc⁡(𝒐t)),𝒐t)].\mathcal{L}_{\mathrm{dec}}(\psi)=\mathbb{E}_{{\bm{o}}_{t}}\!\left[\operatorname{MSE}\bigl(d_{\psi}(\operatorname{Enc}({\bm{o}}_{t})),{\bm{o}}_{t}\bigr)\right]. (12)

End-to-end LeWAM

Refer to caption

LeWAM with frozen V-JEPA 2.1

Refer to caption
Figure 16: Matched visual reconstructions and recorded-action latent rollouts for the scene camera. Within each model, rows show the ground-truth (GT) video, independent frame reconstructions, and decoded autoregressive predictions.

Appendix E Discussion of the Role of Joint Prediction

The appendix explains how joint prediction can shape representations and improve proposal quality. Table 13 compares training requirements and capabilities across related models.

Table 13: Modeling and Control Capabilities. FDM denotes forward dynamics conditioned on clean actions; Policy denotes learned action prediction. Entries describe reported training recipes or the goal-reaching configurations used here.
Method FDM Policy Joint FDM and Policy End-to-end from scratch
World models
DINO-WM (Zhou et al., 2024) ✓ ✗ ✗ ✗
V-JEPA 2-AC (Assran et al., 2025) ✓ ✗ ✗ ✗
LeWM (CEM) (Maes et al., 2026b) ✓ ✗ ✗ ✓
World action models
UVA (Li et al., 2025) ✓ ✓ ✗ ✗
UWM (Zhu et al., 2025) ✓ ✓ ✗ ✗
DreamZero (Ye et al., 2026) ✗ ✓ ✗ ✗
Fast-WAM (Yuan et al., 2026) ✗ ✓ ✗ ✗
Cosmos 3 (NVIDIA, 2026) ✓ ✓ ✗ ✗
LeWAM (ours) ✓ ✓ ✓ ✓
Goal-conditioned policy
LeWM (IDM) (Nguyen et al., 2026) ✓ ✓ ✗ ✗
GCBC (Ghosh et al., 2021) ✗ ✓ ✗ ✗

Joint requires clean-action FDM and policy losses on the same sample in one training pass. From scratch includes visual encoding and prediction without pretrained or frozen components. UWM uses a frozen pretrained variational autoencoder even when its robot backbone is trained from scratch.

E.1 Representation and Dynamics

For squared flow-matching loss, the optimal velocity is a conditional mean. The conditional-projection identity below shows that removing visual history increases optimal risk when history changes that mean. Thus action supervision discourages representations that discard action-relevant distinctions. SIGReg separately constrains the global latent distribution; this argument is not a guarantee that every useful state variable is recovered.

Refer to caption
Figure 17: Conceptual roles of SIGReg and action flow matching in shaping the latent space.
Equation 13
ℛFM​(constant)−ℛFM​(𝒛≤t)\displaystyle\mathcal{R}_{\mathrm{FM}}(\mathrm{constant})-\mathcal{R}_{\mathrm{FM}}({\bm{z}}_{\leq t}) (13)
=1C∑j=tt+C−1𝔼[MSE(𝔼[𝒂j−ϵj∣ℐj,𝒛≤t],𝔼[𝒂j−ϵj∣ℐj])]≥0.\displaystyle=\frac{1}{C}\sum_{j=t}^{t+C-1}\mathbb{E}\!\left[\operatorname{MSE}\!\left(\mathbb{E}[{\bm{a}}_{j}-{\bm{\epsilon}}_{j}\mid\mathcal{I}_{j},{\bm{z}}_{\leq t}],\mathbb{E}[{\bm{a}}_{j}-{\bm{\epsilon}}_{j}\mid\mathcal{I}_{j}]\right)\right]\geq 0.

E.2 Planning with Learned Action Proposals

Contact transitions can make nearby actions produce different outcomes. A broad sampling prior may miss useful contacts, leaving little goal-cost signal for ranking plans. Learned action proposals improve the chance of starting in useful regions; predicted dynamics then evaluates and refines them.

The paper presents this explanation as a heuristic for difficult contact-rich search. A rigorous general verification remains future work (Pang et al., 2023) (Brudermüller et al., 2025).

Refer to caption
Figure 18: Action Proposals and Predicted Consequences. (a) Smooth rollouts; (b) candidate rollouts branch at contact; (c) LeWAM pairs learned action proposals with predicted outcomes. Dashed links denote search; the solid link denotes joint prediction.

Appendix F Author Contribution Statement

Minghao Fu initiated and led technical development and benchmarks. Tavis Siebert refined architecture, attention, planning, experiments, and released code. Eryk Halicki led the physical-robot system, data, and evaluation. Randall Balestriero initiated and supervised the project. All contributed to discussion, writing, and editing.

Concept demonstration

Open image ↗