Literature
Display
← All papers

What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?

Basile Terver · Tsung-Yen Yang · Jean Ponce · Adrien Bardes · Yann LeCun

· Transactions on Machine Learning Research ↗

World models12 min text · CondensedOriginal paper ↗
Contents

Overview

The work has an open review record at OpenReview. The study compares technical choices within explicit latent world-model planning.

At a glance

Intuition. A world model can imagine convincing future frames and still choose the wrong action: the representation may miss small task-relevant differences, or the planner may get stuck before finding a useful route. This study varies the encoder, predictor, training objective, observation context and planning algorithm within a common family of explicit latent JEPA world models. It then combines favorable choices into stronger task-specific recipes. The key result is practical and conditional: proprioception, short rollout training, suitable image features and the optimizer all matter, while scaling and architecture preferences change with the domain. Results include simulation and offline real-robot action comparisons. They do not prove a universal optimal JEPA architecture or that lower prediction loss necessarily yields higher closed-loop success.

1 Introduction

The work studies implementation choices within a specified model family rather than proposing a new general JEPA principle. A frozen visual encoder supplies state features, an action-conditioned predictor learns their dynamics, and a goal-distance planner selects actions. Isolating these components reveals why promising predictive representations can fail as controllers.

Refer to caption
Figure 1: Left: Training of JEPA-WM: the encoder Eϕ,θE_{\phi,\theta} embeds video and optionally proprioceptive observation, which is fed to the predictor PθP_{\theta}, along with actions, to predict (in parallel across timesteps) the next state embedding. Right: Planning with JEPA-WM: sample action sequences, unroll the predictor on them, compute a planning cost LpL^{p} for each trajectory, and use this cost to iteratively refine the action sampling. The action encoder AθA_{\theta} and proprioceptive encoder Eθp​r​o​pE_{\theta}^{prop} are not explicitly displayed in this figure for readability.

2 Related work

World modeling and planning.

World-model planning uses learned dynamics to evaluate candidate futures. The JEPA-WM family here avoids pixel reconstruction and reward training for its main predictive objective. Related generative and policy-learning systems differ in supervision and deployment procedure.

Explicit and implicit world models.

Explicit models generate successive latent states and can be paired with different costs and search methods. Implicit approaches fold temporal information or value-like structure into representations. These are different modeling commitments, not interchangeable implementations of the same rollout.

Goal-conditioned RL.

Goal-conditioned reinforcement learning learns how to reach goals through values, policies or representation distances. Its offline reward-free data can resemble JEPA training data, but amortized policy execution differs from optimizing fresh action sequences through an explicit predictor.

Robotics.

Classical model-predictive control, visual servoing and vision-language-action policies provide complementary robotics context. They differ in dynamics knowledge, task supervision, sensors and whether computation occurs during training or planning.

Open and Move Up Close and Move Up

Refer to caption
Refer to caption

D-WM

Refer to caption
Refer to caption

Ours

Refer to caption
Refer to caption
Figure 2: Comparison of different methods on the counterfactual Franka arm lift cup task, where we hardcode 2 actions, either “open and move up" or “close and move up". Each shows 5 model actions in open-loop rollout. Left: “open and move up" action. Right: “close and move up". First row: V-JEPA-2-AC. Second row: DINO-WM. Third row: our best model, described in Section 5.3.

3 Background

The background formalizes the common training and deployment pipeline. It is the shared experimental framework rather than a new algorithmic contribution.

Training method.

A fixed visual encoder and optional learned proprioceptive encoder produce state embeddings. An action encoder and predictor learn next-state features with teacher forcing. Frozen visual targets avoid co-adaptation collapse but inherit the pretrained encoder’s information and geometry.

Equation 1
ℒ=1B∑b=1BL[Pθ(Eϕ,θ(ot−w:tb),Aθ(at−w:tb)),Eϕ,θ(ot+1b)],\mathcal{L}=\frac{1}{B}\sum_{b=1}^{B}L[P_{\theta}\left(E_{\phi,\theta}(o_{t-w:t}^{b}),A_{\theta}(a_{t-w:t}^{b})\right),E_{\phi,\theta}\left(o_{t+1}^{b}\right)], (1)

Planning.

The planner searches action sequences whose rolled-out final representation matches the encoded goal, with optional proprioceptive weighting. A sliding context window supplies recent predicted states. Goal distance is a heuristic task cost whose usefulness depends on both perception and search.

Equations 2–4
Lαp(ot,at:t+H−1,og)=(Lv​i​s+αLp​r​o​p)(Gϕ,θ(ot,at:t+H−1),Eϕ,θ(og)),L^{p}_{\alpha}(o_{t},a_{t:t+H-1},o_{g})=(L_{vis}+\alpha L_{prop})(G_{\phi,\theta}(o_{t},a_{t:t+H-1}),E_{\phi,\theta}(o_{g})), (2)
Fϕ,θ:(ot,at−w:t+k−1)\displaystyle F_{\phi,\theta}:(o_{t},a_{t-w:t+k-1}) ↦z^t+k,\displaystyle\mapsto\hat{z}_{t+k}, (3)
z^i+1\displaystyle\hat{z}_{i+1} =Pθ(z^i−w:i,Aθ(ai−w:i)),i=t,…,t+k−1,zt=Eϕ,θ(ot)\displaystyle=P_{\theta}(\hat{z}_{i-w:i},A_{\theta}(a_{i-w:i})),\quad i=t,\dots,t+k-1,\quad z_{t}=E_{\phi,\theta}(o_{t}) (4)
Table 1: Summary of all candidate design choices studied in this work and recommended recipe per task category. Each row lists the options evaluated; the two rightmost columns report the empirically best option for simulated navigation and real-world manipulation, respectively.
Component Candidates Best: [1pt] Simu. Nav. Best: [1pt] Real Manip.
Encoder
   Visual encoder DINOv2 (ViT-S/B/L), DINOv3 (L), V-JEPA (L), V-JEPA-2 (L) DINOv2-S DINOv3-L
Predictor
   Architecture Feat. cond. + sincos, Seq. cond. + RoPE, Feat. cond. + RoPE, AdaLN + RoPE AdaLN + RoPE
AdaLN + RoPE, AdaLN-zero + RoPE
   Depth 3, 6, 9, 12 6 12
Training
   Rollout steps 1-step (teacher forcing), 2-step, 3-step, 6-step 2 6
   Context length (WW) 1, 2, 3, 5, 7, 9, 14 3 5
   Proprioception With / Without ✓ ✓\bm{\checkmark}
Planning
   Optimizer CEM, NG (NeverGrad), Adam, GD CEM CEM
   Cost L1L_{1}, L2L_{2} 𝑳𝟐\bm{L_{2}} 𝑳𝟐\bm{L_{2}}

4 Studied design choices

The study first tunes planning choices, then varies training and architecture from a stated baseline, and finally tests scaling and combined recipes. This staged protocol is more informative than assuming a named model’s success follows from its high-level objective alone.

Planner.

Sampling-based CEM and Nevergrad optimization are compared with gradient descent and Adam through the same predictive cost. Horizons, context, candidate populations and iteration counts control runtime and success. Nevergrad’s meta-optimizer chooses its internal search method from the parameterization and budget.

Multistep rollout training.

Teacher forcing sees real context, while rollout training also sees the model’s own predicted context. Additional rollout losses aim to reduce the train–deployment mismatch. Truncated derivatives limit backpropagation depth; more rollout steps need not monotonically improve local accuracy or control.

Equation 5
ℒk=1B∑b=1BL[Pθ(z^t−w:t+k−1b,Aθ(at−w:t+k−1b)),Eϕ,θ(ot+kb)],\mathcal{L}_{k}=\frac{1}{B}\sum_{b=1}^{B}L[P_{\theta}(\hat{z}_{t-w:t+k-1}^{b},A_{\theta}(a_{t-w:t+k-1}^{b})),E_{\phi,\theta}\left(o_{t+k}^{b}\right)], (5)

Proprioception.

Proprioceptive inputs add precise state measurements to image features. Both visual and proprioceptive predictions are trained where used. A vision-only versus augmented comparison therefore changes the available information, not merely the architecture.

Training context size.

Training context lengths are varied from one to fourteen frames. Longer history can reveal velocity or acceleration, but changes available trajectory slices and prediction difficulty.

Encoder type.

Frozen DINOv2, DINOv3, V-JEPA and V-JEPA2 encoders are compared with token counts and preprocessing controlled as described. Video encoders are applied to duplicated-frame inputs in the selected setting, so this is not an unrestricted comparison of every possible video encoding procedure.

Predictor architecture.

Action information can enter as concatenated features, separate tokens or per-layer adaptive normalization. Positional encoding and zero initialization are also varied. Action-to-visual capacity ratios can confound comparisons, motivating additional matched-ratio controls.

Model and data scaling.

Encoder size, predictor width/depth and data fraction are varied. Larger capacity can help complex real-world data while making optimization harder or providing little benefit on simpler simulations.

Original figure
Original figure
Figure 3: (a) Comparison of planning optimizers: NG is the Nevergrad-based interface that we introduce, compared to the Cross-Entropy Method (CEM), Adam, and Gradient Descent (GD), with L1L_{1} or L2L_{2} distance. (b) Effect of adding multistep rollout loss terms: models are trained with total loss ℒ1+⋯+ℒk\mathcal{L}_{1}+\dots+\mathcal{L}_{k}. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa.

5 Experiments

5.1 Evaluation Setup.

Datasets.

The study uses simulated navigation/manipulation datasets and DROID-trained models evaluated on offline Franka videos and RoboCasa transfer. The DROID action score compares planned commands with recorded commands; it is not a measured real-robot task-completion rate.

Goal definition.

Goal states come from expert trajectories, held-out observations or simulator samplers depending on the task. Proprioceptive goal costs are weighted differently where state conventions align, and omitted for the stated cross-domain comparison.

Metrics.

Success is the main control metric, supplemented by latent rollout error, decoded-state error and image reconstruction diagnostics. These supplementary measures inspect model behavior but do not guarantee that a particular optimizer finds a useful action sequence.

Statistical significance.

Final recipes are trained with three seeds and evaluated across multiple episodes and checkpoints. Smoothing across training epochs and sampling multiple task pairs reduces noise but should be distinguished from independent training replications.

Original figure
Original figure
Figure 4: (a) Models trained with proprioceptive input are denoted “prop", while pure visual world models are named “no-prop". (b) Comparison of JEPA-WMs trained on top of various pretrained visual encoders, all of size ViT-L for fair comparison. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa.

5.2 Results

Accurate autoregressive prediction is not sufficient for planning success. The subsequent controls locate failures in objective geometry, observation precision, optimization and domain shift.

Comparing planning optimizers.

CEM with squared-distance cost is strongest overall, while Adam is especially effective on smoother Metaworld objectives. Gradient methods struggle with non-greedy obstacle and manipulation landscapes. Nevergrad is competitive on some robot-data settings with less tuning. No single optimizer wins uniformly.

Multistep rollout predictor training.

A short rollout-training horizon helps several tasks, but longer training rollouts can reduce one-step accuracy or alter data coverage. The appendix’s Lipschitz bounds explain potential error amplification rather than asserting that every actual error grows exponentially. Training and planning context choices must be interpreted together.

Original figure
Original figure
Figure 5: (a) Comparing predictor architectures: we denote positional embedding in the predictor as sincos or RoPE; the feature conditioning technique as “ftcond" and the sequence conditioning as “seqcond". The Adaptive LayerNorm conditioning technique is denoted “AdaLN". (b) Maximum number of timesteps of state embedding seen by the predictor at train time in equation 1, the predictor takes up to (Eϕ,θ(ot−W+1:t),Aθ(at−W+1:t))\left(E_{\phi,\theta}(o_{t-W+1:t}),A_{\theta}(a_{t-W+1:t})\right) as context. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa.

Impact of proprioception.

Measured state information improves precise goal reaching when visual patch features cannot resolve small displacements. Cross-domain transfer can invalidate proprioceptive coordinates, so the benefit does not transfer unchanged between DROID and RoboCasa.

Maximum context size.

Two frames can supply motion information missing from a single image, but very long context does not consistently help. Deployment contexts longer than those seen in training can degrade rollouts. Larger context also changes the number and diversity of available training slices.

Original figure
Original figure
Figure 6: (a) Comparison of model size: we vary from ViT-S to ViT-L the visual encoder size, as well as the predictor embedding dimension, keeping predictor depth constant at 6. (b) Comparison of predictor depth: we vary the predictor depth from 3 to 12, keeping the encoder fixed to DINOv2-S. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa.

Encoder type.

DINO features are stronger in these controlled planning comparisons, and DINOv3 helps especially on photorealistic data. The proposed explanation concerns object boundaries and spatially localized changes. This is evidence for the examined frozen-feature planning setup, not a general ranking of all video-understanding or representation-learning tasks.

Predictor architecture.

Adaptive normalization with RoPE gives a useful average recipe, but differences are small and task-dependent; feature conditioning is best in some settings. Zero initialization and positional encoding do not produce a universal improvement. Matched action-ratio controls help separate conditioning from capacity.

Model scaling.

Simple simulated tasks often saturate or degrade with larger encoders and deeper predictors, while DROID benefits more consistently. Explanations include task complexity, harder latent optimization and fixed-compute undertraining. The study does not isolate one universal scaling mechanism.

Data scaling.

More data generally helps within the studied distributions, but transfer gains to RoboCasa are weaker because of domain shift. A larger training corpus cannot by itself guarantee camera, action or scene calibration in a new environment.

Original figure
Original figure
Figure 7: Data scaling ablation: we display planning performance for models trained on 2%, 10%, 50% or 100% of the entire dataset, fixing the number of iterations across runs, independently of dataset size, with dataset metrics in Table 8. (a) Models trained on simulated environments, namely Metaworld, Maze, Push-T and Wall. (b) Models trained on the real-world robotic DROID dataset, evaluated on real-world videos (DROID) and on the Robocasa simulator. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa.

5.3 Our proposed optimum in the class of JEPA-WMs

The combined recipe uses short rollout training, appropriate context, selected image encoders and action conditioning, with smaller simulation models and larger photorealistic models. It improves most examined comparisons with DINO-WM and V-JEPA2-AC. Optimal here means best among the tested choices and tasks, not a theorem over all JEPA designs.

Table 2: Comparison of our best model to DINO-WM and V-JEPA-2-AC. MW-R and MW-RW denote the Reach and Reach-Wall tasks of Metaworld. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa. V-JEPA-2-AC numbers are obtained from our retrained model with the rollout-loss bug fix described in Appendix C, not from the public checkpoint. Best model is in bold.
Model Maze Wall Push-T MW-R MW-RW Rc-R Rc-Pl DROID
DWM 81.6 (3.4) 64.1 (4.6) 66.0 (4.7) 44.8 (8.9) 35.1 (9.4) 19.1 (13.4) 21.7 (7.2) 39.4 (2.1)
VJ2AC — — — — — 16.2 (8.3) 33.1 (7.2) 42.9 (2.5)
Ours 83.9 (2.3) 78.8 (3.9) 70.2 (2.8) 58.2 (9.3) 41.6 (10.0) 25.4 (16.6) 30.7 (8.0) 48.2 (1.8)

6 Conclusion

Planning outcomes emerge from representation, model training and search together. Deterministic regression is also a limitation when several futures are genuinely possible: predicting their conditional mean can be a poor physical future. Stochastic extensions and broader evaluation remain necessary beyond these largely deterministic benchmarks.

Broader Impact Statement

The impact statement discusses applications and risks of learned physical planning. The tested benchmarks do not certify safe deployment.

Author Contributions

The source specifies author contributions to methodology, implementation, experiments and writing. These disclosures remain separate from the evidence for individual design choices.

Acknowledgments

Acknowledgments and research support are retained as source context rather than treated as an additional result.

Appendix

The appendices retain the full implementation recipes, dataset definitions, planner algorithms, theoretical bounds, extra ablations and failure examples needed to qualify the main recommendations.

Appendix A Notation

The full symbol table defines training context, planning context, horizons, modalities and optimizer budgets. Distinguishing these quantities avoids conflating a longer training slice with a longer executed plan.

Table 3: Summary of notation used throughout this paper.
Symbol Description
Observations, actions and embeddings
oto_{t} Observation (image frame) at time tt
ata_{t} Action at time tt
ztz_{t} State embedding at time tt, i.e. zt=Eϕ,θ​(ot)z_{t}=E_{\phi,\theta}(o_{t})
z^t\hat{z}_{t} Predicted state embedding at time tt
Model components
Eϕ,θE_{\phi,\theta} Encoder (visual, and optionally proprioceptive)
Eθp​r​o​pE_{\theta}^{prop} Proprioceptive encoder
PθP_{\theta} Predictor
AθA_{\theta} Action encoder
Fϕ,θF_{\phi,\theta} Predictor unrolling function (Equations 3 and 4)
Gϕ,θG_{\phi,\theta} Planning evaluation function
Parameters
ϕ\phi Encoder parameters (frozen)
θ\theta Learnable parameters (predictor, action encoder, proprioceptive encoder)
Losses and objectives
ℒ\mathcal{L} Teacher-forcing training loss (Equation 1)
ℒk\mathcal{L}_{k} kk-step rollout loss (Equation 5)
LαpL^{p}_{\alpha} Planning objective (Equation 2)
Lv​i​sL_{vis}, Lp​r​o​pL_{prop} Visual and proprioceptive dissimilarity metrics
α\alpha Proprioceptive loss weight in planning
Dimensional and architectural parameters
DD Embedding dimension of encoder / predictor
AA Action dimension
ff Frameskip factor
faf_{a} Action embedding dimension (feature conditioning)
h×wh\times w Number of spatial patches of the visual encoder
BB Training batch size
Training parameters
WW Maximum teacher-forcing context length at training time
WtW^{t} Maximum rollout context length at training time
Planning parameters (see also Table 10)
WpW^{p} Maximum rollout context length at planning time
HH Planning horizon (number of predicted steps)
NN Number of candidate action trajectories
JJ Number of optimizer iterations
KeK_{e} Number of top trajectories retained (CEM)
mm Number of planned actions stepped in environment
MM Maximum number of steps per planning episode

Appendix B Extended Related Work

Alternative latent-space planning paradigms.

Locally linear models, control-oriented latent spaces and alternative latent planning methods impose different representations or objectives. The survey defines the narrower explicit JEPA-WM family studied here rather than subsuming every model-based controller.

Explicit and implicit world models.

Autoregressive latent predictors permit new rollout costs and search procedures at test time. Implicit predictive representations may encode longer-term structure without exposing the same explicit transition interface, changing what planning operations are available.

Reward- and reconstruction-based latent world models vs. JEPA-WMs.

Generative and reward-based latent models often jointly learn reconstruction, rewards, values or policies. Their success is not an isolated test of reconstruction-free next-embedding prediction, and the paper details those differences model by model.

Appendix C Training details

Predictor.

The appendix retains the exact predictor, action encoder and proprioceptive encoder optimization schedule. These settings make the controlled design comparisons reproducible.

Action conditioning of the predictor.

Feature concatenation, action-token concatenation, AdaLN and AdaLN-zero distribute conditioning differently across dimensions and layers. The complete architectural definitions prevent treating the labels as equivalent implementations.

Table 4: Training hyperparameters of some of the studied models common to all environments. If left empty, the hyperparameter value is the same as the leftmost column. WM-V refers to models trained with V-JEPA and V-JEPA2 encoders.
Hyperparameter WM WM-L WM-V
data
WW 3 3 3
ff 5 - -
resolution 224 224 256
optimization
lr 5e-4 - -
start_weight_decay 1e-7 - -
final_weight_decay 1e-6 - -
AdamW β1\beta_{1} 0.9 - -
AdamW β2\beta_{2} 0.995 - -
clip_grad 1 - -
architecture
patch_size 14 - 16
pred_depth 6 - -
pred_embed_dim 384 1024 1024
enc_embed_dim 384 1024 1024
hardware
dtype bfloat16 - -
accelerator H100 80G - -
Table 5: Environment-specific training hyperparameters. proprio_embed_dim is used only for models using proprioception. For WMW\text{WM}_{W}-6-step, the batch size is half the default batch size displayed here. We do not train but only evaluate DROID models on Robocasa.
Hyperparameter Metaworld Push-T Maze Wall DROID
optimization
batch_size 256 256 128 128 128
epochs 50 50 50 50 315
architecture
action_dim 20 10 10 10 7
action_embed_dim 20 10 10 10 10
proprio_dim 4 4 4 4 7
proprio_embed_dim 20 20 20 10 10

Train time.

Measured per-epoch times expose training-cost differences across models and datasets. They are tied to the stated hardware and data pipeline.

Table 6: Model-specific training times in minutes per epoch on 16 H100 80 GB GPUs for Maze and Wall, on 32 H100 GPUs for Push-T and Metaworld. We denote WM-B, WM-L the variants of the base model with size ViT-B and ViT-L, WM-prop the variant with proprioception and WM-V the variant with V-JEPA encoders. For DROID, we display the train time for 10 epochs since we train for 315 epochs.
Model Metaworld Push-T Maze Wall DROID
1-step 23 48 5 1 7
2-step 23 49 5 1 8
3-step 23 50 5 1 9
6-step 30 64 16 2 17
W=7W=7 20 42 5 1 13
WM-B 23 50 5 1 8
WM-L 25 50 5 1 8
WM-prop 24 50 5 1 7
WM-V 25 60 7 2 9

Visual decoder.

A separately trained visual decoder reconstructs each encoder’s representations for diagnostic plots. Decoder quality and perceptual loss can affect apparent image fidelity, so the visualization does not directly measure task success.

State decoder.

A supervised state decoder probes physical variables from frozen features. It allows evaluating represented state along imagined rollouts but does not add those labels to the main predictive training objective.

V-JEPA-2-AC reproduction.

The reproduction identifies and corrects a particular rollout-loss indexing issue in the referenced action-conditioned implementation. Reported reproduction settings and fixes matter; this finding should not be generalized into a claim that all V-JEPA2 results are invalid.

Data augmentation ablations.

Cropping and image aspect ratios affect which scene information remains visible. The augmentation ablations distinguish realistic robot-data preprocessing from synthetic square images.

Ablations on models trained with video encoders.

Video encoder frame duplication and loss choices are explicitly tested before selecting the comparative recipe. The result depends on that adaptation; it is not a proof that video pretraining cannot support good control.

Encoder comparison details.

Token counts, patch sizes and preprocessing are documented along with encoder weights. Equal token counts improve comparability but do not make the pretrained data and objectives identical.

Table 7: Detailed comparison of encoder configurations used in our experiments. All encoders use frozen weights during predictor training.
Configuration DINOv2 ViT-L DINOv3 ViT-L V-JEPA ViT-L V-JEPA2 ViT-L
Encoder Architecture
Encoder type Image Image Video Video
Model size ViT-L/14 ViT-L/16 ViT-L/16 ViT-L/16
Patch embedding Conv2d(14, 14) Conv2d(16, 16) Conv3d(2, 16, 16) Conv3d(2, 16, 16)
Embedding dimension 1024 1024 1024 1024
Patches per timestep 16×16=25616\times 16=256 16×16=25616\times 16=256 16×16=25616\times 16=256 16×16=25616\times 16=256
Input normalization ImageNet stats ImageNet stats ImageNet stats ImageNet stats
Positional encoding Sincos RoPE Sincos RoPE
Attention mask Full Full Full Full
Input Preprocessing
Input resolution 224×224224\times 224 256×256256\times 256 256×256256\times 256 256×256256\times 256
Input frame count 1 per timestep 1 per timestep 2 per timestep 2 per timestep
Frame duplication No No Yes (duplicate each) Yes (duplicate each)

Multistep rollout variants ablations.

Several teacher-forced and self-fed context variants change where prediction errors are measured and where derivatives stop. The diagrams retain the precise schedules, because a nominal two-step rollout loss can otherwise refer to materially different procedures.

Original figure
Original figure
Figure 8: Two rollout strategies with any predictor network predicting the next timestep (frame) simultaneously for all timesteps. Predictor is used as an RNN by recursively feeding it a mix of its previous predictions and groundtruth embeddings. We vertically represent the “predictor unrolling step" dimension. For a predicted embedding zt(k)z_{t}^{(k)}, we denote the timestep it corresponds to as tt and its prediction order as kk. The embeddings that enter in the loss computation are in grey whereas those which do not are in light grey. Left: “Last-gradient-only" strategy. We sample a random groundtruth embedding prefix, (z1,…,zt)(z_{1},\dots,z_{t}) (in this figure t=2t=2), and concatenate only the latest prediction to the predictor context at the next unrolling step. Strategy used in V-JEPA-2-AC with a groundtruth embedding prefix always equal to z1z_{1}. Right: “All-gradients" strategy, we compute all available prediction tasks without redundancies, e.g. we exclude from loss computation prediction tasks that have already been included in loss computation at previous timesteps, i.e. L⁡(zt(t−1),zt)L(z_{t}^{(t-1)},z_{t}).

Appendix D Error propagation in autoregressive latent prediction

A recurrence bounds rollout error by accumulated one-step error amplified through the predictor’s Lipschitz constant. The upper bound is geometric for an expansive predictor, linear when the constant equals one and bounded when contractive. Tight examples and stronger lower-bound assumptions are distinguished from a universal claim of exponential observed error. Context windows, training error and robustness create additional tradeoffs.

Equations 6–17
z^i+1=Pθ(z^i−w:i,Aθ(ai−w:i)),i=t,…,t+H−1,z^t=Eϕ,θ(ot).\hat{z}_{i+1}=P_{\theta}(\hat{z}_{i-w:i},\,A_{\theta}(a_{i-w:i})),\quad i=t,\dots,t+H-1,\quad\hat{z}_{t}=E_{\phi,\theta}(o_{t}). (6)
ϵ(H)≤{δ​ΛH−1Λ−1if ​Λ≠1,H​δif ​Λ=1.\epsilon^{(H)}\leq\begin{cases}\delta\,\dfrac{\Lambda^{H}-1}{\Lambda-1}&\text{if }\Lambda\neq 1,\\[6.0pt] H\,\delta&\text{if }\Lambda=1.\end{cases} (7)
ϵ(k+1)\displaystyle\epsilon^{(k+1)} =‖z^t+k+1−zt+k+1‖\displaystyle=\|\hat{z}_{t+k+1}-z_{t+k+1}\|
=‖Pθ​(z^t+k,Aθ​(at+k))−zt+k+1‖\displaystyle=\|P_{\theta}(\hat{z}_{t+k},A_{\theta}(a_{t+k}))-z_{t+k+1}\|
≤‖Pθ​(z^t+k,Aθ​(at+k))−Pθ​(zt+k,Aθ​(at+k))‖+‖Pθ​(zt+k,Aθ​(at+k))−zt+k+1‖\displaystyle\leq\|P_{\theta}(\hat{z}_{t+k},A_{\theta}(a_{t+k}))-P_{\theta}(z_{t+k},A_{\theta}(a_{t+k}))\|+\|P_{\theta}(z_{t+k},A_{\theta}(a_{t+k}))-z_{t+k+1}\|
≤Λ​ϵ(k)+δ.\displaystyle\leq\Lambda\,\epsilon^{(k)}+\delta. (8)
ek+1\displaystyle e_{k+1} =Pθ​(z^t+k)−zt+k+1\displaystyle=P_{\theta}(\hat{z}_{t+k})-z_{t+k+1}
=[Pθ​(z^t+k)−Pθ​(zt+k)]+[Pθ​(zt+k)−zt+k+1]⏟rk.\displaystyle=\bigl[P_{\theta}(\hat{z}_{t+k})-P_{\theta}(z_{t+k})\bigr]+\underbrace{\bigl[P_{\theta}(z_{t+k})-z_{t+k+1}\bigr]}_{r_{k}}. (9)
‖ek+1‖≥‖JPθ​(ξk)​ek‖−‖rk‖≥ℓ​‖ek‖−δ.\|e_{k+1}\|\;\geq\;\|J_{P_{\theta}}(\xi_{k})\,e_{k}\|-\|r_{k}\|\;\geq\;\ell\,\|e_{k}\|-\delta. (10)
‖eH‖\displaystyle\|e_{H}\| ≥ℓH−1​‖e1‖−δ​∑j=0H−2ℓj\displaystyle\geq\ell^{H-1}\,\|e_{1}\|-\delta\sum_{j=0}^{H-2}\ell^{\,j}
=ℓH−1​‖e1‖−δ​ℓH−1−1ℓ−1\displaystyle=\ell^{H-1}\,\|e_{1}\|-\delta\,\frac{\ell^{H-1}-1}{\ell-1}
≥ℓH−1​δ′−δ​ℓH−1−1ℓ−1\displaystyle\geq\ell^{H-1}\,\delta^{\prime}-\delta\,\frac{\ell^{H-1}-1}{\ell-1}
=(δ′−δℓ−1)​ℓH−1+δℓ−1.\displaystyle=\Bigl(\delta^{\prime}-\frac{\delta}{\ell-1}\Bigr)\,\ell^{H-1}+\frac{\delta}{\ell-1}. (11)
ϵ(k+1)≤∑j=0wΛj​ϵ(k−j)+δ,\epsilon^{(k+1)}\leq\sum_{j=0}^{w}\Lambda_{j}\,\epsilon^{(k-j)}+\delta, (12)
𝐂=(Λ0Λ1⋯Λw10⋯0⋱0⋯10).\mathbf{C}=\begin{pmatrix}\Lambda_{0}&\Lambda_{1}&\cdots&\Lambda_{w}\\ 1&0&\cdots&0\\ \vdots&\ddots&&\vdots\\ 0&\cdots&1&0\end{pmatrix}. (13)
ϵ(k+1)\displaystyle\epsilon^{(k+1)} =‖z^t+k+1−zt+k+1‖\displaystyle=\|\hat{z}_{t+k+1}-z_{t+k+1}\|
≤∥Pθ(z^t+k−w:t+k,Aθ(at+k−w:t+k))−Pθ(zt+k−w:t+k,Aθ(at+k−w:t+k))∥+δ.\displaystyle\leq\|P_{\theta}(\hat{z}_{t+k-w:t+k},\,A_{\theta}(a_{t+k-w:t+k}))-P_{\theta}(z_{t+k-w:t+k},\,A_{\theta}(a_{t+k-w:t+k}))\|+\delta. (14)
vj(m)={zt+k−jif ​j<m,z^t+k−jif ​j≥m,j=0,…,w.v^{(m)}_{j}=\begin{cases}z_{t+k-j}&\text{if }j<m,\\ \hat{z}_{t+k-j}&\text{if }j\geq m,\end{cases}\qquad j=0,\dots,w. (15)
‖Pθ​(v(0))−Pθ​(v(w+1))‖\displaystyle\|P_{\theta}(v^{(0)})-P_{\theta}(v^{(w+1)})\| ≤∑m=0w‖Pθ​(v(m))−Pθ​(v(m+1))‖\displaystyle\leq\sum_{m=0}^{w}\|P_{\theta}(v^{(m)})-P_{\theta}(v^{(m+1)})\|
≤∑j=0wΛj​‖z^t+k−j−zt+k−j‖=∑j=0wΛj​ϵ(k−j).\displaystyle\leq\sum_{j=0}^{w}\Lambda_{j}\,\|\hat{z}_{t+k-j}-z_{t+k-j}\|\;=\;\sum_{j=0}^{w}\Lambda_{j}\,\epsilon^{(k-j)}. (16)
𝐞¯k+1=𝐂​𝐞¯k+δ​𝐛,\bar{\mathbf{e}}_{k+1}=\mathbf{C}\,\bar{\mathbf{e}}_{k}+\delta\,\mathbf{b}, (17)

Appendix E Planning environments and datasets

Actions and proprioception are normalized per dataset and denormalized for execution. Shared parameter shapes do not imply shared physical coordinates across datasets.

Data slicing.

Splitting occurs by trajectories before constructing temporal slices, reducing adjacent-frame leakage across train and validation. Slice length and stride change training coverage as context grows.

Table 8: Datasets statistics. We denote the number of trajectories in the dataset under Dataset Size, the length of trajectories under Traj. Len. LL.
Dataset Size Traj. Len. TT
PointMaze 2000 100
Push-T 18500 100-300
Wall 1920 50
Metaworld 12600 100
DROID 8000 20-50

DROID.

DROID uses measured pose deltas and selected exterior camera views rather than joint multi-view input. Its offline action comparison is sensitive to coordinate and action calibration.

Robocasa.

RoboCasa provides simulated household manipulation for transfer from DROID-trained models. Camera and action-domain differences can make a plausible predicted movement physically inaccurate in the new simulator.

Metaworld.

Metaworld data is gathered with trained agents across the listed tasks. Evaluation includes pose-reaching and object-interaction settings, whose requirements differ despite sharing the same model family.

Task Description
turn on faucet Rotate the faucet counter-clockwise. Randomize faucet positions
sweep Sweep a puck off the table. Randomize puck positions
assemble nut Pick up a nut and place it onto a peg. Randomize nut and peg positions
turn off faucet Rotate the faucet clockwise. Randomize faucet positions
push Push the puck to a goal. Randomize puck and goal positions
pull lever Pull a lever down 9090 degrees. Randomize lever positions
push with stick Grasp a stick and push a box using the stick. Randomize stick positions.
get coffee Push a button on the coffee machine. Randomize the position of the coffee machine
pull handle side Pull a handle up sideways. Randomize the handle positions
pull with stick Grasp a stick and pull a box with the stick. Randomize stick positions
disassemble nut pick a nut out of the a peg. Randomize the nut positions
place onto shelf pick and place a puck onto a shelf. Randomize puck and shelf positions
press handle side Press a handle down sideways. Randomize the handle positions
hammer Hammer a screw on the wall. Randomize the hammer and the screw positions
slide plate Slide a plate into a cabinet. Randomize the plate and cabinet positions
slide plate side Slide a plate into a cabinet sideways. Randomize the plate and cabinet positions
press button wall Bypass a wall and press a button. Randomize the button positions
press handle Press a handle down. Randomize the handle positions
pull handle Pull a handle up. Randomize the handle positions
soccer Kick a soccer into the goal. Randomize the soccer and goal positions
retrieve plate side Get a plate from the cabinet sideways. Randomize plate and cabinet positions
retrieve plate Get a plate from the cabinet. Randomize plate and cabinet positions
close drawer Push and close a drawer. Randomize the drawer positions
press button top Press a button from the top. Randomize button positions
reach reach a goal position. Randomize the goal positions
press button top wall Bypass a wall and press a button from the top. Randomize button positions
reach with wall Bypass a wall and reach a goal. Randomize goal positions
insert peg side Insert a peg sideways. Randomize peg and goal positions
pull Pull a puck to a goal. Randomize puck and goal positions
push with wall Bypass a wall and push a puck to a goal. Randomize puck and goal positions
pick out of hole Pick up a puck from a hole. Randomize puck and goal positions
pick&place w/ wall Pick a puck, bypass a wall and place the puck. Randomize puck and goal positions
press button Press a button. Randomize button positions
pick&place Pick and place a puck to a goal. Randomize puck and goal positions
unplug peg Unplug a peg sideways. Randomize peg positions
close window Push and close a window. Randomize window positions
open door Open a door with a revolving joint. Randomize door positions
close door Close a door with a revolving joint. Randomize door positions
open drawer Open a drawer. Randomize drawer positions
close box Grasp the cover and close the box with it. Randomize the cover and box positions
lock door Lock the door by rotating the lock clockwise. Randomize door positions
pick bin Grasp the puck from one bin and place it into another bin. Randomize puck positions
Table 9: A list of all of the Meta-World tasks and a description of each task.

Push-T.

Push-T requires accurate contact between a pusher and a T-shaped block. Small action-position errors can cause failure even when the visual future appears broadly plausible.

PointMaze.

PointMaze includes velocity and inertia in a force-actuated navigation task. Random start–goal tests require enough state information and search ability to navigate the layout.

Wall.

Wall requires moving through a doorway between rooms. Its non-greedy geometry tests whether a planner escapes directly goal-directed local minima.

Appendix F Planning Optimization

The same latent-goal objective is paired with several planning algorithms. Their complete pseudocode and budgets are retained so runtime and success comparisons can be assessed independently of model training.

Model Predictive Control.

The study uses repeated model-predictive replanning on Metaworld as specified; other reported protocols should not automatically be assumed to have the same feedback opportunities.

Cross-Entropy Method.

CEM samples time-indexed actions from a factorized Gaussian, ranks candidates and refits using elites. Its useful exploration depends on variance, population and stopping settings.

Algorithm 1 Cross-Entropy Method
1: μ0∈ℝH×A\mu^{0}\in\mathbb{R}^{H\times A} is zero and covariance matrix σ0​I∈ℝ(H×A)2\sigma^{0}\textrm{I}\in\mathbb{R}^{(H\times A)^{2}} is the identity. Number of optimisation steps JJ.
2: for j=1j=1 to JJ do
3:   Sample NN independent trajectories ({at,…,at+H−1})∼𝒩⁡(μj,(σj)2​I)(\{a_{t},\ldots,a_{t+H-1}\})\sim\mathcal{N}(\mu^{j},(\sigma^{j})^{2}\textrm{I})
4:   For each of the NN trajectories, unroll predictor to predict the resulting trajectory, z^i=Pθ(z^i−1,ai−1),i=t+1,…,t+H\hat{z}_{i}=P_{\theta}(\hat{z}_{i-1},a_{i-1}),\quad i=t+1,\ldots,t+H. Compute cost Lαp(ot,at:t+H−1,og)L^{p}_{\alpha}(o_{t},a_{t:t+H-1},o_{g}) for each candidate trajectory.
5:   Select top KeK_{e} action sequences with the lowest cost, denote them ({at,…,at+H−1})1,…,Ke(\{a_{t},\ldots,a_{t+H-1}\})_{1,\dots,K_{e}}. Update
Equation
μj+1=1Ke​∑k=1Ke({at,…,at+H−1})k\displaystyle\mu^{j+1}=\frac{1}{K_{e}}\sum_{k=1}^{K_{e}}(\{a_{t},\ldots,a_{t+H-1}\})_{k}
σj+1=1Ke−1​∑k=1Ke[({at,…,at+H−1})k−μj+1]2\displaystyle\sigma^{j+1}=\sqrt{\frac{1}{K_{e}-1}\sum_{k=1}^{K_{e}}\left[(\{a_{t},\ldots,a_{t+H-1}\})_{k}-\mu^{j+1}\right]^{2}}
6: end for
7: Step the first mm actions of μJ\mu^{J}, where m≤Hm\leq H is a planning hyperparameter in the environment. If we are in MPC mode, the process then repeats at the next time step with the new context observation.
Original figure
Original figure
Figure 9: Planning a 100-steps Metaworld episode with the base DINO-WM at the end of training of WM, for the same Metaworld environment episode seed, with our two planners. We display the average objective of the top KeK_{e} imagined trajectories and its standard deviation (orange), and the best imagined trajectory’s planning loss LαpL_{\alpha}^{p} (blue). Bottom: Planning with CEM. Top: Planning with NG. Failure episode for both: with NG the arms stays stuck against the wall, hence the higher planning objective, whereas with CEM this episode fails because of imprecision around the goal position.

NG Planner.

The Nevergrad interface supplies the same predictive cost and a comparable evaluation budget. Its meta-optimizer selects a diagonal evolutionary strategy in these experiments, rather than an entirely fixed universal algorithm.

Algorithm 2 NeverGrad planner
1: optimizer chosen by nevergrad.optimizers.NGOpt from budget N×JN\times J, on space ℝH×A\mathbb{R}^{H\times A}, with NN workers.
2: for j=1j=1 to JJ do
3:   optimizer.ask() NN trajectories sequentially.
4:   For each of the NN trajectories, unroll predictor to predict the resulting trajectory, z^i=Pθ(z^i−1,ai−1),i=t+1,…,t+H\hat{z}_{i}=P_{\theta}(\hat{z}_{i-1},a_{i-1}),\quad i=t+1,\ldots,t+H. Compute cost Lαp(ot,at:t+H−1,og)L^{p}_{\alpha}(o_{t},a_{t:t+H-1},o_{g}) for each candidate trajectory.
5:   optimizer.tell() the cost Lαp(ot,at:t+H−1,og)L^{p}_{\alpha}(o_{t},a_{t:t+H-1},o_{g}) of the NN trajectories sequentially.
6: end for
7: Step the first mm actions of optimizer.provide_recommendation(), where m≤Hm\leq H is a planning hyperparameter in the environment. If we are in MPC mode, the process then repeats at the next time step with the new context observation.

Gradient Descent Planner.

Gradient descent differentiates final goal distance through the latent rollout. It can refine smooth local paths efficiently but is sensitive to initialization and non-greedy local minima.

Adam Planner.

Adam changes the action optimizer’s directional scaling through moving gradient moments. It uses the same learned predictive objective, so changing the planner does not independently improve representation quality.

Planning hyperparameters.

Horizon, context, candidate population, elite count and iteration budgets are reported per environment. Success is sensitive to these settings even for an unchanged world model.

Table 10: Environment-specific hyperparameters for planning, corresponding to the notations of Appendix F. The number of steps per planning episode is denoted MM and the frameskip is denoted ff. HH is the planning horizon, mm the number of actions to step in the environment, KeK_{e} the number of top actions in CEM, NN the number of trajectories evaluated in parallel, JJ the number of iterations of the optimizer. The total number of replanning steps for an evaluation episode is Mf​m\frac{M}{fm}.
NN HH mm KeK_{e} JJ WpW^{p} ff MM
PointMaze 300 6 6 10 30 2 5 30
Push-T 300 6 6 10 30 2 5 30
Wall 300 6 6 10 30 2 5 30
Metaworld 300 6 3 10 15 2 5 100
Robocasa 300 3 1 10 15 2 5 60
DROID 300 3 3 10 15 2 1 mm

Appendix G Additional experiments

G.1 Additional results

Equalized action ratio experiments.

Equalized action-to-visual ratios control a capacity confound in comparing conditioning schemes. Remaining preferences still depend on the task, limiting universal architecture conclusions.

Additional planner ablations.

Additional optimizer and seed tables test how the chosen combined recipe behaves across planners and initializations. Reproducibility at a checkpoint and stability across training are distinct forms of variability.

Table 11: Comparison of different models across all planner configurations. MW-R and MW-RW denote the Reach and Reach-Wall tasks of Metaworld. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa. In bold is the best of the three models for a given planning optimizer, underlined is the best across all planning optimizers.
Model Planner Maze Wall Push-T MW-R MW-RW Rc-R Rc-Pl DROID
CEM L2L_{2} DWM 81.6 (3.4) 64.1 (4.6) 66.0 (4.7) 44.8 (8.9) 35.1 (9.4) 19.1 (13.4) 21.7 (7.2) 39.4 (2.1)
VJ2AC — — — — — 16.2 (8.3) 33.1 (7.2) 42.9 (2.5)
Ours 83.9 (2.3) 78.8 (3.9) 70.2 (2.8) 58.2 (9.3) 41.6 (10.0) 25.4 (16.6) 30.7 (8.0) 48.2 (1.8)
CEM L1L_{1} DWM 78.8 (2.9) 48.7 (4.0) 61.6 (4.5) 45.1 (10.4) 34.0 (9.1) 14.4 (11.3) 19.6 (7.4) 41.7 (2.7)
VJ2AC — — — — — 13.2 (10.3) 30.9 (7.3) 38.5 (5.7)
Ours 79.7 (3.1) 46.7 (3.6) 63.4 (2.1) 55.1 (8.5) 40.8 (9.2) 17.6 (12.3) 30.5 (8.7) 47.2 (1.8)
NG L2L_{2} DWM 54.2 (3.8) 25.3 (4.2) 47.6 (5.4) 28.1 (7.8) 27.9 (10.0) 25.8 (16.7) 27.1 (8.6) 36.0 (3.6)
VJ2AC — — — — — 22.1 (8.9) 31.1 (7.0) 36.2 (3.2)
Ours 72.7 (4.5) 35.4 (5.9) 48.0 (3.0) 29.3 (8.6) 21.0 (7.7) 30.5 (10.7) 36.9 (9.1) 39.9 (1.9)
NG L1L_{1} DWM 52.3 (4.0) 24.6 (5.2) 46.2 (5.1) 27.5 (8.5) 28.6 (8.9) 21.6 (15.5) 26.0 (8.0) 35.4 (3.3)
VJ2AC — — — — — 18.4 (11.8) 29.4 (5.1) 33.7 (3.9)
Ours 69.5 (2.1) 30.9 (4.1) 48.2 (5.1) 27.8 (8.2) 21.3 (8.9) 25.7 (12.0) 32.0 (9.9) 40.0 (2.2)
Adam L2L_{2} DWM 14.8 (1.5) 0.1 (0.3) 8.0 (2.4) 62.0 (8.3) 49.9 (9.9) 2.8 (3.7) 21.5 (8.6) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 5.9 (5.1) 0.0 (0.0)
Ours 17.7 (3.3) 1.1 (1.1) 9.0 (2.1) 55.3 (9.9) 39.7 (9.7) 0.0 (0.0) 10.5 (5.0) 0.0 (0.0)
Adam L1L_{1} DWM 12.6 (2.8) 0.1 (0.3) 2.1 (1.5) 50.7 (8.5) 37.2 (8.3) 0.9 (1.4) 22.1 (7.5) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 3.5 (2.8) 0.0 (0.0)
Ours 16.1 (3.5) 0.0 (0.2) 0.9 (0.9) 53.2 (7.4) 43.1 (6.4) 0.2 (0.7) 7.5 (3.9) 0.0 (0.0)
GD L2L_{2} DWM 14.5 (1.9) 0.1 (0.3) 8.1 (2.4) 62.2 (8.2) 50.0 (10.1) 2.9 (3.8) 21.9 (8.4) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 8.1 (3.6) 0.0 (0.0)
Ours 17.7 (3.3) 1.1 (1.1) 9.1 (1.9) 55.3 (9.9) 39.7 (9.7) 0.0 (0.0) 10.5 (5.0) 0.0 (0.0)
GD L1L_{1} DWM 12.4 (2.6) 0.1 (0.3) 2.0 (1.3) 50.9 (8.5) 37.1 (8.4) 1.0 (1.4) 22.2 (7.3) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 4.0 (2.6) 0.0 (0.0)
Ours 16.1 (3.5) 0.0 (0.2) 0.9 (0.9) 53.3 (7.3) 40.1 (12.3) 0.2 (0.7) 7.5 (3.9) 0.0 (0.0)
Table 12: Comparison of the final checkpoint of three methods, displaying the variance across three seeds, for all planner configurations. MW-R and MW-RW denote the Reach and Reach-Wall tasks of Metaworld. Rc-Pl and Rc-R denote the Place and Reach tasks of Robocasa. In bold is the best of the three models for a given planning optimizer, underlined is the best across all planning optimizers.
Model Planner Maze Wall Push-T MW-R MW-RW Rc-R Rc-Pl DROID
CEM L2L_{2} DWM 79.2 (2.9) 63.1 (2.3) 63.3 (3.9) 44.4 (5.5) 31.9 (2.6) 11.5 (7.8) 20.8 (9.7) 40.4 (1.6)
VJ2AC — — — — — 7.3 (6.4) 33.3 (5.9) 43.2 (1.9)
Ours 83.3 (3.1) 80.9 (4.9) 69.4 (4.0) 49.0 (9.4) 29.2 (11.5) 22.9 (21.4) 34.4 (7.7) 46.5 (0.4)
CEM L1L_{1} DWM 78.3 (2.1) 50.0 (3.7) 60.6 (2.2) 49.3 (5.5) 38.2 (10.4) 13.5 (8.2) 14.6 (5.3) 41.6 (1.8)
VJ2AC — — — — — 8.3 (3.9) 31.2 (6.8) 40.9 (3.1)
Ours 81.2 (4.7) 47.6 (2.1) 55.9 (3.0) 48.4 (10.7) 30.7 (7.0) 20.8 (22.9) 29.2 (11.8) 45.0 (0.3)
NG L2L_{2} DWM 55.6 (2.3) 25.2 (4.3) 48.8 (3.0) 25.7 (2.0) 27.1 (1.7) 20.8 (7.8) 33.3 (8.2) 35.4 (2.2)
VJ2AC — — — — — 18.8 (4.4) 29.2 (2.9) 35.7 (2.1)
Ours 76.4 (0.5) 38.9 (4.2) 49.7 (2.7) 23.4 (8.4) 13.0 (5.6) 26.0 (16.6) 35.4 (6.4) 39.8 (0.7)
NG L1L_{1} DWM 52.3 (2.0) 23.1 (6.6) 42.1 (4.4) 31.2 (6.1) 29.2 (3.4) 24.0 (8.2) 28.1 (15.9) 38.4 (3.2)
VJ2AC — — — — — 9.4 (2.6) 28.1 (2.6) 34.0 (4.9)
Ours 73.6 (3.0) 30.6 (1.3) 43.8 (0.9) 27.6 (6.5) 12.5 (6.4) 18.8 (19.9) 32.3 (12.6) 39.9 (1.3)
Adam L2L_{2} DWM 14.6 (3.1) 0.0 (0.0) 8.9 (0.5) 60.4 (2.9) 44.4 (7.7) 7.3 (2.9) 24.0 (10.3) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 12.5 (5.1) 0.0 (0.0)
Ours 21.5 (4.2) 1.0 (0.9) 8.3 (0.9) 45.8 (3.9) 25.5 (10.6) 3.1 (3.1) 9.4 (3.1) 0.0 (0.0)
Adam L1L_{1} DWM 12.0 (1.6) 0.0 (0.0) 3.1 (1.0) 45.8 (6.8) 34.7 (5.2) 3.1 (4.4) 21.9 (0.0) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 4.2 (1.5) 0.0 (0.0)
Ours 17.7 (1.5) 0.0 (0.0) 1.0 (0.0) 45.8 (2.6) 33.9 (3.1) 0.0 (0.0) 7.8 (4.7) 0.0 (0.0)
GD L2L_{2} DWM 13.5 (2.1) 0.0 (0.0) 8.3 (1.0) 61.8 (2.0) 45.8 (6.8) 7.3 (2.9) 25.0 (9.2) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 5.2 (1.5) 0.0 (0.0)
Ours 21.5 (4.2) 1.0 (0.9) 8.3 (0.9) 45.8 (3.9) 25.5 (10.6) 3.1 (3.1) 9.4 (3.1) 0.0 (0.0)
GD L1L_{1} DWM 12.0 (1.6) 0.0 (0.0) 2.1 (0.0) 45.8 (6.8) 34.7 (5.2) 3.1 (4.4) 21.9 (0.0) 0.0 (0.0)
VJ2AC — — — — — 0.0 (0.0) 4.2 (1.5) 0.0 (0.0)
Ours 17.7 (1.5) 0.0 (0.0) 1.0 (0.0) 45.8 (2.6) 33.9 (3.1) 0.0 (0.0) 7.8 (4.7) 0.0 (0.0)
Refer to caption
Refer to caption
Original figure
Refer to caption
Refer to caption
Original figure
Figure 10: Two typical failure cases with the Gradient-Descent (GD) planner on the Wall task. For each failure case, we show: (top) the planned trajectory visualization, (bottom left) initial and goal states, (bottom right) planning cost evolution throughout gradient descent iterations. First failure case (top 3 subfigures): the agent finds a local planning cost minimum by going to the borders of the image when starting close to them. Second failure case (bottom 3 subfigures): the agent goes into the wall without being able to pass the door.
Original figure
Original figure
Figure 11: Comparison of planning optimizers. NG is the Nevergrad-based interface for trajectory optimization that we introduce, CEM is the Cross-Entropy Method, and GD is the Gradient Descent Planner with L1L_{1} or L2L_{2} distance. Left: comparison of the GD, CEM and NG planners on DINO-WM without proprioception. Right: comparison of CEM and NG planners on DINO-WM with proprioception.

Object manipulation on Robocasa and DROID.

Selected RoboCasa and DROID examples expose calibration errors and differing success on reaching, placing and picking. A favorable latent objective can coexist with the wrong physical arm location.

Refer to caption
Refer to caption
Figure 12: Planning at horizon 3 with our proposed JEPA-WM on Robocasa, with our best model presented in Section 5.3. The model is trained on DROID and evaluated zero-shot on Robocasa on the “Reach" task, where the goal is to reach the object. Top: the model’s visual decoding of the action plan. Bottom: the ground-truth action stepping in simulator. The model predicts a state shifted to the left compared to the ground-truth.
Refer to caption
Refer to caption
Figure 13: Planning episode with our proposed JEPA-WM on Robocasa, with DINOv2 ViT-S encoder, 2-step rollout, AdaLN predictor. The model is trained on DROID and evaluated zero-shot on Robocasa on the ’place’ task. The planning cost is the embedding space distance to the goal frame embedding.
Refer to caption
Figure 14: Unrolling a trajectory with our proposed JEPA-WM on some of our collected Franka arm trajectories, with our best model presented in Section 5.3. For each pair of rows, the top one is the model’s visual decoding of the action unrolling, the bottom one is the ground-truth trajectory of the dataset. The model sometimes does not grasp well interaction with objects, which is the most frequent failure case.

Object manipulation on Metaworld.

In Metaworld, the model can imagine grasping an object when execution does not grasp it. These failure examples separate visual plausibility and interaction modeling from actual controlled outcome.

Refer to caption
Refer to caption
Refer to caption
Figure 15: Samples of DINO-WM open-loop rollouts on the validation split, each on 18 model actions corresponding to 90 elementary actions in simulator. For each of the three pairs of rows, the groundtruth action stepping in simulator is below the decoding of the predictor rollout.
Refer to caption
Refer to caption
Figure 16: Upper row: Visual decoding of the unrolling of the action plan outputted by the NG planner, at step 1 of the Metaworld episode, on the bin-picking task. The world model’s predictions resulting from the plan indicate that the object is picked. Lower row: Stepping of half of the plan in the simulator. The object is not picked and the plan leads the robotic arm to the target location without the object.

Action precision Push-T.

Push-T needs precise final actions, favoring convergence before execution. Nevergrad can explore useful alternatives yet remain less precise than CEM under the stated budgets.

Refer to caption
Original figure
Original figure
Refer to caption
Original figure
Original figure
Refer to caption
Original figure
Original figure
Refer to caption
Original figure
Original figure
Figure 17: Convergence of planning optimization for all 2D environments and DROID. The model evaluated is the base WM model, at the end of training. Left: episode initial and goal state. Center: CEM planner. Right: NG planner. We display the planning cost of the best trajectory throughout the optimization steps.

Embedding space and model size.

A larger feature space can make relative goal-distance differences smaller. This can complicate optimization despite greater representation capacity, providing one explanation for why scaling is task-dependent.

Refer to caption
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 18: Same Metaworld reach-wall task setup: trajectory of the Base model (bottom center), the Large model (bottom right) and the expert policy (all other subfigures). Left: expert’s executed episode first and last state at top, expert’s distance of arm to goal position in the simulator space at bottom. Center: WM ViT-S encoder embedding space L2L_{2} distance to goal, expert trajectory on top, WM planned episode at bottom. Right: WM-L encoder embedding space, expert trajectory on top, WM planned episode at bottom.

G.2 Evaluation metrics

Statistical significance.

The statistical protocol averages multiple task pairs and training checkpoints and reports seed variability where available. Correlated checkpoints should not be mistaken for fully independent training runs.

Embedding space error throughout unrolling.

Latent prediction error is evaluated at several autoregressive horizons on validation trajectories. It measures the forecasting problem on those trajectories rather than whether an optimizer can solve a new goal.

Proprioceptive decoding error throughout unrolling.

A fixed state decoder maps imagined latents to physical-state predictions, testing rollout content with a supervised probe. Its own approximation error limits interpretation.

Visual decoder.

Visual decoders act frame by frame to reduce artificial temporal consistency from a powerful decoder. Reconstructed video remains a diagnostic rather than direct proof of the model’s physical accuracy.

Action error.

Offline action error compares one planned command sequence with a recorded one. Multiple action sequences may reach the same goal, and out-of-distribution action magnitudes can distort this proxy.

Refer to caption
Original figure
Refer to caption
Original figure
Figure 19: Top row: model at the end of the first training epoch. Bottom row: model at the end of training. Left: visual decoding of the horizon 1 plan. Right: comparison of the actions outputted by the planner (blue) and the groundtruth actions (orange) for the 7 action dimensions. The first three dimensions correspond to end-effector position control, the three next to end-effector orientation control, the last one to the gripper closure. The action error mostly comes from the gripper and orientation control dimensions. Hence, although only the model at the end of training correctly plans to approach the gripper from the box, its total action error is higher than at the beginning of training, if we consider all 7 dimensions.

G.3 Is there a proxy for success rate?

Validation losses correlate with success inconsistently across tasks and planners. Forecasting under held-out trajectories differs from goal-directed search, whose optimizer can exploit out-of-distribution actions or get stuck. The study therefore retains direct control evaluation rather than treating any single loss as a universal success proxy.

Table 13: Negative Spearman Correlation Coefficients between smoothed success rate and several validation metrics on Metaworld. In bold is highest value, underlined is 2nd highest. We denote the visual embedding prediction errors Vis Emb and the proprioceptive decoding error Proprio dec, at horizons HH from one to three. We display the mean success rate over the last 10 epochs averaged over the four eval setups in the last row.
Model name WM WMW\text{WM}_{W} WM-prop WM-2-step WM-L Mean
Proprio dec H=1H=1 0.40 0.44 0.40 0.26 0.20 0.34
Proprio dec H=2H=2 0.41 0.34 0.45 0.27 0.21 0.34
Proprio dec H=3H=3 0.38 0.44 0.40 0.33 0.18 0.35
Vis emb L2L_{2} H=1H=1 0.44 0.51 0.62 0.39 0.23 0.44
Vis emb L2L_{2} H=2H=2 0.42 0.46 0.62 0.39 0.24 0.42
Vis emb L2L_{2} H=3H=3 0.36 0.44 0.59 0.37 0.20 0.39
Vis emb L1L_{1} H=1H=1 0.45 0.55 0.69 0.41 0.24 0.47
Vis emb L1L_{1} H=2H=2 0.45 0.52 0.72 0.43 0.25 0.47
Vis emb L1L_{1} H=3H=3 0.42 0.52 0.72 0.42 0.22 0.46
SR 29.7 ±3.8 24.9 ±7.6 39.2 ±4.1 28.7 ±5.8 19.4 ±5.8
Table 14: Negative Spearman Correlation Coefficients across data points of the four eval setups between smoothed success rate and several validation metrics on the Push-T task. The rightmost mean column is the average of Spearman correlation of each model. In bold is highest value, underlined is 2nd highest. We display the mean success rate over the last 10 epochs in the last row.
Model name WM WMW\text{WM}_{W} WM-prop WM-2-step WM-L Mean
Proprio dec H=1H=1 0.70 0.73 0.85 0.71 0.81 0.76
Proprio dec H=2H=2 0.78 0.66 0.87 0.79 0.84 0.79
Proprio dec H=3H=3 0.76 0.70 0.78 0.83 0.86 0.79
Vis emb L2L_{2} H=1H=1 0.80 0.73 0.88 0.82 0.87 0.82
Vis emb L2L_{2} H=2H=2 0.85 0.76 0.91 0.84 0.87 0.85
Vis emb L2L_{2} H=3H=3 0.86 0.70 0.89 0.80 0.87 0.82
Vis emb L1L_{1} H=1H=1 0.87 0.79 0.92 0.83 0.90 0.86
Vis emb L1L_{1} H=2H=2 0.88 0.78 0.93 0.84 0.88 0.86
Vis emb L1L_{1} H=3H=3 0.87 0.75 0.91 0.83 0.87 0.85
Table 15: Negative Spearman Correlation Coefficients across data points of the eight eval setups between smoothed success rate and several validation metrics on the Wall task. In bold is highest value, underlined is 2nd highest. We denote the visual embedding prediction errors Vis Emb and the proprioceptive decoding error Proprio dec, at horizons HH from one to three.
Model name WM WMW\text{WM}_{W} WM-prop WM-2-step WM-3-step WM-L Mean
Proprio dec H=1H=1 0.25 0.38 0.19 0.23 0.26 0.16 0.25
Proprio dec H=2H=2 0.23 0.51 0.22 0.30 0.14 0.28 0.28
Proprio dec H=3H=3 0.22 0.49 0.39 0.26 0.34 0.23 0.32
Vis emb L2L_{2} H=1H=1 0.72 0.94 0.74 0.73 0.69 0.75 0.76
Vis emb L2L_{2} H=2H=2 0.69 0.94 0.69 0.70 0.61 0.74 0.73
Vis emb L2L_{2} H=3H=3 0.68 0.94 0.65 0.69 0.56 0.70 0.70
Vis emb L1L_{1} H=1H=1 0.78 0.96 0.81 0.77 0.73 0.80 0.81
Vis emb L1L_{1} H=2H=2 0.74 0.96 0.76 0.72 0.69 0.79 0.78
Vis emb L1L_{1} H=3H=3 0.73 0.96 0.73 0.72 0.64 0.75 0.75
Table 16: Negative Spearman Correlation Coefficients across data points of the eight eval setups between smoothed success rate and several validation metrics on the Point Maze environment. The rightmost mean column is the average of Spearman correlation of each model. In bold is highest value, underlined is 2nd highest.
Model name WM WMW\text{WM}_{W} WM-prop WM-2-step WM-3-step WM-L Mean
Proprio dec H=1H=1 0.15 0.19 0.15 0.14 0.25 0.09 0.16
Proprio dec H=2H=2 0.13 0.21 0.10 0.35 0.23 0.21 0.21
Proprio dec H=3H=3 0.10 0.34 0.25 0.10 0.27 0.10 0.19
Vis emb L2L_{2} H=1H=1 0.43 0.50 0.82 0.53 0.42 0.19 0.48
Vis emb L2L_{2} H=2H=2 0.23 0.24 0.74 0.50 0.53 0.03 0.38
Vis emb L2L_{2} H=3H=3 0.21 0.30 0.67 0.46 0.49 0.17 0.38
Vis emb L1L_{1} H=1H=1 0.58 0.68 0.86 0.54 0.38 0.36 0.57
Vis emb L1L_{1} H=2H=2 0.37 0.49 0.78 0.55 0.50 0.19 0.48
Vis emb L1L_{1} H=3H=3 0.29 0.38 0.74 0.51 0.50 0.29 0.45

G.4 Success over epochs

The complete success-over-training grids retain favorable and unfavorable dynamics across tasks, encoders and rollout settings. They contextualize final averages and expose instability hidden by a single checkpoint.

Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 20: Success rate evolution for several evaluation setups on all tasks, comparing image and video encoders. At each epoch, we evaluate the success rate on 96 independent episodes and report the average. We denote WM the base model for the design choice study, namely DINO-WM (Zhou et al., 2024a) without proprioception, and WM-L its Vit-L version. We display the results for the models learned on top of V-JEPA and V-JEPA2.
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 21: Success rate evolution for several evaluation setups on all tasks, comparing multistep rollout, proprioception and model size. We denote WM-B, WM-L the variants of the base model with size ViT-B and ViT-L, WM-prop the variant with proprioception, and the multistep rollout models as WM-kk-step. Row 1: Metaworld reach, row 2: Metaworld reach-wall, row 3: Wall, row 4: Push-T, row 5: Point Maze. At each epoch, we evaluate the success rate on 96 independent episodes and report the average.
Concept demonstration

Open image ↗