Literature
Display
← All papers

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

Gaoyue Zhou · Hengkai Pan · Yann LeCun · Lerrel Pinto

· International Conference on Machine Learning (ICML 2025) ↗

World models8 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition: learn a cheap simulator for what a camera sees, then try possible action sequences inside it before moving. DINO-WM freezes a pretrained DINOv2 image encoder and trains a predictor of its spatial patch features on recorded observation–action trajectories. A planner searches for actions whose predicted final features resemble a supplied goal image. Across six simulated environment suites, patch features support stronger manipulation and generalization than the tested global image representations. Zero-shot here means selecting new goals without training a task-specific policy; the dynamics model still learns from action-labeled data for the environment family. The experiments do not establish unrestricted real-world transfer or that visual similarity always defines the correct control objective.

1 Introduction

A policy stores an action-selection rule, while a predictive model can evaluate a fresh plan at deployment. The paper targets the latter using offline data. Its practical bet is that large-scale image pretraining supplies useful perception before dynamics training starts. Separating perception, future prediction and planning makes new visual goals possible without a new reward model. Generalization still depends on the offline data covering relevant interactions.

Refer to caption
Figure 1: We present DINO-WM, a method for training visual models by using pretrained DINOv2 embeddings of image frames (a). Once trained, given a target observation oTo_{T}, we can directly optimize agent behavior by planning through DINO-WM using model predictive control (b). The use of pretrained embeddings significantly improves performance over prior state-of-the-art world models (c).

2 Related Work

The closest context is visual model-based control, video generation and pretrained visual representations. DINO-WM predicts useful image features rather than generating pixels during planning. This removes a decoder from the control loop, but inherits the encoder’s blind spots. Comparisons with reward-based systems modify their original objectives to fit the reward-free setting, so they compare these adaptations rather than each baseline at its intended best use.

3 DINO World Models

The input is a current camera observation, recent observations and actions, and a target image. The objective is to reach a physical state that looks like that target. Partial observability motivates temporal context: one image may reveal position without revealing velocity. The proposed representation and planner are described separately below.

Refer to caption
Figure 2: Architecture of DINO-WM. Given observations ot−k:to_{t-k:t}, we optimize the sequence of actions at:T−1a_{t:T-1} to minimize the predicted loss to the desired goal ogo_{g}. All forward computation is done in the latent space zz. Here pθp_{\theta} indicates DINO-WM’s dynamics model, which is used for making future predictions.

3.1 DINO-based World Models ( DINO-WM )

Three modules have distinct jobs: a frozen encoder extracts spatial features, a learned action-conditioned model predicts their evolution, and an optional decoder converts features back to images for inspection. Only the first two are needed for planning. Training the decoder separately prevents its reconstruction objective from changing the predictive representation.

Equation
Observation model: zt∼encθ​(zt∣ot)\displaystyle\quad z_{t}\sim\text{enc}_{\theta}(z_{t}\mid o_{t})
Transition model: zt+1∼pθ(zt+1∣zt−H:t,at−H:t)\displaystyle\quad z_{t+1}\sim p_{\theta}(z_{t+1}\mid z_{t-H:t},a_{t-H:t})
Decoder model: o^t∼qθ​(ot∣zt)\displaystyle\quad\hat{o}_{t}\sim q_{\theta}(o_{t}\mid z_{t})
(optional for visualization)

3.1.1 Observation Model

DINOv2 produces a grid of patch vectors, retaining where things appear in the scene. Its parameters remain fixed during both training and deployment. This supplies a stable prediction target and useful image semantics, while avoiding collapse caused by a jointly learned encoder returning a constant. It does not guarantee that every hidden physical property is represented.

3.1.2 Transition Model

The predictor consumes recent patch grids plus embedded actions and, where available, proprioception. A transformer predicts the next entire grid. Its causal attention mask permits only earlier frames as evidence; all patches of the next frame are predicted together. Training uses real previous observations as context, a procedure called teacher forcing, and minimizes squared error to the frozen encoder’s next-frame features. At planning time, later inputs are the model’s own predictions, so errors can accumulate.

Equation 1
ℒp​r​e​d=‖pθ(encθ(ot−H:t),ϕ(at−H:t))−encθ(ot+1)‖2\mathcal{L}_{pred}=\left\|p_{\theta}\left(\text{enc}_{\theta}(o_{t-H:t}),\phi(a_{t-H:t})\right)-\text{enc}_{\theta}\left(o_{t+1}\right)\right\|^{2} (1)

3.1.3 Decoder for Interpretability

The optional decoder learns to reconstruct images from encoded features. It is a diagnostic for inspecting predicted object positions and motions, not a requirement for choosing actions. A plausible decoded frame can help spot failures, but visual quality alone does not establish correct dynamics or successful control.

Equation 2
ℒr​e​c=‖qθ​(zt)−ot‖2,wherezt=encθ​(ot)\mathcal{L}_{rec}=\left\|q_{\theta}(z_{t})-o_{t}\right\|^{2},\quad\text{where}\quad z_{t}=\text{enc}_{\theta}(o_{t}) (2)

3.2 Visual Planning with DINO-WM

Encode the current and goal images, simulate candidate actions, and minimize the final latent distance to the goal. The cross-entropy method repeatedly samples action sequences, keeps the best ones and refits their sampling distribution. Model predictive control executes only a prefix and replans after receiving new observations. Gradient descent through the differentiable predictor is possible, but performs worse in the reported comparison; differentiability alone does not make this search easy.

Equation
𝒞=‖z^T−zg‖2,wherez^t=p⁡(z^t−1,at−1),z^0=enc​(o0),zg=enc​(og).\mathcal{C}=\left\|\hat{z}_{T}-z_{g}\right\|^{2},\quad\text{where}\quad\begin{aligned} \hat{z}_{t}&=p(\hat{z}_{t-1},a_{t-1}),\\ \hat{z}_{0}&=\text{enc}(o_{0}),\\ z_{g}&=\text{enc}(o_{g}).\end{aligned}

4 Experiments

The evaluation asks whether offline training yields usable dynamics, whether those dynamics support planning, how much the encoder matters, and whether performance transfers to changed configurations. All six suites are simulated. The control metric differs between rigid navigation/manipulation and deformable objects.

4.1 Environments and Tasks

The suites cover maze navigation, a two-room wall, arm reaching, pushing a T-shaped block, moving a rope and gathering particles. A target image specifies each task. Feasible Push-T goals are selected within a short horizon; this is a bounded goal-reaching problem rather than arbitrary long-horizon manipulation.

Refer to caption
Figure 3: We evaluate DINO-WM on six environment suites, from left to right, top to bottom: Maze, Reach, Wall, Push-T, Rope Manipulation, and Granular Manipulation.

4.2 Baselines

IRIS, DreamerV3 and TD-MPC2 are adapted to train offline without their usual reward information, then used with planning. An action-conditioned AVDC video generator supplies another comparison. Reward removal is a substantial change for methods whose representations normally depend on rewards. Their poor performance here should not be read as a general ranking of the original algorithms.

Refer to caption
Figure 4: Open-loop rollouts of world models on Push-T and Granular. Given the first frame and action sequence, each model predicts future frames, reconstructed by its decoder. For each environment, the bottom row denotes the ground truth. DINO-WM (Ours) rollouts are bolded and are visually indistinguishable from the ground truth observations.

4.3 Optimizing Behaviors with DINO-WM

DINO-WM is strong on reaching and manipulation, while DreamerV3 slightly exceeds it on the two easiest navigation settings. Success rates use fifty start–goal pairs; rope and granular tasks instead report Chamfer distance over ten instances. Lower Chamfer distance means the final point configurations are closer. These distances cannot be compared numerically with success percentages.

Table 1: Planning results for offline world models on six control environments.
Model Maze Wall Reach PushT Rope Granular
SR ↑\uparrow SR ↑\uparrow SR ↑\uparrow SR ↑\uparrow CD ↓\downarrow CD ↓\downarrow
IRIS 0.74 0.04 0.18 0.32 1.11 0.37
DreamerV3 1.00 1.00 0.64 0.30 2.49 1.05
TD-MPC2 0.00 0.00 0.00 0.00 2.52 1.21
Ours 0.98 0.96 0.92 0.90 0.41 0.26

4.4 Does pre-trained visual representations matter?

Replacing patch features with R3M, ImageNet ResNet or DINO’s single global token hurts several spatially precise tasks. The controlled comparison supports retaining local spatial information. It does not isolate patch structure from every pretraining or architecture difference among all encoders.

Table 2: Planning results for world models with various pre-trained encoders.
Model Maze Wall Reach PushT Rope Granular
SR ↑\uparrow SR ↑\uparrow SR ↑\uparrow SR ↑\uparrow CD ↓\downarrow CD ↓\downarrow
R3M 0.94 0.34 0.40 0.42 1.13 0.95
ResNet 0.98 0.12 0.06 0.20 1.08 0.90
DINO CLS 0.96 0.58 0.60 0.44 0.84 0.79
DINOPatch (Ours) 0.98 0.96 0.92 0.90 0.41 0.26

4.5 Generalizing to Novel Environment Configurations

Held-out wall and door positions, unseen block shapes and changed particle counts test transfer within related environment families. DINO-WM leads the compared methods, but PushObj success is only 34%: unfamiliar contact geometry remains difficult. The explanation that patch features tolerate particle-count changes is a plausible interpretation rather than an independently verified mechanism.

Refer to caption
Figure 5: Training and testing visualizations for WallRandom, PushObj and GranularRandom. Test setups are highlighted in blue boxes.
Table 3: Planning results for offline world models on three suites with unseen environment configurations.
Model WallRandom PushObj GranularRandom
SR ↑\uparrow SR ↑\uparrow CD ↓\downarrow
IRIS 0.06 0.14 0.86
DreamerV3 0.76 0.18 1.53
R3M 0.40 0.16 1.12
ResNet 0.40 0.14 0.98
DINO CLS 0.64 0.18 1.36
Ours 0.82 0.34 0.63

4.6 Qualitative comparisons with generative video models

Selected AVDC generations can look realistic while violating the action-conditioned physical trajectory. The additional action-conditioned comparison also shows long-rollout divergence. These examples distinguish visual plausibility from control usefulness; they do not prove that every generative world model must fail.

Refer to caption
Figure 6: Plans generated by DINO-WM and AVDC.

4.7 Decoding and Interpreting the Latents

A separately trained decoder lets the authors measure image-space agreement after latent prediction. Lower LPIPS and higher SSIM indicate better decoded perceptual and structural similarity. These are auxiliary diagnostics; the planning results remain the direct test of whether predicted features support behavior.

Table 4: Comparison of world models on LPIPS metrics.
Method PushT Wall Rope Granular
R3M 0.045 0.008 0.023 0.080
ResNet 0.063 0.002 0.025 0.080
DINO CLS 0.039 0.004 0.029 0.086
AVDC 0.046 0.030 0.060 0.106
Ours 0.007 0.0016 0.009 0.035

5 Conclusion

The contribution is a simple frozen-feature dynamics pipeline that supports planning without reward labels or a learned task policy. Its main dependencies are sufficient state–action coverage, recorded action labels and a useful visual goal representation. Broader real-world deployment, learning from videos without actions and hierarchical long-horizon control remain future directions.

Impact Statement

The authors identify general advances in machine learning and do not discuss application-specific societal impacts in detail.

Acknowledgements

The source acknowledges research discussions, institutional grants and fellowship support.

Appendix A Appendix

A.1 Environments and Dataset Generation

Dataset construction differs by task. Several navigation, arm and deformable-object datasets use random actions; Push-T includes noisy replays of expert trajectories. Thus the broad claim of not requiring expert demonstrations does not mean that no evaluated dataset contains expert-derived behavior. The suite also uses different success conditions and trajectory lengths, all retained in the source details.

A.2 Environment families for testing generalization

Generalization tests alter door positions, replace training block shapes with held-out shapes, or change the quantity and arrangement of particles. They test specified shifts within these families. No result here demonstrates transfer to an unrelated control system.

A.3 Pretraining features

R3M and ImageNet encoders provide global ResNet representations, while DINO CLS uses DINOv2’s global token. They form the comparison against its patch grid. Keeping this distinction clear explains why a strong image descriptor can still discard detail needed for contact control.

A.4 Ablations

A.4.1 Scaling Laws of DINO-WM

Increasing Push-T data from 200 to 18,500 trajectories improves both planning success and decoded prediction metrics. This is an empirical data-size trend within one setup, not a fitted universal scaling law. Small datasets remain a major limitation.

Table 5: Planning performance and prediction quality on PushT with DINO-WM trained on datasets of various sizes. SSIM and LPIPS are measured on the predicted future latents after decoding. We observe consistent improvement in performance as we increase the dataset size.
Dataset Size SR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow
n=200 0.08 0.949 0.056
n=1000 0.48 0.973 0.013
n=5000 0.72 0.981 0.007
n=10000 0.88 0.984 0.006
n=18500 0.92 0.987 0.005

A.4.2 DINO-WM with vs. without Causal Attention Mask

Without a temporal causal mask, a multi-frame training input can reveal the frame the model is supposed to predict. Training then becomes easier in a way unavailable during rollout. The mask prevents this shortcut: increasing history helps the masked model while strongly hurting the unmasked one. At history length one, the two constructions coincide.

Table 6: Comparison of DINO-WM with and without causal attention mask on PushT. We train models with varying history hh, representing the number of past observations the model takes as input.
h=1h=1 h=2h=2 h=3h=3
w/o mask 0.76 0.36 0.08
with mask 0.76 0.88 0.92

A.4.3 DINO-WM with Reconstruction Loss

Backpropagating a pixel reconstruction loss through the predictor lowers reported Push-T success from 92% to 80%. The experiment favors separate decoder training in this setting. It does not establish that every reconstruction objective is harmful for all world models.

Table 7: Comparison of DINO-WM trained with and without loss from the decoder on PushT, highlighting the advantage of disentangling feature learning from reconstruction objectives.
Success Rate
w/o decoder loss 0.92
with decoder loss 0.80

A.5 Planning Optimization

The appendix makes the planning procedures explicit and compares executing a complete optimized plan with replanning from new observations.

A.5.1 Model Predictive Control with Cross-Entropy Method

CEM scores many sampled action sequences by their predicted final goal error. It updates a Gaussian proposal using the best candidates, repeats for a fixed budget, executes a chosen prefix and starts again from the new observation. Replanning corrects some prediction and execution errors; it cannot repair an encoder or dynamics model that misses a crucial interaction.

Equations 3–5
z^0=enc​(o0),zg=enc​(og).\hat{z}_{0}=\text{enc}(o_{0}),\quad z_{g}=\text{enc}(o_{g}). (3)
𝒞=‖z^T−zg‖2,wherez^t=p⁡(z^t−1,at−1),z^0=enc​(o0).\mathcal{C}=\left\|\hat{z}_{T}-z_{g}\right\|^{2},\quad\text{where}\quad\hat{z}_{t}=p(\hat{z}_{t-1},a_{t-1}),\quad\hat{z}_{0}=\text{enc}(o_{0}). (4)
z^t=p(z^t−1,at−1),t=1,…,T.\hat{z}_{t}=p(\hat{z}_{t-1},a_{t-1}),\quad t=1,\ldots,T. (5)

A.5.2 Gradient Descent:

Gradient descent changes each proposed action using the derivative of final goal error through the predicted rollout. This trades large candidate populations for sensitivity to the loss landscape and initialization. The same model and latent cost do not guarantee the same search quality.

Equations 6–8
z^0=enc​(o0),zg=enc​(og).\hat{z}_{0}=\text{enc}(o_{0}),\quad z_{g}=\text{enc}(o_{g}). (6)
𝒞=‖z^T−zg‖2,wherez^t=p⁡(z^t−1,at−1),z^0=enc​(o0).\mathcal{C}=\left\|\hat{z}_{T}-z_{g}\right\|^{2},\quad\text{where}\quad\hat{z}_{t}=p(\hat{z}_{t-1},a_{t-1}),\quad\hat{z}_{0}=\text{enc}(o_{0}). (7)
at←at−η∂𝒞∂at,t=0,…,T−1,a_{t}\leftarrow a_{t}-\eta\frac{\partial\mathcal{C}}{\partial a_{t}},\quad t=0,\ldots,T-1, (8)

A.5.3 Planning Results

The table distinguishes open-loop CEM, open-loop gradient descent and CEM inside model predictive control. Replanning improves the reported tasks. Rope and granular entries under MPC are Chamfer distances, inherited from their task metrics; they are not success rates despite sharing the comparison table.

Table 8: Planning results of DINO-WM
PointMaze Push-T Wall Rope Granular
CEM 0.8 0.86 0.74 NA NA
GD 0.22 0.28 NA NA NA
MPC 0.98 0.90 0.96 0.41 0.26

A.6 Comparison with Action-Conditioned Generative Models

The action-conditioned AVDC variant predicts a next observation from the current observation and action. Its selected open-loop rollouts drift from the true sequence. This is a stronger control-oriented comparison than a text-only video generator, but remains a particular trained baseline.

Refer to caption
Figure 7: Open-loop rollout on PushT with DINO-WM and action-conditioned AVDC (AVDC-AC). For each trajectory, the model is given the first frame as well as sequence of actions. The world models performs open-loop rollout with these actions.

A.7 Decoding the Latents: LPIPS and SSIM Metrics

SSIM measures structural image similarity and LPIPS measures distance between perceptual features. The reported decoded rollouts favor DINO-WM across these tasks. The decoder introduces its own approximation error, so these scores probe the combined prediction-and-decoding pipeline.

Table 9: Comparison of world models across different environments on LPIPS and SSIM metrics.
LPIPS ↓\downarrow SSIM ↑\uparrow
Method PushT Wall Rope Granular PushT Wall Rope Granular
R3M 0.045 0.008 0.023 0.080 0.956 0.994 0.982 0.917
ResNet 0.063 0.002 0.025 0.080 0.950 0.996 0.980 0.915
DINO CLS 0.039 0.004 0.029 0.086 0.973 0.996 0.980 0.912
AVDC 0.046 0.030 0.060 0.106 0.959 0.983 0.979 0.909
Ours 0.007 0.0016 0.009 0.035 0.985 0.997 0.985 0.940

A.8 Inference Time

One forward pass and a complete plan have very different costs. The source reports 0.014 seconds for a batched inference step but 53 seconds for CEM planning with 100 samples and ten iterations on an A6000. Fast latent simulation therefore does not imply real-time replanning. The simulator timing uses a different batch size and should not be treated as an equal-work hardware benchmark.

Table 10: Inference time and planning time for DINO-WM. Inference time represents the time required for a single forward pass for one step, while environment rollout time measures the simulator’s speed for advancing one step. Planning time corresponds to Cross-Entropy Method (CEM) with 100 samples per iteration and 10 optimization steps.
Metric Time (s)
Inference (Batch 32) 0.014
Simulation Rollout (Batch 1) 3.0
Planning (CEM, 100x10) 53.0

A.9 Hyperparameters and implementation

The predictor has roughly 19 million parameters and uses task-dependent frame skips to avoid learning only tiny visual changes. The prose describes 196-pixel encoder input, while its shared table lists image size 224; both printed details are retained rather than silently reconciled. Task-specific trajectory counts and lengths also deserve attention when reproducing the experiments.

Table 11: Environment-dependent hyperparameters for DINO-WM training. We report the number of trajectories in the dataset under Dataset Size, and the length of trajectories under Traj. Len.
HH Frameskip Dataset Size Traj. Len.
PointMaze 3 5 2000 100
Reacher 3 5 3000 100
Push-T 3 5 18500 100-300
PushObj 3 5 20000 100
Wall 1 5 1920 50
WallRandom 1 5 10240 50
Rope 1 1 1000 5
Granular 1 1 1000 5
Table 12: Shared hyperparameters for DINO-WM training
Name Value
Image size 224
Optimizer AdamW
Decoder lr 3e-4
Predictor lr 5e-5
Action encoder lr 5e-4
Action emb dim 10
Epochs 100
Batch size 32

A.10 Additional Planning Visualizations

The final visualizations compare actual executed trajectories with decoded imagined trajectories and test different goals from the same initial state. They illustrate controllability and characteristic errors, while the quantitative evaluation determines how often those examples generalize.

Refer to caption
Figure 8: Planning visualizations for PointMaze, Push-T, and Granular, on randomly sampled initial and goal configurations. The task is defined by Start and Goal, denoting the initial and goal observations. Final shows the final state the system arrives at after planning with each world model. For comparison, we show the best performing world models DINO CLS and DreamerV3.
Refer to caption
Figure 9: Trajectories planned with DINO-WM on all six environments. For each environment, the top (shaded) row shows the environment’s observation after executing the planned actions, and the bottom row shows the world model’s imagined observations.
Refer to caption
Figure 10: Trajectories planned with DINO-WM on PushT with the same initial states but different goal states.
Refer to caption
Figure 11: Trajectories planned with DINO-WM on PointMaze with the same initial states but different goal states.
Concept demonstration

Open image ↗