Literature
Display
← All papers

Back to the Features: DINO as a Foundation for Video World Models

Federico Baldassarre · Marc Szafraniec · Basile Terver · Vasil Khalidov · Francisco Massa · Yann LeCun · Patrick Labatut · Maximilian Seitzer · Piotr Bojanowski

Preprint posted · arXiv (preprint) ↗

World models9 min text · CondensedOriginal paper ↗
Contents

At a glance

Intuition: pretrain a model to forecast the meaningful visual features of ordinary videos, then add action inputs when using it as a controller. DINO-world freezes DINOv2 and learns a causal predictor of future patch features on roughly 66 million unlabeled videos. It accepts explicit timestamps, so prediction intervals can vary. Dense forecasting and intuitive-physics tests probe the learned dynamics, while action-conditioned fine-tuning supports planning in three simulated environments. This is a direct frozen-feature companion to JEPA, not a jointly learned JEPA encoder. Its deterministic predictions become inaccurate at longer horizons, and physics plausibility scores are not proof of general physical reasoning. The large private training corpus and simulation-only planning limit reproduction and deployment claims.

1 Introduction

Predicting every pixel spends capacity on details that may not matter for decisions. This work instead starts from an image representation already trained to capture semantics and geometry, then uses video to learn temporal evolution. Large-scale video pretraining and small action-conditioned adaptation are separate stages. That separation reduces the need for action-labeled videos but does not remove it for control.

2 Related work

The paper uses world model broadly for temporal prediction, with or without action inputs. It distinguishes predicting observations, predicting features and using those predictions to select behavior. These capabilities should be evaluated separately.

2.1 Applications

A world model can train a policy in imagined experience, search actions online, generate videos or forecast labels such as segmentation and depth. Strong performance on one use does not guarantee another. Here forecasting and physics provide representation diagnostics, while the final experiments explicitly test action-conditioned planning.

2.2 Design choices

Frozen foundation features avoid jointly learning perception and dynamics from scratch. Unconditional video pretraining uses plentiful videos, while later action blocks adapt the predictor to limited labeled trajectories. The tradeoff is dependence on what the original encoder retains. Comparisons with generative systems involve different training data, objectives and compute.

3 Method

A frozen frame encoder supplies spatial patch tokens. A timestamp-conditioned cross-attention predictor forecasts future tokens from past tokens. Added action blocks adapt that same predictive architecture to controlled trajectories. No pixel decoder is required by this model.

Refer to caption
Figure 1: Latent video world model architecture. A frozen DINOv2 encoder maps video frames to patch tokens in latent space. The predictor is a stack of cross-attention blocks trained to predict the patch token 𝒙^t+1,i′,j′\hat{{\bm{x}}}_{t+1,i^{\prime},j^{\prime}} at a future timestamp τt+1\tau_{t+1} and location (i′,j′)(i^{\prime},j^{\prime}) from all past tokens 𝒙1:t,⋅,⋅{\bm{x}}_{1:t,\cdot,\cdot} and timestamps 𝓣1:t\bm{\mathcal{T}}_{1:t}. Spatiotemporal coordinates are injected into each token via a 3-axial RoPE. The block-triangular attention mask allows to train the model in parallel for all patches of all frames, respecting temporal causality. For conditional fine-tuning, we add zero-initialized action blocks that update each query with the corresponding action 𝒂t{\bm{a}}_{t}.

3.1 Observations and state space

Each observation has both an image and a time in seconds. DINOv2 converts it to a feature grid. A prediction query specifies the desired future time and patch position, allowing the model to request one part of a future feature map. Explicit timestamps distinguish, for example, the same visual motion sampled at different frame rates. Architectural flexibility does not guarantee accurate extrapolation to arbitrary times.

Equation 1
(𝐗1:t,𝓣1:t,(τt′,i′,j′))→𝒙t′,i′,j′∀(i′,j′)∈{1,…,H}×{1,…,W},∀t′>t,\left(\mathbf{X}_{1:t},\bm{\mathcal{T}}_{1:t},(\tau_{t^{\prime}},i^{\prime},j^{\prime})\right)\rightarrow{\bm{x}}_{t^{\prime},i^{\prime},j^{\prime}}\qquad\forall(i^{\prime},j^{\prime})\in\{1,\ldots,H\}{\times}\{1,\ldots,W\},\ \forall t^{\prime}>t, (1)

3.2 Predictor architecture and training

A learned query attends to past patch features through successive transformer blocks. Three-axis rotary encodings carry time, horizontal position and vertical position. A causal mask prevents future observations leaking into a prediction. Smooth L 1 loss matches every next-frame patch to its frozen target, and teacher forcing permits parallel training. Random time gaps expose the predictor to different forecast intervals; rollout still feeds back its own imperfect predictions.

Equations 2–4
𝒒\displaystyle{\bm{q}} ←𝒒+CrossAttention​(LN​(𝒒),{𝒙t,i,j∣τt<τt′}),\displaystyle\leftarrow{\bm{q}}+\textsc{CrossAttention}\left({\textsc{LN}\left({{\bm{q}}}\right)},{\{\,{\bm{x}}_{t,i,j}\mid\tau_{t}<\tau_{t^{\prime}}\,\}}\right), (2)
𝒒\displaystyle{\bm{q}} ←𝒒+MLP​(LN​(𝒒)).\displaystyle\leftarrow{\bm{q}}+\textsc{MLP}\left({\textsc{LN}\left({{\bm{q}}}\right)}\right). (3)
minθℒ(𝒙t+1,i′,j′,Predictorθ(𝑿1:t,𝓣1:t,(τt+1,i′,j′)))∀i′,j′,t∈{1,…,T−1},\min_{\theta}\mathcal{L}\bigl({\bm{x}}_{t+1,i^{\prime},j^{\prime}},\textsc{Predictor}_{\theta}\bigl({\bm{X}}_{1:t},\bm{\mathcal{T}}_{1:t},(\tau_{t+1},i^{\prime},j^{\prime})\bigl)\bigl)\qquad\forall i^{\prime},j^{\prime},t\in\{1,\ldots,T{-}1\}, (4)

3.3 Action-conditioned fine-tuning

Each new residual action block combines an action embedding with a prediction query. Zero initialization initially leaves the pretrained computation unchanged. Fine-tuning can update only these blocks or the full predictor. This provides a way to reuse video knowledge without requiring the original unlabeled videos to contain action annotations.

4 Experiments

The main setup uses a frozen DINOv2 base encoder, a 1.1-billion-parameter predictor and a large private collection of uncurated videos. Forecasting probes use trained linear output heads; planning uses separate per-environment action fine-tuning. Smaller open video datasets are studied as controls but do not reproduce the scale or diversity of the main corpus.

4.1 Dense feature forecasting and intuitive physics

Forecasting predicts segmentation or depth roughly 0.2 or 0.5 seconds ahead. DINO-world performs strongly on VSPW, while domain-specialized DINO-Foresight is better on some Cityscapes and KITTI settings. Physics tests rank plausible versus implausible videos by prediction surprise. The authors treat those noisy scores as sanity checks, not comprehensive evidence of physical understanding. Encoder quality, head training and prediction quality all affect the measurements.

Table 1: Dense forecasting. For each method, we train a “present-time” linear head for segmentation or depth estimation, which we then apply to the output of the world model at a prediction distance of ∼200{\sim}200ms (short) or ∼0.5{\sim}0.5 seconds (mid). A small gap between present and forecasting performance indicates a strong world model.
VSPW mIoU (↑\uparrow) Cityscapes mIoU (↑\uparrow) KITTI RMSE (↓\downarrow)
Encoder Present Short Mid Present Short Mid Present Short Mid
Copy Last ViT-B 52.8 47.9 42.1 68.6 53.2 39.7 2.963 3.778 4.745
COSMOS-4B ViT-B 52.8 46.6 40.2 68.6 55.4 46.2 2.963 4.178 4.742
COSMOS-12B ViT-B 52.8 46.6 40.7 68.6 55.6 45.9 2.963 4.157 4.617
V-JEPA ViT-L 29.1 08.2 07.7 48.8 15.5 14.0 3.502 7.217 7.491
V-JEPA ViT-H 28.0 04.9 04.6 49.7 13.3 12.2 3.402 5.458 5.785
DINO-Foresight ViT-B 50.6 44.7 37.7 66.9 64.5 57.2 2.882 3.562 3.740
DINO-world ViT-B 52.8 51.6 47.0 68.6 64.7 55.1 2.963 3.214 4.268
Table 2: Intuitive physics benchmarks. For each dataset, we identify physically implausible videos by tracking the amount of “surprise”, i.e. the difference between encoded and predicted features. We report the mean relative accuracy across all video categories. The detailed breakdown per categories is provided in the appendix.
Encoder Predictor IntPhys GRASP InfLevel
COSMOS-4B VAE 4B 99.5 60.1 44.8
V-JEPA ViT-L 022M 92.2 67.0 58.9
V-JEPA ViT-H 022M 89.4 73.0 59.9
DINO-Foresight ViT-B 193M 87.8 64.9 62.8
DINO-world ViT-B 1.1B 91.3 76.0 63.7

4.2 Ablation studies

Larger predictors and the broad video corpus improve the tested metrics. Alternative encoders perform worse in the reported configurations, but these comparisons also vary representation geometry and capacity requirements. Direct prediction is better at short intervals; intermediate autoregressive steps help farther ahead, yet forecasting becomes inaccurate near one second. The deterministic model tends to blur uncertain futures rather than explicitly representing multiple possibilities.

Table 3: Ablation studies. For each setup, we report physical understanding on IntPhys, and mid-term segmentation on Cityscapes and VSPW. Predictor size (left): We train a base (86M), large (304M), and giant (1.1B) predictor to assess scaling trends w.r.t. model capacity. Training data (center): We compare the smaller and more specialized datasets Cityscapes (CS) and Something-Something V2 (SSv2) vs. our large-scale collection of web videos. Visual encoder (right): We compare encoders of different sizes trained for pixel reconstruction (VAE), vision-language alignment (SigLIP2), or representation learning (DINOv2).
Predictor IntPhys CS VSPW
Base 84.9 47.7 45.4
Large 89.1 51.9 46.4
Giant 90.6 53.2 46.8
Dataset IntPhys CS VSPW
CS 66.7 45.6 23.1
SSv2 79.3 44.9 45.2
Ours 66M 90.6 53.2 46.8
Encoder IntPhys CS VSPW
SD3.5 VAE – 13.0 01.5
SigLIP2 80.7 50.5 41.0
DINOv2 90.6 53.2 46.8
Refer to caption
Refer to caption
Figure 2: Autoregressive predictions. For each video, from top to bottom: frames with timestamps, encoder features, autoregressive predictions in latent space. The predictor has access to “ground-truth” encoder features for the first 8–10 frames (mid row, above the gray squares), and then to its own predictions (bottom row, past the gray squares). Latent features are visualized through a PCA projection computed on the encoder outputs.
Original figure
Figure 3: How far can the model predict? Cityscapes segmentation forecasting performance as context frames are progressively shifted back in time, further away from the target frame 19. We compare direct prediction vs. 1- or 2-step autoregressive prediction.
Refer to caption
Refer to caption
Refer to caption
Model PushT Wall PointMaze
Scratch 46.9 87.1 59.4
Action-only 49.4 91.1 61.6
Fine-tuned 59.4 93.8 68.7
Table 4: Planning evaluations. Success rate of a planner that refines candidate trajectories by rolling them out in the latent space of our action-conditioned world model. conditioned world model. conditioned world model.

4.3 Action-conditioned fine-tuning and planning evaluations

Action adaptation improves planning relative to training a predictor from scratch. These experiments use the smaller base predictor, RGB images and actions in Push-T, Wall and PointMaze, evaluated over 512 episodes each. Full fine-tuning outperforms training action blocks alone. The evaluation executes an entire 25-action plan without replanning, so these scores are not interchangeable with closed-loop MPC results from other papers.

5 Conclusion

The study supports reusable latent video prediction followed by action adaptation. It also exposes the remaining gap between short-term feature forecasting and dependable long-horizon control. Stochastic futures, curated data, real-world planning and language conditioning are future work.

Appendix A Implementation and training

A.1 Architecture

Video encoder.

The main encoder produces a grid of 768-dimensional patch tokens; its global classification token and register tokens are discarded. Alternative SigLIP and VAE encoders need representation adjustments. The predictor therefore learns in a particular pretrained feature space rather than a shared universal state space.

Unconditional predictor.

Input and output projections connect encoder dimensions to the cross-attention stack. Rotary position encoding uses separate groups of dimensions for time and two spatial axes, leaving four dimensions unchanged in each head. Relative spatial coordinates support changing resolution, while timestamps retain real temporal units.

Equation 5
𝝎=10linspace​(−2, 2,steps=10)\bm{\omega}=10^{\texttt{linspace}(-2,\ 2,\ \texttt{steps}=10)} (5)

Action blocks.

Action embeddings enter residual MLP blocks after each transformer block. Their initial zero layer scales preserve the pretrained model, then learning gives actions an effect. Adding these blocks increases base-model parameters by about 22 million.

A.2 Optimization

Training objective.

Smooth L 1 behaves quadratically for small errors and linearly for larger ones. It is applied elementwise to predicted and target patch features with transition parameter 0.1. This is feature regression, not a probability distribution over possible next frames.

Equation 6
ℒ(xt+1,i′,j′,x^t+1,i′,j′,)={12​β​(x−x^)2,if ​|x−x^|<β|x−x^|−β2,otherwisewith β=0.1.\mathcal{L}(x_{t+1,i^{\prime},j^{\prime}},\hat{x}_{t+1,i^{\prime},j^{\prime}},)=\begin{cases}\frac{1}{2\beta}(x-\hat{x})^{2},&\text{if }|x-\hat{x}|<\beta\\ |x-\hat{x}|-\frac{\beta}{2},&\text{otherwise}\end{cases}\qquad\text{with }\beta=0.1. (6)

Hyperparameters.

Training uses AdamW, a warmup to a constant learning rate and fixed weight decay. The appendix reports little benefit from the tested schedule alternatives. These are implementation choices, not required properties of frozen-feature prediction.

Compute resources.

The base, large and giant predictors contain 86 million,304 million and 1.1 billion parameters. Training uses multi-node H100 systems; the table distinguishes nodes from GPUs, with eight GPUs per node. Resource efficiency is relative to the compared large generative models, not a claim that the main training is inexpensive.

Table 5: Model sizes and compute resources. For each model size, we report the embedding dimension, number of transformer blocks, number of attention heads, MLP ratio, and total number of parameters. We also report the number of GPU nodes and training hours used to train the models.
Embedding Blocks Heads MLP ratio Params. Num. nodes Training hours
Base 0768 12 12 4.0 086M 04 45.3
Large 1024 24 16 4.0 304M 08 70.2
Giant 1536 40 24 4.0 01.1B 16 95.6

Appendix B Datasets

Pre-training datasets.

The main dataset contains roughly 66 million videos across varied activities, durations, resolutions and frame rates. Its private status limits exact reproduction. Cityscapes and Something-Something V 2 provide smaller public comparisons, but size and content diversity change together, so their results do not isolate either factor alone.

Table 6: Pre-training dataset statistics. We report the number of videos, duration, frames per second (FPS), and resolution of the datasets used to pre-train our video world model.
Domain Num. videos Duration (s) FPS Resolution (H×WH{\times}W)
Cityscapes Driving 002,975 ∼{\sim}2 16 1024×20481024{\times}2048
SSv2 Object manipulation 168,913 2–6 12 240×240240{\times}240 – 240×463240{\times}463
Ours General 00066M 5–60 10–60 Misc.
Refer to caption
Refer to caption
Figure 4: Pre-training dataset statistics. For our 66M video dataset, we report the joint histogram of height vs. width with highlighted aspect ratios 16:916{:}9, 1:11{:}1, and 9:169{:}16, as well as the distribution of number of frames vs. duration with highlighted 10, 30, 60 FPS.

Figures and assets.

The paper identifies the provenance of displayed videos and states that none of the illustrated clips are in the pretraining dataset. Those figures are retained as source evidence rather than substituted with newly generated examples.

Appendix C Unconditional world model evaluations

C.1 Evaluation datasets

Cityscapes.

Cityscapes has one annotated frame per sequence. The evaluation selects earlier context frames and forecasts that fixed target at short or mid-range intervals. Consequently, each sequence supplies one labeled evaluation case rather than labels for every generated frame.

VSPW.

VSPW annotates every frame, allowing multiple overlapping subsequences per video. Its short and mid-range intervals are 200 and 600 milliseconds in this protocol. Different sampling and class sets matter when comparing its scores with Cityscapes.

KITTI.

KITTI evaluates monocular depth at 200- and 600-millisecond prediction intervals after specified cropping and resizing. A lower depth RMSE is better. It measures future depth estimation, not direct downstream vehicle-control performance.

C.2 Baseline models

COSMOS.

COSMOS has fixed image size, restricted context lengths and a fixed output frame rate. Its evaluation must select suitable generated frames rather than querying an arbitrary timestamp. Those constraints introduce a protocol difference from DINO-world.

DINO-Foresight.

DINO-Foresight predicts compressed features from several intermediate encoder layers at a trained image shape. The protocol reverses its PCA projection and selects compatible features before applying a task head. This differs from predicting only DINOv2’s final-layer features.

V-JEPA.

V-JEPA’s online and target encoders play different roles: present-time heads use normalized target features, while future prediction uses the online encoder and predictor. Tubelet grouping and position interpolation require additional handling. Its predictor was trained as a representation-learning head, so this forecasting adaptation should not be treated as its original training objective.

Equation 7
𝒙^\displaystyle\hat{{\bm{x}}} =LN​(TargetEncoder​(𝒗))\displaystyle=\textsc{LN}\left({\textsc{TargetEncoder}\left({\bm{v}}\right)}\right) (7)
𝒙\displaystyle{\bm{x}} =Predictor​(Encoder​(𝒗))\displaystyle=\textsc{Predictor}\left({\textsc{Encoder}\left({{\bm{v}}}\right)}\right)
ℒ\displaystyle\mathcal{L} =MSE⁡(𝒙,𝒙^)\displaystyle=\mathrm{MSE}\left({{\bm{x}}},{\hat{{\bm{x}}}}\right)

C.3 Evaluation protocols

DINOv2 dense prediction heads.

Linear segmentation and depth heads are trained on actual present-time encoder features, then applied to predicted future features. Sliding windows and upsampling handle dense outputs. This tests whether predicted features remain useful for a downstream head, separating feature prediction from training a specialized future-label decoder.

DINO-Foresight on Cityscapes.

For DINO-Foresight, the head is trained on features processed through its PCA compression and inverse. This yields better results than using untouched final-layer features. The paper therefore uses a baseline-compatible head protocol rather than assuming all feature spaces are identical.

COSMOS on Cityscapes.

COSMOS receives selected context frames, generates a sequence, and the corresponding future frame is encoded with DINOv2 before segmentation. Resizing, padding and frame selection align it with the labeled target. Its short and mid-range predictions are both generated autoregressively.

COSMOS on VSPW.

The VSPW protocol uses overlapping valid subsequences and resizes generated frames to the segmentation head’s expected scale. This follows its denser annotation structure rather than copying Cityscapes sampling unchanged.

COSMOS on KITTI.

KITTI similarly removes padding and resizes COSMOS output to match the depth head. The reported result includes this generation-to-encoding evaluation pipeline.

DINO-Foresight on KITTI.

DINO-Foresight’s positional encoding is interpolated for KITTI’s wide frame shape, and the head uses PCA-processed features. These adaptations are necessary to compare a fixed-shape predictor on the depth benchmark.

V-JEPA on Cityscapes.

Using target-encoder features for head training improves V-JEPA forecasting scores relative to online-encoder features. The protocol assumes the predictor is aligned with those target features. Head choice materially affects a comparison of world-model representations.

Appendix D Action-conditioned fine-tuning and planning

D.1 Environments

Action fine-tuning reuses DINO-WM’s released simulated trajectories. This study ignores proprioception and uses only RGB observations and recorded actions. Its control input is therefore narrower than some later world-model comparisons.

Push-T.

Push-T uses noisy replays of expert-derived trajectories. Both the pusher and object must match the target configuration. The presence of action-labeled, expert-derived data should remain visible when interpreting the training requirements.

Point Maze.

PointMaze uses random trajectories but retains inertia and acceleration, so temporal evidence is necessary for predicting motion. It is simulated image-based navigation.

Wall.

Wall has two rooms and a doorway, with random movement trajectories. The retained dataset table records trajectory counts and lengths separately for all three tasks.

Table 7: Datasets used for action-conditioned fine-tuning and planning evaluations. We report the frame resolution, the total number of trajectories and the typical length of the trajectories. When using a “frameskip”, the length of each trajectory is effectively divided by the number of skipped frames.
Resolution Num. trajectories Traj. length
PointMaze 224 02000 100
Push-T 224 18500 100-300
Wall 224 01920 50

D.2 Action-conditioned fine-tuning.

A 90/10 trajectory split separates fine-tuning from planning evaluation. One prediction consumes a chunk of five actions and skips intermediate image observations. Skipping frames is not stacking them as additional evidence. The comparison tests from-scratch training, frozen-predictor action blocks and full predictor fine-tuning.

D.3 Planning algorithm

Planning evaluation.

A plan minimizes the final predicted feature distance to a supplied goal image. The horizon is 25 physical actions but only five predictor calls because actions are chunked in groups of five. Goals are selected to be reachable within that horizon. All actions execute without feedback replanning in the reported evaluation.

Equation 8
Lp(𝒗t,at:t+H,𝒗goal)=∥𝒙^t+H−𝒙goal∥2,L^{p}({\bm{v}}_{t},a_{t:t+H},{\bm{v}}_{\text{goal}})=\lVert\hat{{\bm{x}}}_{t+H}-{\bm{x}}_{\text{goal}}\rVert_{2}, (8)

Cross-Entropy Method.

CEM samples 300 candidate sequences per iteration and fits its Gaussian proposal to the ten best, repeating for thirty iterations. All 25 actions are then executed. Although the generic procedure permits executing a shorter prefix, the actual hyperparameters choose the full horizon; this is open-loop planning, not repeated MPC feedback.

Equation 9
𝒙^i+1=Predictor(𝒙^i,ai),∀i=t,…,t+H−1.\hat{{\bm{x}}}_{i+1}=\textsc{Predictor}\left({\hat{{\bm{x}}}_{i},a_{i}}\right),\qquad\forall i=t,\ldots,t+H-1. (9)
Table 8: Environment-specific hyperparameters for planning: NN is the number of action sequences evaluated in parallel in each planning iteration, HH is the planning horizon, ff is the frameskip, H′H^{\prime} is the number of calls to the predictor, mm is the number of actions stepped in the environment, KK is the number of top trajectories selected at each iteration to update the proposal distribution, and JJ is the number of optimization iterations.
NN HH ff H′H^{\prime} mm KK JJ
Push-T 300 25 5 5 25 10 30
PointMaze 300 25 5 5 25 10 30
Wall 300 25 5 5 25 10 30

Appendix E Results

E.1 Qualitative visualizations

Feature-color visualizations use a PCA projection fitted to encoder features. They show plausible local motions and increasing blur as rollout uncertainty grows. Selected attention maps suggest tracking of previous object locations, but attention inspection alone does not establish an object-centric causal mechanism.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 5: Unconditional autoregressive rollouts in latent space. For each clip, we feed the model a few initial frames, either 4 or 6, as processed by the encoder. We then roll out the predictor autoregressively, predicting one frame at a time and feeding it back to the predictor. From top to bottom, the three rows are: video frames with indices and timestamps, visualization of encoder features, visualization of autoregressive predictions of the world model. For each video, the predictor has access to the initial encoder features (middle row, all frames with a gray square below), and then to its own predictions (bottom row, past the gray squares).
Refer to caption
Refer to caption
Figure 6: Visualization of cross attentions. We visualize the cross-attention of a single query, to all patches of all previous frames, for two intermediate blocks of the predictor. The selected blocks 6 and 16, zero-indexed, are those where the attention maps are the most informative. The model appears to track previous locations of the object of interest and its movement through the frames. Top row: five past frames in color and full resolution, and the future frame to be predicted low resolution and grayscale colors. The location of the query used for the visualization is marked in red in the future frame. Second and third rows: attention maps of the two selected blocks. The spatial location of the query is also marked in the attention maps for ease of visualization.

E.2 Intuitive physics

The category tables reveal uneven intuitive-physics performance beneath average scores. Some categories are near perfect while others remain near chance or worse. These diagnostics support a limited plausibility capability, not universal understanding of permanence, gravity or solidity.

Table 9: Intuitive physics evaluation. Average accuracy per benchmark and category.
(a) IntPhys benchmark, three categories.
Encoder Object Permanence Unchangeableness Spatiotemporal Continuity
DINO-Foresight ViT-B 78.3 090.0 095.0
V-JEPA ViT-L 85.0 098.3 093.3
V-JEPA ViT-H 78.3 095.0 095.0
COSMOS 4B 98.4 100.0 100.0
COSMOS 12B 98.4 100.0 100.0
DINO-world ViT-B 83.3 095.3 095.3
(b) GRASP benchmark, sixteen categories, split in two rows for readability.
Encoder Collision Continuity Gravity Gravity Gravity Gravity Gravity Inertia Inertia2
Continuity Inertia Inertia2 Support
DINO-Foresight ViT-B 00.0 71.1 73.4 46.1 14.8 53.1 100.0 44.5 60.2
V-JEPA ViT-L 54.7 47.7 55.5 64.1 75.0 53.1 100.0 47.7 53.9
V-JEPA ViT-H 28.9 53.9 58.6 71.9 78.9 56.2 100.0 52.3 68.0
COSMOS 4B 56.3 51.6 56.3 57.0 89.8 51.6 100.0 48.4 46.1
DINO-world ViT-B 00.0 79.7 89.1 28.9 59.4 59.4 100.0 57.8 71.1
Object Object Object Solidity Solidity Unchangeableness Unchangeableness2
Permanence Permanence2 Permanence3 Continuity Continuity2
100.0 46.9 99.2 71.9 83.6 100.0 73.4
027.3 82.0 99.2 87.5 60.9 091.4 71.9
075.8 94.5 99.2 88.3 81.2 093.0 67.2
000.0 53.9 96.9 79.7 23.4 088.3 61.7
100.0 83.6 99.2 96.9 93.0 100.0 98.4
(c) InfLevel benchmark, three categories.
Encoder Continuity Gravity Solidity
DINO-Foresight ViT-B 86.3 51.0 50.7
V-JEPA ViT-L 71.4 52.8 52.4
V-JEPA ViT-H 76.7 51.4 51.6
COSMOS 4B 35.6 49.7 49.2
DINO-world ViT-B 88.5 51.1 51.3
Concept demonstration

Open image ↗