Literature
Display
← All papers

Intuitive physics understanding emerges from self-supervised pretraining on natural videos

Quentin Garrido · Nicolas Ballas · Mahmoud Assran · Adrien Bardes · Laurent Najman · Michael Rabbat · Emmanuel Dupoux · Yann LeCun

Preprint posted · arXiv (preprint) ↗

Contents

Overview

The motivating question is whether ordinary video prediction can produce useful expectations about objects without explicit labels for physical rules. Here “understanding” is operational: a frozen model should assign greater prediction error to an impossible event than to its carefully matched possible counterpart. The study does not directly test a robot’s ability to act safely or establish how human infants learn.

Refer to caption
Figure 1: Video prediction in representation space (V-JEPA) achieves an understanding of intuitive physics. (A) Video models are evaluated on three intuitive physics datasets using the Violation of Expectation paradigm (IntPhys, GRASP, and InfLevel). V-JEPA is significantly more ‘surprised’ by implausible videos. Random initializations of V-JEPA (untrained networks) show near-chance performance, and state-of-the-art video models based on text or pixel prediction are much closer to chance. Confidence intervals at 95% are obtained via bootstrapping, except for untrained networks (n=20n=20) which use a normal distribution assumption. (B) V-JEPA is trained to ’inpaint’ natural videos in a learned representation space. Starting from a video and a corrupted version, representations are first extracted. The goal is then to predict the representation of the original video from the representation of the corrupted ones. (C) From a trained V-JEPA, we compute a surprise metric by predicting representations of N future frames based on M past ones and comparing the predictions to the representations of observed events. The surprise metric is then used to decide which of the two videos contains a physical violation.

At a glance

Intuition. An object rolls behind a screen and never emerges. A predictor that expects it to continue existing may find this outcome harder to predict than the matching ordinary event. The paper uses this difference in prediction error as a model’s “surprise,” rather than asking it to describe a physical rule in words.

V-JEPA performs well on several object-focused violation tests after natural-video pretraining, without finetuning on those tests. Performance is uneven across properties, and the context-window settings are selected for the evaluation. These limits matter when interpreting the claim of emergent intuitive physics.

Measuring intutive physics understanding

Violation-of-expectation tests compare closely matched possible and impossible scenes. For a predictive model, surprise is the error between predicted future features and encoded observed features. Higher error on the impossible member is counted as evidence that the model distinguishes that violation. Paired testing controls many visual differences, but prediction error can also respond to unfamiliar appearance or motion. The exact dataset design and untrained baselines therefore matter.

Representation prediction learns to detect violations of intuitive physics

The study evaluates frozen V-JEPA, normalized-pixel VideoMAEv2 and selected multimodal language models on IntPhys, GRASP and InfLevel-lab. Together these probe properties such as permanence, continuity, constancy and object interactions across simulated and filmed scenes. V-JEPA’s latent-prediction scores outperform the tested alternatives on several benchmarks. The comparison concerns the particular models and evaluation procedures used, not every possible language or generative model.

Original figure
Figure 2: V-JEPA accuracy increase relative to randomly-initialized models and humans across different physical properties and benchmarks. (A) Because some benchmarks contain low-level biases, we test the model performance against a set of randomly initialized networks (n=20n=20). V-JEPA models (n=5n=5) have higher relative classification accuracy on intuitive physics benchmarks for most, but not all concepts. (B) V-JEPA relative (left) and absolute (right) accuracy on the IntPhys test set across different conditions compared to naive human performance, showing a high correlation between human and machine errors. The V-JEPA score uses the maximum surprise from each video, which generalizes better for single-video classification. Human data are taken from (Riochet et al., 2022).

Per property analysis of V-JEPA

Per-property comparisons show strong gains over untrained networks for permanence, continuity and shape constancy on IntPhys. Other properties are weaker, especially interactions such as solidity or collisions, and color is not consistently captured. Multiple trained seeds, random-network baselines and uncertainty estimates help separate a learning effect from a dataset shortcut. A high aggregate score should not hide these differences.

Keys to intuitive physics understanding

The ablations vary pretraining data, model size and masking task. The physics evaluation always predicts future frames from past context even when pretraining masked spatial blocks across the whole clip. Making pretraining explicitly causal is not automatically better; random masking can retain useful physics-test performance despite harming ordinary video classification more strongly. Data diversity and scale contribute, but above-chance results also appear with much less unique video than the full dataset. Repeated training exposure and unique footage are different quantities.

Refer to caption
Figure 3: Influence of type of mask, type and amount of training data, and model size on V-JEPA IntPhys scores. (A) When pretrained on VM2M, V-JEPA exhibits an understanding of intuitive physics with every masking strategy. (B) Of the three training datasets, two give high accuracies when trained separately (K710 and Howto100M). High scores are found with only 1289 hours of Howto100M (the largest dataset), and even 128h gives better than chance performance. (C) While larger encoders improve performance, we find that the performance remains non-trivial across sizes when pretraining on HowTo100M. Confidence intervals obtained via bootstrapping.

Discussion

The engineering evidence is that a general latent-prediction objective can acquire useful expectations for several tested event classes without explicit rule supervision. It does not resolve the biological question of innate knowledge. Current models have short memory and no action conditioning in this study; interaction-heavy events and earlier contextualizing information remain difficult. Hierarchy, longer memory and interaction data are proposed directions, not demonstrated solutions here.

Appendix A Materials and Methods

A.1 Unsupervised pretraining of V-JEPA

A context encoder reads the visible video, a slowly updated target encoder supplies target features and a predictor estimates missing features. At evaluation, changing the mask to expose only past frames turns the same machinery into a future-prediction error. The target still comes from the actually observed future, which is needed to measure surprise.

Equation S1
‖pϕ​(fθ​(VC))−fθE​M​A​(VC¯)‖1.\|p_{\phi}\left(f_{\theta}\left(V_{C}\right)\right)-f_{\theta^{EMA}}(\bar{V_{C}})\|_{1}. (S1)

A.2 Pretraining Data

The original VideoMix2M combines action clips and instructional videos; many analyses here use HowTo100M alone. Dataset labels are not the physics-training targets. The amount of distinct source video should be distinguished from the much larger total of repeatedly sampled clips seen by the optimizer.

A.3 V-JEPA pretraining hyperparameters

The appendix specifies encoders, a smaller 12-block predictor, three-axis rotary positions, optimization schedules and mask variants. Standard inputs cover at most about three seconds through 16 sampled frames. Block, causal-block and random masks change what must be predicted; these are ablations rather than extra physical-rule labels.

Table S1: Pretraining hyper-parameters for V-JEPA. Table structure and values identical to the original V-JEPA paper (Bardes et al., 2024), apart from positional embedding where we rely on RoPE (Su et al., 2021).
Hyper-parameter ViT-B/16 ViT-L/16 ViT-H/16
positional embeddings
Type RoPE RoPE RoPE
theta 10000 10000 10000
data
resolution 224 224 224
num_frames 16 16 16
framerate 5.33 fps 5.33 fps 5.33 fps
horizontal_flip true true true
random_resize_scale (0.3, 1.0) (0.3, 1.0) (0.3, 1.0)
random_resize_aspect_ratio (0.75, 1.35) (0.75, 1.35) (0.75, 1.35)
masking
block_aspect_ratio (0.75, 1.5) (0.75, 1.5) (0.75,1.5)
shortrange_mask_num_blocks 8 8 8
shortrange_mask_spatial_scale 0.15 0.15 0.15
longrange_mask_num_blocks 2 2 2
longrange_mask_spatial_scale 0.7 0.7 0.7
optimization
optimizer AdamW AdamW AdamW
batch_size 3072 3072 3072
total_number_of_iterations 90000 90000 90000
scheduler Linear + Cosine Linear + Cosine Linear + Cosine
warmup_iterations 12000 12000 12000
learning_rate 6.25×10−4\times 10^{-4} 6.25×10−4\times 10^{-4} 6.25×10−4\times 10^{-4}
start_lr 2×10−4\times 10^{-4} 2×10−4\times 10^{-4} 2×10−4\times 10^{-4}
final_lr 1×10−6\times 10^{-6} 1×10−6\times 10^{-6} 1×10−6\times 10^{-6}
start_momentum 0.998 0.998 0.998
final_momentum 1.0 1.0 1.0
start_weight_decay 0.04 0.04 0.04
final_weight_decay 0.4 0.4 0.4
scheduler_scale_factor 1.25 1.25 1.25
architecture
patch_size 16 16 16
tubelet_size 2 2 2
pred_depth 12 12 12
pred_embed_dim 384 384 384
hardware
dtype bfloat16 bfloat16 bfloat16
accelerator A100 80G A100 80G A100 80G

A.4 Evaluation Data

IntPhys provides carefully aligned simulated pairs and a private challenge test set. GRASP covers more properties but was not originally designed for this exact paired protocol, and some untrained-model results reveal shortcuts. InfLevel-lab uses real scenes in which earlier demonstrations can change what counts as possible. Models with a short context may miss that necessary information.

Table S2: Summary of datasets used for evaluation. IntPhys, GRASP and InfLevel-lab provide qualitatively different data sources to perform a more holistic evaluation of models.
Dataset Realistic Diverse scenes Size Number of Properties
IntPhys No Yes ∼\sim 360 3
GRASP No No ∼\sim 4000 10
InfLevel-lab Yes No ∼\sim 4000 3

A.5 Properties

The tested properties are operationally distinct: permanence concerns continued existence, continuity a smooth path, constancy unchanged attributes, gravity and support falling or stability, solidity non-interpenetration, and inertia or collision changes in motion. The examples define benchmark manipulations, rather than a complete physical theory.

A.6 Baselines

VideoMAEv2 supplies a closely related pixel-prediction comparator. Qwen2-VL and Gemini 1.5 Pro supply video-capable language-model comparisons. Differences in temporal sampling, model accessibility and response procedures complicate a pure architecture-only interpretation.

A.7 Evaluation Protocol

Prediction-based scores compare a forecast with the target representation or normalized pixels, aggregate surprise over sliding windows and compare paired events. Language models use the paper’s question/response protocol instead. Evaluation choices such as stride, aggregation, context length and temporal subsampling influence the result and should be kept with the checkpoint when reproducing it.

Original figure
Original figure
Figure S1: Different surprise measures are better suited for different tasks. Focusing on IntPhys, we find that looking at the average surprise over a video leads to better performance when comparing pairs of videos. A one-sample t-test was performed to see if the relative surprises are greater than zero (left). However, when looking at individual videos’ surprise, choosing the maximum surprise over a video leads to a better separation between possible and impossible videos. A two-sample t-test was performed to see if impossible videos have higher surprise than possible ones. (rigt).
Equations S2–S4
St=∥pϕ(fθ(Vt:t+C))−gψ(Vt:t+C+M)∥1.S_{t}=\|p_{\phi}\left(f_{\theta}\left(V_{t:t+C}\right)\right)-g_{\psi}\left(V_{t:t+C+M}\right)\|_{1}. (S2)
AvgSurprise=1T​∑t∈{1,1+s,…,T−(C+M)}StorMaxSurprise=maxt∈{1,1+s,…,T−(C+M)}⁡St.\texttt{AvgSurprise}=\frac{1}{T}\sum_{t\in\{1,1+s,...,T-(C+M)\}}S_{t}\quad\textbf{or}\quad\texttt{MaxSurprise}=\max_{t\in\{1,1+s,...,T-(C+M)\}}S_{t}. (S3)
P=P⁡("​1​")P⁡("​1​")+P⁡("​2​")o​rP⁡("​2​")P⁡("​1​")+P⁡("​2​")P=\frac{P("1")}{P("1")+P("2")}\quad or\quad\frac{P("2")}{P("1")+P("2")} (S4)
Original figure
Figure S2: Normalized probabilities output by Qwen2-VL-72B. When presented with a pair of videos, we find that the model outputs similar probabilities for possible and impossible videos.

A.8 Evaluation hyperparameters

Context and temporal sampling are selected from specified ranges, sometimes separately by property. IntPhys also permits a minimum-surprise aggregation that filters ordinary events that are difficult to predict. “No finetuning” therefore does not mean a single fixed, uncalibrated evaluation recipe was used everywhere.

Table S3: Evaluation hyperparameters.
Dataset Method Frame skip FPS Window size Window Stride
IntPhys V-JEPA 2 7.5 16 2
VideoMAEv2 2 7.5 16 2
Qwen-2-VL-72b 5 3 All N/A
Gemini-1.5-pro 2 7.5 All N/A
GRASP V-JEPA 10 5 16 2
VideoMAEv2 10 5 16 2
Qwen-2-VL-72b 10 5 All N/A
Gemini-1.5-pro 10 5 All N/A
InfLevel-lab V-JEPA 5 6 32 2
VideoMAEv2 10 3 16 2
Qwen-2-VL-72b 20 1.5 All N/A
Gemini-1.5-pro 30 1 All N/A

Appendix B Choice of prediction hyperparameters and influence on 0-shot performance

The authors report attainable performance after choosing context lengths per property, motivated by the absence of suitable validation splits and calibration. They also analyze sensitivity. These results are useful for comparing capabilities under the chosen settings, but are not an unbiased estimate for an untouched future test suite with arbitrary fixed hyperparameters.

Original figure
Figure S3: Models perform suboptimally with a fixed context size. Due to limitations in how long of a video models can process, we find drops in performance when using a single context size across all properties and datasets. Performance remains non-trivial for V-JEPA in this scenario.
Refer to caption
Refer to caption
Figure S4: Variation of performance when changing context size for predictions. While models tend to perform better with smaller context sizes, we find the optimal context size to be dependent on the property and dataset. GRASP exhibits the most variation whereas IntPhys and InfLevel-lab are less sensitive in general.

Appendix C Influence of semantic and motion diversity in videos

Reducing the number of distinct videos changes scene diversity; shortening every video changes temporal/motion diversity. The ablation separates these reductions instead of treating equal total footage as equivalent training information.

Original figure
Figure S5: Influence of motion and scene diversity. By pretraining V-JEPA-L on subsets of HowTo100M, we investigate how the diversity in motion and scenes affects performance on IntPhys.(left) By subsampling videos, we reduce the diversity in scenes, where we find that the model can still reach good performance with 128h of unique videos. (right) By subsampling frames in videos, we reduce the diversity of motions in each scene. Here we find lower performance than when subsampling videos, but the model still achieves good performance with 2% of the frames (2579h).

Appendix D Results on the IntPhys challenge

The private IntPhys challenge test supports the development-set findings on its tested properties. Paired and single-video judgments differ: the larger model helps more on the harder single-video task. Claims of matching human baselines apply to these particular benchmark protocols, not general human physical reasoning.

Table S4: Pairwise error rates on IntPhys’ test set. For pairs of videos, taking either the maximum or average surprise from a video leads to high performance, surpassing the human results reported in (Riochet et al., 2022).
Method Surprise Object permanence Shape constancy Continuity
Visible Occluded All Visible Occluded All Visible Occluded All
V-JEPA-H Max 0.6% 8.2% 4.4% 0.8% 8.1% 4.4% 0.19% 25.6% 12.87%
V-JEPA-H Avg 0.0% 0.56% 0.28% 0.0% 0.0% 0.0% 0.0% 0.19% 0.09%
V-JEPA-L Max 5.2% 35.4% 0.20% 8.8% 35.0% 21.9% 5.9% 41.5% 23.8%
V-JEPA-L Avg 0.9% 1.8% 1.4% 2.5% 3.5% 3.1% 0.7% 3.3% 2.0%
Riochet et al. (2020) 5.0% 19.0% 12.0% 11.0% 31.0% 21.0% 26.0% 47.0% 41.0%
Human 10.0% 15.0% 12.5% 13.0% 16.0% 14.5% 20.0% 40.0% 30.0%
Table S5: Single video classification error rates (1-AUROC) on IntPhys’ test set. For single videos, we see that the maximum surprise of video leads to the highest performance, surpassing the human baselines reported in (Riochet et al., 2022). Here, the average surprise of a video is not a good metric, possibly due to values being too dependent on the experimental setup. We report the metric as percentages for legibility.
Method Surprise Object permanence Shape constancy Continuity
Visible Occluded All Visible Occluded All Visible Occluded All
V-JEPA-H Max 8.0% 28.1% 19.2% 11.9% 29.7% 21.9% 7.8% 43.9% 29.67%
V-JEPA-H Avg 27.8% 38.9% 38.3% 31.2% 39.3% 39.2% 28.4% 31.3% 37.05%
V-JEPA-L Max 25.5% 47.8% 40.0% 29.9% 47.8% 41.8% 26.0% 49.0% 41.6%
V-JEPA-L Avg 33.4% 41.7% 41.5% 37.0% 42.5% 42.7% 34.4% 38.8% 41.5%
Human 18.0% 30.0% 24.0% 22.0% 30.0% 26.0% 28.0% 47.0% 38.0%

Appendix E Importance of contextualization events on InfLevel

InfLevel sometimes first demonstrates that a cup lacks a bottom or an object has a hidden opening. Without remembering that event, an ordinary-looking later outcome may be misclassified. Relabeling under an unchanged-object assumption improves scores even for random networks, showing that the altered task can introduce shortcuts rather than establish that the original memory problem is solved.

Original figure
Figure S6: Relabeling InfLevel to remove contextualization events. Gravity and solidity both require to remember the properties about the containers shown in a video before the actual experiment. By relabeling the videos such that the prefix video is not necessary, we find a significant increase in performance for both V-JEPA and VideoMAE. However, this relabeling breaks the assumption that the possible and impossible videos have the same difficulty.

Appendix F Per-property performance of methods

The preserved per-property figures expose strengths and failures hidden by aggregate accuracy. Read them together with trained versus untrained comparisons and the evaluation-setting qualifications.

Original figure
Figure S7: Complete results for V-JEPA-L. The models (n=5n=5) achieve accuracies higher than untrained networks on most properties. Black dots represent the performance of 5 seeds.
Original figure
Figure S8: Complete results for V-JEPA-H. The model achieves accuracies higher than untrained networks on most properties. Gray dots represent the performance of the 20 untrained networks. Confidence intervals obtained via bootstrapping.
Original figure
Figure S9: Complete results for VideoMAEv2. The model achieves performance on par or slightly higher than untrained networks across properties, apart from solidity and collision. Gray dots represent the performance of the 20 untrained networks. Confidence intervals obtained via bootstrapping.
Original figure
Figure S10: Complete results for Qwen2-VL-72B. The model achieves performance on par or slightly higher than untrained networks across properties, except for color constancy and support. Gray dots represent the performance of the 20 untrained networks. Confidence intervals obtained via bootstrapping.
Original figure
Figure S11: Complete results for Gemini 1.5 pro. The model achieves performance on par or slightly higher than untrained networks across properties. Gray dots represent the performance of the 20 untrained networks. Confidence intervals obtained via bootstrapping.
Concept demonstration

Open image ↗