Overview
The motivating question is whether ordinary video prediction can produce useful expectations about objects without explicit labels for physical rules. Here “understanding” is operational: a frozen model should assign greater prediction error to an impossible event than to its carefully matched possible counterpart. The study does not directly test a robot’s ability to act safely or establish how human infants learn.
At a glance
Intuition. An object rolls behind a screen and never emerges. A predictor that expects it to continue existing may find this outcome harder to predict than the matching ordinary event. The paper uses this difference in prediction error as a model’s “surprise,” rather than asking it to describe a physical rule in words.
V-JEPA performs well on several object-focused violation tests after natural-video pretraining, without finetuning on those tests. Performance is uneven across properties, and the context-window settings are selected for the evaluation. These limits matter when interpreting the claim of emergent intuitive physics.
Measuring intutive physics understanding
Violation-of-expectation tests compare closely matched possible and impossible scenes. For a predictive model, surprise is the error between predicted future features and encoded observed features. Higher error on the impossible member is counted as evidence that the model distinguishes that violation. Paired testing controls many visual differences, but prediction error can also respond to unfamiliar appearance or motion. The exact dataset design and untrained baselines therefore matter.
Representation prediction learns to detect violations of intuitive physics
The study evaluates frozen V-JEPA, normalized-pixel VideoMAEv2 and selected multimodal language models on IntPhys, GRASP and InfLevel-lab. Together these probe properties such as permanence, continuity, constancy and object interactions across simulated and filmed scenes. V-JEPA’s latent-prediction scores outperform the tested alternatives on several benchmarks. The comparison concerns the particular models and evaluation procedures used, not every possible language or generative model.
Per property analysis of V-JEPA
Per-property comparisons show strong gains over untrained networks for permanence, continuity and shape constancy on IntPhys. Other properties are weaker, especially interactions such as solidity or collisions, and color is not consistently captured. Multiple trained seeds, random-network baselines and uncertainty estimates help separate a learning effect from a dataset shortcut. A high aggregate score should not hide these differences.
Keys to intuitive physics understanding
The ablations vary pretraining data, model size and masking task. The physics evaluation always predicts future frames from past context even when pretraining masked spatial blocks across the whole clip. Making pretraining explicitly causal is not automatically better; random masking can retain useful physics-test performance despite harming ordinary video classification more strongly. Data diversity and scale contribute, but above-chance results also appear with much less unique video than the full dataset. Repeated training exposure and unique footage are different quantities.
Discussion
The engineering evidence is that a general latent-prediction objective can acquire useful expectations for several tested event classes without explicit rule supervision. It does not resolve the biological question of innate knowledge. Current models have short memory and no action conditioning in this study; interaction-heavy events and earlier contextualizing information remain difficult. Hierarchy, longer memory and interaction data are proposed directions, not demonstrated solutions here.
Appendix A Materials and Methods
A.1 Unsupervised pretraining of V-JEPA
A context encoder reads the visible video, a slowly updated target encoder supplies target features and a predictor estimates missing features. At evaluation, changing the mask to expose only past frames turns the same machinery into a future-prediction error. The target still comes from the actually observed future, which is needed to measure surprise.
Equation S1
| (S1) |
A.2 Pretraining Data
The original VideoMix2M combines action clips and instructional videos; many analyses here use HowTo100M alone. Dataset labels are not the physics-training targets. The amount of distinct source video should be distinguished from the much larger total of repeatedly sampled clips seen by the optimizer.
A.3 V-JEPA pretraining hyperparameters
The appendix specifies encoders, a smaller 12-block predictor, three-axis rotary positions, optimization schedules and mask variants. Standard inputs cover at most about three seconds through 16 sampled frames. Block, causal-block and random masks change what must be predicted; these are ablations rather than extra physical-rule labels.
| Hyper-parameter | ViT-B/16 | ViT-L/16 | ViT-H/16 |
|---|---|---|---|
| positional embeddings | |||
| Type | RoPE | RoPE | RoPE |
| theta | 10000 | 10000 | 10000 |
| data | |||
| resolution | 224 | 224 | 224 |
| num_frames | 16 | 16 | 16 |
| framerate | 5.33 fps | 5.33 fps | 5.33 fps |
| horizontal_flip | true | true | true |
| random_resize_scale | (0.3, 1.0) | (0.3, 1.0) | (0.3, 1.0) |
| random_resize_aspect_ratio | (0.75, 1.35) | (0.75, 1.35) | (0.75, 1.35) |
| masking | |||
| block_aspect_ratio | (0.75, 1.5) | (0.75, 1.5) | (0.75,1.5) |
| shortrange_mask_num_blocks | 8 | 8 | 8 |
| shortrange_mask_spatial_scale | 0.15 | 0.15 | 0.15 |
| longrange_mask_num_blocks | 2 | 2 | 2 |
| longrange_mask_spatial_scale | 0.7 | 0.7 | 0.7 |
| optimization | |||
| optimizer | AdamW | AdamW | AdamW |
| batch_size | 3072 | 3072 | 3072 |
| total_number_of_iterations | 90000 | 90000 | 90000 |
| scheduler | Linear + Cosine | Linear + Cosine | Linear + Cosine |
| warmup_iterations | 12000 | 12000 | 12000 |
| learning_rate | 6.25 | 6.25 | 6.25 |
| start_lr | 2 | 2 | 2 |
| final_lr | 1 | 1 | 1 |
| start_momentum | 0.998 | 0.998 | 0.998 |
| final_momentum | 1.0 | 1.0 | 1.0 |
| start_weight_decay | 0.04 | 0.04 | 0.04 |
| final_weight_decay | 0.4 | 0.4 | 0.4 |
| scheduler_scale_factor | 1.25 | 1.25 | 1.25 |
| architecture | |||
| patch_size | 16 | 16 | 16 |
| tubelet_size | 2 | 2 | 2 |
| pred_depth | 12 | 12 | 12 |
| pred_embed_dim | 384 | 384 | 384 |
| hardware | |||
| dtype | bfloat16 | bfloat16 | bfloat16 |
| accelerator | A100 80G | A100 80G | A100 80G |
A.4 Evaluation Data
IntPhys provides carefully aligned simulated pairs and a private challenge test set. GRASP covers more properties but was not originally designed for this exact paired protocol, and some untrained-model results reveal shortcuts. InfLevel-lab uses real scenes in which earlier demonstrations can change what counts as possible. Models with a short context may miss that necessary information.
| Dataset | Realistic | Diverse scenes | Size | Number of Properties |
|---|---|---|---|---|
| IntPhys | No | Yes | 360 | 3 |
| GRASP | No | No | 4000 | 10 |
| InfLevel-lab | Yes | No | 4000 | 3 |
A.5 Properties
The tested properties are operationally distinct: permanence concerns continued existence, continuity a smooth path, constancy unchanged attributes, gravity and support falling or stability, solidity non-interpenetration, and inertia or collision changes in motion. The examples define benchmark manipulations, rather than a complete physical theory.
A.6 Baselines
VideoMAEv2 supplies a closely related pixel-prediction comparator. Qwen2-VL and Gemini 1.5 Pro supply video-capable language-model comparisons. Differences in temporal sampling, model accessibility and response procedures complicate a pure architecture-only interpretation.
A.7 Evaluation Protocol
Prediction-based scores compare a forecast with the target representation or normalized pixels, aggregate surprise over sliding windows and compare paired events. Language models use the paper’s question/response protocol instead. Evaluation choices such as stride, aggregation, context length and temporal subsampling influence the result and should be kept with the checkpoint when reproducing it.
Equations S2–S4
| (S2) |
| (S3) |
| (S4) |
A.8 Evaluation hyperparameters
Context and temporal sampling are selected from specified ranges, sometimes separately by property. IntPhys also permits a minimum-surprise aggregation that filters ordinary events that are difficult to predict. “No finetuning” therefore does not mean a single fixed, uncalibrated evaluation recipe was used everywhere.
| Dataset | Method | Frame skip | FPS | Window size | Window Stride |
|---|---|---|---|---|---|
| IntPhys | V-JEPA | 2 | 7.5 | 16 | 2 |
| VideoMAEv2 | 2 | 7.5 | 16 | 2 | |
| Qwen-2-VL-72b | 5 | 3 | All | N/A | |
| Gemini-1.5-pro | 2 | 7.5 | All | N/A | |
| GRASP | V-JEPA | 10 | 5 | 16 | 2 |
| VideoMAEv2 | 10 | 5 | 16 | 2 | |
| Qwen-2-VL-72b | 10 | 5 | All | N/A | |
| Gemini-1.5-pro | 10 | 5 | All | N/A | |
| InfLevel-lab | V-JEPA | 5 | 6 | 32 | 2 |
| VideoMAEv2 | 10 | 3 | 16 | 2 | |
| Qwen-2-VL-72b | 20 | 1.5 | All | N/A | |
| Gemini-1.5-pro | 30 | 1 | All | N/A |
Appendix B Choice of prediction hyperparameters and influence on 0-shot performance
The authors report attainable performance after choosing context lengths per property, motivated by the absence of suitable validation splits and calibration. They also analyze sensitivity. These results are useful for comparing capabilities under the chosen settings, but are not an unbiased estimate for an untouched future test suite with arbitrary fixed hyperparameters.


Appendix C Influence of semantic and motion diversity in videos
Reducing the number of distinct videos changes scene diversity; shortening every video changes temporal/motion diversity. The ablation separates these reductions instead of treating equal total footage as equivalent training information.
Appendix D Results on the IntPhys challenge
The private IntPhys challenge test supports the development-set findings on its tested properties. Paired and single-video judgments differ: the larger model helps more on the harder single-video task. Claims of matching human baselines apply to these particular benchmark protocols, not general human physical reasoning.
| Method | Surprise | Object permanence | Shape constancy | Continuity | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Visible | Occluded | All | Visible | Occluded | All | Visible | Occluded | All | ||
| V-JEPA-H | Max | 0.6% | 8.2% | 4.4% | 0.8% | 8.1% | 4.4% | 0.19% | 25.6% | 12.87% |
| V-JEPA-H | Avg | 0.0% | 0.56% | 0.28% | 0.0% | 0.0% | 0.0% | 0.0% | 0.19% | 0.09% |
| V-JEPA-L | Max | 5.2% | 35.4% | 0.20% | 8.8% | 35.0% | 21.9% | 5.9% | 41.5% | 23.8% |
| V-JEPA-L | Avg | 0.9% | 1.8% | 1.4% | 2.5% | 3.5% | 3.1% | 0.7% | 3.3% | 2.0% |
| Riochet et al. (2020) | 5.0% | 19.0% | 12.0% | 11.0% | 31.0% | 21.0% | 26.0% | 47.0% | 41.0% | |
| Human | 10.0% | 15.0% | 12.5% | 13.0% | 16.0% | 14.5% | 20.0% | 40.0% | 30.0% | |
| Method | Surprise | Object permanence | Shape constancy | Continuity | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Visible | Occluded | All | Visible | Occluded | All | Visible | Occluded | All | ||
| V-JEPA-H | Max | 8.0% | 28.1% | 19.2% | 11.9% | 29.7% | 21.9% | 7.8% | 43.9% | 29.67% |
| V-JEPA-H | Avg | 27.8% | 38.9% | 38.3% | 31.2% | 39.3% | 39.2% | 28.4% | 31.3% | 37.05% |
| V-JEPA-L | Max | 25.5% | 47.8% | 40.0% | 29.9% | 47.8% | 41.8% | 26.0% | 49.0% | 41.6% |
| V-JEPA-L | Avg | 33.4% | 41.7% | 41.5% | 37.0% | 42.5% | 42.7% | 34.4% | 38.8% | 41.5% |
| Human | 18.0% | 30.0% | 24.0% | 22.0% | 30.0% | 26.0% | 28.0% | 47.0% | 38.0% | |
Appendix E Importance of contextualization events on InfLevel
InfLevel sometimes first demonstrates that a cup lacks a bottom or an object has a hidden opening. Without remembering that event, an ordinary-looking later outcome may be misclassified. Relabeling under an unchanged-object assumption improves scores even for random networks, showing that the altered task can introduce shortcuts rather than establish that the original memory problem is solved.
Appendix F Per-property performance of methods
The preserved per-property figures expose strengths and failures hidden by aggregate accuracy. Read them together with trained versus untrained comparisons and the evaluation-setting qualifications.