At a glance
First learn what changes in videos, then learn how robot actions change a scene. V-JEPA 2 pretrains a video encoder using masked-feature prediction on a large mixture of videos and images. A separate action-conditioned predictor, V-JEPA 2-AC, is then trained on fewer than 62 hours of robot interaction recordings while the encoder stays frozen. At deployment it tests candidate actions in feature space, compares predicted outcomes with a goal image and replans after acting. The robot results require recorded robot states/actions, chosen camera placement and supplied goal images; they do not arise from passive video alone. Classification, action-anticipation and language-alignment experiments evaluate additional capabilities separately.
1 Introduction
The central contribution is a staged route from observational video pretraining to an action-conditioned world model. Scaling improves visual features; robot interaction data then connects those features to control commands. Experiments cover frozen recognition, human-action anticipation, language-aligned video question answering and manipulation in two new labs. Those outcomes use different downstream heads and training signals, so the paper should be read as a family of connected systems rather than one model used identically for every task.
2 V-JEPA 2: Scaling Self-Supervised Video Pretraining
V-JEPA 2 expands masked-feature pretraining through more data, larger encoders, longer training and progressive spatial/temporal resolution. The first stage remains action-free.
2.1 Methodology
Mask-Denoising in Representation Space.
A context encoder processes a masked clip, and a predictor estimates features at the hidden patch locations. A momentum target encoder supplies targets without receiving the same gradient update. Loss is applied to masked patches.
Equation 1
| (1) |
Architecture.
Video tubelets span two frames and 16-by-16 pixels. Three-dimensional rotary positional embeddings encode time, height and width separately and improve stability for large models.
Key Scaling Ingredients.
Four scaling changes work together: a larger data mixture, encoders reaching roughly one billion parameters, longer training and a late increase in resolution and clip length. The experiments quantify their cumulative effect.
Evaluation Protocol.
Design choices are compared using an average over six motion and appearance tasks. Each score trains a task-specific attentive head above a frozen encoder; the average is a chosen benchmark summary.
2.2 Scaling Self-Supervised Video Learning
Each scaling intervention improves the selected six-task average in the staged comparison. Because the interventions are applied cumulatively, the plot describes the tested recipe progression rather than proving independent universal gains.
2.3 Pretraining Dataset
The pretraining mixture combines public video sources with images, and uses retrieval-based curation to increase useful coverage.
| Source | Samples | Type | Total Hours | Apply Curation | Weight |
|---|---|---|---|---|---|
| SSv2 (Goyal et al., 2017) | 168K | EgoVideo | 168 | No | 0.056 |
| Kinetics (Carreira et al., 2019) | 733K | ExoVideo | 614 | No | 0.188 |
| Howto100M (Miech et al., 2019) | 1.1M | ExoVideo | 134K | No | 0.318 |
| YT-Temporal-1B (Zellers et al., 2022) | 19M | ExoVideo | 1.6M | Yes | 0.188 |
| ImageNet (Deng et al., 2009) | 1M | Images | n/a | No | 0.250 |
Scaling Dataset Size.
VideoMix 22M combines Kinetics, Something-Something-v 2, instructional and general videos, plus ImageNet images repeated across frames. Sampling weights are manually tuned. The mixture contains about 22 million training items; dataset size, video hours and repeated training exposure are distinct quantities.
Data Curation.
A pretrained image encoder embeds video scenes, then clustering retrieves scenes near selected target datasets. Removing validation-set overlaps and reweighting clusters are part of curation. Improved average scores show the value of this selected distribution, while larger models benefit differently from the full mixture.
2.4 Pretraining Recipe
Scaling Model Size.
Encoder size increases from roughly 300 million to one billion parameters while the pretraining predictor stays fixed. Motion and appearance benchmarks improve within this tested range.
Training Schedule.
A warmup–constant–decay schedule supports branching cooldown runs from a common checkpoint. Weight decay and target momentum remain fixed, simplifying the earlier V-JEPA schedules.
Efficient Progressive-Resolution Training.
Most training uses short, lower-resolution clips. Only the final cooldown raises spatial resolution and duration, saving substantial compute relative to using the final large inputs throughout training. The reported 8.4-fold saving is specific to that comparison.
Scaling temporal and spatial video resolution.
Longer clips help both pretraining and evaluation, but these are separate sources of improvement. Extending beyond 64 frames did not improve the selected understanding tasks in the reported experiments.
3 V-JEPA 2-AC: Learning an Action-Conditioned World Model
The pretrained masked predictor does not know which changes an agent can cause. V-JEPA 2-AC adds a frame-causal predictor conditioned on robot actions and end-effector state above the frozen visual encoder.
3.1 Action-Conditioned World Model Training
Robot recordings supply images and end-effector states. Actions are constructed from changes between successive states. “Unlabeled” here means no added task, reward or success labels; the method still uses robot interaction signals. Training uses short tabletop-arm clips from DROID.
Model inputs.
Inputs are four-second clips sampled at 4 frames per second, with seven-dimensional end-effector states describing position, orientation and gripper state. The action vector represents the change between adjacent states in the robot’s coordinate frame.
Loss function.
The frozen encoder turns individual frames into spatial feature maps. A predictor learns next-frame features from past features, states and actions. Training combines teacher-forced one-step prediction with a two-step rollout loss, so it sees some of the errors caused by feeding predictions back into itself.
Equations 2–4
| (2) |
| (3) |
| (4) |
Architecture.
A roughly 300 million-parameter transformer interleaves action, state and visual tokens. Block-causal attention exposes the current and previous timesteps while hiding future observations. Position embeddings track spatial and temporal locations.
3.2 Inferring Actions by Planning
Energy minimization.
Planning searches candidate action sequences for a predicted final feature map close to the encoded goal image. The cross-entropy method repeatedly samples and refines candidates. Only the first selected action is executed before a new observation triggers replanning. The objective measures learned feature similarity, so success depends on what the encoder retains.
Equation 5
| (5) |
4 Planning: Zero-shot Robot Control
Robot evaluations test reaching, grasping and pick-and-place in new laboratories using image goals and model-predictive control.
4.1 Experimental Setup
Baselines.
The comparisons include an Octo imitation-learning policy and an action-conditioned Cosmos video-generation model. Both are adapted using their stated DROID procedures. Their training histories, architectures and planning costs differ, making these system-level comparisons.
Robot deployment.
The same model weights and inference code run on Franka arms in two labs absent from DROID. A monocular RGB camera supplies observations. Actions are constrained to modest movements because large commands are outside the model’s well-covered distribution.
Start Frame Goal Frame
Start Frame Goal Frame
Start Frame Goal Frame
4.2 Results
Single-goal reaching.
Three reaching examples move the end-effector to within 4 cm of the visual goal. The illustrated energy landscape favors useful control directions. These examples demonstrate visual-feedback control, while camera-coordinate errors can make individual actions inefficient.
Prehensile manipulation.
Manipulation success is measured over ten trials per task with varied starting configurations. Pick-and-place receives two intermediate subgoal images plus the final goal, with a fixed switching schedule. This supplies part of the long-horizon task structure.
V-JEPA 2-AC leads the tested systems on these tasks, but performance varies with object geometry and gripper accuracy. Its reported planning cost is about 16 seconds per action; the evaluated Cosmos planner is much slower. These measurements are tied to the chosen hardware, sampling budget and control setup.



| Grasp | Reach w/ Obj. | Pick-&-Place | ||||||
|---|---|---|---|---|---|---|---|---|
| Method | Reach | Cup | Box | Cup | Box | Cup | Box | |
| Octo (Octo Model Team et al., 2024) | Lab 1 | 100% | 20% | 0% | 20% | 70% | 20% | 10% |
| Lab 2 | 100% | 10% | 0% | 10% | 70% | 10% | 10% | |
| Avg | 100% | 15% | 0% | 15% | 70% | 15% | 10% | |
| V-JEPA 2-AC (ours) | Lab 1 | 100% | 70% | 30% | 90% | 80% | 80% | 80% |
| Lab 2 | 100% | 60% | 20% | 60% | 70% | 80% | 50% | |
| Avg | 100% | 65% | 25% | 75% | 75% | 80% | 65% | |
| Lab 2 | Planning Details | Grasp | Pick-&-Place | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | #Samples | Iter. | Horizon | Time | Reach | Cup | Box | Cup | Box |
| Cosmos (Agarwal et al., 2025) | 80 | 10 | 1 | 4 min. | 80% | 0% | 20% | 0% | 0% |
| V-JEPA 2-AC (ours) | 800 | 10 | 1 | 16 sec. | 100% | 60% | 20% | 80% | 50% |
4.3 Limitations
Sensitivity to camera positioning.
Without camera calibration, the model must infer how image directions correspond to robot-coordinate actions. Camera placement was tried manually before selecting one for the experiments. Transfer to a new lab therefore does not imply arbitrary camera-view robustness.
Long horizon planning.
Longer rollouts accumulate prediction errors and make action search harder. The supplied subgoals and short planning horizons reduce that difficulty; autonomous long-horizon decomposition remains unresolved.
Image goals.
Goals are specified as images. Converting natural-language instructions into planning targets requires another alignment mechanism and is future work here.
5 Understanding: Probe-based Classification
A world model can only plan with information its representation preserves. Six classification tasks therefore test motion and appearance separately, using supervised probes above frozen encoders.
Attentive Probe.
A four-layer attentive probe combines token features and predicts task labels. Multiple temporal clips and spatial views improve coverage. This is a trained nonlinear readout rather than a parameter-free or strictly linear test.
| Motion Understanding | Appearance Understanding | |||||||
| Method | Param. | Avg. | SSv2 | Diving-48 | Jester | K400 | COIN | IN1K |
| Results Reported in the Literature | ||||||||
| VideoMAEv2 (Wang et al., 2023) | 1B | – | 56.1 | – | – | 82.8 | – | 71.4 |
| InternVideo2-1B (Wang et al., 2024b) | 1B | – | 67.3 | – | – | 87.9 | – | – |
| InternVideo2-6B (Wang et al., 2024b) | 6B | – | 67.7 | – | – | 88.8 | – | – |
| VideoPrism (Zhao et al., 2024) | 1B | – | 68.5 | 71.3 | – | 87.6 | – | – |
| Image Encoders Evaluated Using the Same Protocol | ||||||||
| DINOv2 (Darcet et al., 2024) | 1.1B | 81.1 | 50.7 | 82.5 | 93.4 | 83.6 | 90.7 | 86.1 |
| PEcoreG (Bolya et al., 2025) | 1.9B | 82.3 | 55.4 | 76.9 | 90.0 | 88.5 | 95.3 | 87.6∗ |
| SigLIP2 (Tschannen et al., 2025) | 1.2B | 81.1 | 49.9 | 75.3 | 91.0 | 87.3 | 95.1 | 88.0 |
| Video Encoders Evaluated Using the Same Protocol | ||||||||
| V-JEPA ViT-H (Bardes et al., 2024) | 600M | 85.2 | 74.3 | 87.9 | 97.7 | 84.5 | 87.1 | 80.0 |
| InternVideo2s2-1B (Wang et al., 2024b) | 1B | 87.0 | 69.7 | 86.4 | 97.0 | 89.4 | 93.8 | 85.8 |
| V-JEPA 2 ViT-L | 300M | 86.0 | 73.7 | 89.0 | 97.6 | 85.1 | 86.8 | 83.5 |
| V-JEPA 2 ViT-H | 600M | 86.4 | 74.0 | 89.8 | 97.7 | 85.3 | 87.9 | 83.8 |
| V-JEPA 2 ViT-g | 1B | 87.5 | 75.3 | 90.1 | 97.7 | 86.6 | 90.7 | 84.6 |
| V-JEPA 2 ViT-g384 | 1B | 88.2 | 77.3 | 90.2 | 97.8 | 87.3 | 91.1 | 85.1 |
Evaluation protocol.
A common evaluation recipe is applied to encoders trained with different data and supervision. This makes readouts more comparable while leaving the results as system-level comparisons. Some additional published scores use different heads.
Results.
The reported encoder improves strongly on motion-sensitive tasks and remains competitive on appearance tasks. Longer, higher-resolution inputs further improve the chosen average, so input coverage is part of the strongest configuration.
6 Prediction: Probe-based Action Anticipation
Action anticipation asks which labeled human action will begin after the visible context. It is evaluated separately from robot-action planning.
| Method | Param. | Action Anticipation | ||
|---|---|---|---|---|
| Verb | Noun | Action | ||
| InAViT (Roy et al., 2024) | 160M | 51.9 | 52.0 | 25.8 |
| Video-LLaMA (Zhang et al., 2023) | 7B | 52.9 | 52.0 | 26.0 |
| PlausiVL (Mittal et al., 2024) | 8B | 55.6 | 54.2 | 27.6 |
| Frozen Backbone | ||||
| V-JEPA 2 ViT-L | 300M | 57.8 | 53.8 | 32.7 |
| V-JEPA 2 ViT-H | 600M | 59.2 | 54.6 | 36.5 |
| V-JEPA 2 ViT-g | 1B | 61.2 | 55.7 | 38.0 |
| V-JEPA 2 ViT-g384 | 1B | 63.6 | 57.1 | 39.7 |
Task.
EPIC-KITCHENS-100 uses kitchen videos and fixed verb–noun action categories. The default prediction is one second before the action begins. Mean-class recall at five credits a correct category among five guesses and accounts for class imbalance.
Anticipation Probe.
An attentive head uses the frozen encoder’s observed features together with predicted features near the future action. Supervised training maps these features to the benchmark’s noun, verb and action labels.
Baselines.
Task-specific anticipation models and other visual encoders form the comparison set. Their additional modules and pretraining histories differ from the simple frozen-feature probe.
Results.
Scaling the encoder improves the reported anticipation scores, reaching 39.7 mean-class recall at five for actions. This metric is not 39.7% exact top-one prediction and does not establish open-vocabulary forecasting.
Limitations.
Errors remain in nouns, verbs and complete actions. Performance declines for longer anticipation times, and the fixed kitchen vocabulary excludes unseen action categories. Generalization to unrelated environments is not demonstrated by this benchmark.
7 Understanding : Video Question Answering
Language capabilities are added by aligning visual features with a pretrained language model using large paired image/video–text datasets. The initial encoder pretraining has no language supervision, but the resulting question-answering system receives substantial language alignment data.
7.1 Experiment Setup
Video Question Answering Tasks.
The benchmark suite includes physical, temporal, memory and general video questions. Some tasks can rely heavily on single-frame appearance; using several benchmarks helps expose that difference.
Visual Instruction Tuning.
A learnable projector converts visual patch features into language-model inputs. Progressive image-caption, image-question and video-question training aligns the modalities. Freezing the encoder isolates its initial features more clearly; unfreezing can improve the complete system.
| Method | Params Enc / LLM | Avg. |
PerceptionTest SFT / Acc |
Paired-Acc |
TempCompass multi-choice |
TVBench Acc |
Acc |
MVBench Acc | |
|---|---|---|---|---|---|---|---|---|---|
| Off-the-shelf image encoders | |||||||||
| DINOv2 ViT-g518 | 1.1B/7B | 45.7 | 67.1 | 22.4 | 62.3 | 26.8 | 47.6 | 32.0 | 61.8 |
| SigLIP2 ViT-g384 | 1.1B/7B | 48.1 | 72.4 | 26.2 | 66.8 | 25.7 | 48.7 | 33.2 | 64.0 |
| PE ViT-G/14448 | 1.9B/7B | 49.1 | 72.3 | 26.7 | 67.0 | 27.5 | 51.6 | 34.0 | 64.7 |
| V-JEPA 2 ViT-g512 | 1B/7B | 52.3 | 72.0 | 31.1 | 69.2 | 33.3 | 55.9 | 37.0 | 67.7 |
| Method | Params Enc / LLM | Avg. |
PerceptionTest SFT / Acc |
Paired-Acc |
TempCompass multi-choice |
TVBench Acc |
Acc |
MVBench Acc | |
|---|---|---|---|---|---|---|---|---|---|
| End-to-end Evaluation | |||||||||
| V-JEPA 2 ViT-L256 | 300M/7B | 51.7 | 74.6 | 32.3 | 70.1 | 30.2 | 50.9 | 36.5 | 67.1 |
| V-JEPA 2 ViT-H256 | 600M/7B | 52.0 | 74.7 | 30.6 | 70.9 | 29.8 | 54.6 | 35.1 | 68.0 |
| V-JEPA 2 ViT-g256 | 1B/7B | 52.3 | 75.5 | 31.9 | 70.7 | 28.3 | 54.2 | 37.3 | 68.3 |
| V-JEPA 2 ViT-g384 | 1B/7B | 54.0 | 76.5 | 33.0 | 71.7 | 33.1 | 56.5 | 39.0 | 68.5 |
| V-JEPA 2 ViT-g512 | 1B/7B | 54.4 | 77.7 | 33.7 | 71.6 | 32.3 | 57.5 | 38.5 | 69.5 |
7.2 Comparing with Image Encoders
The controlled comparison keeps the language backbone and 18 million paired samples fixed while changing visual encoders. V-JEPA 2 performs competitively, especially on temporal tests, with an exception on PerceptionTest. This is evidence that language-free visual pretraining can be aligned successfully afterward.
7.3 Scaling Vision Encoder Size and Input Resolution
Larger encoders and higher spatial resolution improve several video-question benchmarks within the controlled setup. They also change compute and visual token counts, which must be included in a practical tradeoff.
| Method | Params Enc / LLM | Avg. |
PerceptionTest Test Acc |
Paired-Acc |
TempCompass multi-choice |
Acc |
TVBench Acc |
MVBench Acc | |
|---|---|---|---|---|---|---|---|---|---|
| Video Language Models Results Reported in the Literature | |||||||||
| InternVL-2.5 (Chen et al., 2024) | 300M/7B | 52.1 | 68.9 | 39.9 | 68.3 | 24.3 | 29.4 | 61.6 | 72.6 |
| Qwen2VL (Wang et al., 2024a) | 675M/7B | 47.0 | 66.9 | 29.2 | 67.9 | 20.4 | 31.5 | 46.0 | 67.0 |
| Qwen2.5VL (Qwen Team et al., 2025) | 1B/7B | 49.7 | 70.5 | 36.7 | 71.7 | 24.5 | 24.6 | 50.5 | 69.6 |
| PLM 8B (Cho et al., 2025) | 1B/8B | 56.7 | 82.7 | 39.7 | 72.7 | 28.3 | 33.2 | 63.5 | 77.1 |
| V-JEPA 2 ViT-g384 LLama 3.1 8B | 1B/8B | 59.5 | 84.0 | 44.5 | 76.9 | 36.7 | 40.3 | 60.6 | 73.5 |
7.4 Improving the State-of-the-art by Scaling Data
The larger alignment experiment uses 88.5 million pairs and a different language backbone from the controlled study. It improves several reported benchmarks while leaving some PerceptionLM comparisons behind. Its gains therefore reflect the complete scaled system, not just the visual pretraining objective.
8 Related Work
World models and planning.
The work connects representation-space dynamics with model-predictive control. Its distinction is large observational pretraining followed by action-conditioned learning and evaluation on new real-robot environments. Planning and policy learning can be complementary rather than mutually exclusive.
Vision-Language-Action models for Robotic Control.
Imitation policies learn demonstrated actions; this world model learns transition effects from recordings that may include unsuccessful interactions. That difference motivates exploring candidate actions at inference time, but does not prove planning can extrapolate safely beyond its training distribution.
Vision Foundation Models.
Visual understanding benchmarks measure important perceptual skills, while control tests whether a representation supports interaction. The paper uses both to examine different requirements of a world model.
9 Conclusion
The combined results support a staged approach from large-scale self-supervised perception to a limited action-conditioned planning system. Strong benchmarks coexist with camera sensitivity, explicit visual goals, slow planning and short-horizon manipulation.
Future work.
Future work includes hierarchical prediction, longer horizons, language-defined goals and larger encoders. None of those extensions should be assumed solved by the current robot demonstrations.
Acknowledgements
Appendix A V-JEPA 2 Pretraining
A.1 Pretraining Hyperparameters
Training switches from constant learning rate to cooldown after periodic task evaluations show diminishing gains. Cooldown also increases resolution and clip length. Several ablations instead use an abbreviated earlier-style schedule; their compute budgets differ from the final models.
| Parameter | Primary Phase | Cooldown Phase |
|---|---|---|
| Number of frames | 16 | 64 |
| Frames per Second | 4.0 | 4.0 |
| Crop Size | 256 | [256, 384, 512] |
| Random Resize Aspect Ratio | [0.75 1.35] | [0.75, 1.35] |
| Random Resize Scale | [0.3, 1.0] | [0.3, 1.0] |
| Steps | Variable | 12000 |
| Warmup Steps | 12000 | N/A |
| Batch Size (global) | 3072 | 3072 |
| Starting Learning Rate | 1e-4 | 5.25e-4 |
| Final Learning Rate | 5.25e-4 | 1e-6 |
| Weight Decay | 0.04 | 0.04 |
| EMA | 0.99925 | 0.99925 |
| Spatial Mask Scale | [0.15, 0.7] | [0.15, 0.7] |
| Temporal Mask Scale | [1.0, 1.0] | [1.0, 1.0] |
| Mask Aspect Ratio | [0.75 1.5] | [0.75, 1.5] |
| Tubelet Size | 2 | 2 |
| Patch Size | 16 | 16 |
| Parameter | Abbreviated Recipe |
|---|---|
| Number of frames | 16 |
| Frames per Second | 4.0 |
| Crop Size | 256 |
| Random Resize Aspect Ratio | [0.75 1.35] |
| Random Resize Scale | [0.3, 1.0] |
| Steps | 90000 |
| Warmup Steps | 12000 |
| Batch Size (global) | 3072 |
| Starting Learning Rate | 2e-4 |
| Larning Rate | 6.25e-4 |
| Final Learning Rate | 1e-6 |
| Starting Weight Decay | 0.04 |
| Final Weight Decay | 0.4 |
| Starting EMA | 0.999 |
| Final EMA | 1.0 |
| Spatial Mask Scale | [0.15, 0.7] |
| Temporal Mask Scale | [1.0, 1.0] |
| Mask Aspect Ratio | [0.75 1.5] |
| Tubelet Size | 2 |
| Patch Size | 16 |
A.2 Pretraining data
Scene extraction and DINOv 2 embeddings support cluster-based retrieval from YT1B. Retained clusters are weighted toward selected target datasets. This makes curation a designed training distribution, rather than a neutral sample of all internet video.
| Retrieval Target | Cluster Count | Number of Scenes | Retrieval Weight |
|---|---|---|---|
| Uncurated YT1B | 1.5M | 316M | |
| K710 | 170k | 100M | 0.7 |
| SSv2 | 41k | 19M | 0.125 |
| COIN | 37k | 21M | 0.125 |
| EpicKitchen | 4k | 13k | 0.05 |
| Final Curated (includes duplicates) | 210k | 115M |
A.3 Scaling Model Size
The appendix specifies encoder widths, depths and head counts across model sizes. The pretraining predictor is held fixed, separating encoder scaling from predictor-capacity changes.
| Model | Params | Width | Depth | Heads | MLP | Embedder |
| Encoders: | ||||||
| ViT-L | 300M | 1024 | 24 | 16 | 4096 | strided conv |
| ViT-H | 600M | 1280 | 32 | 16 | 5120 | strided conv |
| ViT-g | 1B | 1408 | 40 | 22 | 6144 | strided conv |
| Predictor: | ||||||
| ViT-s | 22M | 384 | 12 | 12 | 1536 | N.A. |
A.4 Additional Results
A.4.1 Effect of Data Curation
Curation effects vary with model size and mixture. At smaller scale, adding curated data does not improve every task; at larger scale and longer training, the curated full mixture is more consistently beneficial.
| Training Data | IN1K | COIN | SSv2 | K400 |
|---|---|---|---|---|
| ViT-L | ||||
| Uncurated YT1B | 80.6 | 83.2 | 70.9 | 82.9 |
| Curated YT1B | 80.8 | 86.5 | 73.1 | 84.6 |
| Mixed+Uncurated YT1B | 82.9 | 86.25 | 73.3 | 83.0 |
| VM22M | 82.9 | 86.0 | 72.8 | 83.7 |
| ViT-g | ||||
| Uncurated YT1B | 81.8 | 86.4 | 73.6 | 85.1 |
| Curated YT1B | 81.7 | 88.4 | 74.8 | 86.5 |
| Mixed+Uncurated YT1B | 83.7 | 88.5 | 75.5 | 85.9 |
| VM22M | 83.9 | 89.2 | 75.6 | 86.2 |
A.4.2 Effect of Long Training Schedule and cooldown
The strongest gains appear after cooldown, which combines learning-rate reduction with longer clips. A checkpoint before cooldown should not be treated as the completed final recipe.
| Training Stage | IN1K | COIN | SSv2 | K400 |
|---|---|---|---|---|
| Phase 1 (epoch 800, no cooldown) | 83.8 | 89.1 | 75.1 | 85.8 |
| Phase 2 (annealed, resolution) | 84.6 | 90.7 | 75.3 | 86.6 |
| Phase 2 (annealed, resolution) | 85.1 | 90.2 | 76.5 | 87.3 |
A.4.3 Effect of Video Length at Evaluation.
Longer evaluation clips improve the tested single-clip protocol substantially. This ablation intentionally differs from standard multi-clip evaluation because of memory limits.
Appendix B V-JEPA 2-AC Post-training
B.1 Post-Training Hyperparameters
Post-training uses four-second DROID clips and separate predictor optimization. Only left-camera views are used; mixing left and right views without conditioning on camera position degraded performance. This anticipates the deployment camera-sensitivity result.
B.2 Robot Task Definitions
The manipulation setup specifies goal images and a fixed subgoal schedule. The reported planner uses 800 candidates, ten refinement rounds and a one-step horizon. That short horizon works with the selected relatively greedy tasks and supplied subgoals.
B.3 Visualizing World Model Predictions
A separate deterministic frame decoder visualizes frozen encoder features and predicted robot rollouts. It is not used for control. Examples distinguish closed- and open-gripper outcomes, while later rollout frames show accumulated prediction error.
B.4 Assessing Sensitivity to Camera Position
Sweeping camera angles reveals systematic errors in the model’s inferred action coordinate frame. A fitted rotation could calibrate those errors, but the paper states that such a calibration procedure was not used in its reported task experiments. Camera selection itself was manual.
Equations
Appendix C Visual Classification
The classification appendix specifies probe capacity, task-specific view coverage and hyperparameter selection.
C.1 Hyperparameters
Probe Architecture.
Four attention blocks and a final classifier are trained above frozen encoder tokens. A learned query in the last cross-attention block produces the task representation.
Evaluation setup parameters.
Tasks use different clip counts, frame rates and spatial views. The largest configuration also receives higher input resolution and longer clips, so compare scores with their full evaluation settings.
ImageNet evaluation.
ImageNet images are repeated into still clips and use a single view. No new temporal content is created by this formatting step.
Jester and Diving-48 evaluation.
Jester and Diving-48 probes combine several encoder layers and train longer. Their evaluation recipe is not identical to the other classification tasks.
Optimization.
Several classifier heads with different learning rates and weight decay are trained, and the best is reported. This hyperparameter search is part of the downstream evaluation budget.
| Parameter | Default (K400) | ImageNet | SSv2 | COIN | Jester/Diving-48 |
|---|---|---|---|---|---|
| Number of frames | 16 | 16 | 16 | 32 | 32 |
| Segments / Clip | 8 | 1 | 2 | 8 | 4 |
| Views / Segment | 3 | 1 | * | * | * |
| Frame Step | 4 | n/a | * | * | 2 |
| Epochs | 20 | * | * | * | 100 |
| Batch Size (global) | 256 | 1024 | * | 128 | 128 |
| Resolution | * | * | * | * | |
| Classifier Heads | 20 (4x5) | * | * | * | 3 (3x1) |
| Classifier Learning Rates | [5e-3 3e-3 1e-3 3e-4 1e-4] | * | * | * | [1e-3 3e-4 1e-4] |
| Classifier Weight Decay | [.8 .4 .1 .01] | * | * | * | [.8] |
| Encoder | # Layers | Attended Layers |
|---|---|---|
| ViT-L | 24 | 17, 19, 21, 23 |
| ViT-H | 32 | 25, 27, 29, 31 |
| ViT-g | 40 | 24, 29, 34, 39 |
C.2 Additional Results
Probe Size.
The four-layer probe generally improves over a single cross-attention block, with task-dependent exceptions. Better readout capacity can increase a frozen encoder’s measured score.
Impact of encoder multilayer.
Combining intermediate and final encoder features benefits the tested motion tasks. Choosing an extraction layer is therefore another meaningful evaluation decision.
| Model | Encoder Layers | Diving-48 | Jester |
|---|---|---|---|
| ViT-g | 1 | 82.9 | 96.1 |
| ViT-g | 4 | 86.7 | 97.6 |
| Motion Understanding | Appearance Understanding | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Probe Layers | Avg. | SSv2 | Diving-48 | Jester | K400 | COIN | IN1K |
| ViT-L | 1 | 84.0 | 72.0 | 83.2 | 97.7 | 83.3 | 85.9 | 81.8 |
| ViT-L | 4 | 85.6 | 73.6 | 87.1 | 97.7 | 85.1 | 86.8 | 83.5 |
| ViT-g | 1 | 86.0 | 74.8 | 85.3 | 97.8 | 85.6 | 88.9 | 83.5 |
| ViT-g | 4 | 87.0 | 75.6 | 86.7 | 97.6 | 86.6 | 90.7 | 84.6 |
Appendix D Action Anticipation
The anticipation appendix separates head design, training-time sampling and the default one-second validation protocol.
D.1 Hyperparameters
Probe Architecture.
A four-block attentive head uses learned queries for the benchmark’s classification outputs. The encoder and predictor supply features rather than being replaced by a task-specific generative decoder.
Evaluation setup parameters.
Focal loss addresses imbalanced action classes. Training varies anticipation timing, while validation fixes a one-second gap. Context length, frame rate and resolution are explicitly specified.
| Parameter | EK100 |
|---|---|
| Train Anticipation time | 0.25s – 1.75s |
| Train Anticipation point | 0.0 – 0.25 |
| Val Anticipation time | 1s |
| Val Anticipation point | 0.0 |
| Number of frames | 32 |
| Frames per second | 8 |
| Epochs | 20 |
| Warmup epochs | 0 |
| Batch Size (global) | 128 |
| Classifier Heads | 20 (4x5) |
| Classifier Learning Rates | [5e-3 3e-3 1e-3 3e-4 1e-4] |
| Classifier Weight Decay | [1e-4 1e-3 1e-2 1e-1] |
D.2 Additional results
Impact of Architecture.
Observed encoder features account for much of the anticipation performance. Adding predicted features gives a smaller consistent gain. This limits the extent to which a high benchmark score can be attributed to future simulation alone.
| Action Anticipation | ||||
| Encoder | Predictor | Verb | Noun | Action |
| ✓ | 61.3 | 57.0 | 39.1 | |
| ✓ | 48.7 | 34.7 | 20.2 | |
| ✓ | ✓ | 63.6 | 57.1 | 39.7 |
Impact of Input Resolution.
Additional context, frame rate and resolution help up to saturation. The selected recipe balances temporal coverage against input cost.
Longer-term Prediction.
Recall declines sharply as the anticipation gap grows from one to several seconds. A more distant future is increasingly ambiguous in this dataset.
Analysis of Failure Cases.
Failures are broken down by noun, verb and combined action. Correctly identifying one component does not necessarily recover the complete action category.
Appendix E Video Question Answering
The language appendix distinguishes a controlled 18 million-pair study from the 88.5 million-pair scaling study. Different language backbones and training setups mean these are not a single-variable data-size experiment.
E.1 Processing Images and Videos as Input
High-resolution images are processed with a tiled strategy rather than only duplicated at low resolution. Video token counts are limited through frame selection or pooling to fit the language-model budget.
E.2 Controlled Setup
Training details
The controlled setup progressively trains image alignment and then video tasks using Qwen 2-7B-Instruct. It explores frozen and unfrozen vision encoders; the language model and projector are trained in both cases.
Baselines.
Image-based encoders process video frames separately at their native resolutions. Pooling ratios are adjusted to keep language input lengths similar, while pretraining histories still differ.
| Model | Pooling Ratio | Vision Tokens | BS | Stage 1/1.5 LR | Stage 2/3 LR | WD | Frames |
| V-JEPA 2 ViT-L256 | 4 | 4096 | 256 | 1e-5 | 5e-6 | 0.0 | 128 |
| V-JEPA 2 ViT-H256 | 4 | 4096 | 256 | ||||
| V-JEPA 2 ViT-g256 | 4 | 4096 | 256 | ||||
| V-JEPA 2 ViT-g384 | 4 | 9216 | 256 | ||||
| V-JEPA 2 ViT-g512 | 8 | 8192 | 256 | ||||
| DINOv2518 | 16 | 10952 | 256 | 1e-5 | 5e-6 | 0.0 | 128 |
| SigLIP2384 | 16 | 5832 | 256 | ||||
| PE448 | 16 | 8192 | 256 |
Evaluation.
The controlled study uses uniform frame sampling and the lmms-eval framework. PerceptionTest additionally fine-tunes on its training split, which should be distinguished from untouched zero-shot benchmark evaluation.
Impact of Video Duration
Increasing frame coverage helps the tested V-JEPA 2 language system more than its frozen DINOv 2 comparison. The result belongs to this aligned model and benchmark setup.
E.3 Data scaling setup
| Parameter | Values |
|---|---|
| Common parameters | |
| Crop Size | 384 |
| Video Frames per Second | 1 |
| Sampling method | Uniform |
| Seed | 777 |
| Stage 1 | |
| Steps | 16000 |
| Warmup Steps | 96 |
| Batch Size (global) | 128 |
| Learning Rate | 1e-4 |
| Final Learning Rate | 1e-6 |
| Weight Decay | 0.05 |
| Max sequence length | 1920 |
| Stage 2 | |
| Steps | 35000 |
| Warmup Steps | 200 |
| Batch Size (global) | 2048 |
| Learning Rate | 4e-5 |
| Final Learning Rate | 4e-7 |
| Weight Decay | 0.05 |
| Max sequence length | 6400 |
| Image tiles | 16 |
| Video frames | 16 |
| Stage 3 | |
| Steps | 28000 |
| Early stopping step | 22000 |
| Warmup Steps | 168 |
| Batch Size (global) | 2048 |
| Learning Rate | 1e-5 |
| Final Learning Rate | 1e-7 |
| Weight Decay | 0.05 |
| Max sequence length | 12800 |
| Image tiles | 32 |
| Video frames | 32 |
Training details
The scaling setup uses Llama 3.1-8B-Instruct, a simple projector and the full 88.5 million paired samples. Its larger training infrastructure and different backbone are part of the strongest reported system.
Baselines.
Most large-model baseline numbers are taken from their papers; MVP is rerun by the authors. These mixed sources should be considered when assessing small score differences.
Evaluation.
The scaling evaluation uses 32 frames, unlike the 128-frame controlled evaluation. The appendix makes that protocol difference explicit.