Literature
Display
← All papers

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

Mahmoud Assran · Adrien Bardes · David Fan · Quentin Garrido · Russell Howes · Mojtaba Komeili · Matthew Muckley · Ammar Rizvi · Claire Roberts · Koustuv Sinha · Artem Zholus · Sergio Arnaud · Abha Gejji · Ada Martin · Francois Robert Hogan · Daniel Dugas · Piotr Bojanowski · Vasil Khalidov · Patrick Labatut · Francisco Massa · Marc Szafraniec · Kapil Krishnakumar · Yong Li · Xiaodong Ma · Sarath Chandar · Franziska Meier · Yann LeCun · Michael Rabbat · Nicolas Ballas

Preprint posted · arXiv (preprint) ↗

World models13 min text · CondensedOriginal paper ↗
Contents

At a glance

First learn what changes in videos, then learn how robot actions change a scene. V-JEPA 2 pretrains a video encoder using masked-feature prediction on a large mixture of videos and images. A separate action-conditioned predictor, V-JEPA 2-AC, is then trained on fewer than 62 hours of robot interaction recordings while the encoder stays frozen. At deployment it tests candidate actions in feature space, compares predicted outcomes with a goal image and replans after acting. The robot results require recorded robot states/actions, chosen camera placement and supplied goal images; they do not arise from passive video alone. Classification, action-anticipation and language-alignment experiments evaluate additional capabilities separately.

1 Introduction

The central contribution is a staged route from observational video pretraining to an action-conditioned world model. Scaling improves visual features; robot interaction data then connects those features to control commands. Experiments cover frozen recognition, human-action anticipation, language-aligned video question answering and manipulation in two new labs. Those outcomes use different downstream heads and training signals, so the paper should be read as a family of connected systems rather than one model used identically for every task.

Refer to caption
Figure 1: V-JEPA 2 Overview. Leveraging 1M hours of internet-scale video and 1M images, we pretrain the V-JEPA 2 video model using a visual mask denoising objective (Bardes et al., 2024; Assran et al., 2023), and leverage this model for downstream tasks such as action classification, object recognition, action anticipation, and Video Question Answering by aligning the model with an LLM backbone. After pretraining, we can also freeze the video encoder and train a new action-conditioned predictor with a small amount of robot interaction data on top of the learned representations, and leverage this action-conditioned model, V-JEPA 2-AC, for downstream robot manipulation tasks using planning within a model predictive control loop.
Refer to caption
 
Refer to caption
Figure 2: Multistage training. (Left) We first pretrain the V-JEPA 2 video encoder on internet-scale image and video data using a visual mask denoising objective (Bardes et al., 2024; Assran et al., 2023). A video clip is patchified into a sequence of tokens and a mask is applied by dropping a subset of the tokens. The encoder then processes the masked video sequence and outputs an embedding vector for each input token. Next, the outputs of the encoder are concatenated with a set of learnable mask tokens that specify the position of the masked patches, and subsequently processed by the predictor. The outputs of the predictor are then regressed to the prediction targets using an L1 loss. The prediction targets are computed by an ema-encoder, the weights of which are defined as an exponential moving average of the encoder weights. (Right) After pretraining, we freeze the video encoder and learn a new action-conditioned predictor, V-JEPA 2-AC, on top of the learned representation. We leverage an autoregressive feature prediction objective that involves predicting the representations of future video frames conditioned on past video frames, actions, and end-effector states. Our action-conditioned predictor uses a block-causal attention pattern such that each patch feature at a given time step can attend to the patch features, actions, and end-effector states from current and previous time steps.

2 V-JEPA 2: Scaling Self-Supervised Video Pretraining

V-JEPA 2 expands masked-feature pretraining through more data, larger encoders, longer training and progressive spatial/temporal resolution. The first stage remains action-free.

2.1 Methodology

Mask-Denoising in Representation Space.

A context encoder processes a masked clip, and a predictor estimates features at the hidden patch locations. A momentum target encoder supplies targets without receiving the same gradient update. Loss is applied to masked patches.

Equation 1
minimizeθ,ϕ,Δy∥Pϕ​(Δy,Eθ​(x))−sg​(Eθ¯​(y))∥1,\text{minimize}_{\theta,\phi,\Delta_{y}}\quad\lVert P_{\phi}(\Delta_{y},E_{\theta}(x))-\text{sg}(E_{\overline{\theta}}(y))\rVert_{1}, (1)

Architecture.

Video tubelets span two frames and 16-by-16 pixels. Three-dimensional rotary positional embeddings encode time, height and width separately and improve stability for large models.

Key Scaling Ingredients.

Four scaling changes work together: a larger data mixture, encoders reaching roughly one billion parameters, longer training and a late increase in resolution and clip length. The experiments quantify their cumulative effect.

Original figure
Figure 3: Scaling Ingredients. The effects of scaling interventions on average accuracy across 6 image and video classification tasks (SSv2, Diving-48, Jester, Kinetics, COIN, ImageNet) using a ViT-L/16 model as baseline.

Evaluation Protocol.

Design choices are compared using an average over six motion and appearance tasks. Each score trains a task-specific attentive head above a frozen encoder; the average is a chosen benchmark summary.

2.2 Scaling Self-Supervised Video Learning

Each scaling intervention improves the selected six-task average in the staged comparison. Because the interventions are applied cumulatively, the plot describes the tested recipe progression rather than proving independent universal gains.

2.3 Pretraining Dataset

The pretraining mixture combines public video sources with images, and uses retrieval-based curation to increase useful coverage.

Table 1: VideoMix22M (VM22M) Pretraining Dataset. To build our observation pretraining dataset, we combined four different video sources and one image dataset. We use a source-specific sampling probability during training and apply retrieval-based curation on YT1B to reduce noisy content (e.g., cartoon- or clipart-style).
Source Samples Type Total Hours Apply Curation Weight
SSv2 (Goyal et al., 2017) 168K EgoVideo 168 No 0.056
Kinetics (Carreira et al., 2019) 733K ExoVideo 614 No 0.188
Howto100M (Miech et al., 2019) 1.1M ExoVideo 134K No 0.318
YT-Temporal-1B (Zellers et al., 2022) 19M ExoVideo 1.6M Yes 0.188
ImageNet (Deng et al., 2009) 1M Images n/a No 0.250

Scaling Dataset Size.

VideoMix 22M combines Kinetics, Something-Something-v 2, instructional and general videos, plus ImageNet images repeated across frames. Sampling weights are manually tuned. The mixture contains about 22 million training items; dataset size, video hours and repeated training exposure are distinct quantities.

Original figure
Original figure
Figure 4: Data Scaling & Curation. We train and compare models on different data-mixes. Models are ViT-L/16 trained for 90K iterations using a cosine learning schedule following Bardes et al. (2024). (Left) We compare the performance of a ViT-L/16 model pretrained on the VM2M dataset and our VM22M dataset. Training on the VM22M dataset leads to a +1+1 point improvement in average performance. Performance improvement is more pronounced on appearance-based tasks such as Kinetics-400, COIN, and ImageNet (Right) We compare the performance of a ViT-L/16 model pretrained on YT1B and a model pretrained on our Curated-YT1B dataset, which leverages our cluster-based curation. Training on the curated dataset leads to a +1.4+1.4 point improvement on average performances, showing the effectiveness of data-curation.

Data Curation.

A pretrained image encoder embeds video scenes, then clustering retrieves scenes near selected target datasets. Removing validation-set overlaps and reweighting clusters are part of curation. Improved average scores show the value of this selected distribution, while larger models benefit differently from the full mixture.

2.4 Pretraining Recipe

Scaling Model Size.

Encoder size increases from roughly 300 million to one billion parameters while the pretraining predictor stays fixed. Motion and appearance benchmarks improve within this tested range.

Original figure
Original figure
Original figure
Figure 5: Model Scaling. We explore the impact of scaling model size and input video resolution. All models are trained on the VideoMix22M pretraining dataset. (Left) Average performance across six understanding tasks as a function of model scale. Models are trained with a constant learning rate until performance plateaus on downstream tasks. We then cool down the model using 64 frames at 256×256256\times 256 resolution and report post-cooldown performance. Scaling the model size from 300M to 1B parameters yields a +1.7+1.7 point average improvement. (Middle) Training times (GPU-days) for ViT-g on A100 GPUs when training videos at 384×384384\times 384 resolution with different numbers of frames per clip. We compare progressive resolution training (252K iterations at 16 frames / 256×256256\times 256 resolution, followed by 12K cooldown iterations at 384×384384\times 384 resolution) to the projected time for full-resolution training. Progressive training provides up to 8×\times speedup, significantly reducing the pretraining compute requirement. (Right) Effect of inscreasing video duration at cooldown on downstream performance for ViT-g. Even when only using 16-frame clips during inference/evaluation, increasing video duration during the cooldown phase of training improves average task performance by +0.7+0.7 points.

Training Schedule.

A warmup–constant–decay schedule supports branching cooldown runs from a common checkpoint. Weight decay and target momentum remain fixed, simplifying the earlier V-JEPA schedules.

Efficient Progressive-Resolution Training.

Most training uses short, lower-resolution clips. Only the final cooldown raises spatial resolution and duration, saving substantial compute relative to using the final large inputs throughout training. The reported 8.4-fold saving is specific to that comparison.

Scaling temporal and spatial video resolution.

Longer clips help both pretraining and evaluation, but these are separate sources of improvement. Extending beyond 64 frames did not improve the selected understanding tasks in the reported experiments.

3 V-JEPA 2-AC: Learning an Action-Conditioned World Model

The pretrained masked predictor does not know which changes an agent can cause. V-JEPA 2-AC adds a frame-causal predictor conditioned on robot actions and end-effector state above the frozen visual encoder.

3.1 Action-Conditioned World Model Training

Robot recordings supply images and end-effector states. Actions are constructed from changes between successive states. “Unlabeled” here means no added task, reward or success labels; the method still uses robot interaction signals. Training uses short tabletop-arm clips from DROID.

Model inputs.

Inputs are four-second clips sampled at 4 frames per second, with seven-dimensional end-effector states describing position, orientation and gripper state. The action vector represents the change between adjacent states in the robot’s coordinate frame.

Loss function.

The frozen encoder turns individual frames into spatial feature maps. A predictor learns next-frame features from past features, states and actions. Training combines teacher-forced one-step prediction with a two-step rollout loss, so it sees some of the errors caused by feeding predictions back into itself.

Equations 2–4
ℒteacher-forcing​(ϕ)≔1T​∑k=1T∥z^k+1−zk+1∥1=1T​∑k=1T‖Pϕ​((at,st,E⁡(xt))t≤k)−E⁡(xk+1)‖1,\mathcal{L}_{\text{teacher-forcing}}(\phi)\coloneqq\frac{1}{T}\sum^{T}_{k=1}\lVert\hat{z}_{k+1}-z_{k+1}\rVert_{1}=\frac{1}{T}\sum^{T}_{k=1}\left\lVert P_{\phi}\left(\left(a_{t},s_{t},E(x_{t})\right)_{t\leq k}\right)-E(x_{k+1})\right\rVert_{1}, (2)
ℒrollout(ϕ)≔∥Pϕ(a1:T,s1,z1)−zT+1∥1.\mathcal{L}_{\text{rollout}}(\phi)\coloneqq\lVert P_{\phi}(a_{1:T},s_{1},z_{1})-z_{T+1}\rVert_{1}. (3)
L⁡(ϕ)≔ℒteacher-forcing​(ϕ)+ℒrollout​(ϕ),L(\phi)\coloneqq\mathcal{L}_{\text{teacher-forcing}}(\phi)+\mathcal{L}_{\text{rollout}}(\phi), (4)
Refer to caption
 
Refer to caption
Figure 6: V-JEPA 2-AC training. V-JEPA 2-AC is trained in an autoregressive fashion, utilizing a teacher forcing loss and a rollout loss. (Left) In the teacher forcing loss, the predictor takes the encoding of the current frame representation as input and learns to predict the representation of the next timestep. (Right) The rollout loss involves feeding the predictor’s output back as input, allowing the model to be trained to predict several timesteps ahead. By optimizing the sum of these two losses, V-JEPA 2-AC enhances its ability to accurately forecast the future by reducing error accumulation during rollouts.

Architecture.

A roughly 300 million-parameter transformer interleaves action, state and visual tokens. Block-causal attention exposes the current and previous timesteps while hiding future observations. Position embeddings track spatial and temporal locations.

3.2 Inferring Actions by Planning

Energy minimization.

Planning searches candidate action sequences for a predicted final feature map close to the encoded goal image. The cross-entropy method repeatedly samples and refines candidates. Only the first selected action is executed before a new observation triggers replanning. The objective measures learned feature similarity, so success depends on what the encoder retains.

Equation 5
ℰ(a^1:T;zk,sk,zg)≔∥P(a^1:T;sk,zk)−zg∥1,\mathcal{E}(\hat{a}_{1:T};\ z_{k},s_{k},z_{g})\coloneqq\lVert P(\hat{a}_{1:T};s_{k},z_{k})-z_{g}\rVert_{1}, (5)

4 Planning: Zero-shot Robot Control

Robot evaluations test reaching, grasping and pick-and-place in new laboratories using image goals and model-predictive control.

Refer to caption
Figure 7: Planning. We plan an action sequence for a fixed time horizon TT by minimizing the L1 distance between the world model’s imagined state representation TT steps into the future and its goal representation. The L1 loss is optimized with respect to the actions (ak)k∈[T](a_{k})_{k\in[T]} using the cross-entropy method (Rubinstein, 1997). Specifically, in each planning step, we sample the action coordinates at each point in the planning horizon from a sequence of Gaussian distributions initialized with zero mean and unit variance. The population statistics of the top-k actions trajectories are used to update the mean and variance of the Gaussian distributions. This process is repeated for several iterations before finally returning the mean of the sequence of Gaussians as the selected action trajectory.

4.1 Experimental Setup

Baselines.

The comparisons include an Octo imitation-learning policy and an action-conditioned Cosmos video-generation model. Both are adapted using their stated DROID procedures. Their training histories, architectures and planning costs differ, making these system-level comparisons.

Robot deployment.

The same model weights and inference code run on Franka arms in two labs absent from DROID. A monocular RGB camera supplies observations. Actions are constrained to modest movements because large commands are outside the model’s well-covered distribution.

Start Frame   Goal Frame
Refer to caption Refer to caption
Original figure

Start Frame   Goal Frame
Refer to caption Refer to caption
Original figure

Start Frame   Goal Frame
Refer to caption Refer to caption
Original figure

Figure 8: Single-Goal Reaching. Single-goal reaching involves moving the end-effector to a desired location in space based on a single goal image. This task measures for a basic understanding of actions as well as a 3D spatial understanding of the scene, including depth, from the monocular RGB camera. In each step, we use V-JEPA 2-AC to plan a sequence of actions by minimizing the L1 distance between the model’s imagined future state representations and its representation of the goal frame. The first action is then executed before re-planning in the next time step. During planning, we only sample individual actions in the L1-Ball of radius 0.0750.075 centered at the origin. Thus, the maximum achievable decrease in cartesian distance to the goal in a single step is 0.130.13 (∼\sim13 cm).

4.2 Results

Single-goal reaching.

Three reaching examples move the end-effector to within 4 cm of the visual goal. The illustrated energy landscape favors useful control directions. These examples demonstrate visual-feedback control, while camera-coordinate errors can make individual actions inefficient.

Original figure
Figure 9: V-JEPA 2-AC Energy Landscape. Energy landscape for single-goal reaching task with respect to end-effector cartesian-control action (sweeping Δ​x\Delta x and Δ​y\Delta y while holding Δ​z=0\Delta z=0 fixed); ground truth action relating goal image to start frame is located at (Δ​x,Δ​y)=(0,−0.1)(\Delta x,\Delta y)=(0,-0.1). We see that the energy function achieves its minimum around (Δ​x,Δ​y)≈(0,−0.05)(\Delta x,\Delta y)\approx(0,-0.05), indicating that the model has learned to reasonably infer the effect of actions without requiring precision sensing.

Prehensile manipulation.

Manipulation success is measured over ten trials per task with varied starting configurations. Pick-and-place receives two intermediate subgoal images plus the final goal, with a fixed switching schedule. This supplies part of the long-horizon task structure.

V-JEPA 2-AC leads the tested systems on these tasks, but performance varies with object geometry and gripper accuracy. Its reported planning cost is about 16 seconds per action; the evaluated Cosmos planner is much slower. These measurements are tied to the chosen hardware, sampling budget and control setup.

Figure 10: Pick-&-Place. Closed-loop robot execution of V-JEPA 2-AC for multi-goal pick-&-place tasks. Highlighted frames indicate when the model achieves a sub-goal and switches to the next goal. The first goal image shows the object being grasped, the second goal image shows the object in the vicinity of the desired location, and the third goal image shows the object placed in the desired position. The model first optimizes actions with respect to the first sub-goal for 4 time-steps before automatically switching to the second sub-goal for the next 10 time-steps, and finally the third goal for the last 4 time-steps. Robot actions are inferred through goal-conditioned planning. The V-JEPA 2-AC model is able to perform zero-shot pick-&-place tasks on two Franka arms in different labs, with various object configurations and cluttered environments.
Refer to caption
Refer to caption
Refer to caption
Table 2: Zero-Shot Robot Manipulation. All models are deployed zero-shot on two Franka arms with RobotiQ grippers located in different labs. Given image-goals for each considered task, all models run closed loop to infer a sequence of actions to achieve the goal. Success rates are reported out of 10 trials with various permutations to the task across trials (e.g., object location, starting pose, etc.).
Grasp Reach w/ Obj. Pick-&-Place
Method Reach Cup Box Cup Box Cup Box
Octo (Octo Model Team et al., 2024) Lab 1 100% 20% 0% 20% 70% 20% 10%
Lab 2 100% 10% 0% 10% 70% 10% 10%
Avg 100% 15% 0% 15% 70% 15% 10%
V-JEPA 2-AC (ours) Lab 1 100% 70% 30% 90% 80% 80% 80%
Lab 2 100% 60% 20% 60% 70% 80% 50%
Avg 100% 65% 25% 75% 75% 80% 65%
Table 3: Planning Performance. Comparing closed-loop robot manipulation using MPC with V-JEPA 2-AC world model and Cosmos world model. In both cases we leverage the cross-entropy method (Rubinstein, 1997) for optimizing the sequence of actions using a single NVIDIA RTX 4090 GPU. For each robot skill, we evaluate each model across 10 tasks and average the results. With 80 samples, 10 refinement steps, and a planning horizon of 1, it takes 4 minutes to compute a single action in each planning step with Cosmos, which is an action-conditioned video generation model based on latent diffusion. Note that under a planning time of 4 minutes per action, a full pick & place trajectory takes over one hour. By contrast, with 10×\times more samples in each refinement step, the V-JEPA 2-AC world model requires only 16 seconds per action and leads to higher performance across all considered robot skills.
Lab 2 Planning Details Grasp Pick-&-Place
Method #Samples Iter. Horizon Time Reach Cup Box Cup Box
Cosmos (Agarwal et al., 2025) 80 10 1 4 min. 80% 0% 20% 0% 0%
V-JEPA 2-AC (ours) 800 10 1 16 sec. 100% 60% 20% 80% 50%

4.3 Limitations

Sensitivity to camera positioning.

Without camera calibration, the model must infer how image directions correspond to robot-coordinate actions. Camera placement was tried manually before selecting one for the experiments. Transfer to a new lab therefore does not imply arbitrary camera-view robustness.

Long horizon planning.

Longer rollouts accumulate prediction errors and make action search harder. The supplied subgoals and short planning horizons reduce that difficulty; autonomous long-horizon decomposition remains unresolved.

Image goals.

Goals are specified as images. Converting natural-language instructions into planning targets requires another alignment mechanism and is future work here.

5 Understanding: Probe-based Classification

A world model can only plan with information its representation preserves. Six classification tasks therefore test motion and appearance separately, using supervised probes above frozen encoders.

Attentive Probe.

A four-layer attentive probe combines token features and predicts task labels. Multiple temporal clips and spatial views improve coverage. This is a trained nonlinear readout rather than a parameter-free or strictly linear test.

Table 4: Action and Object Classification. We report the classification performance of V-JEPA 2 models pretrained on 64 frames at resolution 256×256256\times 256 for all models, except V-JEPA 2 ViT-g384 which was pretrained at resolution 384×384384\times 384, on action and object classification, and compare their performance with state-of-art image and video encoders. All models follow the same evaluation protocol except for V-JEPA 2 ViT-g384. We use 256×256256\times 256 resolution with 16×2×316\times 2\times 3 inputs for SSv2 (16 frames clip, 2 temporal crops, 3 spatial crops), 16×8×316\times 8\times 3 for K400, 32×8×332\times 8\times 3 for COIN and 32×4×332\times 4\times 3 for Diving-48 and Jester. V-JEPA 2 ViT-g384 uses a higher resolution of 384×384384\times 384 for all six tasks, and additionally uses 64×2×364\times 2\times 3 inputs for SSv2 and 32x8x3 inputs for COIN. Our V-JEPA 2 ViT-g significantly outperforms other vision encoders on motion understanding tasks and is competitive on appearance tasks. It achieves the best average performance of 87.587.5 across all image and videos encoders. V-JEPA 2 ViT-g384 further improves results consistently across tasks, reaching 88.288.2 average performance. ∗*: PEcoreG achieves an accuracy 89.8%89.8\% on ImageNet using an attentive probe and input resoluton of 448448px (Bolya et al., 2025). We use an input resolution of 256256px and a different probe architecture in our case.
Motion Understanding Appearance Understanding
Method Param. Avg. SSv2 Diving-48 Jester K400 COIN IN1K
Results Reported in the Literature
VideoMAEv2 (Wang et al., 2023) 1B – 56.1 – – 82.8 – 71.4
InternVideo2-1B (Wang et al., 2024b) 1B – 67.3 – – 87.9 – –
InternVideo2-6B (Wang et al., 2024b) 6B – 67.7 – – 88.8 – –
VideoPrism (Zhao et al., 2024) 1B – 68.5 71.3 – 87.6 – –
Image Encoders Evaluated Using the Same Protocol
DINOv2 (Darcet et al., 2024) 1.1B 81.1 50.7 82.5 93.4 83.6 90.7 86.1
PEcoreG (Bolya et al., 2025) 1.9B 82.3 55.4 76.9 90.0 88.5 95.3 87.6∗
SigLIP2 (Tschannen et al., 2025) 1.2B 81.1 49.9 75.3 91.0 87.3 95.1 88.0
Video Encoders Evaluated Using the Same Protocol
V-JEPA ViT-H (Bardes et al., 2024) 600M 85.2 74.3 87.9 97.7 84.5 87.1 80.0
InternVideo2s2-1B (Wang et al., 2024b) 1B 87.0 69.7 86.4 97.0 89.4 93.8 85.8
V-JEPA 2 ViT-L 300M 86.0 73.7 89.0 97.6 85.1 86.8 83.5
V-JEPA 2 ViT-H 600M 86.4 74.0 89.8 97.7 85.3 87.9 83.8
V-JEPA 2 ViT-g 1B 87.5 75.3 90.1 97.7 86.6 90.7 84.6
V-JEPA 2 ViT-g384 1B 88.2 77.3 90.2 97.8 87.3 91.1 85.1

Evaluation protocol.

A common evaluation recipe is applied to encoders trained with different data and supervision. This makes readouts more comparable while leaving the results as system-level comparisons. Some additional published scores use different heads.

Results.

The reported encoder improves strongly on motion-sensitive tasks and remains competitive on appearance tasks. Longer, higher-resolution inputs further improve the chosen average, so input coverage is part of the strongest configuration.

6 Prediction: Probe-based Action Anticipation

Action anticipation asks which labeled human action will begin after the visible context. It is evaluated separately from robot-action planning.

Table 5: Prediction: Human Action Anticipation. Comparison with the state-of-the-art on the EK100 Action Anticipation benchmark. We report mean-class recall-at-5 for verb, noun and action on the validation set of EK100. V-JEPA 2 performance scales linearly with model size and outperforms previous state-of-the-art across all model sizes.
Method Param. Action Anticipation
Verb Noun Action
InAViT (Roy et al., 2024) 160M 51.9 52.0 25.8
Video-LLaMA (Zhang et al., 2023) 7B 52.9 52.0 26.0
PlausiVL (Mittal et al., 2024) 8B 55.6 54.2 27.6
Frozen Backbone
V-JEPA 2 ViT-L 300M 57.8 53.8 32.7
V-JEPA 2 ViT-H 600M 59.2 54.6 36.5
V-JEPA 2 ViT-g 1B 61.2 55.7 38.0
V-JEPA 2 ViT-g384 1B 63.6 57.1 39.7

Task.

EPIC-KITCHENS-100 uses kitchen videos and fixed verb–noun action categories. The default prediction is one second before the action begins. Mean-class recall at five credits a correct category among five guesses and accounts for class imbalance.

Anticipation Probe.

An attentive head uses the frozen encoder’s observed features together with predicted features near the future action. Supervised training maps these features to the benchmark’s noun, verb and action labels.

Baselines.

Task-specific anticipation models and other visual encoders form the comparison set. Their additional modules and pretraining histories differ from the simple frozen-feature probe.

Results.

Scaling the encoder improves the reported anticipation scores, reaching 39.7 mean-class recall at five for actions. This metric is not 39.7% exact top-one prediction and does not establish open-vocabulary forecasting.

Figure 11: Visualization of EK100 prediction. (Left): four selected frames from the context frames. (Middle): model predictions, ordered by likelihood. (Right): following frame after the 1 second anticipation time. We show two examples where the model is successful and one example where the model fails.
Refer to caption

Limitations.

Errors remain in nouns, verbs and complete actions. Performance declines for longer anticipation times, and the fixed kitchen vocabulary excludes unseen action categories. Generalization to unrelated environments is not demonstrated by this benchmark.

7 Understanding : Video Question Answering

Language capabilities are added by aligning visual features with a pretrained language model using large paired image/video–text datasets. The initial encoder pretraining has no language supervision, but the resulting question-answering system receives substantial language alignment data.

7.1 Experiment Setup

Video Question Answering Tasks.

The benchmark suite includes physical, temporal, memory and general video questions. Some tasks can rely heavily on single-frame appearance; using several benchmarks helps expose that difference.

Visual Instruction Tuning.

A learnable projector converts visual patch features into language-model inputs. Progressive image-caption, image-question and video-question training aligns the modalities. Freezing the encoder isolates its initial features more clearly; unfreezing can improve the complete system.

Table 6: Comparison between off-the-shelf image encoders and V-JEPA 2 in frozen encoder setting. All experiments use the same LLM backbone (Qwen2-7B-Instruct), data, and training setup with a frozen vision encoder. PerceptionTest accuracy is reported on the validation set post SFT.
Method Params Enc / LLM Avg.

PerceptionTest

SFT / Acc

MVP

Paired-Acc

TempCompass

multi-choice

TemporalBench

(MBA-short QA)

TVBench

Acc

TOMATO

Acc

MVBench

Acc

Off-the-shelf image encoders
DINOv2 ViT-g518 1.1B/7B 45.7 67.1 22.4 62.3 26.8 47.6 32.0 61.8
SigLIP2 ViT-g384 1.1B/7B 48.1 72.4 26.2 66.8 25.7 48.7 33.2 64.0
PE ViT-G/14448 1.9B/7B 49.1 72.3 26.7 67.0 27.5 51.6 34.0 64.7
V-JEPA 2 ViT-g512 1B/7B 52.3 72.0 31.1 69.2 33.3 55.9 37.0 67.7
Table 7: Scaling Vision Encoder Size and Resolution. We scale the vision encoder from 300 million to 1 billion parameters and input resolution from 256 pixels to 512 pixels. All experiments use the same LLM backbone (Qwen2-7B-Instruct), data, and end-to-end training (unfrozen vision encoder) setup. PerceptionTest accuracy is reported on the validation set post SFT. Increasing V-JEPA 2 encoder scale and resolution improve average performances on VidQA tasks.
Method Params Enc / LLM Avg.

PerceptionTest

SFT / Acc

MVP

Paired-Acc

TempCompass

multi-choice

TemporalBench

(MBA-short QA)

TVBench

Acc

TOMATO

Acc

MVBench

Acc

End-to-end Evaluation
V-JEPA 2 ViT-L256 300M/7B 51.7 74.6 32.3 70.1 30.2 50.9 36.5 67.1
V-JEPA 2 ViT-H256 600M/7B 52.0 74.7 30.6 70.9 29.8 54.6 35.1 68.0
V-JEPA 2 ViT-g256 1B/7B 52.3 75.5 31.9 70.7 28.3 54.2 37.3 68.3
V-JEPA 2 ViT-g384 1B/7B 54.0 76.5 33.0 71.7 33.1 56.5 39.0 68.5
V-JEPA 2 ViT-g512 1B/7B 54.4 77.7 33.7 71.6 32.3 57.5 38.5 69.5

7.2 Comparing with Image Encoders

The controlled comparison keeps the language backbone and 18 million paired samples fixed while changing visual encoders. V-JEPA 2 performs competitively, especially on temporal tests, with an exception on PerceptionTest. This is evidence that language-free visual pretraining can be aligned successfully afterward.

7.3 Scaling Vision Encoder Size and Input Resolution

Larger encoders and higher spatial resolution improve several video-question benchmarks within the controlled setup. They also change compute and visual token counts, which must be included in a practical tradeoff.

Table 8: Comparison with state-of-the-art. We use the full 88.5M-sample alignment dataset and train using the same methodology as PLM 8B Cho et al. (2025), using a Llama 3.1 backbone. We observe significant improvements in downstream evaluations, obtaining state-of-the-art results in the 8B model class. PerceptionTest accuracy is reported on the test set with SFT for V-JEPA 2; all other results are zero-shot.
Method Params Enc / LLM Avg.

PerceptionTest

Test Acc

MVP

Paired-Acc

TempCompass

multi-choice

TemporalBench

(MBA-short QA)

TOMATO

Acc

TVBench

Acc

MVBench

Acc

≤8​B\leq 8B Video Language Models Results Reported in the Literature
InternVL-2.5 (Chen et al., 2024) 300M/7B 52.1 68.9 39.9 68.3 24.3 29.4 61.6 72.6
Qwen2VL (Wang et al., 2024a) 675M/7B 47.0 66.9 29.2 67.9 20.4 31.5 46.0 67.0
Qwen2.5VL (Qwen Team et al., 2025) 1B/7B 49.7 70.5 36.7 71.7 24.5 24.6 50.5 69.6
PLM 8B (Cho et al., 2025) 1B/8B 56.7 82.7 39.7 72.7 28.3 33.2 63.5 77.1
V-JEPA 2 ViT-g384 LLama 3.1 8B 1B/8B 59.5 84.0 44.5 76.9 36.7 40.3 60.6 73.5

7.4 Improving the State-of-the-art by Scaling Data

The larger alignment experiment uses 88.5 million pairs and a different language backbone from the controlled study. It improves several reported benchmarks while leaving some PerceptionLM comparisons behind. Its gains therefore reflect the complete scaled system, not just the visual pretraining objective.

8 Related Work

World models and planning.

The work connects representation-space dynamics with model-predictive control. Its distinction is large observational pretraining followed by action-conditioned learning and evaluation on new real-robot environments. Planning and policy learning can be complementary rather than mutually exclusive.

Vision-Language-Action models for Robotic Control.

Imitation policies learn demonstrated actions; this world model learns transition effects from recordings that may include unsuccessful interactions. That difference motivates exploring candidate actions at inference time, but does not prove planning can extrapolate safely beyond its training distribution.

Vision Foundation Models.

Visual understanding benchmarks measure important perceptual skills, while control tests whether a representation supports interaction. The paper uses both to examine different requirements of a world model.

9 Conclusion

The combined results support a staged approach from large-scale self-supervised perception to a limited action-conditioned planning system. Strong benchmarks coexist with camera sensitivity, explicit visual goals, slow planning and short-horizon manipulation.

Future work.

Future work includes hierarchical prediction, longer horizons, language-defined goals and larger encoders. None of those extensions should be assumed solved by the current robot demonstrations.

Acknowledgements

Appendix A V-JEPA 2 Pretraining

A.1 Pretraining Hyperparameters

Training switches from constant learning rate to cooldown after periodic task evaluations show diminishing gains. Cooldown also increases resolution and clip length. Several ablations instead use an abbreviated earlier-style schedule; their compute budgets differ from the final models.

Table 9: Pretraining Hyperparameters. Common parameters for pretraining large computer vision models. We report these parameters for both the primary training phase and the cooldown phase.
Parameter Primary Phase Cooldown Phase
Number of frames 16 64
Frames per Second 4.0 4.0
Crop Size 256 [256, 384, 512]
Random Resize Aspect Ratio [0.75 1.35] [0.75, 1.35]
Random Resize Scale [0.3, 1.0] [0.3, 1.0]
Steps Variable 12000
Warmup Steps 12000 N/A
Batch Size (global) 3072 3072
Starting Learning Rate 1e-4 5.25e-4
Final Learning Rate 5.25e-4 1e-6
Weight Decay 0.04 0.04
EMA 0.99925 0.99925
Spatial Mask Scale [0.15, 0.7] [0.15, 0.7]
Temporal Mask Scale [1.0, 1.0] [1.0, 1.0]
Mask Aspect Ratio [0.75 1.5] [0.75, 1.5]
Tubelet Size 2 2
Patch Size 16 16
Table 10: Abbreviated Pretraining Hyperparameters. Common parameters for pretraining large computer vision models, targeting our abbreviated recipe.
Parameter Abbreviated Recipe
Number of frames 16
Frames per Second 4.0
Crop Size 256
Random Resize Aspect Ratio [0.75 1.35]
Random Resize Scale [0.3, 1.0]
Steps 90000
Warmup Steps 12000
Batch Size (global) 3072
Starting Learning Rate 2e-4
Larning Rate 6.25e-4
Final Learning Rate 1e-6
Starting Weight Decay 0.04
Final Weight Decay 0.4
Starting EMA 0.999
Final EMA 1.0
Spatial Mask Scale [0.15, 0.7]
Temporal Mask Scale [1.0, 1.0]
Mask Aspect Ratio [0.75 1.5]
Tubelet Size 2
Patch Size 16

A.2 Pretraining data

Scene extraction and DINOv 2 embeddings support cluster-based retrieval from YT1B. Retained clusters are weighted toward selected target datasets. This makes curation a designed training distribution, rather than a neutral sample of all internet video.

Table 11: Data Curation Statistics. We summarized the number of extracted scenes and hours of videos across clusters extracted from YT1B. The final line includes duplicates among retrievals of K710, SSv2, COIN, and EpicKitchen.
Retrieval Target Cluster Count Number of Scenes Retrieval Weight
Uncurated YT1B 1.5M 316M
K710 170k 100M 0.7
SSv2 41k 19M 0.125
COIN 37k 21M 0.125
EpicKitchen 4k 13k 0.05
Final Curated (includes duplicates) 210k 115M

A.3 Scaling Model Size

The appendix specifies encoder widths, depths and head counts across model sizes. The pretraining predictor is held fixed, separating encoder scaling from predictor-capacity changes.

Table 12: Model architecture details. Family of encoders and predictor architectures used during V-JEPA 2 pretraining, with some of the major parameters.
Model Params Width Depth Heads MLP Embedder
Encoders: Eθ​(⋅)E_{\theta}(\cdot)
ViT-L 300M 1024 24 16 4096 2×16×162\times 16\times 16 strided conv
ViT-H 600M 1280 32 16 5120 2×16×162\times 16\times 16 strided conv
ViT-g 1B 1408 40 22 6144 2×16×162\times 16\times 16 strided conv
Predictor: Pϕ​(⋅)P_{\phi}(\cdot)
ViT-s 22M 384 12 12 1536 N.A.

A.4 Additional Results

A.4.1 Effect of Data Curation

Curation effects vary with model size and mixture. At smaller scale, adding curated data does not improve every task; at larger scale and longer training, the curated full mixture is more consistently beneficial.

Table 13: Effects of Data Curation on Video Understanding. Results are reported at both the ViT-L and ViT-g model scales. Models at both scales were pretrained using the abbreviated schedule of Bardes et al. (2024).
Training Data IN1K COIN SSv2 K400
ViT-L
Uncurated YT1B 80.6 83.2 70.9 82.9
Curated YT1B 80.8 86.5 73.1 84.6
Mixed+Uncurated YT1B 82.9 86.25 73.3 83.0
VM22M 82.9 86.0 72.8 83.7
ViT-g
Uncurated YT1B 81.8 86.4 73.6 85.1
Curated YT1B 81.7 88.4 74.8 86.5
Mixed+Uncurated YT1B 83.7 88.5 75.5 85.9
VM22M 83.9 89.2 75.6 86.2
Original figure
Figure 12: Effect of data curation for V-JEPA 2 pre-training. We show model performance averaged across the IN1K, COIN, SSv2, and K400 tasks as a function of pre-training “epochs” (equivalent to 300 optimization steps). Models trained with and without uncurated data achieve similar performance until epoch 600, at which point the performance of the model trained with uncurated YT1B beings degrading.

A.4.2 Effect of Long Training Schedule and cooldown

The strongest gains appear after cooldown, which combines learning-rate reduction with longer clips. A checkpoint before cooldown should not be treated as the completed final recipe.

Table 14: Effects of Long Training and cooldown. Results are reported at ViT-g model with cooldown at different resolutions.
Training Stage IN1K COIN SSv2 K400
Phase 1 (epoch 800, no cooldown) 83.8 89.1 75.1 85.8
Phase 2 (annealed, 256×256256\times 256 resolution) 84.6 90.7 75.3 86.6
Phase 2 (annealed, 384×384384\times 384 resolution) 85.1 90.2 76.5 87.3

A.4.3 Effect of Video Length at Evaluation.

Longer evaluation clips improve the tested single-clip protocol substantially. This ablation intentionally differs from standard multi-clip evaluation because of memory limits.

Original figure
Figure 13: Effect of video duration during evaluation. Task performance further improves by running inference on longer video clips. All evaluations use ViT-g models that were annealed with 64 frames at resolution 256×256256\times 256. Due to memory constraints, results are reported using a single clip evaluation protocol. Increasing the number of frames processed at inference time boosts average performance by up to +9.7+9.7 points.

Appendix B V-JEPA 2-AC Post-training

B.1 Post-Training Hyperparameters

Post-training uses four-second DROID clips and separate predictor optimization. Only left-camera views are used; mixing left and right views without conditioning on camera position degraded performance. This anticipates the deployment camera-sensitivity result.

B.2 Robot Task Definitions

The manipulation setup specifies goal images and a fixed subgoal schedule. The reported planner uses 800 candidates, ten refinement rounds and a one-step horizon. That short horizon works with the selected relatively greedy tasks and supplied subgoals.

Refer to caption
(a) Grasp Cup
Refer to caption
(b) Reach with Cup
Refer to caption
(c) Pick and Place Cup
Figure 14: Prehensile Manipulation Task Definition. Start and goal frames for prehensile manipulation tasks with a cup in Lab 1. For the grasp and reach with object tasks the model is shown a single goal image. For the pick-and-place tasks we present two sub-goal images to the model in addition to the final goal. The first goal image shows the object being grasped, the second goal image shows the object in the vicinity of the goal position. The model first optimizes actions with respect to the first sub-goal for 4 time-steps before automatically switching to the second sub-goal for the next 10 time-steps, and finally the third goal for the last 4 time-steps.

B.3 Visualizing World Model Predictions

A separate deterministic frame decoder visualizes frozen encoder features and predicted robot rollouts. It is not used for control. Examples distinguish closed- and open-gripper outcomes, while later rollout frames show accumulated prediction error.

Robot observations

Reconstructions

from V-JEPA 2

Reconstructions

from V-JEPA 2-AC

Refer to caption
(a) Comparing accuracy of predictions to ground truth trajectory. (Top Row) Video frames of a ground-truth trajectory from a robot in our lab. (Middle row) Each frame is encoded by V-JEPA 2 encoder, and then decoded using the feedforward frame decoder. Reconstructions of the V-JEPA 2 representations show that the encoder captures the salient parts of the scene necessary for vision-based control; blurry background generation can be partially attributed to the low-capacity of our feedforward frame decoder. (Bottom Row) Autoregressive rollout produced by V-JEPA 2-AC world model using the ground-truth action sequence given first frame as context, and then decoded using feedforward frame decoder. Reconstructions of the V-JEPA 2-AC rollout show that the action-conditioned world model successfully animates the robot while keeping the background and non-interactated objects (e.g., the shelf) unaffected. However, we do observe error accumulation as the world model predicts the location of the cup to be slightly lower than that of the real trajectory in the final frame.

V-JEPA 2-AC imagination: move with closed gripper

V-JEPA 2-AC imagination: move with open gripper

Refer to caption
(b) Ablating predictions with open versus closed gripper. We explore how the V-JEPA 2-AC predictions change when driving the model with identical action sequences, but in one cause using a closed gripper (top row) and in the other with an open gripper (bottom row). The world model predicts the location of the cup to be unchanged across time steps when using an open gripper action sequence, suggesting a reasonable understanding of intuitive physics (e.g., object constancy, shape constancy, and gravity).
Figure 15: Decoding representations. To visualize the model’s predictions, we train a frame decoder on the Droid dataset that maps the V-JEPA 2 representations to human-interpretable pixels. The decoder is a feedforward network (fully deterministic regression model that does not use any sampling internally) trained with a mean-squared error pixel reconstruction loss. By applying the frame decoder, trained on the V-JEPA 2 encoder, to the representations produced by the V-JEPA 2-AC predictor, we can visualize world model rollouts for various action sequences.

B.4 Assessing Sensitivity to Camera Position

Sweeping camera angles reveals systematic errors in the model’s inferred action coordinate frame. A fitted rotation could calibrate those errors, but the paper states that such a calibration procedure was not used in its reported task experiments. Camera selection itself was manual.

Original figure
Figure 16: Sensitivity to camera position. Rotation error (in the x-y plane) of the action coordinate axis inferred by V-JEPA 2-AC as a function of camera position, with 0 degrees corresponding to a camera located at the robot base, and 90 degrees corresponding to a camera located left of the robot base. While ideally, the model’s inferred coordinate axis would be invariant to camera position, here we observe that the model’s inferred coordinate axis is sensitive to the camera position.
Equations
W⋆=argminW∈ℝ2×2​∥A​W−B∥2.W^{\star}=\underset{W\in\mathbb{R}^{2\times 2}}{\text{argmin}}\ \lVert AW-B\rVert_{2}.
W⋆≈W¯⋆=[cos⁡θ−sin⁡θsin⁡θcos⁡θ],W^{\star}\approx\overline{W}^{\star}=\begin{bmatrix}\cos\theta&-\sin\theta\\ \sin\theta&\cos\theta\end{bmatrix},

Appendix C Visual Classification

The classification appendix specifies probe capacity, task-specific view coverage and hyperparameter selection.

C.1 Hyperparameters

Probe Architecture.

Four attention blocks and a final classifier are trained above frozen encoder tokens. A learned query in the last cross-attention block produces the task representation.

Evaluation setup parameters.

Tasks use different clip counts, frame rates and spatial views. The largest configuration also receives higher input resolution and longer clips, so compare scores with their full evaluation settings.

ImageNet evaluation.

ImageNet images are repeated into still clips and use a single view. No new temporal content is created by this formatting step.

Jester and Diving-48 evaluation.

Jester and Diving-48 probes combine several encoder layers and train longer. Their evaluation recipe is not identical to the other classification tasks.

Optimization.

Several classifier heads with different learning rates and weight decay are trained, and the best is reported. This hyperparameter search is part of the downstream evaluation budget.

Table 15: Visual Classification eval params. Default parameters used for the visual classification evaluations, with non-default values for each eval (* denotes default). All attentive probes use 4 transformer blocks with 16 heads.
Parameter Default (K400) ImageNet SSv2 COIN Jester/Diving-48
Number of frames 16 16 16 32 32
Segments / Clip 8 1 2 8 4
Views / Segment 3 1 * * *
Frame Step 4 n/a * * 2
Epochs 20 * * * 100
Batch Size (global) 256 1024 * 128 128
Resolution 256×256256\times 256 * * * *
Classifier Heads 20 (4x5) * * * 3 (3x1)
Classifier Learning Rates [5e-3 3e-3 1e-3 3e-4 1e-4] * * * [1e-3 3e-4 1e-4]
Classifier Weight Decay [.8 .4 .1 .01] * * * [.8]
Table 16: Input layers for Jester/Diving-48. For each encoder size, indices of the four encoder layers whose tokens are used as input to the linear classifier in the Jester and Diving-48 evaluations.
Encoder # Layers Attended Layers
ViT-L 24 17, 19, 21, 23
ViT-H 32 25, 27, 29, 31
ViT-g 40 24, 29, 34, 39

C.2 Additional Results

Probe Size.

The four-layer probe generally improves over a single cross-attention block, with task-dependent exceptions. Better readout capacity can increase a frozen encoder’s measured score.

Impact of encoder multilayer.

Combining intermediate and final encoder features benefits the tested motion tasks. Choosing an extraction layer is therefore another meaningful evaluation decision.

Table 17: Encoder Multilayer Ablation. We vary the number of encoder layers fed to the attentive probe. We report the classification performances of attentive probes trained on top of V-JEPA 2 with 16 frames at 256×256256\times 256 resolution.
Model Encoder Layers Diving-48 Jester
ViT-g 1 82.9 96.1
ViT-g 4 86.7 97.6
Table 18: Probe Size Ablation. We vary the number of layers in the attentive probe. We report the classification performances of attentive probes trained on top of V-JEPA 2 with 16 frames at 256×256256\times 256 resolution.
Motion Understanding Appearance Understanding
Model Probe Layers Avg. SSv2 Diving-48 Jester K400 COIN IN1K
ViT-L 1 84.0 72.0 83.2 97.7 83.3 85.9 81.8
ViT-L 4 85.6 73.6 87.1 97.7 85.1 86.8 83.5
ViT-g 1 86.0 74.8 85.3 97.8 85.6 88.9 83.5
ViT-g 4 87.0 75.6 86.7 97.6 86.6 90.7 84.6

Appendix D Action Anticipation

The anticipation appendix separates head design, training-time sampling and the default one-second validation protocol.

D.1 Hyperparameters

Probe Architecture.

A four-block attentive head uses learned queries for the benchmark’s classification outputs. The encoder and predictor supply features rather than being replaced by a task-specific generative decoder.

Evaluation setup parameters.

Focal loss addresses imbalanced action classes. Training varies anticipation timing, while validation fixes a one-second gap. Context length, frame rate and resolution are explicitly specified.

Table 19: Action Anticipation Evaluation params. Default parameters used for the EK100 Action Anticipation evaluation.
Parameter EK100
Train Anticipation time 0.25s – 1.75s
Train Anticipation point 0.0 – 0.25
Val Anticipation time 1s
Val Anticipation point 0.0
Number of frames 32
Frames per second 8
Epochs 20
Warmup epochs 0
Batch Size (global) 128
Classifier Heads 20 (4x5)
Classifier Learning Rates [5e-3 3e-3 1e-3 3e-4 1e-4]
Classifier Weight Decay [1e-4 1e-3 1e-2 1e-1]

D.2 Additional results

Impact of Architecture.

Observed encoder features account for much of the anticipation performance. Adding predicted features gives a smaller consistent gain. This limits the extent to which a high benchmark score can be attributed to future simulation alone.

Table 20: EK100: Impact of Anticipation Probe Inputs. We investigate the impact of providing the outputs of the V-JEPA 2 encoder, predictor, or both, to the action anticipation probe. Using encoder outputs already leads to competitive performance on the EK100 task. Adding the predictor provides a small but consistent improvement across action, verb and object categories.
Action Anticipation
Encoder Predictor Verb Noun Action
✓ 61.3 57.0 39.1
✓ 48.7 34.7 20.2
✓ ✓ 63.6 57.1 39.7
Original figure
Original figure
Original figure
Figure 17: Protocol ablation for action anticipation on EK100. (Left) Performance with respect to the number of context frames used for action anticipation. (Middle) Performance with respect to the frame rate (fps) used for inference; number of context frames fixed at 32. (Right) Performance with respect to the spatial resolution (height and width) of the context frames used for action anticipation.
Original figure
Original figure
Figure 18: (Left): Impact of longer-term anticipation times. Performance on EK100 action anticipation, at several recall values and anticipation times. (Right): Distribution of success and failure cases of V-JEPA 2. Calculated on the validation set of EK100. VNA means that verb, noun and action are all correctly classified by the model. An X symbol means that the corresponding attribute is not correctly classified by the model. Note: the probe is composed of 3 independent classifiers for verb, noun and action, hence why the model can have a different prediction for the action and for the verb/noun pair.

Impact of Input Resolution.

Additional context, frame rate and resolution help up to saturation. The selected recipe balances temporal coverage against input cost.

Longer-term Prediction.

Recall declines sharply as the anticipation gap grows from one to several seconds. A more distant future is increasingly ambiguous in this dataset.

Analysis of Failure Cases.

Failures are broken down by noun, verb and combined action. Correctly identifying one component does not necessarily recover the complete action category.

Appendix E Video Question Answering

The language appendix distinguishes a controlled 18 million-pair study from the 88.5 million-pair scaling study. Different language backbones and training setups mean these are not a single-variable data-size experiment.

E.1 Processing Images and Videos as Input

High-resolution images are processed with a tiled strategy rather than only duplicated at low resolution. Video token counts are limited through frame selection or pooling to fit the language-model budget.

E.2 Controlled Setup

Training details

The controlled setup progressively trains image alignment and then video tasks using Qwen 2-7B-Instruct. It explores frozen and unfrozen vision encoders; the language model and projector are trained in both cases.

Baselines.

Image-based encoders process video frames separately at their native resolutions. Pooling ratios are adjusted to keep language input lengths similar, while pretraining histories still differ.

Table 21: Hyperparameters for controlled comparison of vision encoders. We use each vision encoder with its native pretrained input resolution.
Model Pooling Ratio Vision Tokens BS Stage 1/1.5 LR Stage 2/3 LR WD Frames
V-JEPA 2 ViT-L256 4 4096 256 1e-5 5e-6 0.0 128
V-JEPA 2 ViT-H256 4 4096 256
V-JEPA 2 ViT-g256 4 4096 256
V-JEPA 2 ViT-g384 4 9216 256
V-JEPA 2 ViT-g512 8 8192 256
DINOv2518 16 10952 256 1e-5 5e-6 0.0 128
SigLIP2384 16 5832 256
PE448 16 8192 256
Original figure
Figure 19: Impact of video duration during visual instruction tuning. We investigate the effect of increasing the number of frames during visual instruction tuning, where the encoder is frozen. We observe that with more frames, V-JEPA 2 performance linearly increases compared to DINOv2, an SSL-based image encoder, showing the potential of V-JEPA 2 to scale with more frames.

Evaluation.

The controlled study uses uniform frame sampling and the lmms-eval framework. PerceptionTest additionally fine-tunes on its training split, which should be distinguished from untouched zero-shot benchmark evaluation.

Impact of Video Duration

Increasing frame coverage helps the tested V-JEPA 2 language system more than its frozen DINOv 2 comparison. The result belongs to this aligned model and benchmark setup.

E.3 Data scaling setup

Table 22: Data scaling training parameters.
Parameter Values
Common parameters
Crop Size 384
Video Frames per Second 1
Sampling method Uniform
Seed 777
Stage 1
Steps 16000
Warmup Steps 96
Batch Size (global) 128
Learning Rate 1e-4
Final Learning Rate 1e-6
Weight Decay 0.05
Max sequence length 1920
Stage 2
Steps 35000
Warmup Steps 200
Batch Size (global) 2048
Learning Rate 4e-5
Final Learning Rate 4e-7
Weight Decay 0.05
Max sequence length 6400
Image tiles 16
Video frames 16
Stage 3
Steps 28000
Early stopping step 22000
Warmup Steps 168
Batch Size (global) 2048
Learning Rate 1e-5
Final Learning Rate 1e-7
Weight Decay 0.05
Max sequence length 12800
Image tiles 32
Video frames 32

Training details

The scaling setup uses Llama 3.1-8B-Instruct, a simple projector and the full 88.5 million paired samples. Its larger training infrastructure and different backbone are part of the strongest reported system.

Baselines.

Most large-model baseline numbers are taken from their papers; MVP is rerun by the authors. These mixed sources should be considered when assessing small score differences.

Evaluation.

The scaling evaluation uses 32 frames, unlike the 128-frame controlled evaluation. The appendix makes that protocol difference explicit.

Concept demonstration

Open image ↗