Overview
Intuition: several valid captions may describe the same event with different words. VL-JEPA predicts a vector representing the answer’s meaning, then decodes that vector into text only when needed. Candidate labels and retrieved descriptions can instead be compared directly in the shared embedding space. This edition follows arXiv v 2, whose author list includes Yejin Bang. All original figures, detailed comparisons and bibliography are retained.
1 Introduction
VL-JEPA separates semantic prediction from the final choice of words. A visual encoder and a query-conditioned predictor produce an answer embedding; a text encoder defines the target space; an optional decoder turns a predicted embedding into language. This can support classification, retrieval and selected text outputs with one architecture. The paper motivates the design using streaming video, where repeatedly decoding nearly unchanged content can waste work.
2 Methodology
The visual input and textual query feed a predictor trained to match an encoding of the target answer. The target representation is learned with a contrastive objective that discourages collapse. The text decoder is not the main prediction target: it translates the predicted semantic vector into words at inference. This division allows a downstream decision without decoding when an application only needs a candidate label or similarity score. The training signal still comes from image/video–text pairs.
3 Implementation of VL-JEPA
Implementation choices determine how much computation is saved and what prior knowledge is inherited. The next sections separate the model components from the data and two-stage training procedure.
3.1 Model Architecture
The system combines a pretrained visual encoder, a smaller transformer predictor, a sentence-level text encoder and a lightweight text decoder. The predictor is initialized using selected language-model layers. Parameter counts should distinguish the full system from the trainable predictor; reusing pretrained components also means the study is not learning every modality from scratch.
3.2 Two-stage Training
A first stage learns broad image/video-to-text alignment without a task query. A second stage adds query-conditioned supervised tasks such as visual questions. This staging gives the semantic space a broad starting point before task adaptation. The preserved settings identify data mixtures, update schedules and the lower learning rate used for the text encoder. Later ablations show that the initial alignment stage is consequential.
| Video Classification (Top-1 Accuracy) | Text-to-video Retrieval (Recall@1) | ||||||||||||||||||||||
| Model |
# Parameters |
# Samples Seen |
Zero-shot |
Generalist Model |
Average |
SSv2
(Goyal et al., 2017)
|
EgoExo4D
(Grauman et al., 2024)
|
Kinetics-400
(Kay et al., 2017)
|
COIN (SR)
(Tang et al., 2019)
|
COIN (TR)
(Tang et al., 2019)
|
CrossTask (SR)
(Zhukov et al., 2019)
|
CrossTask (TR)
(Zhukov et al., 2019)
|
Average
|
ActivityNet
(Caba Heilbron et al., 2015)
|
DiDeMo
(Anne Hendricks et al., 2017)
|
YouCook2
(Zhou et al., 2018)
|
PVD-Bench
(Bolya et al., 2025)
|
Dream-1k
(Wang et al., 2024)
|
VDC-1k
(Chai et al., 2024)
| ||||
| RN50 | 75M | 12.8B | 21.8 | 2.1 | 1.5 | 2.1 | 41.4 | 8.6 | 39.0 | 10.9 | 68.7 | 28.3 | 28.7 | 17.7 | 24.7 | 29.7 | 5.1 | 27.6 | 47.2 | 46.0 | |||
| ViT-B | 124M | 12.8B | 25.4 | 3.1 | 1.3 | 2.8 | 49.5 | 11.2 | 47.3 | 16.2 | 71.5 | 29.3 | 31.0 | 19.5 | 25.7 | 34.0 | 6.1 | 27.0 | 48.5 | 42.9 | |||
| CLIP | ViT-L | 389M | 12.8B | ✓ | ✓ | 30.7 | 3.8 | 3.7 | 2.6 | 58.3 | 14.7 | 63.5 | 20.8 | 78.5 | 35.3 | 35.9 | 23.4 | 30.7 | 41.9 | 7.9 | 36.7 | 56.8 | 49.3 |
| ViT-B | 375M | 40B | 33.9 | 5.2 | 2.3 | 4.5 | 57.8 | 20.6 | 69.9 | 27.7 | 82.9 | 39.6 | 40.2 | 25.0 | 32.1 | 48.6 | 13.8 | 52.1 | 60.9 | 43.7 | |||
| ViT-L | 882M | 40B | 38.6 | 5.9 | 4.5 | 6.4 | 63.6 | 24.2 | 78.5 | 35.1 | 90.8 | 45.4 | 41.6 | 32.7 | 35.1 | 53.5 | 19.0 | 59.2 | 71.6 | 50.9 | |||
| SigLIP2 | ViT-g | 1.9B | 40B | ✓ | ✓ | 39.8 | 6.1 | 6.1 | 5.6 | 68.0 | 26.0 | 80.4 | 35.1 | 90.8 | 47.5 | 43.4 | 33.9 | 38.9 | 56.0 | 22.2 | 60.4 | 73.0 | 52.5 |
| ViT-B | 448M | 58B | 37.2 | 5.8 | 3.3 | 6.0 | 65.4 | 21.5 | 77.1 | 26.9 | 91.8 | 44.9 | 46.5 | 35.4 | 35.3 | 49.1 | 15.2 | 59.8 | 68.7 | 49.2 | |||
| ViT-L | 671M | 58B | 42.9 | 9.3 | 6.0 | 11.6 | 73.4 | 27.1 | 83.3 | 37.5 | 95.3 | 50.2 | 48.9 | 41.7 | 40.8 | 56.2 | 22.5 | 64.7 | 75.9 | 51.0 | |||
| PE-Core | ViT-G | 2.3B | 86B | ✓ | ✓ | 44.7 | 9.0 | 6.4 | 13.6 | 76.4 | 29.0 | 86.0 | 40.3 | 97.2 | 58.1 | 51.6 | 49.1 | 44.5 | 58.7 | 26.0 | 77.0 | 89.2 | 68.5 |
| VL-JEPA | ViT-L | 1.6B | 3.3B | ✓ | ✓ | 52.5 | 19.3 | 21.8 | 33.2 | 64.8 | 47.4 | 79.4 | 64.5 | 89.6 | 63.7 | 40.0 | 64.9 | 50.0 | 49.0 | 40.4 | 83.1 | 93.3 | 88.8 |
| VL-JEPA | ViT-L | 1.6B | 3.6B | ✗ | ✓ | 75.4 | 73.2 | 44.6 | 68.1 | 84.8 | 66.4 | 90.3 | 79.8 | 96.2 | 63.8 | 46.2 | 62.4 | 52.3 | 52.3 | 37.8 | 82.6 | 89.3 | 87.7 |
| Previous SoTA (including specialist models) | ✗ | ✗ | - | 77.5 | 56.4 | 47.8 | 92.1 | 67.3 | 95.3 | 64.5 | 96.0 | - | 62.8 | 74.1 | 74.2 | 61.4 | 28.9 | 77.0 | 89.2 | 68.5 | |||
4 Experiments
The evaluation covers multiple uses of the same embedding space. Zero-shot label selection, text-to-video retrieval, visual question answering, transition matching and fine-tuned action anticipation are different tests and should not be collapsed into a single claim of world-model competence.
4.1 Classification and Retrieval
Classification compares a predicted visual embedding with encoded candidate labels; retrieval compares it with candidate descriptions. The paper tests eight video-classification and eight retrieval datasets and separates the base and task-adapted variants. This evaluates semantic matching under the given candidate sets, not unrestricted answer generation. The original per-dataset results retain differences that an overall mean can conceal.
| GQA: compositional visual reasoning | TallyQA: complex object counting | POPE: object hallucination | POPEv2: object hallucination | |||||||
| Model | Accuracy | Model | Accuracy | Model | Accuracy | Model | Accuracy | |||
| BLIP-2 (OPT-2.7B) | 33.9 | SmolVLM-256M | 32.3 | SmolVLM2-256M | 56.4 | SmolVLM-256M | 62.3 | |||
| BLIP-2 (FlanT5XXL) | 41.0 | SmolVLM-500M | 44.8 | SmolVLM-256M | 57.9 | LLaVA-1.5-13B | 72.7 | |||
| InstructBLIP (FlanT5XL) | 48.4 | PaLI-700M | 62.3 | LLaVA-7B | 72.9 | InternVL2-8B | 74.5 | |||
| InstructBLIP (Vicuna-13B) | 49.5 | SmolVLM-2B | 64.7 | InstructBLIP (Vicuna-13B) | 79.0 | InternVL2-26B | 76.1 | |||
| Qwen-VL-Chat-7B | 57.5 | PaLI-3B | 65.8 | Video-LLaVA (7B) | 83.4 | Qwen2-VL-72B | 79.4 | |||
| Qwen-VL-7B | 59.3 | InstructBLIP (Vicuna-13B) | 68.0 | SmolVLM-500M | 85.8 | SmolVLM-500M | 83.8 | |||
| InternVL-Chat (Vicuna-7B) | 59.5 | PaLI-17B | 71.9 | LLaVA-1.5-7B | 85.9 | Qwen2-VL-7B | 87.0 | |||
| LLaVA-1.5 (Vicuna-7B) | 62.0 | LLaVA-1.5 (Vicuna-13B) | 72.3 | LLaVA-1.5-13B-HD | 86.3 | SmolVLM-2B | 88.8 | |||
| InternVL-Chat (Vicuna-13B) | 66.6 | PaliGemma (3B) | 76.8 | SmolVLM-2B | 87.5 | Qwen2-VL-2B | 91.3 | |||
| VL-JEPA (1.6B) | 61.5 | VL-JEPA (1.6B) | 69.9 | VL-JEPA (1.6B) | 85.7 | VL-JEPA (1.6B) | 86.3 | |||
| Vision Language Models | Socratic LLMs (w/ Qwen2.5-VL-72B captions) | VL-JEPA | |||||||||||||||||
| InternVL2.5 | Qwen2.5-VL | Llama-3.1 | Llama-4 | Qwen2.5 | GPT-4o | Claude-3.5 | Gemini-2 | BASE | SFT | ||||||||||
| 2B | 4B | 26B | 38B | 3B | 7B | 32B | 72B | 8B | 70B | 109B | 400B | 3B | 7B | 72B | N/A | N/A | N/A | 1.6B | 1.6B |
| 20.0 | 29.8 | 30.2 | 50.3 | 21.6 | 45.5 | 49.0 | 57.0 | 48.7 | 49.8 | 52.7 | 53.6 | 44.0 | 49.1 | 48.5 | 52.0 | 53.3 | 55.6 | 63.9 | 65.7 |
4.2 Visual Question Answering
For discriminative visual questions, candidate answers can be ranked in the embedding space. The reported comparisons include established generative vision-language models, but training recipes and inherited components vary across published systems. The task results therefore complement, rather than replace, the controlled same-data comparison later in the paper.
4.3 WorldPrediction-WM
The WorldPrediction-WM benchmark supplies initial and final images and asks which of four candidate actions explains the transition. VL-JEPA compares state-pair and candidate-action embeddings. This is a transition-recognition task with a finite answer set. High accuracy here should not be equated with rolling out arbitrary future physical states or executing a robot plan.
4.4 Action Anticipation
Action anticipation predicts a future action from a video prefix. Results include different anticipation gaps on EPIC-KITCHENS and next-step forecasting on COIN. Some comparisons match the visual encoder while the largest V-JEPA 2 variant uses a stronger encoder. Reading both rows matters: a gain against a matched smaller backbone does not imply dominance over every larger baseline at every horizon.
4.5 Embedding Prediction vs. Token Prediction: A Controlled Comparison
The controlled study fixes visual encoder, training data, iterations, batch size and scheduling, then compares embedding prediction with token prediction. The predictor sizes differ as part of the proposed efficiency design. The results support a better accuracy–compute tradeoff in the tested captioning and classification regime. They do not establish superiority for all language-heavy reasoning or agent tasks.
4.6 Effectiveness of Selective Decoding
Selective decoding watches how the predicted embedding changes over time and invokes text decoding at informative points. The benchmark compares annotation coverage against the number of decoding operations. The reported roughly 2.85-fold reduction concerns those operations at similar measured performance; it is not automatically a 2.85-fold end-to-end speedup once visual encoding and other work are included.
| Model |
|
|
SugarCrepe++ (Dumpala et al., 2024a) | VISLA (Dumpala et al., 2024b) | ||||||||||||||
| Average |
|
|
|
|
|
Average | Generic | Spatial | ||||||||||
| CLIP | ViT-L | 389M | 85M | 44.5 | 56.7 | 83.0 | 42.5 | 27.0 | 13.5 | 34.5 | 37.6 | 31.3 | ||||||
| SigLIP2 | ViT-g | 1.9B | 708M | 56.5 | 66.9 | 74.4 | 52.1 | 58.4 | 30.6 | 40.4 | 48.7 | 32.1 | ||||||
| PE-Core | ViT-G | 2.3B | 537M | 58.6 | 73.6 | 90.6 | 48.9 | 53.2 | 26.5 | 38.3 | 45.2 | 31.4 | ||||||
| VL-JEPA | ViT-L | 1.6B | 300M | 63.9 | 72.2 | 90.1 | 52.2 | 62.9 | 42.0 | 42.9 | 49.8 | 35.9 | ||||||
| VL-JEPA | ViT-L | 1.6B | 300M | 58.4 | 68.5 | 90.9 | 47.4 | 55.4 | 29.8 | 39.5 | 44.8 | 34.2 | ||||||
4.7 Evaluation of Y-Encoder
Hard-negative text triplets test whether the answer encoder preserves differences in objects, relations and attributes. The base JEPA encoder performs strongly, while supervised adaptation changes the pattern of results. This is evidence about particular semantic distinctions in the learned text space, not proof that every paraphrase or contradiction is represented correctly.
|
|
| ||||||||||
| VL-JEPA | 75.4 | 63.8 | 74.2 | |||||||||
| (a) Effectiveness of pretraining stage on caption data | ||||||||||||
| w/ Pretraining | 49.0 | 47.5 | 46.1 | |||||||||
| w/o Pretraining | 27.3 | (-21.7) | 30.2 | (-17.3) | 42.5 | (-3.6) | ||||||
| (b) Learning rate multiplier for Y-Encoder | ||||||||||||
| multiplier = 0.05 | 27.3 | 30.2 | 42.5 | |||||||||
| multiplier = 1.00 | 23.7 | (-3.6) | 28.8 | (-1.4) | 40.7 | (-1.8) | ||||||
| multiplier = 0.10 | 26.9 | (-0.4) | 30.2 | (-0.0) | 42.9 | (+0.4) | ||||||
| multiplier = 0.01 | 25.6 | (-1.7) | 27.7 | (-2.5) | 41.0 | (-1.5) | ||||||
| multiplier = 0.00 | 20.0 | (-7.3) | 25.9 | (-4.3) | 41.4 | (-1.1) | ||||||
| (c) Loss function (with no projection head on top frozen text encoder) | ||||||||||||
| InfoNCE | 23.3 | 30.3 | 44.3 | |||||||||
| Cosine | 16.5 | (-6.8) | 20.2 | (-10.1) | 46.6 | (+2.3) | ||||||
| L1 | 14.8 | (-8.5) | 15.5 | (-14.8) | 41.9 | (-2.4) | ||||||
| L2 | 13.5 | (-9.8) | 11.7 | (-18.6) | 43.7 | (-0.6) | ||||||
|
|
| ||||||||||
| (d) Predictor architecture and initialization | ||||||||||||
| Layer 8-16 | 27.3 | 30.2 | 42.5 | |||||||||
| Layer 0-2 | 24.3 | (-3.0) | 27.8 | (-2.4) | 40.1 | (-2.4) | ||||||
| Layer 0-4 | 25.1 | (-2.2) | 28.9 | (-1.3) | 43.6 | (+1.1) | ||||||
| Layer 0-8 | 27.2 | (-0.1) | 29.3 | (-0.9) | 43.4 | (+0.9) | ||||||
| Layer 0-16 | 27.4 | (+0.1) | 31.0 | (+0.8) | 45.5 | (+3.0) | ||||||
| w/o Bi-direction Attention | 26.7 | (-0.6) | 31.2 | (+1.0) | 40.6 | (-1.9) | ||||||
| w/o Llama-3 Initialization | 28.1 | (+0.8) | 30.4 | (+0.2) | 40.6 | (-1.9) | ||||||
| (e) Y-Encoder (trainable linear projection on top of frozen text encoder) | ||||||||||||
| EmbeddingGemma-300M | 19.5 | 24.1 | 42.5 | |||||||||
| Qwen3-Embedding-0.6B | 24.5 | (+5.0) | 24.5 | (+0.4) | 41.5 | (-1.0) | ||||||
| Qwen3-Embedding-4B | 27.7 | (+8.2) | 26.6 | (+2.5) | 38.1 | (-4.4) | ||||||
| Qwen3-Embedding-8B | 29.6 | (+10.1) | 29.5 | (+5.4) | 41.9 | (-0.6) | ||||||
| PE-B (356M) | 29.4 | (+9.9) | 34.5 | (+10.4) | 35.9 | (-6.6) | ||||||
| PE-L (356M) | 29.0 | (+9.5) | 34.2 | (+10.1) | 42.9 | (+0.4) | ||||||
| PE-G (539M) | 33.9 | (+14.4) | 32.0 | (+7.9) | 41.8 | (-0.7) | ||||||
4.8 Ablation Study
Ablations vary pretraining, text-encoder learning rate, objective, predictor depth, attention direction and initialization. Omitting broad alignment pretraining substantially hurts classification and retrieval. Contrastive regularization matters when the target text encoder is trainable; a simple similarity loss cannot automatically replace it. The complete table shows that different tasks can prefer different choices.
5 Related Works
Related work separates predictive joint embeddings from ordinary image–text alignment and from token-generative multimodal models. VL-JEPA combines a query-conditioned predictor with a semantic text target, retaining a decoder for applications that require language. Its relationship to action-conditioned world models is architectural; those models and these finite-candidate recognition tasks do not have identical evaluation requirements.
6 Conclusion
The study supports semantic embedding prediction as an efficient design for the tested video-language applications, especially streaming and selective decoding. It explicitly leaves broader reasoning, tool use and agentic behavior to future evaluation. The central engineering question is when an application needs a semantic decision now and when it actually needs newly generated text.
Acknowledgments
The acknowledgments record research feedback, baseline-performance assistance and real-time demonstration support. These contributions are attribution rather than additional experimental evidence.