Literature
Display
← All papers

VL-JEPA: Joint Embedding Predictive Architecture for Vision-language

Delong Chen · Mustafa Shukor · Theo Moutakanni · Willy Chung · Jade Yu · Tejaswi Kasarla · Yejin Bang · Allen Bolourchi · Yann LeCun · Pascale Fung

Preprint posted · ICLR 2026 ↗

Contents

Overview

Intuition: several valid captions may describe the same event with different words. VL-JEPA predicts a vector representing the answer’s meaning, then decodes that vector into text only when needed. Candidate labels and retrieved descriptions can instead be compared directly in the shared embedding space. This edition follows arXiv v 2, whose author list includes Yejin Bang. All original figures, detailed comparisons and bibliography are retained.

1 Introduction

VL-JEPA separates semantic prediction from the final choice of words. A visual encoder and a query-conditioned predictor produce an answer embedding; a text encoder defines the target space; an optional decoder turns a predicted embedding into language. This can support classification, retrieval and selected text outputs with one architecture. The paper motivates the design using streaming video, where repeatedly decoding nearly unchanged content can waste work.

Original figure
Figure 1: VL-JEPA model architecture
Refer to caption
Figure 2: Left: VL-JEPA Architecture. It learns to predict the target embedding SYS_{Y}, instead of reconstructing the raw target YY in token space as in classical VLMs. Right: VL-JEPA Applications: It handles vision-text-to-text generation tasks (e.g., captioning) with selective decoding mechanism natively supported. Furthermore, VL-JEPA’s embedding space facilitates discriminative VQA, open-vocabulary classification and text-to-video retrieval tasks using a single unified model architecture.

2 Methodology

The visual input and textual query feed a predictor trained to match an encoding of the target answer. The target representation is learned with a contrastive objective that discourages collapse. The text decoder is not the main prediction target: it translates the predicted semantic vector into words at inference. This division allows a downstream decision without decoding when an application only needs a candidate label or similarity score. The training signal still comes from image/video–text pairs.

3 Implementation of VL-JEPA

Implementation choices determine how much computation is saved and what prior knowledge is inherited. The next sections separate the model components from the data and two-stage training procedure.

3.1 Model Architecture

The system combines a pretrained visual encoder, a smaller transformer predictor, a sentence-level text encoder and a lightweight text decoder. The predictor is initialized using selected language-model layers. Parameter counts should distinguish the full system from the trainable predictor; reusing pretrained components also means the study is not learning every modality from scratch.

3.2 Two-stage Training

A first stage learns broad image/video-to-text alignment without a task query. A second stage adds query-conditioned supervised tasks such as visual questions. This staging gives the semantic space a broad starting point before task adaptation. The preserved settings identify data mixtures, update schedules and the lower learning rate used for the text encoder. Later ablations show that the initial alignment stage is consequential.

Table 1: Video classification and text-to-video retrieval. Best zero-shot performance in each dataset are highlighted. Samples seen == training step ×\times effective batch size.
Video Classification (Top-1 Accuracy) Text-to-video Retrieval (Recall@1)
Model

# Parameters

# Samples Seen

Zero-shot

Generalist Model

Average

Kinetics-400 (Kay et al., 2017)
CrossTask (SR) (Zhukov et al., 2019)
CrossTask (TR) (Zhukov et al., 2019)
Average
YouCook2 (Zhou et al., 2018)
Dream-1k (Wang et al., 2024)
RN50 75M 12.8B 21.8 2.1 1.5 2.1 41.4 8.6 39.0 10.9 68.7 28.3 28.7 17.7 24.7 29.7 5.1 27.6 47.2 46.0
ViT-B 124M 12.8B 25.4 3.1 1.3 2.8 49.5 11.2 47.3 16.2 71.5 29.3 31.0 19.5 25.7 34.0 6.1 27.0 48.5 42.9
CLIP ViT-L 389M 12.8B ✓ ✓ 30.7 3.8 3.7 2.6 58.3 14.7 63.5 20.8 78.5 35.3 35.9 23.4 30.7 41.9 7.9 36.7 56.8 49.3
ViT-B 375M 40B 33.9 5.2 2.3 4.5 57.8 20.6 69.9 27.7 82.9 39.6 40.2 25.0 32.1 48.6 13.8 52.1 60.9 43.7
ViT-L 882M 40B 38.6 5.9 4.5 6.4 63.6 24.2 78.5 35.1 90.8 45.4 41.6 32.7 35.1 53.5 19.0 59.2 71.6 50.9
SigLIP2 ViT-g 1.9B 40B ✓ ✓ 39.8 6.1 6.1 5.6 68.0 26.0 80.4 35.1 90.8 47.5 43.4 33.9 38.9 56.0 22.2 60.4 73.0 52.5
ViT-B 448M 58B 37.2 5.8 3.3 6.0 65.4 21.5 77.1 26.9 91.8 44.9 46.5 35.4 35.3 49.1 15.2 59.8 68.7 49.2
ViT-L 671M 58B 42.9 9.3 6.0 11.6 73.4 27.1 83.3 37.5 95.3 50.2 48.9 41.7 40.8 56.2 22.5 64.7 75.9 51.0
PE-Core ViT-G 2.3B 86B ✓ ✓ 44.7 9.0 6.4 13.6 76.4 29.0 86.0 40.3 97.2 58.1 51.6 49.1 44.5 58.7 26.0 77.0 89.2 68.5
VL-JEPABASE{}_{\texttt{BASE}} ViT-L 1.6B 3.3B ✓ ✓ 52.5 19.3 21.8 33.2 64.8 47.4 79.4 64.5 89.6 63.7 40.0 64.9 50.0 49.0 40.4 83.1 93.3 88.8
VL-JEPASFT{}_{\texttt{SFT}} ViT-L 1.6B 3.6B ✗ ✓ 75.4 73.2 44.6 68.1 84.8 66.4 90.3 79.8 96.2 63.8 46.2 62.4 52.3 52.3 37.8 82.6 89.3 87.7
Previous SoTA (including specialist models) ✗ ✗ - 77.5 56.4 47.8 92.1 67.3 95.3 64.5 96.0 - 62.8 74.1 74.2 61.4 28.9 77.0 89.2 68.5

4 Experiments

The evaluation covers multiple uses of the same embedding space. Zero-shot label selection, text-to-video retrieval, visual question answering, transition matching and fine-tuned action anticipation are different tests and should not be collapsed into a single claim of world-model competence.

4.1 Classification and Retrieval

Classification compares a predicted visual embedding with encoded candidate labels; retrieval compares it with candidate descriptions. The paper tests eight video-classification and eight retrieval datasets and separates the base and task-adapted variants. This evaluates semantic matching under the given candidate sets, not unrestricted answer generation. The original per-dataset results retain differences that an overall mean can conceal.

Table 2: VQA benchmarks. We report accuracy on GQA (Hudson and Manning, 2019), TallyQA (Acharya et al., 2019), POPE (Li et al., 2023b), and POPEv2 (Li et al., 2025b). Scores lower than our model are marked in red. Scores from SmolVLM are obtained by our evaluation, while other baselines are reported in the literature.
GQA: compositional visual reasoning TallyQA: complex object counting POPE: object hallucination POPEv2: object hallucination
Model Accuracy Model Accuracy Model Accuracy Model Accuracy
BLIP-2 (OPT-2.7B) 33.9 SmolVLM-256M 32.3 SmolVLM2-256M 56.4 SmolVLM-256M 62.3
BLIP-2 (FlanT5XXL) 41.0 SmolVLM-500M 44.8 SmolVLM-256M 57.9 LLaVA-1.5-13B 72.7
InstructBLIP (FlanT5XL) 48.4 PaLI-700M 62.3 LLaVA-7B 72.9 InternVL2-8B 74.5
InstructBLIP (Vicuna-13B) 49.5 SmolVLM-2B 64.7 InstructBLIP (Vicuna-13B) 79.0 InternVL2-26B 76.1
Qwen-VL-Chat-7B 57.5 PaLI-3B 65.8 Video-LLaVA (7B) 83.4 Qwen2-VL-72B 79.4
Qwen-VL-7B 59.3 InstructBLIP (Vicuna-13B) 68.0 SmolVLM-500M 85.8 SmolVLM-500M 83.8
InternVL-Chat (Vicuna-7B) 59.5 PaLI-17B 71.9 LLaVA-1.5-7B 85.9 Qwen2-VL-7B 87.0
LLaVA-1.5 (Vicuna-7B) 62.0 LLaVA-1.5 (Vicuna-13B) 72.3 LLaVA-1.5-13B-HD 86.3 SmolVLM-2B 88.8
InternVL-Chat (Vicuna-13B) 66.6 PaliGemma (3B) 76.8 SmolVLM-2B 87.5 Qwen2-VL-2B 91.3
VL-JEPASFT{}_{\texttt{SFT}} (1.6B) 61.5 VL-JEPASFT{}_{\texttt{SFT}} (1.6B) 69.9 VL-JEPASFT{}_{\texttt{SFT}} (1.6B) 85.7 VL-JEPASFT{}_{\texttt{SFT}} (1.6B) 86.3
Table 3: WorldPrediction-WM benchmark results. We compare the accuracy between large VLMs, socratic LLMs, and VL-JEPA. VL-JEPASFT{}_{\texttt{SFT}} achieves a new SoTA at 65.7%.
Vision Language Models Socratic LLMs (w/ Qwen2.5-VL-72B captions) VL-JEPA
InternVL2.5 Qwen2.5-VL Llama-3.1 Llama-4 Qwen2.5 GPT-4o Claude-3.5 Gemini-2 BASE SFT
2B 4B 26B 38B 3B 7B 32B 72B 8B 70B 109B 400B 3B 7B 72B N/A N/A N/A 1.6B 1.6B
20.0 29.8 30.2 50.3 21.6 45.5 49.0 57.0 48.7 49.8 52.7 53.6 44.0 49.1 48.5 52.0 53.3 55.6 63.9 65.7

4.2 Visual Question Answering

For discriminative visual questions, candidate answers can be ranked in the embedding space. The reported comparisons include established generative vision-language models, but training recipes and inherited components vary across published systems. The task results therefore complement, rather than replace, the controlled same-data comparison later in the paper.

4.3 WorldPrediction-WM

The WorldPrediction-WM benchmark supplies initial and final images and asks which of four candidate actions explains the transition. VL-JEPA compares state-pair and candidate-action embeddings. This is a transition-recognition task with a finite answer set. High accuracy here should not be equated with rolling out arbitrary future physical states or executing a robot plan.

4.4 Action Anticipation

Action anticipation predicts a future action from a video prefix. Results include different anticipation gaps on EPIC-KITCHENS and next-step forecasting on COIN. Some comparisons match the visual encoder while the largest V-JEPA 2 variant uses a stronger encoder. Reading both rows matters: a gain against a matched smaller backbone does not imply dominance over every larger baseline at every horizon.

Table 4: Finetuning results on EPIC-KITCHENS-100 action anticipation (Recall@5) for different anticipation time tt. VL-JEPA outperforms V-JEPA2 with a ViT-L-256px vision encoder across all anticipation time.
Model Vision Encoder t=1​st=1s t=2​st=2s t=4​st=4s t=10​st=10s
V-JEPA2 ViT-g-384px 39.7 28.6 19.3 8.2
V-JEPA2 ViT-L-256px 32.7 23.4 15.9 7.1
VL-JEPA ViT-L-256px 34.2 26.0 19.4 11.7
Table 5: Finetuning result on COIN next step forecasting. VL-JEPA established a new SoTA at 56.2.
Model
VideoLLM
-online (8B)
VideoLLM
-MoD (8B)
ProVideLLM
-8B/11+
VL-JEPA
(1.6B)
Accuracy
49.1 49.7 53.6 56.2
Refer to caption
Figure 3: Comparison of embedding prediction (VL-JEPA) and token prediction (VLM). We conduct a fair comparison of under strictly aligned training settings (encoder, data, batchsize, etc.). Left: Zero-shot video captioning CIDEr score averaged over 3 datasets and zero-shot classification accuracy (top-5) averaged over 3 benchmarks. Right: Comparing the trainable parameters and average inference time cost.

4.5 Embedding Prediction vs. Token Prediction: A Controlled Comparison

The controlled study fixes visual encoder, training data, iterations, batch size and scheduling, then compares embedding prediction with token prediction. The predictor sizes differ as part of the proposed efficiency design. The results support a better accuracy–compute tradeoff in the tested captioning and classification regime. They do not establish superiority for all language-heavy reasoning or agent tasks.

4.6 Effectiveness of Selective Decoding

Selective decoding watches how the predicted embedding changes over time and invokes text decoding at informative points. The benchmark compares annotation coverage against the number of decoding operations. The reported roughly 2.85-fold reduction concerns those operations at similar measured performance; it is not automatically a 2.85-fold end-to-end speedup once visual encoding and other work are included.

Refer to caption
Figure 4: Evaluation of selective decoding. Left: We compare uniform sampling of decoding points at fixed intervals (red) and embedding-guided selective decoding (blue). Performance is measured by the average CIDEr score between each annotation yy and its closest decoded output y^\hat{y}. Right: Results on EgoExo4D show that selective decoding achieves a Pareto improvement over uniform sampling: for the same performance level, it requires fewer decoding operations.
Table 6: Comparison of text-encoders performance. We report triplet-based accuracy (%) on SugarCrepe++ and VISLA datasets.
Model
# Params.
(total)
# Params.
(text encoder)
SugarCrepe++ (Dumpala et al., 2024a) VISLA (Dumpala et al., 2024b)
Average
Replace
Attribute
Replace
Object
Replace
Relation
Swap
Attribute
Swap
Object
Average Generic Spatial
CLIP ViT-L 389M 85M 44.5 56.7 83.0 42.5 27.0 13.5 34.5 37.6 31.3
SigLIP2 ViT-g 1.9B 708M 56.5 66.9 74.4 52.1 58.4 30.6 40.4 48.7 32.1
PE-Core ViT-G 2.3B 537M 58.6 73.6 90.6 48.9 53.2 26.5 38.3 45.2 31.4
VL-JEPABASE{}_{\texttt{BASE}} ViT-L 1.6B 300M 63.9 72.2 90.1 52.2 62.9 42.0 42.9 49.8 35.9
VL-JEPASFT{}_{\texttt{SFT}} ViT-L 1.6B 300M 58.4 68.5 90.9 47.4 55.4 29.8 39.5 44.8 34.2

4.7 Evaluation of Y-Encoder

Hard-negative text triplets test whether the answer encoder preserves differences in objects, relations and attributes. The base JEPA encoder performs strongly, while supervised adaptation changes the pattern of results. This is evidence about particular semantic distinctions in the learned text space, not proof that every paraphrase or contradiction is represented correctly.

Table 7: Ablation studies results. The default setting adopted by VL-JEPA is marked in blue. We calculate ±\pmdelta within each group of ablations in comparison with the default setting.
Classification
(Accuracy)
Retrieval
(Recall@1)
VQA
(Accuracy)
VL-JEPASFT{}_{\texttt{SFT}} 75.4 63.8 74.2
(a) Effectiveness of pretraining stage on caption data
w/ Pretraining 49.0 47.5 46.1
w/o Pretraining 27.3 (-21.7) 30.2 (-17.3) 42.5 (-3.6)
(b) Learning rate multiplier for Y-Encoder
multiplier = 0.05 27.3 30.2 42.5
multiplier = 1.00 23.7 (-3.6) 28.8 (-1.4) 40.7 (-1.8)
multiplier = 0.10 26.9 (-0.4) 30.2 (-0.0) 42.9 (+0.4)
multiplier = 0.01 25.6 (-1.7) 27.7 (-2.5) 41.0 (-1.5)
multiplier = 0.00 20.0 (-7.3) 25.9 (-4.3) 41.4 (-1.1)
(c) Loss function (with no projection head on top frozen text encoder)
InfoNCE 23.3 30.3 44.3
Cosine 16.5 (-6.8) 20.2 (-10.1) 46.6 (+2.3)
L1 14.8 (-8.5) 15.5 (-14.8) 41.9 (-2.4)
L2 13.5 (-9.8) 11.7 (-18.6) 43.7 (-0.6)
Classification
(Accuracy)
Retrieval
(Recall@1)
VQA
(Accuracy)
(d) Predictor architecture and initialization
Layer 8-16 27.3 30.2 42.5
Layer 0-2 24.3 (-3.0) 27.8 (-2.4) 40.1 (-2.4)
Layer 0-4 25.1 (-2.2) 28.9 (-1.3) 43.6 (+1.1)
Layer 0-8 27.2 (-0.1) 29.3 (-0.9) 43.4 (+0.9)
Layer 0-16 27.4 (+0.1) 31.0 (+0.8) 45.5 (+3.0)
w/o Bi-direction Attention 26.7 (-0.6) 31.2 (+1.0) 40.6 (-1.9)
w/o Llama-3 Initialization 28.1 (+0.8) 30.4 (+0.2) 40.6 (-1.9)
(e) Y-Encoder (trainable linear projection on top of frozen text encoder)
EmbeddingGemma-300M 19.5 24.1 42.5
Qwen3-Embedding-0.6B 24.5 (+5.0) 24.5 (+0.4) 41.5 (-1.0)
Qwen3-Embedding-4B 27.7 (+8.2) 26.6 (+2.5) 38.1 (-4.4)
Qwen3-Embedding-8B 29.6 (+10.1) 29.5 (+5.4) 41.9 (-0.6)
PEcore{}_{\text{core}}-B (356M) 29.4 (+9.9) 34.5 (+10.4) 35.9 (-6.6)
PEcore{}_{\text{core}}-L (356M) 29.0 (+9.5) 34.2 (+10.1) 42.9 (+0.4)
PEcore{}_{\text{core}}-G (539M) 33.9 (+14.4) 32.0 (+7.9) 41.8 (-0.7)

4.8 Ablation Study

Ablations vary pretraining, text-encoder learning rate, objective, predictor depth, attention direction and initialization. Omitting broad alignment pretraining substantially hurts classification and retrieval. Contrastive regularization matters when the target text encoder is trainable; a simple similarity loss cannot automatically replace it. The complete table shows that different tasks can prefer different choices.

5 Related Works

Related work separates predictive joint embeddings from ordinary image–text alignment and from token-generative multimodal models. VL-JEPA combines a query-conditioned predictor with a semantic text target, retaining a decoder for applications that require language. Its relationship to action-conditioned world models is architectural; those models and these finite-candidate recognition tasks do not have identical evaluation requirements.

Table 8: Task coverage comparison.
CLIP VLM VL-JEPA
Generation ✗ ✓ ✓
Retrieval ✓ ✗ ✓

6 Conclusion

The study supports semantic embedding prediction as an efficient design for the tested video-language applications, especially streaming and selective decoding. It explicitly leaves broader reasoning, tool use and agentic behavior to future evaluation. The central engineering question is when an application needs a semantic decision now and when it actually needs newly generated text.

Acknowledgments

The acknowledgments record research feedback, baseline-performance assistance and real-time demonstration support. These contributions are attribution rather than additional experimental evidence.

Concept demonstration

Open image ↗