Literature
Display
← All papers

Proving and Improving the Platonic Representation Hypothesis with JEPAs

Randall Balestriero · Yann LeCun

Preprint posted · arXiv (preprint) ↗

Learning theory20 min text · CondensedOriginal paper ↗
Contents

At a glance

The Platonic Representation Hypothesis proposes that independently trained models can converge toward a shared representation of the world. This paper gives sufficient conditions for JEPAs to agree up to a rotation, then tests whether learning from different observations of the same scene improves image–text alignment.

Good-view text JEPAs outperform matched next-token models on the tested alignment metrics. A 109M-parameter model also exceeds the evaluated language models up to 72B parameters on selected benchmarks. Applications include captioning through a map fitted on image–caption pairs and a separate method that estimates the map from unpaired data.

The proof concerns idealized population optima, not arbitrary trained networks. Experiments use grouped captions, often generated by a vision–language model; the strongest captioner fits its map on 30,000 pairs. Alignment scores do not establish general language ability.

Figure 1: original paper visual, restored from the PDF
Figure 1: Good views make the PRH a theorem, and alignment 42 times more sample-efficient. Left: the PRH picture [22], one world zz observed as images and as texts. When a JEPA’s positive pairs are two observations of one scene (good views: two photographs, two captions), the encoders of every modality learn the same embedding up to one rotation (Section 2.1); bad views, two corruptions of one observation, and next-token prediction, a prefix and its continuation, align only partly (Sections 2.3 and 2.4). Right: alignment with DINOv2 on the PRH paper’s benchmark (WIT-1024, mutual 10-NN) of text models trained on the texts of web images, against the samples they saw (means of three seeds, one for the 109M bad-view JEPA; lines: log-linear trends). With 21.6M samples a good-view JEPA matches next-token prediction on the same corpus at 900M samples (.109 and .111; 14 times fewer text tokens), and with 900M it reaches .184, above Qwen2.5-72B and Llama-3-70B, trained on 15–18T tokens (dashed). This demonstrates that with good views, alignment needs far less scale.

1 Introduction

The PRH describes increasing agreement between representations learned independently from different modalities [22]. The proposed mechanism is shared positive relationships: if two training objectives treat the same underlying world states as similar, their optimal geometries can coincide.

This suggests a practical distinction. Different captions or frames of one scene can remove observation-specific details; repeatedly corrupting one caption or image can preserve them. The paper combines a finite-world analysis, controlled Flickr and video experiments, web-scale text training, and applications of the resulting alignment. Its guarantees and experimental evidence have different scopes.

2 Shared Positives Provably Make Representations Platonic

The theory studies identity-predictor JEPAs with rotation-invariant objectives. It separates three questions: when independently learned spaces coincide, how differing positive pairs change them, and when nuisance information prevents agreement.

2.1 The Platonic Representation Theorem

Under shared positive pairs, each modality's population objective can be rewritten as the same optimization over world-state embeddings. If that objective has a unique family of minimizers up to rotation, every optimal modality encoder belongs to that family.

World, modalities, and positive pairs.

The world has finitely many states with positive probabilities. Each modality renders every state injectively; its symmetric positive-pair distribution has the world distribution as both marginals. The loss and distributional regularizer are rotation invariant, and optimization ranges over unrestricted encoders.

Theorem 1 requires shared world-level positives and uniqueness of the optimum up to rotation. Proposition 1 establishes that uniqueness for whitened MSE when the selected nonconstant eigenspace of the normalized positive-pair kernel has a spectral gap. Its solution consists of the leading eigenfunctions [17].

SIGReg is rotation invariant in expectation, but uniqueness for its full-distribution constraint is assumed rather than proved. A finite world cannot exactly realize a continuous Gaussian distribution. Learned or action-conditioned JEPA predictors are outside this identity-predictor result.

Equation 1
ℒm​(f)=𝔼(z,z′)∼πm​ℓ​(f⁡(gm​(z)),f⁡(gm​(z′)))+Ω⁡(f),\mathcal{L}_{m}(f)=\mathbb{E}_{(z,z^{\prime})\sim\pi_{m}}\,\ell\big(f(g_{m}(z)),f(g_{m}(z^{\prime}))\big)+\Omega(f), (1)

Theorem 1: Platonic Representation Theorem

Let M≥2M\geq 2 modalities share one positive-pair distribution, π1=⋯=πM=π\pi_{1}=\dots=\pi_{M}=\pi, and let the minimizers of ℒπ\mathcal{L}_{\pi} be unique up to rotation. Then there is one embedding of the world h⋆:𝒵→ℝkh^{\star}:\mathcal{Z}\to\mathbb{R}^{k} such that every minimizer fm⋆f_{m}^{\star} of ℒm\mathcal{L}_{m}, for every modality mm, satisfies

Equationfm⋆∘gm=𝑸m​h⋆for some ​𝑸m∈O⁡(k),f_{m}^{\star}\circ g_{m}=\boldsymbol{Q}_{m}\,h^{\star}\quad\text{for some }\boldsymbol{Q}_{m}\in\mathrm{O}(k),
(2)

so any two modalities agree up to the single rotation 𝑸m​𝑸m′⊤\boldsymbol{Q}_{m}\boldsymbol{Q}_{m^{\prime}}^{\top}. (Proof in Appendix A.)

Proposition 1: Spectral JEPAs Are Unique Up to Rotation

Let ℓ⁡(a,b)=‖a−b‖2\ell(a,b)=\|a-b\|^{2}, let Ω\Omega whiten the embeddings (𝔼p​h=0\mathbb{E}_{p}h=0, 𝔼p​h​h⊤=𝑰k\mathbb{E}_{p}hh^{\top}=\boldsymbol{I}_{k}), and let 1≤k<|𝒵|1\leq k<|\mathcal{Z}|. Then min⁡ℒπ=2​∑i=1k(1−μi)\min\mathcal{L}_{\pi}=2\sum_{i=1}^{k}(1-\mu_{i}), and if μk>μk+1\mu_{k}>\mu_{k+1} (with μ|𝒵|:=−∞\mu_{|\mathcal{Z}|}:=-\infty) the minimizers are exactly 𝑸​𝚽π\boldsymbol{Q}\boldsymbol{\Phi}_{\pi}, 𝑸∈O⁡(k)\boldsymbol{Q}\in\mathrm{O}(k), with the spectral embedding 𝚽π​(z)=(v1​(z),…,vk​(z))/p⁡(z)\boldsymbol{\Phi}_{\pi}(z)=\big(v_{1}(z),\dots,v_{k}(z)\big)/\sqrt{p(z)}. (Proof in Appendix A.)

2.2 When Positives Differ

For whitened MSE, Theorem 2 bounds the best rotational mismatch by the difference between the two normalized positive-pair kernels, divided by a spectral gap. The modalities must share the same world-state marginal. Thus approximately shared positives give approximately shared representations when the relevant eigenspace is stable; a small gap weakens the guarantee.

Theorem 2: Kernel Mismatch Bounds Misalignment

Let π1,π2\pi_{1},\pi_{2} share the marginal pp, let 𝚽1,𝚽2\boldsymbol{\Phi}_{1},\boldsymbol{\Phi}_{2} be their spectral embeddings (any orthonormal top-kk eigenvectors if 𝑲π1\boldsymbol{K}_{\pi_{1}} has ties), and let δ=μk−μk+1>0\delta=\mu_{k}-\mu_{k+1}>0 be the eigengap of 𝑲π2\boldsymbol{K}_{\pi_{2}}. Then

Equationmin𝑸∈O⁡(k)⁡(𝔼p​‖𝚽1​(z)−𝑸​𝚽2​(z)‖2)1/2≤2​2δ​min⁡(k​‖𝑲π1−𝑲π2‖2,‖𝑲π1−𝑲π2‖F).\begin{split}&\min_{\boldsymbol{Q}\in\mathrm{O}(k)}\Big(\mathbb{E}_{p}\big\|\boldsymbol{\Phi}_{1}(z)-\boldsymbol{Q}\boldsymbol{\Phi}_{2}(z)\big\|^{2}\Big)^{\!1/2}\\ &\quad\leq\frac{2\sqrt{2}}{\delta}\min\big(\sqrt{k}\,\|\boldsymbol{K}_{\pi_{1}}{-}\boldsymbol{K}_{\pi_{2}}\|_{2},\,\|\boldsymbol{K}_{\pi_{1}}{-}\boldsymbol{K}_{\pi_{2}}\|_{F}\big).\end{split}
(3)

(Proof in Appendix A, empirical validation in Figure 2.)

Original figure
Figure 2: Kernel mismatch bounds misalignment (Section 2.2). Each point is one randomly generated finite world (8–39 states, random marginal, embedding dimension kk from 1 to 5) with a pair of kernels on it: the world’s kernel, and a second kernel of the same marginal that mixes in an unrelated kernel with weight 10−410^{-4} to 11, as if a second modality had partly different positives. For each pair we compute the two spectral embeddings (Section 2.1) and plot their distance after the best rotation (vertical) against the bound of Section 2.2 (horizontal); of the 300 worlds, half are clustered, with small spectral gaps. We clearly observe that the error lies below the bound in every world (dashed: equality), at 18% of it at most.

What pairs are for.

Whitened unpaired clouds have identical first and second moments, so those moments cannot identify their relative rotation. Enough paired states spanning the embedding space determine it by orthogonal Procrustes [55]. Higher-order geometry can also identify a map without pairs, but this needs additional structure.

The experiments often map unequal-dimensional spaces with a row-orthonormal matrix. Such rectangular maps are useful operationally but are not the equal-dimensional rotations covered by the exact theorem.

Proposition 2: kk Pairs Identify the Rotation

In the setting of Sections 2.1 and 2.1, (i) every rotation of a whitened embedding is whitened, so no statistic of the first two moments of unpaired samples from two modalities identifies the rotation between them; (ii) if the embeddings of n≥kn\geq k world states span ℝk\mathbb{R}^{k}, the rotation 𝑸\boldsymbol{Q} with f1⋆∘g1=𝑸​f2⋆∘g2f_{1}^{\star}\circ g_{1}=\boldsymbol{Q}f_{2}^{\star}\circ g_{2} is the unique minimizer over 𝑹∈O⁡(k)\boldsymbol{R}\in\mathrm{O}(k) of ∑i≤n‖f1⋆​(g1​(zi))−𝑹​f2⋆​(g2​(zi))‖2\sum_{i\leq n}\|f_{1}^{\star}(g_{1}(z_{i}))-\boldsymbol{R}f_{2}^{\star}(g_{2}(z_{i}))\|^{2} (orthogonal Procrustes). (Proof in Appendix A.)

2.3 Good-View and Bad-View JEPAs

Let an observation contain a shared world state and independent modality-specific nuisance. Theorem 3 lower-bounds rotational mismatch using how much embedding variation is explained by the world state. Retaining nuisance therefore obstructs exact alignment.

Good views independently resample nuisance while preserving the state. Under the stated spectral condition, their leading eigenfunctions ignore nuisance. For the analyzed bad-view product kernel, persistent nuisance enters the representation when its eigenvalues outrank state directions; each selected nuisance direction contributes to an irreducible mismatch. Bad views do not necessarily fail: the relative correlations determine the outcome.

Theorem 3: Nuisance Costs Alignment

Let f1f_{1} and f2f_{2} be centered, whitened encoders of two modalities into ℝk\mathbb{R}^{k}. Over paired observations (x1,x2)(x_{1},x_{2}) of one world state with independent nuisances,

Equationmin𝑸∈O⁡(k)⁡𝔼​‖𝑸​f1​(x1)−f2​(x2)‖2≥2​k​(1−minm⁡ρ⁡(fm)),\min_{\boldsymbol{Q}\in\mathrm{O}(k)}\mathbb{E}\big\|\boldsymbol{Q}f_{1}(x_{1})-f_{2}(x_{2})\big\|^{2}\geq 2k\big(1-\min_{m}\rho(f_{m})\big),
(4)

and ρ⁡(f)=1\rho(f)=1 if and only if f⁡(x)f(x) is a function of the world state alone. (Proof in Appendix A.)

Proposition 3: Good Views Ignore Nuisance, Bad Views Capture It

Take the objective of Section 2.1. (i) If μk>max⁡(μk+1,0)\mu_{k}>\max(\mu_{k+1},0), a good-view JEPA learns a function of the world state alone (ρ=1\rho=1): good-view JEPAs of any number of modalities with the same world kernel π\pi agree up to one rotation each on every observation, although every state has many. (ii) Let a bad-view JEPA corrupt state and nuisance independently, let αk\alpha_{k} be the correlation that two corruptions of one observation keep along the kk-th direction of the state, and β1\beta_{1} the largest they keep along a direction of the nuisance. If its kernel has a gap after its kk-th eigenvalue, it spends rr of its kk directions on nuisance, so that its alignment error with any centered, whitened encoder of another modality is at least 2​r2r, and r≥1r\geq 1 if and only if β1>αk\beta_{1}>\alpha_{k}. (Formal statement and proof in Appendix A.)

Figure 3: original paper visual, restored from the PDF
Figure 3: What counts as a positive, and which observations it connects. Eight captions of two scenes (dots), joined as each objective’s positives join them. (a) Good views join every description of a scene and never another, so the world kernel they induce is the world’s own (Section 2.3). (b) Weak corruptions mostly pair a caption with itself, keeping its wording, and join other captions only where their corruptions overlap, which takes dense data. (c) Strong corruptions also coincide across scenes, so the induced kernel mixes scenes whatever the amount of data. (d) Next-token prediction pairs a prefix with its own continuation, and implicitly contexts followed by the same token (Section 2.4): narrow links within scenes, plus spill through frequent tokens. That is, good views fix the world kernel by design, while bad views and next-token prediction get it from the data.

Views and data together.

Theorem 4 extends the analysis to general observations by separating state and nuisance components of the positive-pair operator. Agreement is controlled by differences between the induced world kernels, state–nuisance mixing, and spectral separation from nuisance directions. The bound is informative only when the required gaps dominate these perturbations.

Theorem 4: Views and Data Set the World Kernel

Take the objective of Section 2.1 on the observations of each modality mm, and let the kk-th eigenvalue of 𝑲πWm\boldsymbol{K}_{\pi_{W_{m}}} exceed both the next one and the share of nuisance its positives keep. (i) Every spectral solution fmf_{m} is within τm\tau_{m} of a rotation of the spectral embedding of πWm\pi_{W_{m}}, where τm\tau_{m} grows with how much positive pairs mix world state with nuisance and vanishes when they do not. (ii) Over paired observations of two modalities,

Equation|min𝑸∈O⁡(k)⁡(𝔼​‖f1​(x1)−𝑸​f2​(x2)‖2)1/2−d|≤τ1+τ2,\Big|\min_{\boldsymbol{Q}\in\mathrm{O}(k)}\Big(\mathbb{E}\big\|f_{1}(x_{1})-\boldsymbol{Q}f_{2}(x_{2})\big\|^{2}\Big)^{\!1/2}-d\Big|\leq\tau_{1}+\tau_{2},
(5)

with dd the misalignment of the spectral embeddings of πW1\pi_{W_{1}} and πW2\pi_{W_{2}}, which Section 2.2 bounds. (Formal statement and proof in Appendix A.)

How much data, and which.

Views and dataset coverage jointly determine which observations become connected. Weak augmentations may need dense data to link observations of one state; augmentations that connect different states can impose a ceiling even with more data. Synthetic experiments illustrate both effects, including a reduced bad-view ceiling when qualitatively different data are mixed.

Figure 4: original paper visual, restored from the PDF
Figure 4: Positives that join less of a scene need more data (Section 2.3). A synthetic world of five classes of six states (literal data, as in Figure 12): alignment of a text JEPA with the image side against the samples per state NN and the reach Δ\Delta of its bad views (means over three samplings; a state has width 1/61/6). White: the 0.9 contour, and Δ∝log⁡N/N\Delta\propto\log N/N (dashed, one fitted constant), the scale of the largest gap between a state’s samples. Reach large enough to cross classes caps alignment (.95), and good views (top strip) align exactly at any NN. We observe that weaker positives need more data for the same alignment.

2.4 Next-Token Prediction is a Bad-View JEPA

Next-token cross-entropy can be written exactly as a generalized two-encoder JEPA objective: a context–target similarity term plus a log-sum-exp regularizer. At population optimum with sufficient rank and full support, logits recover conditional token probabilities up to a context-dependent offset.

This factorization is identifiable up to an invertible change of coordinates, not merely a rotation. The argument that a prefix and continuation retain shared linguistic nuisance is a proposed explanation of the experiments. It is not a corollary of the one-space, whitened spectral theorem, nor a result about all predictive world models.

Proposition 4: Next-Token Prediction Has the Form of a JEPA

Let a context cc be embedded by h⁡(c)∈ℝkh(c)\in\mathbb{R}^{k} and tokens by a table {wv}v∈V\{w_{v}\}_{v\in V}. The next-token loss with a linear softmax head is

Equation−log⁡e⟨h⁡(c),wy⟩∑ve⟨h⁡(c),wv⟩=−⟨h⁡(c),wy⟩⏟prediction+log∑ve⟨h⁡(c),wv⟩⏟anti-collapse ​A​(h⁡(c)).\begin{split}-\log\frac{e^{\langle h(c),w_{y}\rangle}}{\sum_{v}e^{\langle h(c),w_{v}\rangle}}={}&\underbrace{-\langle h(c),w_{y}\rangle}_{\text{prediction}}\\ &+\underbrace{\log\textstyle\sum_{v}e^{\langle h(c),w_{v}\rangle}}_{\text{anti-collapse }A(h(c))}.\end{split}
(6)

(i) This has the structure of a JEPA, generalizing (1) to two encoders and an anti-collapse term that depends on the target encoder: context encoder hh, target encoder y↦wyy\mapsto w_{y}, inner-product prediction, positives (c,y)(c,y) drawn from within one sequence. (ii) AA does not depend on the target token yy, and t​A​(h/t)→maxv⁡⟨h,wv⟩tA(h/t)\to\max_{v}\langle h,w_{v}\rangle as t→0t\to 0, so the temperature-scaled loss approaches the margin maxv⁡⟨h,wv⟩−⟨h,wy⟩≥0\max_{v}\langle h,w_{v}\rangle-\langle h,w_{y}\rangle\geq 0, which vanishes iff the target attains the maximal logit. (iii) With full-support conditionals and kk large enough, minimizers satisfy ⟨h⁡(c),wv⟩=log⁡p⁡(v∣c)+b⁡(c)\langle h(c),w_{v}\rangle=\log p(v\mid c)+b(c), a set invariant under GL⁡(k)\mathrm{GL}(k) (h↦𝑮​hh\mapsto\boldsymbol{G}h, w↦𝑮−⁣⊤​ww\mapsto\boldsymbol{G}^{-\top}w), so hh is identified at best up to an invertible linear map, not a rotation. (Proof in Appendix A.)

3 Good Views Improve Alignment

The controlled experiments train image and text encoders separately, varying whether positive views come from one observation or several observations of the same scene.

Setup.

Flickr training uses 29,000 images with five captions each. Image encoders receive crops; text JEPAs receive either masked spans of one caption or different captions describing one image. A 33M-parameter BERT-S backbone also trains with causal next-token prediction, including a control that sees two captions per image.

JEPA models share LeJEPA/SIGReg training components [5]. Default runs use 200 epochs and three seeds. Epoch and image exposure are matched, but JEPA and next-token training process different numbers of text tokens per step.

Metrics.

Mutual 10-NN measures agreement between neighborhood structures without fitting a cross-modal map. Retrieval instead fits an orthogonal map on 1,014 validation pairs and evaluates 1,000 held-out pairs. Chance is 0.01 for the neighborhood score and 0.001 for R@1.

Table 1: Good views align across modalities. Image JEPA vs. each text encoder (epoch 199) on 1,000 Flickr test pairs (backbone features; mean ±\pm half-range over three seeds): good-view JEPA (caption views), bad-view JEPA (masked spans), next-token prediction, and next-token prediction on pairs of captions of one image; R@1 and R@5 use a single rotation fitted on validation pairs. We clearly observe that caption views beat masked spans and next-token prediction on every metric, even when next-token prediction is trained on pairs of captions of the same image.
Text encoder mutual 10-NN R@1 R@5
Good-view JEPA .062±\pm.003 .044±\pm.002 .133±\pm.004
Bad-view JEPA .047±\pm.001 .027±\pm.002 .095±\pm.003
Next-token .042±\pm.001 .021±\pm.002 .077±\pm.005
Next-token, pairs .043±\pm.001 .023±\pm.003 .083±\pm.008
Random init .010 .001 .011
Chance .010 .001 .005
Original figure
Figure 5: Good views align, bad views and next-token prediction do not. Alignment between a separately trained image JEPA and three text encoders on Flickr, at matched epochs (mean of three seeds, band: range; dashed: chance). All start at chance. We clearly observe that the good-view JEPA (caption views) aligns markedly better than the bad-view JEPA (masked spans) and next-token prediction, which nearly coincide, as predicted by Sections 2.2 and 2.4.

3.1 Good Views Align Across Modalities

Good-view training reaches mutual 10-NN 0.062 and R@1 0.044, compared with 0.047/0.027 for bad-view training and 0.042/0.021 for next-token prediction. Giving next-token prediction a second caption scarcely changes these results.

Agreement concentrates in leading principal directions. Estimated state-explained variation is higher for good views, and measured rotation errors exceed the nuisance-based lower bounds. These finite-sample diagnostics are consistent with the proposed mechanism; they do not verify all population assumptions.

Original figure
Figure 6: Alignment lives in the leading spectral directions. Left: after the best rotation, correlation between image and text embeddings along the text embedding’s principal directions (good-view JEPA, at initialization and at epoch 199). Right: variance of the text embedding along the same directions. This suggests that alignment concentrates where the spectrum is well separated, consistent with Section 2.2, although noise in low-variance directions would produce the same trend.

3.2 Trimming Makes Good Views, the Rest Barely Matters

With a frozen DINOv2 image encoder, trimming captions is the largest tested recipe effect: removing it reduces R@1 from 0.246 to 0.185. Other individual changes have smaller mean effects in the reported sweep.

Training stability still matters. Some runs collapse, including three of 36 default runs without gradient clipping. Clipping helps but does not eliminate collapse under every learning-rate and regularization setting. Hyperparameters were tuned for good-view JEPA rather than equally optimized for all objectives.

Original figure
Figure 7: Trimming is the knob that matters. One change at a time to the good-view text JEPA on Flickr: R@1 with frozen DINOv2 through one rotation, last epoch, three seeds where they exist (faint dots; large dot: mean of the seeds that do not collapse; black: default). Top: the two changes to trimming, with the change of their mean from the default’s; whole captions as views lose a quarter of the alignment. Dashed: the bad-view JEPA and next-token prediction with the default recipe. That is, trimming matters most, and every variant of the good-view JEPA that does not collapse stays far above both baselines; gradient clipping prevents most collapses.

3.3 Good Views Help on Both Sides

On MSR-VTT, good image views use different frames and good text views use different captions of one clip. Mutual 10-NN rises from 0.039 with both sides bad to 0.067 with both good; improving only text gives 0.059 and only images 0.043.

Agreement with an independently pretrained image encoder can favor a different recipe: crop-trained images align better with DINOv2 than frame-trained images. Positive-view compatibility matters in addition to within-dataset retrieval.

Original figure
Figure 8: Good views help on both sides. MSR-VTT, 1,000 test clips. Top: image JEPAs trained on different frames of a clip (good) or crops of one frame (bad), against text JEPAs trained on different captions (good) or spans of one caption (bad); dots are the three seeds, dashed is chance. Bottom: alignment of each with frozen DINOv2, whose own positives are crops. This demonstrates that good views on both sides align best in mutual 10-NN, and that DINOv2 prefers the image JEPA whose positives match its own.
Original figure
Figure 9: Good views scale. Left and middle: text JEPAs trained on the texts of 2.16M to 940M web images (Recap-DataComp), never on a Flickr or COCO caption, against frozen DINOv2 ViT-B on WIT-1024 (PRH-paper protocol) and on Flickr (R@1 through one rotation fitted on 30k pairs). Solid lines and circles: BERT-S (33M); dashed lines and diamonds: BERT-B (109M), trained with next-token prediction and bad views on the texts of 940M images only, hence single diamonds there. Lines are means over seeds and faint dots the seeds (one seed for the bad-view JEPA at 940M). Shaded: the language models we evaluated (Qwen2.5 0.5–72B, Llama-3-8B and -70B, Mistral-7B; left); grey dashed: CLIP’s text encoder and Qwen2.5-0.5B under the same protocol (middle). Right: image JEPAs trained from scratch (ViT-S on 2.16M and ViT-B on 10M Recap images, ViT-S on ImageNet-1k) and frozen DINOv2, each against the good-view, next-token and bad-view text JEPAs trained on the same texts of 50M web images (WIT; means of three seeds, dots: seeds). We clearly observe that good views beat bad views and next-token prediction at every scale and with every image encoder we compared, and that a 109M good-view JEPA on the texts of 940M images surpasses every language model we evaluated.

3.4 Good-View JEPAs Beat a Pretrained Language Model and Approach Pair-Trained Encoders

Against frozen DINOv2, good-view Flickr text training reaches R@1 0.246, versus 0.133 for bad views, 0.118 for next-token prediction, and 0.135 for pretrained Qwen2.5-0.5B. These compare in-domain training with an out-of-domain pretrained model.

CLIP and supervised MPNet remain stronger on this retrieval setting at 0.320 and 0.300, although their mutual-neighborhood scores are lower. More fitting pairs, training text, or model capacity can improve retrieval; neither metric alone establishes a universal encoder ranking.

Table 2: Good-view JEPAs agree with DINOv2 more than the pretrained models we tested. Same protocol with a frozen DINOv2 ViT-B/14. Top two rows and bottom two: pretrained models (last layer); the others are ours (BERT-S unless named). All entries use final checkpoints, except the early-stopped row (selected on a loss over the validation and test captions, which can only favor this baseline), with no selection on test data (mean ±\pm half-range over three seeds where shown, seed 0 elsewhere). The last column fits the single map on all 30k Flickr train and validation pairs instead of 1k (–: we kept no epoch-29 checkpoint to embed the training captions with). No network is trained on pairs in either case. For reference, our Flickr image JEPA vs. DINOv2 (image vs. image) reaches mutual 10-NN .101 and R@1 .217. The rows above the first rule are trained on Flickr (and COCO) captions, so this comparison is in-domain for them and out-of-domain for most pretrained models (all-mpnet-base-v2, trained on a billion sentence pairs, saw Flickr and COCO captions); the Recap rows never saw a Flickr or COCO caption. CLIP’s text encoder was trained with 400M images. Bold: best of ours. This demonstrates that good-view JEPAs have the neighborhoods closest to DINOv2’s and, trained on web text alone, match or exceed the retrieval of text encoders trained on pairs.
Text encoder, vs. frozen mutual R@1, rotation fit on
DINOv2 ViT-B/14 10-NN 1k pairs 30k pairs
Qwen2.5-0.5B .098 .135 .240
BERT-base .093 .149 .273
bad-view JEPA .132±\pm.001 .133±\pm.007 .207±\pm.008
next-token .122±\pm.001 .118±\pm.002 .213±\pm.004
early-stopped .124±\pm.002 .134±\pm.008 –
next-token, pairs .132±\pm.002 .137±\pm.004 .247±\pm.016
good-view JEPA .232±\pm.006 .246±\pm.015 .346±\pm.007
good-view JEPA, BERT-B .245 .273 .363
good, BERT-T, +COCO .228 .238 .335
good, +COCO .228 .256 .367
good, BERT-B, +COCO .233 .285 .400
good, BERT-L, +COCO .227 .269 .376
good-view JEPAs on Recap texts, no Flickr or COCO caption
texts of 2.16M images .204±\pm.004 .236±\pm.019 .335±\pm.016
BERT-B .214±\pm.003 .272±\pm.010 .388±\pm.014
texts of 10M images .205±\pm.004 .243±\pm.004 .348±\pm.010
BERT-B .219±\pm.003 .280±\pm.003 .387±\pm.007
texts of 50M images .219±\pm.002 .283±\pm.002 .376±\pm.006
BERT-B .236±\pm.002 .300±\pm.011 .408±\pm.003
texts of 100M images .229±\pm.001 .309±\pm.007 .402±\pm.007
texts of 940M, BERT-B .257 .349 .464
all-mpnet-base-v2 .189 .300 .481
CLIP B/32 text .167 .320 .489
Figure 10: original paper visual, restored from the PDF
Figure 10: Captions without training a network on image–caption pairs. Selected captions of COCO test images (first lines) with the decoder of our demo: frozen DINOv2 and DINOv3 image embeddings, one linear map fitted on the 30k Flickr pairs, and a language model fine-tuned on text only (consensus of 32 samples; Table 10), with one human caption (grey). That is, frozen encoders joined by one rotation describe scenes, objects and actions, although only the rotation is fitted on image–caption pairs.

4 Next-Token Prediction Scales Poorly Compared to Good-View JEPAs

Scaling experiments train text encoders on grouped alt-text and synthetic captions associated with web images. The JEPA never receives pixels, but the grouping and recaptioning carry visual supervision.

4.1 Good Views Scale on Web Text

Recap-DataComp supplies an alt-text and two LLaVA captions per image [34]. A 33M good-view JEPA improves WIT alignment from 0.034 after Flickr training to 0.108 using texts from 2.16M images and about 0.125 at 50–100M images.

Matched-data comparisons favor good views: at 2.16M images, good-view, bad-view, and next-token scores are 0.108, 0.053, and 0.071. Better captions also help independently of the view objective; model width does not improve every alignment metric consistently.

4.2 Good Views Need 42 Times Fewer Samples

The reported 42× comparison matches approximately 0.11 WIT alignment using 21.6M training items for good-view JEPA versus 900M for next-token prediction. These counts include repeated exposure; they are not unique images or tokens. Because JEPA reads three texts per item, the corresponding token ratio is about 14×, while the unique-image ratio is about 435×.

The runs use different learning rates and batch sizes, and no earlier checkpoint of the 900M-item next-token run was evaluated. This is an observed comparison between two training recipes, not a measured minimum sample requirement.

4.3 A 109M Text JEPA Beats Language Models up to 72B Parameters

A 109M good-view JEPA trained for 900M items from the 940M-image text corpus reaches mean WIT alignment 0.184 across three reported seeds, versus 0.111 for matched-scale next-token prediction, 0.088 for bad-view JEPA, and 0.154 for Qwen2.5-72B. Other tested image encoders and alignment metrics support the good-view advantage.

The three-seed mean excludes a fourth run whose effective SIGReg weight was doubled and whose WIT score was 0.121; the four-run mean is 0.168. The regularization choice followed inspection of WIT results. LLaVA recaptioning inherits CLIP-based vision supervision, so these are not encoders trained without any image–text-derived information.

Table 3: At scale, good views beat language models up to 72B parameters. Text encoders against frozen vision models under the PRH protocol: mutual 10-NN and unbiased linear CKA on WIT-1024 with DINOv2 ViT-B, mutual 10-NN on WIT-1024 with CLIP ViT-H/14 and on the 5,000 COCO test pairs with DINOv2 ViT-B, and Flickr R@1 through one rotation fitted on 30k pairs. Our three encoders are BERT-B models (109M) trained with 900M samples from the texts of 940M web images (one seed for the bad-view JEPA); –: not measured. More models in Tables 16, 17 and 18. We clearly observe that the good-view JEPA agrees with both vision models more than any other text encoder, while next-token prediction on the same texts falls far behind.
WIT, DINOv2-B WIT COCO Flickr
Text encoder 10-NN CKA CLIP-H 10-NN R@1
good-view JEPA .184 .507 .230 .161 .478
next-token .111 – .146 – .335
bad-view JEPA .088 – .116 – .318
Qwen2.5-72B .154 .418 .210 .108 –
Llama-3-70B .151 .380 .206 .113 –
CLIP B/32 text .124 .339 .159 .132 .489
all-mpnet-base-v2 .118 .390 .159 .156 .481
Figure 11: original paper visual, restored from the PDF
Figure 11: A scaling law of alignment in good views and samples seen. Flickr test R@1 (DINOv2 ViT-B, 30k-pair rotation) of BERT-S text JEPAs trained on the texts of 10.2M web images, against the samples seen. Each point is a run, colored by its fraction qq of training items with good views (black: q=0q=0, bad views only). Circles: the grid the law is fitted on; diamonds: runs predicted before training. Lines: the law at five values of qq, dashed beyond the fitted range. We observe that R@1 grows linearly in the logarithm of the samples seen at every fraction of good views, faster with more of them, and that the law predicts unseen runs within seed noise up to twice its range.

4.4 A Scaling Law for the PRH

An empirical law relates retrieval to training items, unique images, and the fraction of good views. Fitted on 45 configurations, it predicts 85 subsequent runs with RMSE 0.0093, close to the 0.0096 variation between seed reruns.

Its exponents are weakly identified, and extrapolation to 320M items with repeated data overpredicts good-view retrieval. The formula omits a repeated-data penalty and applies to one studied recipe; it is not derived from the representation theorem.

Equation 7
R​@​1=a+d​log⁡M+b​log⁡(1+qα​Mβ​Nγ/c),\mathrm{R@1}=a+d\log M+b\log\big(1+q^{\alpha}M^{\beta}N^{\gamma}/c\big), (7)

5 Platonic Representations as a Tool

Shared geometry supports two distinct applications: captioning with a map fitted on paired examples, and estimating the map from unpaired embedding clouds.

5.1 Captioning Without Training a Network on Image–Caption Pairs

A text-only decoder reconstructs captions from noisy, frozen JEPA embeddings. At inference, a frozen image encoder and fitted orthogonal map supply its input. Noise compensates for residual image–text mismatch, following an approach related to noise-injected caption decoding [43].

The base captioner obtains Flickr CIDEr 0.396 using 1,014 map-fitting pairs and 0.431 using 30,000. With Flickr pairs only for the map and Flickr plus COCO captions for decoder training, it transfers to COCO at 0.668. CIDEr uses a 0–1 scale throughout this paper.

The strongest configuration combines a pretrained language decoder, a web-trained text JEPA, frozen image encoders, and consensus decoding. It reaches COCO CIDEr 0.838 with its map still fitted on 30,000 Flickr pairs. A rectangular row-orthonormal map performs better on COCO transfer than unconstrained linear or nonlinear maps in the tested comparison. This supports a useful constraint, without proving the exact equal-dimensional theorem applies.

Averaging eight frame embeddings also transfers the photo-trained captioner to video: MSR-VTT CIDEr is 0.246 without video training. It remains below the cited specialized video systems and often misses actions or speech.

5.2 Alignment Without Any Pair

A separate experiment uses the clustering and optimal-transport method of Schnaus et al. to align unpaired clouds [54]. Images and captions come from disjoint COCO training halves; evaluation pairs and class labels are not used to fit the map.

Across three JEPA training seeds and five alignment seeds each, the 109M model reaches FOSCTTM 0.018 and ImageNet-100 top-5 accuracy 64.4%, versus 0.018 and 47.6% for the Qwen3-8B rerun. CLIP still leads on FOSCTTM and ImageNet-100 at 0.006 and 68.7%. Bad-view JEPA also aligns well on COCO, so this setting does not isolate good views as the sole explanation.

On the smaller Flickr pools, only web-trained good-view JEPAs clearly exceed chance in the tested set. Pair-free captioning reaches CIDEr 0.158±0.042, or 0.208 for the map selected by the unpaired training objective, versus 0.430 with a 30,000-pair map. Fully pair-free results should therefore be distinguished from the strongest paired-map captioner.

Table 4: A 109M good-view JEPA beats every pair-free zero-shot accuracy of Schnaus et al. [54]. The protocol and method of Schnaus et al. [54], unchanged: DINOv2 ViT-B/14 and a text encoder aligned with zero pairs on unpaired halves of COCO train2014, scored on the 40,504 val2014 pairs (FOSCTTM, lower is better) and by zero-shot top-1 on CIFAR-10 and top-5 on CIFAR-100 and ImageNet-100 (%). Means over training seeds times the method’s five seeds (std in Appendix C). Their best: the best number of their paper for each metric over its 21 vision–language pairs, on these data and in any of its five settings (paper/data/schnaus2026_reported.csv); our reruns of their MPNet pair (all-mpnet-base-v2) reproduce their reported .04, 70, 46 and 45. Last row: the ceiling of Procrustes on all training pairs, range over all encoders. Our good-view JEPAs are trained on the texts of 940M and 100M web images. We clearly observe that a 109M good-view JEPA beats every zero-shot accuracy of Schnaus et al. [54], matches their best FOSCTTM on these data, and matches or significantly beats their 8B language model under identical conditions, while next-token prediction on the same texts fails; only CLIP’s text encoder, trained on 400M image–text pairs, aligns better on FOSCTTM and ImageNet-100.
Text encoder (runs) FOSCTTM C-10 C-100 IN-100
good-view JEPA, 109M (15) .018 86.0 67.2 64.4
good-view JEPA, 33M (15) .036 83.2 60.9 57.9
bad-view JEPA, 109M (5) .026 78.1 55.3 46.7
next-token, 109M (15) .287 25.5 19.9 20.3
Qwen3-8B, rerun (5) .018 83.4 61.7 47.6
MPNet, rerun (5) .043 70.2 45.6 45.4
their best, these data .02 80 58 46
their best, any setting .01 82 62 56
CLIP B/32 text (5) .006 84.0 67.0 68.7
82,783 pairs .002–.005 83–92 65–85 57–88

6 Related Work

The theory connects spectral accounts of self-supervised learning [4] [19] with prior work on augmentation, identifiability, and shared representations [74] [22]. Earlier cross-space matching methods already use geometry to align independently trained encoders [1] [54].

The experimental contribution is to connect positive-view design to alignment across model scales and to demonstrate useful downstream maps. Prior captioning systems often use jointly trained image–text encoders; this approach separately trains its encoders but still uses visual caption generation and, for its strongest captioner, paired map fitting.

7 Conclusion

Shared positive relationships provide sufficient conditions for common geometry, and different observations of one scene improve alignment in the tested JEPA recipes. They do not establish inevitable convergence for arbitrary models.

No experiment exactly satisfies Theorem 1: observations can be lossy, practical encoders and optimization are restricted, SIGReg differs from exact whitening, and image training often uses views classified as bad by the theory. Synthetic captions inherit multimodal supervision; good-view grouping uses scene correspondences. Hyperparameters were tuned for good-view JEPA, with WIT used to select the largest model's regularization. Evidence remains strongest for the controlled view comparisons and the reported image–text protocols.

Acknowledgments

The acknowledgments credit discussions, infrastructure, and research support. The authors provide code, experiment details, and additional results with the paper.

Appendix A Proofs

The appendix retains the formal objectives, statements, and derivations. The proofs establish sufficient conditions in the specified population models; numerical checks test the algebra rather than real-world applicability.

A.1 Notation and Objectives

Notation.

The notation represents a finite world by its probability diagonal matrix, symmetric positive-pair matrix, and normalized kernel. Embeddings use the probability-weighted inner product; centering removes the constant eigenfunction. Orthogonal alignment is evaluated in this weighted space.

Objectives of the form ( 1 ).

Whitened MSE reduces to maximizing a kernel trace under centered orthonormal coordinates. Rotation-invariant regularizers also include covariance-based objectives and the expected SIGReg penalty. Invariance alone does not establish uniqueness of minimizers; that remains a separate hypothesis outside the whitened spectral case.

A.2 The Platonic Representation Theorem ( Section 2.1 )

Injective renderings give a bijection between modality encoders and world-state embeddings, making their shared-positive objectives identical. Rotation uniqueness then proves Theorem 1. For whitened MSE, the Ky Fan variational principle selects the leading nonconstant eigenspace; the spectral gap makes that subspace unique.

Lemma 1: Renderings Only Change Coordinates

If gmg_{m} is injective, f↦f∘gmf\mapsto f\circ g_{m} is a bijection between encoders on gm​(𝒵)g_{m}(\mathcal{Z}) and functions 𝒵→ℝk\mathcal{Z}\to\mathbb{R}^{k}, and ℒm​(f)=ℒπm​(f∘gm)\mathcal{L}_{m}(f)=\mathcal{L}_{\pi_{m}}(f\circ g_{m}).

Equation
𝔼π​‖h⁡(z)−h⁡(z′)‖2=∑j=1k(2​𝔼p​hj2−2​𝔼π​hj​(z)​hj​(z′))=2​k−2​∑j=1k⟨hj,Tπ​hj⟩p.\mathbb{E}_{\pi}\|h(z)-h(z^{\prime})\|^{2}=\sum_{j=1}^{k}\big(2\mathbb{E}_{p}h_{j}^{2}-2\mathbb{E}_{\pi}h_{j}(z)h_{j}(z^{\prime})\big)=2k-2\sum_{j=1}^{k}\langle h_{j},T_{\pi}h_{j}\rangle_{p}.

A.3 When Positives Differ ( Section 2.2 )

A Davis–Kahan perturbation argument bounds changes in the selected eigenspace when positive-pair kernels differ [71]. Orthogonal Procrustes converts the subspace comparison into the stated embedding error. The appendix also explains why whitening erases the first-two-moment information needed to choose a rotation, whereas full-rank paired observations identify it.

Equation
‖𝑽1​𝑶^−𝑽2‖F≤23/2​min⁡(k​‖𝑲1−𝑲2‖2,‖𝑲1−𝑲2‖F)δ,\|\boldsymbol{V}_{1}\hat{\boldsymbol{O}}-\boldsymbol{V}_{2}\|_{F}\leq\frac{2^{3/2}\min\big(\sqrt{k}\,\|\boldsymbol{K}_{1}-\boldsymbol{K}_{2}\|_{2},\|\boldsymbol{K}_{1}-\boldsymbol{K}_{2}\|_{F}\big)}{\delta},

A.4 Good-View and Bad-View JEPAs ( Section 2.3 )

Conditioning on the world state separates shared variation from nuisance. Covariance and nuclear-norm inequalities yield the nuisance lower bound. Product-kernel eigenfunctions explain when persistent nuisance outranks state information; a block-operator perturbation argument then handles state–nuisance mixing in the general case.

Proposition 3 (Formal): Good Views Ignore Nuisance, Bad Views Capture It

Modality mm has a finite nuisance set 𝒩m\mathcal{N}_{m} with distribution qmq_{m}, qm​(ν)>0q_{m}(\nu)>0, the state and the two nuisances being mutually independent, and renders (z,ν)(z,\nu) injectively as x=gm​(z,ν)x=g_{m}(z,\nu); a paired image and caption are (g1​(z,ν1),g2​(z,ν2))(g_{1}(z,\nu_{1}),g_{2}(z,\nu_{2})) with ν1,ν2\nu_{1},\nu_{2} independent. A good-view JEPA’s positives are (gm​(z,ν),gm​(z′,ν′))(g_{m}(z,\nu),g_{m}(z^{\prime},\nu^{\prime})) with (z,z′)∼π(z,z^{\prime})\sim\pi and ν,ν′∼qm\nu,\nu^{\prime}\sim q_{m} independent; a bad-view JEPA’s are two corruptions x~,x~′∼Γm(⋅∣x)\tilde{x},\tilde{x}^{\prime}\sim\Gamma_{m}(\cdot\mid x) of one observation xx, which share its nuisance. Take the objective of Section 2.1. (i) A good-view JEPA’s kernel is 𝚷⊗qm​qm⊤\boldsymbol{\Pi}\otimes q_{m}q_{m}^{\top}; if μk>max⁡(μk+1,0)\mu_{k}>\max(\mu_{k+1},0), its minimizers are 𝑸​𝚽π​(z)\boldsymbol{Q}\boldsymbol{\Phi}_{\pi}(z), functions of the state alone, and good-view JEPAs of any number of modalities with the same world kernel π\pi agree up to one rotation each on every observation: Section 2.1 when every state has many observations. (ii) Let a bad-view JEPA corrupt state and nuisance independently, Γm=𝑨⊗𝑩m\Gamma_{m}=\boldsymbol{A}\otimes\boldsymbol{B}_{m} with 𝑨\boldsymbol{A} and 𝑩m\boldsymbol{B}_{m} row-stochastic with stationary laws pp and qmq_{m}, so that corrupted views are observations with the law of clean ones, and let 1=α0≥α1≥⋯≥01=\alpha_{0}\geq\alpha_{1}\geq\dots\geq 0 and 1=β0≥β1≥⋯≥01=\beta_{0}\geq\beta_{1}\geq\dots\geq 0 be the eigenvalues of the Markov operators 𝑨∗​𝑨\boldsymbol{A}^{*}\boldsymbol{A} on L2​(p)L^{2}(p) and 𝑩m∗​𝑩m\boldsymbol{B}_{m}^{*}\boldsymbol{B}_{m} on L2​(qm)L^{2}(q_{m}) of two corruptions of one state and of one nuisance (𝑨∗\boldsymbol{A}^{*} the adjoint; αi:=−∞\alpha_{i}:=-\infty for i≥|𝒵|i\geq|\mathcal{Z}|), with eigenfunctions ψi\psi_{i} and χj\chi_{j} (ψ0=χ0=𝟏\psi_{0}=\chi_{0}=\mathbf{1}). Its kernel then has eigenvalues αi​βj\alpha_{i}\beta_{j} with eigenfunctions ψi​(z)​χj​(ν)\psi_{i}(z)\chi_{j}(\nu); let ξ1≥ξ2≥…\xi_{1}\geq\xi_{2}\geq\dots be those with (i,j)≠(0,0)(i,j)\neq(0,0), with multiplicity. If ξk>ξk+1\xi_{k}>\xi_{k+1}, rr of the top kk have j≥1j\geq 1, its minimizers ff have 𝑪f\boldsymbol{C}_{f} with rr zero eigenvalues, i.e., ρ⁡(f)=1−r/k\rho(f)=1-r/k, and by Section 2.3 their alignment error with any centered, whitened encoder of the other modality is at least 2​r2r. Moreover r≥1r\geq 1 if and only if β1>αk\beta_{1}>\alpha_{k}.

Theorem 4 (Formal): Views and Data Set the World Kernel

Let modality mm have finitely many observations xx, each of a world state sm​(x)s_{m}(x), with law ωm\omega_{m} (ωm​(x)>0\omega_{m}(x)>0) of state marginal pp, and let WmW_{m} be any symmetric law on pairs of observations whose marginal is ωm\omega_{m}, the law on which encoders are whitened and evaluated: good views, Wm​(x,x′)=π⁡(sm​(x),sm​(x′))​ωm​(x∣sm​(x))​ωm​(x′∣sm​(x′))W_{m}(x,x^{\prime})=\pi(s_{m}(x),s_{m}(x^{\prime}))\,\omega_{m}(x\mid s_{m}(x))\,\omega_{m}(x^{\prime}\mid s_{m}(x^{\prime})); corruptions Γm\Gamma_{m} of one observation that leave the law of observations invariant (ωm​Γm=ωm\omega_{m}\Gamma_{m}=\omega_{m}, as in Section 2.3); or a mix. Its induced world kernel πWm​(z,z′)\pi_{W_{m}}(z,z^{\prime}) is the probability that a positive pair joins an observation of zz to one of z′z^{\prime}. Let PP be the conditional expectation onto functions of the state in L2​(ωm)L^{2}(\omega_{m}), let ηm\eta_{m} be the largest eigenvalue of the compression (I−P)​TWm​(I−P)(I-P)T_{W_{m}}(I-P) to nuisance functions (P​h=0Ph=0), which measures how much of its own nuisance a positive keeps (ηm=−∞\eta_{m}=-\infty if every state has one observation), and let εm\varepsilon_{m} and εmHS\varepsilon_{m}^{\mathrm{HS}} be the operator and Hilbert–Schmidt norms of (I−P)​TWm​P(I-P)T_{W_{m}}P, which measure how much state and nuisance mix. Take the objective of Section 2.1 with 1≤k≤|𝒵|−11\leq k\leq|\mathcal{Z}|-1, let μ1≥μ2≥…\mu_{1}\geq\mu_{2}\geq\dots be the eigenvalues of TπWmT_{\pi_{W_{m}}} on 𝟏⟂\mathbf{1}^{\perp} (μ|𝒵|:=−∞\mu_{|\mathcal{Z}|}:=-\infty), and suppose δm=μk−max⁡(μk+1,ηm)>0\delta_{m}=\mu_{k}-\max(\mu_{k+1},\eta_{m})>0. (i) Every spectral solution fmf_{m} of WmW_{m} (orthonormal top-kk eigenfunctions of TWmT_{W_{m}} on 𝟏⟂\mathbf{1}^{\perp}) is close to a spectral embedding 𝚽(m)\boldsymbol{\Phi}^{(m)} of its induced world kernel,

Equationmin𝑶∈O⁡(k)⁡‖fm−𝑶​𝚽(m)∘sm‖L2​(ωm)≤τm:=2​2δm​min⁡(k​εm,2​εmHS),\min_{\boldsymbol{O}\in\mathrm{O}(k)}\big\|f_{m}-\boldsymbol{O}\,\boldsymbol{\Phi}^{(m)}\!\circ s_{m}\big\|_{L^{2}(\omega_{m})}\leq\tau_{m}:=\frac{2\sqrt{2}}{\delta_{m}}\min\big(\sqrt{k}\,\varepsilon_{m},\sqrt{2}\,\varepsilon_{m}^{\mathrm{HS}}\big),

and tr​𝑪fm≥k−τm2\mathrm{tr}\,\boldsymbol{C}_{f_{m}}\geq k-\tau_{m}^{2}. (ii) For two modalities of one world, over paired observations from any coupling with z∼pz\sim p and xm∣z∼ωm(⋅∣z)x_{m}\mid z\sim\omega_{m}(\cdot\mid z),

Equation|min𝑸∈O⁡(k)⁡(𝔼​‖f1​(x1)−𝑸​f2​(x2)‖2)1/2−d|≤τ1+τ2,\Big|\min_{\boldsymbol{Q}\in\mathrm{O}(k)}\Big(\mathbb{E}\big\|f_{1}(x_{1})-\boldsymbol{Q}f_{2}(x_{2})\big\|^{2}\Big)^{1/2}-d\Big|\leq\tau_{1}+\tau_{2},

where d=min𝑸⁡‖𝚽(1)−𝑸​𝚽(2)‖L2​(p)d=\min_{\boldsymbol{Q}}\|\boldsymbol{\Phi}^{(1)}-\boldsymbol{Q}\boldsymbol{\Phi}^{(2)}\|_{L^{2}(p)} is at most 2​2δ​min⁡(k​‖𝑲πW1−𝑲πW2‖2,‖𝑲πW1−𝑲πW2‖F)\frac{2\sqrt{2}}{\delta}\min\big(\sqrt{k}\,\|\boldsymbol{K}_{\pi_{W_{1}}}-\boldsymbol{K}_{\pi_{W_{2}}}\|_{2},\|\boldsymbol{K}_{\pi_{W_{1}}}-\boldsymbol{K}_{\pi_{W_{2}}}\|_{F}\big) by Section 2.2, with δ≥δ2\delta\geq\delta_{2} the gap μk−μk+1\mu_{k}-\mu_{k+1} of 𝑲πW2\boldsymbol{K}_{\pi_{W_{2}}}.

Special cases and scope.

Independent nuisance resampling eliminates the nuisance block and mixing term. Bad-view product kernels have no mixing but can retain nuisance eigenvalues large enough to displace state directions. Merging observations of one state can reduce nuisance; connecting different states changes the world kernel. Soft architectural biases and noninjective renderings require analysis beyond the exact theorem.

A.5 Next-Token Prediction is a Bad-View JEPA ( Section 2.4 )

Expanding cross-entropy produces the context–target attraction term and log-sum-exp regularizer. At low temperature the normalized objective approaches a margin form. Full-support optimum logits encode log conditional probabilities, while their two-factor decomposition admits general invertible coordinate changes. These calculations do not import the whitened JEPA nuisance theorem into next-token prediction.

A.6 Numerical Checks

Random finite-world checks verify the stated perturbation and nuisance inequalities, including 300 differing-kernel pairs. Observed errors remain below the analytical bounds. These checks support implementation of the formulas; they do not test their assumptions for trained neural encoders.

Appendix B Experimental details

Code.

Experiments use stable-pretraining's LeJEPA/SIGReg implementation. A multilayer projector and 1,024 random SIGReg projections regularize the latent distribution. The supplied implementation and retained settings specify the reproduction recipe.

Image encoder.

The small image encoder is ViT-S/16 with CLS pooling. Two global and six local crops undergo color and blur augmentations. Frozen pretrained vision encoders are evaluated separately from image JEPAs trained from scratch.

Text encoders.

The main small text backbone has 12 layers, six attention heads, width 384, and a 64-token limit. JEPA uses CLS pooling; causal next-token models use mean pooling for evaluation, which performs better than their last-token representation. Good views sample different captions of one image before trimming and masking.

Optimization.

Default training uses AdamW, batch 256, learning rate 5×10−4, weight decay 0.05, and 200 epochs with warmup and cosine decay. Longer and larger runs use revised rates and gradient clipping. Distributed training changes the effective SIGReg weight unless its scaling is accounted for; the largest-model appendix reports the resulting seed discrepancy.

MSR-VTT.

MSR-VTT uses 7,010 training clips, 1,990 map-fitting clips, and 1,000 retrieval-test clips. Good views sample different frames or captions from one clip; bad views reuse one observation. Runs train for approximately 21,600 updates and average three seeds.

PRH protocol.

The PRH evaluation uses 1,024 WIT image–caption pairs, the first caption per image, and layerwise visual CLS and mean-pooled language features. Features are clipped and normalized before mutual 10-NN evaluation. Reported scores maximize over layer pairs, making protocol and dataset size important when comparing small differences.

Appendix C Additional results

The additional experiments probe training stability, data quality, uncertainty, benchmark overlap, captioning, and alternative encoders. All original result tables and qualitative examples are retained.

Table 5: Good views also make better text encoders. Text encoders alone (caption→\tocaption retrieval among the 5,000 Flickr test captions: is the nearest caption one of the four others describing the same image?) and alignment measured on projector outputs (seed 0; –: no trained projector). We clearly observe that the bad-view JEPA learns little beyond word overlap (below TF-IDF), while the good-view JEPA goes well beyond it. The image JEPA reaches 78.5% 20-NN accuracy on Imagenette [21] (29.1% at initialization).
caption→\tocaption R@1 mutual 10-NN (projector) R@1 (projector) whitened R@1 (projector)
Good-view JEPA .606 .064 .030 .034
Bad-view JEPA .381 .044 .018 .022
Next-token prediction .331 – – –
TF-IDF .442 – – –
Random init .178 – – –

Text encoders alone.

Table 5 evaluates text encoders and projector outputs without jointly retraining the image encoder. Retrieval and neighborhood agreement can rank models differently. Later figures preserve captioning examples, including errors and differences between fitted maps.

Next-token overfitting.

The Flickr next-token model overfits: validation loss rises after epoch 29. Selecting that validation-loss minimum improves R@1 from 0.118 to 0.134, still below good-view JEPA's 0.246. A test-selected checkpoint is reported diagnostically, not used as the main result. JEPA also processes more caption tokens per step.

Nuisance on real encoders.

Real-encoder nuisance estimates whiten principal components and use repeated captions to estimate scene-explained covariance. A finite-caption correction and eigenvalue clipping stabilize the estimate. Higher estimated state information accompanies lower alignment error, but these are finite-sample proxies for the population quantities.

Table 6: Bad views spend their embedding on nuisance. Scene-explained fraction ρ⁡(f)=tr​𝑪f1/2/k\rho(f)=\mathrm{tr}\,\boldsymbol{C}_{f}^{1/2}/k of each text encoder’s top-kk whitened embedding (mean ±\pm half-range over three seeds), the lower bound 2​k​(1−ρ⁡(f))2k(1-\rho(f)) of Section 2.3 on the best-rotation error against DINOv2 ViT-B, and that error (means over seeds; bounds from the unrounded fractions; uncorrelated embeddings give 2​k2k, i.e., 64 and 256). We clearly observe that the bound holds for every encoder, and the scene-explained fraction ranks the encoders as their alignment does.
k=32k=32 k=128k=128
Text encoder (Flickr) scene-explained bound error scene-explained bound error
Good-view JEPA .804±\pm.003 12.6 33.5 .654±\pm.005 88.6 157.0
Bad-view JEPA .655±\pm.005 22.1 39.4 .507±\pm.003 126.1 171.5
Next-token .608±\pm.001 25.1 40.3 .440±\pm.002 143.4 176.6
trained on Recap texts only (seed 0), measured on the same Flickr captions
Good-view JEPA, 50M images .802 12.7 32.4 .642 91.7 156.8
Bad-view JEPA, 50M images .673 20.9 37.6 .506 126.4 168.7
Next-token, 50M images .618 24.4 40.1 .483 132.3 171.0
Good-view JEPA, 100M images .806 12.4 32.0 .661 86.9 153.7
Good-view JEPA, BERT-B, 940M images .830 10.9 30.0 .722 71.1 145.3

Which texts give the gain.

Caption quality and view construction both contribute. Alt-text plus text-only Qwen paraphrases gives WIT alignment 0.060, compared with 0.107 when visual recaptions are available. Even bad-view training benefits from recaptions. The largest gains therefore cannot be attributed to a text-only objective independently of the captioner's visual knowledge.

Table 7: The views and the data both set the alignment. BERT-S text JEPAs, one recipe per scale, three seeds (mean and range), scored on WIT-1024 with DINOv2 ViT-B and on Flickr through one rotation fitted on 30k pairs; scene-explained: ρ⁡(f)\rho(f) on Flickr captions (k=32k=32, seed 0, Table 6). The paraphrase is written from the alt-text alone by a text-only language model, so that row involves no image captioner. We observe that with the same bad views, LLaVA recaptions align far better than alt-texts, and with the same texts, good views double alignment (Section 2.3).
Texts of texts per image views WIT (DINOv2-B) Flickr R@1 scene-explained
2.24M images alt-text, two LLaVA recaptions good .107 (.095–.115) .339 (.336–.344) .793
2.24M images alt-text, text-only paraphrase good .060 (.058–.061) .233 (.230–.235) .654
2.24M images two LLaVA recaptions good .080 (.076–.082) .348 (.346–.351) .778
2.24M images alt-text bad .055 (.054–.056) .180 (.179–.180) .606
50M images alt-text, two LLaVA recaptions good .125 (.118–.129) .376 (.370–.381) .802
50M images two LLaVA recaptions good .096 (.093–.100) .386 (.373–.396) .789
50M images alt-text, two LLaVA recaptions bad .063 (.062–.066) .290 (.283–.301) .673
50M images two LLaVA recaptions bad .092 (.090–.093) .325 (.323–.329) .734
50M images alt-text bad .056 (.056–.057) .199 (.193–.203) .618

Data kind in the synthetic world.

Synthetic worlds separate denser sampling from adding different kinds of observations. More samples can connect nearby views, but mixing data types leaves a bad-view alignment ceiling in the studied construction. Good views recover the intended state geometry under the model's exact assumptions.

Original figure
Figure 12: Weak views need many data, the kind of data sets the ceiling, good views need neither (Section 2.3). A synthetic world of five classes of six states; the image side is a good-view JEPA whose world kernel links states of one class. The text side observes each state through samples on a line, one block per class, with gaps between blocks (left, literal data) or without (right, mixed data); its bad views join two samples with weight e−d2/2Δ2e^{-d^{2}/2\Delta^{2}} (weak Δ=.005\Delta=.005, medium .08.08, strong 11; a state has width 1/61/6). Alignment is the mean canonical correlation of the two spectral solutions over paired observations (k=4k=4; 1: equal up to a rotation); lines are means over five samplings, bands their range.

The scaling law in detail.

The scaling fit evolved from smaller configuration sets to 45 cells; the final held-out exercise covers 85 prospective runs. Several exponent combinations fit similarly, so individual exponents should not be interpreted as established laws. At 320M items, predictions overshoot good-view retrieval after many passes through the same images, exposing the missing repeated-data effect.

Equation
R​@​1=a+d​log⁡M+b​log⁡(1+qα​Mβ​Nγ/c),\mathrm{R@1}=a+d\log M+b\log\!\big(1+q^{\alpha}M^{\beta}N^{\gamma}/c\big),

Robustness of the good-view recipe.

Good-view alignment can collapse after a strong earlier checkpoint. Clipping reduces failures, but high learning rates and some regularization weights still collapse; some failed larger runs are excluded. Very weak regularization can produce a low-rank representation despite reasonable retrieval. Best-checkpoint columns selected using test retrieval are diagnostics rather than the main selection rule.

Table 8: Cropping matters, the rest barely moves the mean, and clipping prevents most collapses. R@1 against frozen DINOv2 on Flickr test pairs with one rotation fitted on validation pairs (chance .001), last epoch, for three seeds where shown (one otherwise); a run collapses (×) when its last-epoch R@1 falls below .05. Default views: 2 global and 6 local spans of 50–100% and 15–50% of a caption. Mean: over the seeds that do not collapse (–: every seed collapses). ∗Best over checkpoints on the test pairs, for reference only; the other columns involve no selection. That is, every run that does not collapse stays far above the bad-view JEPA and next-token prediction, and without gradient clipping 3 of the 36 runs with the default SIGReg weight collapse. In stable-pretraining as we ran it, SIGReg is scaled by the number of embeddings it sees, so the batch and view rows also change its effective weight: ×2\times 2 for batch 512, ×12\times\frac{1}{2} for 2 + 2 views and ×1.5\times 1.5 for 2 + 10 views.
Good-view text JEPA, one change from the default last-epoch R@1, seeds 0 / 1 / 2 mean best∗ (epoch), seed 0
Default: 2 + 6 views, mask 15%, lr 5⋅10−45\cdot 10^{-4}, λ=0.02\lambda=0.02 .248 / .260 / .230 .246 .276 (99)
   BERT-B (768 wide; lr 10−410^{-4}, clipping 1) .273 / .252 / .255 .260 .275 (189)
   BERT-T (192 wide) .244 / .229 / .228 .234 .254 (149)
   no masking .236 / .246 / .244 .242 .259 (109)
   mask 30% .243 / × / .264 .254 .266 (129)
   no trimming (whole captions) .193 / .182 / .180 .185 .242 (79)
   milder trimming (70–100% / 40–80%) .211 / .204 / .191 .202 .268 (59)
   2 + 2 views .247 / × / × .247 .257 (129)
   2 + 2 views, clipping 1 .260 / .258 / .267 .262 .272 (159)
   2 + 10 views .254 / .247 / .243 .248 .262 (119)
   4 + 4 views .242 / .249 / .248 .246 .258 (109)
   lr 2⋅10−42\cdot 10^{-4} .265 / .246 / .253 .255 .270 (169)
   lr 10−310^{-3} (clipping 1) collapses – –
   batch 512 .239 / .243 / .240 .241 .245 (129)
   100 epochs .233 / .261 / .248 .247 .239 (59)
   λ=0.005\lambda=0.005 collapses – .196 (29)
   λ=0.005\lambda=0.005, clipping 1 .238 .238 .242 (119)
   λ=0.05\lambda=0.05 collapses – .242 (89)
   λ=0.05\lambda=0.05, clipping 1 collapses – .162 (19)
   λ=0.1\lambda=0.1 collapses – .191 (79)
   λ=0.1\lambda=0.1, clipping 1 .202 .202 .215 (149)
   1,000 epochs, stopped at 359 (lr 10−410^{-4}, clipping 1) .242 (359) .242 .278 (99)
Table 9: Captioning from independently trained encoders. BLEU-4 [45], CIDEr [62] on Flickr test images. The decoder never sees an image; the image encoder never sees text; pairs only fit one linear map. BLEU-4 rewards generic n-grams, so the random map’s fluent but generic captions score above 1,014-pair retrieval on BLEU-4, not on CIDEr. We clearly observe that decoding through the rotation beats retrieving captions through it by a wide margin.
Captioner (Flickr test) BLEU-4 CIDEr
Oracle: decode the true caption embedding .261 .628
Image →\to rotation (30k pairs) →\to decode .185 .431
Image →\to rotation (1,014 pairs) →\to decode .173 .396
Image →\to rotation (300 pairs) →\to decode .146 .317
Image →\to rotation (100 pairs) →\to decode .119 .244
+ unpaired refinement (100 pairs) .138 .294
+ unpaired refinement (30 pairs) .099 .169
Retrieve nearest caption (30k pairs) .065 .188
Retrieve nearest caption (1,014 pairs) .054 .158
Random rotation .059 .043
Original figure
Figure 13: The captioner’s noise and pairs are predicted, not tuned. Flickr test images. Top: CIDEr against the noise the decoder is trained with; dashed, the noise matched to the residual of the rotation (rotation from 1,014 pairs). Bottom: CIDEr against the number of pairs that fit the one linear map, with and without unpaired refinement, against retrieving the nearest caption through the same map; dashed, decoding the true caption embedding. We clearly observe that transfer peaks at the matched noise, and that a few pairs refined on unpaired data match caption retrieval from 1,014 pairs.
Table 10: The captioner improves without pairs. Building the captioner up one change at a time: Flickr test (1,000 images) and COCO Karpathy test (5,000 images), beam 3 unless stated; consensus rows choose among the greedy caption and 32 samples. The text JEPA is BERT-S trained on the texts of 100M web images; the decoders are trained on its embeddings of Flickr and COCO training captions only, with noise matched to the alignment (σ=1.9\sigma=1.9); the image encoders are frozen. The first row is the captioner of Table 9: on Flickr, the Flickr text JEPA and decoder (σ=3\sigma=3); on COCO, the Flickr and COCO ones (σ=2\sigma=2); it decodes greedily on Flickr and with beam 3 on COCO. Decoding the true caption embedding gives .694 and 1.305 (captions-as-written row), and .743 and 1.369 with the web-pretrained decoder. Only the rotation is fitted on image–caption pairs. Without consensus decoding, the best captioner reaches .496 and .806. This demonstrates that stronger frozen encoders, a pretrained language model and consensus decoding each add to the captioner, from CIDEr .668 to .838 on COCO, while a larger decoder does not.
Captioner (CIDEr; one rotation from the 30k Flickr pairs) Flickr COCO
Small decoder, DINOv2 ViT-B, in-domain text JEPA (Table 9) .431 .668
Qwen2.5-1.5B decoder, web text JEPA, DINOv2 ViT-g (CLS) .466 .720
+ mean patch token .480 .756
+ captions as written (case, punctuation) .479 .765
+ DINOv3 ViT-L, concatenated .496 .806
+ consensus of 32 samples .519 .838
Variants of the DINOv3 ViT-L row
DINOv3 ViT-7B instead of ViT-L .507 .816
decoder first trained on 4M web sentences .504 .802
    map trained on the same pairs instead: any linear map .531 .635
    map trained on the same pairs instead: two-layer MLP .498 .585
    + consensus of 32 samples (the demo) .538 .832
     map trained on the same pairs instead: any linear map .561 .652
     map trained on the same pairs instead: two-layer MLP .511 .597
Qwen2.5-3B decoder (captions-as-written row) .460 .730

Few pairs and no pairs.

Refinement on unpaired clouds helps when the initial paired map is underdetermined: with 100 pairs, Flickr CIDEr improves from 0.244 to 0.294. Thirty pairs reach 0.169. With no pairs, the original Flickr-only encoder and small clouds remain near chance; successful pair-free results use stronger web-trained representations and larger pools.

Video.

Averaging frames improves video alignment and captioning, but photo-derived descriptions frequently omit motion or speech. The main video result uses a map fitted on 30,000 Flickr pairs. A separate demo configuration adds 200,000 web pairs for map fitting and reaches CIDEr 0.268; it is not the same pairing budget.

Table 11: Captions and retrieval for video, from photos alone. Top: zero-shot captioning of the MSR-VTT test clips (20 references each; CIDEr on a 0–1 scale) by our captioner, whose networks never saw a video, a video–caption pair or MSR-VTT text, and by captioners built on CLIP’s encoders without MSR-VTT text, with one trained on MSR-VTT captions for reference (–: not reported). Bottom: alignment (mutual 10-NN fits nothing) and retrieval through one map on the 1k-A split, against zero-shot CLIP. We observe that a rotation fitted on photos captions and retrieves videos, above captioners built on CLIP that use no MSR-VTT text, though CLIP’s jointly trained encoders retrieve better.
MSR-VTT test, 2,990 clips CIDEr BLEU-4 METEOR ROUGE-L
Ours: mean of 8 frames, rotation from 30k Flickr pairs, consensus of 32 .246 .166 .206 .444
   beam 3 .222 .154 .204 .432
   map from 30k Flickr and 200k web pairs (the demo’s) .268 .186 .232 .471
    middle frame instead of the mean of 8 .242 .172 .219 .457
   oracle: decode the text embedding of one human caption .383 .257 .265 .538
CLIP ViT-B/32 encoders (400M image–text pairs), no MSR-VTT text; from Li et al. [33], Tab. 4
ZeroCap [60] .058 – – –
MAGIC [59] .074 – – –
DeCap, decoder on CC3M captions .150 – – –
DeCap, decoder on COCO captions .186 .147 .204 –
DeCap, decoder on MSR-VTT captions .348 – – –
MSR-VTT 1k-A (1,000 clips) map fitted on mutual 10-NN text→\tovideo R@1/5/10 video→\totext R@1
ours, mean of 8 frames 30k Flickr pairs .167 16.5 / 38.8 / 49.1 18.6
ours, mean of 8 frames 9k MSR-VTT clips .167 19.3 / 45.6 / 57.9 20.5
ours, middle frame 30k Flickr, 200k web pairs .148 14.5 / 33.1 / 44.9 17.1
CLIP ViT-B/32 [35] – (jointly trained) – 31.2 / 53.7 / 64.2 27.2

Alignment without any pair, in detail.

Pair-free alignment retains the published method's hyperparameters and five random seeds. Test pairs and labels score results only; method-seed selection uses the unpaired fitting objective. Some encoder and dataset combinations fail across most seeds, and Flickr retrieval varies substantially even for the strongest model.

The 109M results combine five alignment seeds within each of three training seeds. These 15 runs are nested rather than 15 independent trained encoders, which limits interpretation of the reported significance tests. CLIP remains stronger on some COCO metrics, and bad-view JEPA also succeeds there.

Decoder noise.

Decoder noise is chosen to approximate the cosine mismatch between aligned image and text embeddings. The initial calculation used test-set cosine; a later held-out validation calculation gives a similar recommendation near noise scale three. This supports the heuristic but does not make the original choice prospectively validation-only. Small score changes below about 0.01 are within the observed decoding variability.

Scaling the captioner.

The larger decoder fine-tunes Qwen2.5-1.5B on caption text, using a small network to convert the JEPA vector into ten soft-prompt tokens. Consensus decoding selects among sampled captions using agreement between candidates, without human reference captions at selection time [27].

Combining normalized DINOv2 and DINOv3 features improves the tested captioner. Text-only web pretraining improves some reconstruction measures without consistently improving benchmark transfer. The main strongest captioner still uses 30,000 paired examples for its cross-modal map.

Projector outputs.

Whitening projector outputs changes R@1 only slightly, from 0.030 to 0.034 in the reported comparison. Their existing near-isotropy limits the effect; this does not show whitening is irrelevant to other encoders or objectives.

Transfer to COCO.

A map fitted on 30,000 Flickr pairs transfers to COCO at R@1 0.113 among 5,000 candidates. Each modality can be centered using its own unpaired data. Fitting on half the COCO evaluation set and testing on the other half is a separate protocol with fewer candidates, so its score is not directly comparable.

Frozen image encoders.

Larger frozen image encoders can improve retrieval without producing the best captions. DINOv3-7B has strong retrieval, while DINOv2-g is more effective for some captioning settings. Concatenating DINOv2-g and DINOv3-L features yields a useful tested quality–cost compromise, not a general model ranking.

Table 12: Linear-evaluation features and concatenated encoders align best. Flickr test pairs, text JEPAs trained on Recap texts only (seed 0), each against a frozen self-supervised image encoder through one map with orthonormal rows fitted on the 30k Flickr train and validation pairs (chance: mutual 10-NN .01, R@1 .001). Bottom rows: the CLS + mean patch features of two encoders, each rescaled to unit root-mean-square per coordinate, concatenated. No network is trained on pairs. Notably, with DINOv3 ViT-7B, one rotation retrieves the true caption first for more than half of the test images.
BERT-B, texts of 50M images BERT-S, 100M
Frozen image encoder features mutual 10-NN R@1 R@5 R@1 R@5
DINOv2 ViT-B/14 CLS .234 .412 .733 .400 .711
DINOv2 ViT-L/14 CLS .242 .437 .745 .410 .732
DINOv2 ViT-g/14 CLS .245 .437 .769 .425 .741
DINOv2 ViT-g/14 CLS + mean patch .250 .460 .784 .440 .763
DINOv3 ViT-L/16 CLS .209 .436 .722 .414 .703
DINOv3 ViT-L/16 CLS + mean patch .234 .514 .792 .489 .780
DINOv3 ViT-H+/16 CLS .181 .353 .625 .342 .607
DINOv3 ViT-H+/16 CLS + mean patch .228 .479 .779 .486 .775
DINOv3 ViT-7B/16 CLS .207 .455 .752 .452 .730
DINOv3 ViT-7B/16 CLS + mean patch .231 .531 .803 .523 .798
I-JEPA ViT-H/14 (IN-22k) mean patch .109 .188 .451 .183 .436
I-JEPA ViT-g/16 (IN-22k) mean patch .140 .246 .502 .234 .491
DINOv2-g + DINOv3-L both, concatenated .255 .505 .806 .488 .791
DINOv2-g + DINOv3-7B both, concatenated .256 .535 .824 .511 .801
Table 13: Good views help on both sides (numbers behind Figure 8). MSR-VTT, 1,000 test clips, final checkpoints, mean ±\pm half-range over three seeds (chance: mutual 10-NN .01, R@1 .001). R@1 and R@5 use one rotation fitted on the 1,990 validation clips. Every seed ranks the four combinations the same way in mutual 10-NN; the image-side effect is small and reverses on R@5.
Image JEPA Text JEPA mutual 10-NN R@1 R@5
good views (frames) good views (captions) .067±\pm.001 .032±\pm.006 .099±\pm.009
bad views (crops) good views (captions) .059±\pm.002 .024±\pm.002 .105±\pm.008
good views (frames) bad views (spans) .043±\pm.002 .020±\pm.005 .063±\pm.002
bad views (crops) bad views (spans) .039±\pm.003 .017±\pm.002 .072±\pm.009
frozen DINOv2 ViT-B good views (captions) .129±\pm.002 .085±\pm.006 .258±\pm.011
frozen DINOv2 ViT-B bad views (spans) .084±\pm.002 .052±\pm.002 .164±\pm.006
frozen DINOv2 ViT-B image JEPA, good views .132±\pm.002 .219±\pm.010 .462±\pm.011
frozen DINOv2 ViT-B image JEPA, bad views .144±\pm.001 .285±\pm.005 .559±\pm.008

Text-only quality.

Good-view training also improves semantic textual similarity: Flickr-trained good-view, bad-view, and next-token encoders score Spearman correlations 0.579, 0.528, and 0.430. More web text raises good-view performance to 0.632, below supervised MPNet's 0.834. Alignment gains therefore coexist with useful text semantics without implying state-of-the-art language understanding.

Uncertainty on WIT.

WIT contains only 1,024 pairs. Repeated 800-pair subsamples show that small gaps between the 50M-image JEPA and some pretrained language models have confidence intervals spanning zero. Larger matched-data good-view versus next-token gaps remain positive. Close cross-model rankings should be read with this uncertainty.

Image JEPAs trained from scratch.

Image JEPAs trained from scratch on web images or ImageNet generally preserve the ordering of good-view text above next-token and bad-view text. Their image augmentations are crops, classified as bad views in the theory. These experiments broaden empirical support without creating an exact test of its hypotheses.

Scaling to the texts of 940M images.

The largest study distinguishes a corpus associated with 940M unique images from the 900M items actually processed. Batch size, learning rate, and effective regularization differ from the smaller-scale runs.

Three good-view seeds score 0.191, 0.181, and 0.179 on WIT. A fourth seed with doubled effective SIGReg scores 0.121; including it lowers the mean from 0.184 to 0.168. The effective weight was chosen after inspecting WIT, so this benchmark contributed to model selection. Other vision-family comparisons are less decisive when all four runs are included.

Table 14: Alignment keeps growing with the texts of 940M images. Checkpoints of our largest good-view JEPAs (seed 0 unless named; †: mean of three seeds at the end of training; ‡: another seed with twice the effective SIGReg weight, see text; next-token: the same encoder trained to predict the next token; bad views: two masked copies of one text, at the effective SIGReg weight of the first 940M run), scored on WIT-1024 with DINOv2 ViT-B and on Flickr through one rotation fitted on 30k pairs. We observe that WIT alignment jumps between the texts of 200M and 940M images and rises again by 850M samples, while Flickr retrieval rises more slowly. Bold: best per column.
Text JEPA texts of global batch samples seen WIT (DINOv2-B) Flickr R@1
BERT-S 100M images 256 80M .125† .402†
BERT-B 200M images 256 89M .125 .447
BERT-B 200M images 256 117M .126 .447
BERT-B 200M images 256 160M .126 .451
BERT-B 940M images 2,048 430M .182 .410
BERT-B 940M images 2,048 580M .182 .463
BERT-B 940M images 2,048 850M .191 .468
BERT-B 940M images 2,048 900M .191 .464
BERT-B, seed 1 940M images 2,048 900M .181 .484
BERT-B, seed 2 940M images 2,048 900M .179 .485
BERT-B‡ 940M images 2,048 900M .121 .477
BERT-B, next-token, seed 0 940M images 2,048 900M .109 .346
BERT-B, next-token, seed 1 940M images 2,048 900M .111 .332
BERT-B, next-token, seed 2 940M images 2,048 900M .112 .328
BERT-B, bad views 940M images 2,048 900M .088 .318

WIT captions in the training texts.

The overlap search identifies at least 43 of 1,024 WIT captions in the training text. Removing them changes reported scores by at most 0.002. WIT image identities are unavailable, preventing a complete image-level audit.

Recovered image matches also include six Flickr test images and 50 of 5,000 COCO test images. Removing the latter changes alignment by at most 0.001. These checks bound the effect of identified overlap, not undiscovered duplicates or supervision inherited from the recaptioner.

WIT, PRH protocol.

Flickr-only encoders perform poorly on WIT across objectives, around 0.029–0.037. Training on web captions substantially improves this cross-domain measure. The retained table follows the PRH layer-pair maximum protocol, so it should not be equated with retrieval after fitting a single final-layer map.

Table 15: A 109M text JEPA beats language models on the PRH benchmark. PRH protocol of Huh et al. [22]: WIT-1024 pairs, per-layer features, clipping, normalization, maximum over layer pairs. For reference, DINOv2 ViT-B ×\times BERT-base scores .111, all-mpnet-base-v2 .118 and CLIP’s text encoder .124 (.113–.137 across DINOv2 sizes for these two); our Flickr-trained text encoders (good / bad / next-token) score .029–.036 with every DINOv2 size and .023–.024 with our Flickr image JEPA; and our Flickr image JEPA ×\times DINOv2 ViT-B (image vs. image) .241. nt: next-token prediction. Our 33M text JEPAs’ entries are means of three seeds, which differ by at most ±\pm.009 with DINOv2 ViT-B; the 109M column is the mean of three seeds (each above every language model). At least 43 of the 1,024 WIT captions appear among the Recap-DataComp alt-texts; on the other 981 pairs, none of the nine text models we checked with DINOv2 ViT-B (ours, Qwen2.5-3B/7B, Mistral-7B) changes by more than .002. This demonstrates that the 109M good-view JEPA surpasses every language model we evaluated with every vision encoder it was scored with, including our image JEPAs trained from scratch.
Qwen2.5 Mistral ours (33M), on the texts of Recap images ours (109M)
2.16M 10M 50M 100M 940M
Vision 0.5B 1.5B 3B 7B 7B good / bad / nt good good / bad / nt good good
DINOv2 ViT-S/14 .092 .101 .110 .110 .126 .099 / .052 / .065 .102 .114 / .062 / .074 .114 .166
DINOv2 ViT-B/14 .102 .108 .125 .118 .140 .108 / .053 / .071 .111 .125 / .063 / .079 .125 .184
DINOv2 ViT-L/14 .114 .123 .139 .137 .157 .111 / .057 / .075 .115 .130 / .068 / .085 .131 .196
DINOv2 ViT-g/14 .115 .124 .141 .133 .157 .114 / .059 / .076 .117 .132 / .070 / .086 .133 .197
ours, ViT-S, Recap 2M .062 .068 .078 .073 .086 .070 / .042 / .053 .071 .079 / .048 / .057 .080 .106
ours, ViT-B, Recap 10M .064 .070 .079 .075 .087 .071 / .043 / .054 .072 .080 / .048 / .058 .081 .106
ours, ViT-S, ImageNet .074 .078 .091 .087 .101 .082 / .046 / .057 .085 .095 / .053 / .062 .094 .130
ours, ViT-S, Flickr .050 .051 .057 .055 .063 .055 / .034 / .043 .055 .060 / .038 / .047 .060 .078

A second alignment metric.

Unbiased linear CKA provides a second geometry metric and broadly agrees with mutual 10-NN rankings. The largest good-view JEPA reaches 0.507 against DINOv2-B. Agreement between two metrics strengthens the alignment observation, while neither directly measures downstream language competence.

Larger language models.

The language-model comparison extends to Qwen and Llama models up to 72B parameters. The selected good-view JEPA mean exceeds their WIT scores under this protocol. The comparison concerns the evaluated representations and alignment metric, with different training data and supervision histories.

Table 16: No language model up to 72B parameters reaches the 109M good-view JEPA. PRH protocol (per-layer features, maximum over layer pairs) on WIT-1024 and on the 5,000 COCO test pairs, for base language models from 0.5B to 72B parameters and our 109M good-view text JEPA trained on the texts of 940M images (mean of three seeds). We clearly observe that scaling next-token prediction by 660 times in parameters does not reach the alignment that good views give.
WIT-1024 COCO-5k
mutual 10-NN, DINOv2 S / B / L / g CKA, B 10-NN, B CKA, B
Qwen2.5-0.5B .092 .102 .114 .115 .318 .100 .571
Qwen2.5-1.5B .101 .108 .123 .124 .324 .089 .538
Qwen2.5-3B .110 .125 .139 .141 .363 .099 .550
Qwen2.5-7B .110 .118 .137 .133 .353 .101 .535
Qwen2.5-14B .119 .130 .147 .148 .333 .106 .539
Qwen2.5-32B .116 .128 .145 .144 .336 .104 .533
Qwen2.5-72B .138 .154 .166 .169 .418 .108 .534
Mistral-7B .126 .140 .157 .157 .349 .112 .539
Llama-3-8B .130 .145 .165 .164 .377 .109 .599
Llama-3-70B .136 .151 .166 .166 .380 .113 .587
ours: good-view text JEPA, 109M .166 .184 .196 .197 .507 .161 .622

A second benchmark.

COCO supplies a second 5,000-pair alignment benchmark. The large good-view text JEPA leads the tested encoders on the reported geometry metrics. Removing the 50 identified overlapping images has negligible effect, although recaptioner supervision and model-selection differences remain.

Table 17: The ranking holds on COCO. PRH protocol on the 5,000 COCO Karpathy test pairs (first caption), seed 0 for our encoders except the 940M row, the mean of three seeds (each at or above all-mpnet-base-v2); chance mutual 10-NN is .002. We clearly observe that the 109M good-view JEPA aligns best on a second, five times larger benchmark, ahead of language models up to 72B parameters and of text encoders trained on pairs.
mutual 10-NN unbiased CKA
Text encoder (COCO-5k) DINOv2-B DINOv2-g DINOv2-B DINOv2-g
ours, good, BERT-B, 940M images .161 .169 .622 .667
ours, good, 100M images .134 .140 .587 .598
ours, good, 50M images .126 .135 .578 .576
ours, bad, 50M images .085 .090 .447 .423
ours, next-token, 50M images .091 .096 .522 .487
Qwen2.5-0.5B .100 .105 .571 .529
Qwen2.5-1.5B .089 .095 .538 .505
Qwen2.5-3B .099 .104 .550 .525
Qwen2.5-7B .101 .107 .535 .510
Qwen2.5-14B .106 .112 .539 .526
Qwen2.5-32B .104 .109 .533 .521
Qwen2.5-72B .108 .113 .534 .521
Mistral-7B .112 .118 .539 .526
Llama-3-8B .109 .115 .599 .560
Llama-3-70B .113 .118 .587 .529
BERT-base .104 .109 .485 .475
all-mpnet-base-v2 .156 .162 .608 .577
CLIP text encoder .132 .138 .576 .565

Other vision families.

MAE, I-JEPA, and CLIP vision encoders test whether the result depends on DINO features. The retained table generally preserves the advantage of good-view text over matched bad-view and next-token models. Absolute scores and cross-model rankings vary with the vision family.

Table 18: Good views win with every vision family. WIT-1024 (PRH protocol) for frozen vision encoders trained by pixel reconstruction (MAE), latent prediction (I-JEPA), crops (DINOv2) or image–text contrast (CLIP’s image encoders, the vision models with which language models align best in Huh et al. [22]), against our text JEPAs (good / bad / next-token on the texts of 50M images, seed 0; the 109M good-view JEPA on the texts of 940M images, seed 0) and pretrained language models. We clearly observe that good views beat next-token prediction and bad views with every vision encoder, and that the 109M good-view JEPA beats the three language models with each of them. MAE and I-JEPA align alike, both below DINOv2, which saw far more curated images; we draw no conclusion on the vision objective. With CLIP’s image encoders, the 109M good-view JEPA’s three seeds average .215, .219 and .230, against .193, .203 and .210 for the best of all our language models up to 72B (Qwen2.5-72B), and its next-token and bad-view controls on the same texts .137–.146 and .108–.116 (runs/results/wit_clip.txt).
ours (33M), texts of 50M images ours (109M), 940M Qwen2.5 Qwen2.5 Mistral
Vision encoder good bad next-token good 0.5B 7B 7B
MAE ViT-B/16 (IN-1k) .074 .045 .053 .100 .060 .067 .077
MAE ViT-L/16 (IN-1k) .093 .050 .059 .126 .074 .084 .097
MAE ViT-H/14 (IN-1k) .092 .050 .058 .125 .071 .084 .096
I-JEPA ViT-H/14 (IN-22k) .095 .051 .059 .129 .068 .081 .096
DINOv2 ViT-B/14 .129 .066 .079 .191 .102 .118 .140
DINOv2 ViT-L/14 .133 .069 .084 .199 .114 .137 .157
CLIP ViT-B/16 (LAION-2B) .147 .076 .093 .221 .135 .156 .180
CLIP ViT-L/14 (LAION-2B) .148 .078 .098 .224 .144 .164 .194
CLIP ViT-H/14 (LAION-2B) .156 .080 .099 .233 .148 .169 .196
Original figure
Original figure
Figure 14: One rotation superimposes two separately trained encoders. Top: joint t-SNE of test images after the best rotation (blue) and their captions (orange), at initialization and after training (good-view JEPA); grey lines join 60 random true pairs. Bottom: independent t-SNEs of the image and text embeddings, colored by caption keywords used only for coloring. We clearly observe that after training, images and their captions share one geometry.
Figure 15: original paper visual, restored from the PDF
Figure 15: Retrieval through one rotation. Image→\tocaption retrieval (good-view JEPA, epoch 199) for five Flickr test images, with the rank of the true caption among the 1,000 above each list (✓\checkmark would mark it among the top three). We observe that misses are mostly semantically close.
Concept demonstration

Open image ↗