Literature
Display
← All papers

Gaussian Embeddings: How JEPAs Secretly Learn Your Data Density

Randall Balestriero · Nicolas Ballas · Mike Rabbat · Yann LeCun

Preprint posted · arXiv (preprint) ↗

Learning theory3 min text · CondensedOriginal paper ↗
Contents

Overview

The preserved experimental figures compare density-related scores on synthetic distributions, controlled visual data and pretrained image/multimodal encoders. They support the proposed score as a useful diagnostic in those settings. They do not remove the assumptions needed to interpret an encoder-derived score as a normalized input probability.

low probability
 

high probability
 

Refer to captionRefer to captionRefer to captionRefer to caption

MetaCLIP  IJEPA-22k  IJEPA-1k  DINOv2      

Figure 1: Depiction of the 5 least (left) and 5 most (right) likely samples of class 21 from Imagenet as per JEPA-SCORE–JEPAs’ implicit density estimator learned during pretraining. Two striking observations: (i) across all JEPAs (rows) the type of samples with low and high probabilities are alike, and (ii) the same samples (amongst 1,000) are found at those extrema. Random samples from that class are provided in fig. 4

At a glance

Intuition. Imagine stretching a crowded region of data into a more evenly spread feature space. The amount of local stretching contains information about how crowded the original region was. JEPA-SCORE uses this idea: examine the encoder’s local derivative, or Jacobian, when its output distribution is encouraged to be Gaussian.

The result is a density-related scalar that can help with ranking or curation. Recovering a full input density is more delicate when several distinct inputs map to the same embedding. The paper explicitly permits ambiguity within such sets, and its strongest recovery claim assumes an optimally satisfied JEPA objective and a specified view-generation model.

1 Introduction

The paper connects representation learning with density estimation. A Gaussian distribution maximizes differential entropy under a covariance constraint, motivating the study of encoders with Gaussian-like outputs. The new question is what the local transformation into that distribution reveals about the original data. A non-generative training objective may still carry density information, but that is different from providing an unrestricted, calibrated probability for every input.

2 JEPA-SCORE: the Data Density Implicitly Learned by JEPAs

The argument proceeds from high-dimensional Gaussian geometry to a change-of-variables relation and then a JEPA-specific score. Each step adds assumptions; the final estimator should be read together with those assumptions.

2.1 Preliminaries: Gaussian Embeddings Are Uniform on the Hypersphere

A high-dimensional isotropic Gaussian has uniformly distributed directions and a radius concentrated near a predictable value after scaling. The uniform direction is an exact symmetry; concentration near a sphere is a high-dimensional approximation. A finite-dimensional Gaussian is not literally the same distribution as a surface-only uniform law.

Lemma 1.

As KK grows, XX quickly concentrates around the hypersphere of radius 11, converging to a Uniform density over the hypersphere surface. (Proof in section A.1.)

2.2 Producing Gaussian Embeddings Equates Learning an Energy Function

For an invertible map, input and output densities are related by local volume change, measured by the Jacobian. For a many-to-one encoder, the relation integrates over all inputs sharing an embedding. The output distribution therefore constrains density averaged along these level sets, while allowing ambiguity within each set. This is why Gaussian output alone cannot uniquely determine arbitrary pointwise input probabilities.

Equation 1
pf⁡(X)​(f⁡(𝒙))=∫{𝒖∈ℝD|f⁡(𝒖)=f⁡(𝒙)}pX​(x)∏k=1rank⁡(Jf​(𝒙))σk​(Jf​(𝒙))​d​ℋr​(𝒙),p_{f(X)}(f({\bm{x}}))=\int_{\{{\bm{u}}\in\mathbb{R}^{D}|f({\bm{u}})=f({\bm{x}})\}}\frac{p_{X}(x)}{\prod_{k=1}^{\operatorname{rank}(J_{f}({\bm{x}}))}\sigma_{k}(J_{f}({\bm{x}}))}\mathrm{d}\mathcal{H}^{r}({\bm{x}}), (1)
Lemma 2.

In order for f⁡(X)f(X) to be distributed as 𝒩⁡(0,𝐈/K)\mathcal{N}(0,{\bm{I}}/K) for large KK, ff must learn the data density pXp_{X} up to mean-preserving rescaling within each level-set {𝐮∈ℝD|f⁡(𝐮)=f⁡(𝐱)}\{{\bm{u}}\in\mathbb{R}^{D}|f({\bm{u}})=f({\bm{x}})\}. (Proof in section A.2.)

Refer to caption
512 1024 2048 4096
dim
64 0.64 0.69 0.84 0.90
128 0.75 0.85 0.90 0.94
256 0.82 0.83 0.69 0.76
512 0.72 0.75 0.84 0.88
Original figure
Figure 2: Top left: Visual illustration of JEPA-SCORE–the DN f𝜽f_{{\bm{\theta}}} must learn pXp_{X} for its Jacobian matrix to expand or contract the density in order to produce a Uniform density on the hypersphere surface in its embedding space (lemmas 1 and 2). Top right: Pearson correlation between JEPA-SCORE and the true l​o​g​p​(x)logp(x) on a GMM data model for various input dimensions (rows) and number of samples (columns). In all cases, producing Gaussian embeddings make the backbone f𝜽f_{{\bm{\theta}}} internalize the data density which can be easily extracted using our proposed JEPA-SCORE. Bottom: as JEPA-SCORE is an approximation of the true score function, it is possible to perform Langevin sample to recover the true data distribution as shown here in two dimensions.

2.3 JEPA-SCORE: The Data Density Learned by JEPAs

The JEPA-specific derivation models each sample as a stochastic transformation of an underlying generator and links view invariance to encoder level sets. At the stated optimum, the resulting expression estimates the generator density. JEPA-SCORE uses Jacobian singular values and a Monte Carlo estimate, often reported on a log scale. In practical networks, finite training error, approximate Gaussianity, transformation assumptions and numerical estimation qualify the density interpretation. Here “score” names the proposed scalar estimator; it should not be confused with automatically possessing a perfect generative model.

Equations 2–3
ℒ≜\displaystyle\mathcal{L}\triangleq ∑n=1N𝔼(𝒙n(1),𝒙n(2))∼𝒢⁡(𝒙n)​[dist⁡(Pred⁡(Enc⁡(𝒙n(1))),Enc⁡(𝒙n(2)))]\displaystyle\sum_{n=1}^{N}\mathbb{E}_{({\bm{x}}_{n}^{(1)},{\bm{x}}_{n}^{(2)})\sim\mathcal{G}({\bm{x}}_{n})}\left[{\rm dist}\left({\rm Pred}\left({\rm Enc}\left({\bm{x}}_{n}^{(1)}\right)\right),{\rm Enc}\left({\bm{x}}_{n}^{(2)}\right)\right)\right] (predictive invariance)\displaystyle(\text{predictive invariance})
+diversity⁡((Enc⁡(𝒙n))n∈[N]),\displaystyle+{\rm diversity}\left(\left({\rm Enc}\left({\bm{x}}_{n}\right)\right)_{n\in[N]}\right), (anti-collapse),\displaystyle(\text{anti-collapse}), (2)
pX≜pμ⊗pT.\displaystyle p_{X}\triangleq p_{\mu}\otimes p_{T}. (3)
Original figure
Figure 3: Depiction of JEPA-SCORE for 5,0005,000 samples from different datasets (imagenet-1k/a/r, MNIST and Galaxy). We clearly observe that as the pretraining dataset size increases (all models against IJEPA-1k) as MNIST and Galaxy images are seen as lower probability samples, i.e., those images are less and less represented within the overall pretraining dataset. While our score does not rely on singular vectors, we provide some examples in fig. 7. This can be used to assess if a model is ready or not to handle particular data domains at test time for zero-shot tasks.
Theorem 1.

At optimality, JEPA embeddings estimate the data density as per

Equation 4
pμ​(μ)∝𝔼pT​[1∏k=1rank⁡(Jf​(𝒙))σk​(Jf​(μ,T))]−1.p_{\mu}(\mu)\propto\mathbb{E}_{p_{T}}\left[\frac{1}{\prod_{k=1}^{\operatorname{rank}(J_{f}({\bm{x}}))}\sigma_{k}(J_{f}(\mu,T))}\right]^{-1}. (4)
Equation 5
JEPA-SCORE​(𝒙)≜∑k=1rank⁡(Jf​(𝒙))log⁡(σk​(Jf​(𝒙))CLOSE,\displaystyle\text{\bf JEPA-SCORE}({\bm{x}})\triangleq\sum_{k=1}^{\operatorname{rank}(J_{f}({\bm{x}}))}\log\left(\sigma_{k}(J_{f}({\bm{x}})\right), (5)

Appendix A Proofs

A.1 Proof of lemma 1

The proof separates Gaussian radius and direction in spherical coordinates. The normalized squared radius has variance decreasing with dimension, explaining shell concentration. Directional uniformity follows from rotational symmetry.

Equations
f𝒰⁡(𝕊⁡(0,R,K))​(𝒙)\displaystyle f_{\mathcal{U}(\mathbb{S}(0,R,K))}({\bm{x}}) =δ⁡(‖𝒙‖2−R)​Γ⁡(K/2)2​πK/2​RK−1,\displaystyle=\delta(\|{\bm{x}}\|_{2}-R)\frac{\Gamma(K/2)}{2\pi^{K/2}R^{K-1}}, (Cartesian coordinates)
f𝒰⁡(𝕊⁡(0,R,K))​(r,𝜽)\displaystyle f_{\mathcal{U}(\mathbb{S}(0,R,K))}(r,{\bm{\theta}}) =δ⁡(r−R)​Γ⁡(K/2)2​πK/2​∏i=1K−1sin⁡(𝜽i)K−i−1,\displaystyle=\delta(r-R)\frac{\Gamma(K/2)}{2\pi^{K/2}}\prod_{i=1}^{K-1}\sin({\bm{\theta}}_{i})^{K-i-1}, (spherical coordinates),\displaystyle\text{(spherical coordinates)},
f𝒩⁡(0,𝑰/K)​(𝒙)\displaystyle f_{\mathcal{N}(0,{\bm{I}}/K)}({\bm{x}}) =(K2​π)K/2​e−K2​‖𝒙‖22,\displaystyle=\left(\frac{K}{2\pi}\right)^{K/2}e^{-\frac{K}{2}\|{\bm{x}}\|_{2}^{2}}, (Cartesian coordinates)
f𝒩⁡(0,𝑰/K)​(r,𝜽)\displaystyle f_{\mathcal{N}(0,{\bm{I}}/K)}(r,{\bm{\theta}}) =(K2​π)K/2​e−K2​r2​rK−1​∏i=1K−1sin⁡(𝜽i)K−i−1\displaystyle=\left(\frac{K}{2\pi}\right)^{K/2}e^{-\frac{K}{2}r^{2}}r^{K-1}\prod_{i=1}^{K-1}\sin({\bm{\theta}}_{i})^{K-i-1}
=KK22K/2−1​Γ​(K/2)​e−K​r22​Γ⁡(K/2)2​πK/2​rK−1​∏i=1K−1sin⁡(𝜽i)K−i−1\displaystyle=\frac{K^{\frac{K}{2}}}{2^{K/2-1}\Gamma(K/2)}e^{-\frac{Kr^{2}}{2}}\frac{\Gamma(K/2)}{2\pi^{K/2}}r^{K-1}\hskip-4.26773pt\prod_{i=1}^{K-1}\sin({\bm{\theta}}_{i})^{K-i-1}
=KK22K/2−1​Γ​(K/2)​rK−1​e−K​r22⏟scaled Chi-distribution​→K→∞​δ​(r−1)​f𝒰⁡(𝕊⁡(0,R,K))​(r,𝜽),\displaystyle=\underbrace{\frac{K^{\frac{K}{2}}}{2^{K/2-1}\Gamma(K/2)}r^{K-1}e^{-\frac{Kr^{2}}{2}}}_{\text{scaled Chi-distribution}\overset{K\rightarrow\infty}{\rightarrow}\delta(r-1)}f_{\mathcal{U}(\mathbb{S}(0,R,K))}(r,{\bm{\theta}}), (spherical coordinates).\displaystyle\text{(spherical coordinates)}.

A.2 Proof of lemma 2

The proof uses the coarea/change-of-variables relation. With a bijection, local volume change directly relates densities. With a non-injective map, probability can redistribute within a level set while leaving its integrated output contribution unchanged. This ambiguity is part of the result.

A.3 Proof of theorem 1

The theorem connects the assumed transformation-induced view relation with encoder level sets, then substitutes it into the density relation. Its optimum and transformation-model assumptions matter when moving from the formal expression to a trained checkpoint.

Equations
𝔼(mask(1),mask(2))∼(ℳ,ℳ)​[dist⁡(Pred⁡(Enc⁡(mask(1)​(𝒙n))),Enc⁡(mask(2)​(𝒙)))].\displaystyle\mathbb{E}_{({\rm mask}^{(1)},{\rm mask}^{(2)})\sim(\mathcal{M},\mathcal{M})}\left[{\rm dist}\left({\rm Pred}\left({\rm Enc}\left({\rm mask}^{(1)}({\bm{x}}_{n})\right)\right),{\rm Enc}\left({\rm mask}^{(2)}({\bm{x}})\right)\right)\right].
𝔼mask(2)∼ℳ​[dist⁡(Pred⁡(Enc⁡(mask(1)​(𝒙n))),Enc⁡(mask(2)​(𝒙)))]=0,\displaystyle\mathbb{E}_{{\rm mask}^{(2)}\sim\mathcal{M}}\left[{\rm dist}\left({\rm Pred}\left({\rm Enc}\left({\rm mask}^{(1)}({\bm{x}}_{n})\right)\right),{\rm Enc}\left({\rm mask}^{(2)}({\bm{x}})\right)\right)\right]=0,

Appendix B Implementation Details

The preserved implementation blocks give the numerical construction of the proposed estimator. Jacobian computation and singular-value handling are part of its cost and numerical behavior; the absence of a pixel decoder does not make this calculation free.

1 from torch.autograd.functional import jacobian
2
3 # model returns a tensor of shape (num_samples, features_dim)
4 J = jacobian(lambda x: model(x).sum(0), inputs=images)
5 with torch.inference_mode():
6 J = J.flatten(2).permute(1, 0, 2)
7 svdvals = torch.linalg.svdvals(J)
8 jepa_score = svdvals.clip_(eps).log_().sum(1)
Listing 1: JEPA-SCORE implementation in PyTorch. Our empirical ablations demonstrate that JEPA-SCORE is not sensitive to the choice of eps (we pick 1​e−61e-6)

Appendix C Additional Figures

The additional original plots extend the controlled density and representation comparisons. Use them as empirical checks of the stated settings, rather than as proof that any arbitrary self-supervised model supplies calibrated probabilities.

Refer to caption
Figure 4: Random samples from Imagenet-1k training dataset for class 21.

low probability
 

high probability
 

Refer to captionRefer to captionRefer to captionRefer to caption

MetaCLIP  IJEPA-22k  IJEPA-1k  DINOv2      

Random samples
Refer to caption

Figure 5: Top: least and most likely samples based on the proposed JEPA-SCORE for a fe different pretrained backbones on class 141. Bottom:Random samples from Imagenet-1k training dataset for class 141.

low probability
 

high probability
 

Refer to captionRefer to captionRefer to captionRefer to caption

MetaCLIP  IJEPA-22k  IJEPA-1k  DINOv2      

Random samples
Refer to caption

Figure 6: Top: least and most likely samples based on the proposed JEPA-SCORE for a fe different pretrained backbones on class 141. Bottom:Random samples from Imagenet-1k training dataset for class 147.
Refer to caption
Refer to caption
Figure 7: Depiction of the top singular vectors of the Jacobian matrix (columns) for a given input image (left column). The corresponding singular value is provided in the title of each subplot.
Concept demonstration

Open image ↗