Literature
Display
← All papers

LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

Randall Balestriero · Yann LeCun

Preprint posted · arXiv (preprint) ↗

Learning theory10 min text · CondensedOriginal paper ↗
Contents

Overview

Intuition. Making two views produce the same vector is easy if every image produces zero. LeJEPA adds a second requirement: across different examples, embeddings should form a well-spread, approximately Gaussian cloud. Agreement encourages shared information; distribution regularization prevents an uninformative constant solution.

SIGReg checks many one-dimensional projections of that cloud, avoiding a full high-dimensional density estimate. The paper combines this construction with theory about downstream probes and experiments across architectures and domains. Its guarantees depend on the specified statistical assumptions; a Gaussian embedding alone does not identify the semantic features or prove a useful physical world model.

Original figureOriginal figure[Uncaptioned image][Uncaptioned image]
Full FT Frozen
Method 1-sh Full 1-sh Full
LeJEPA (in-domain)
ConvNeXt-V2 Nano 29.42 82.72 28.74 76.52
ResNet-34 24.27 83.28 31.08 78.17
Frontier (transfer)
DINOv2 ViT-S/16 21.05 78.34 27.68 67.62
DINOv3 ViT-S/16 24.71 81.60 30.17 71.38
Refer to caption

f𝜽→\underset{\rightarrow}{f_{{\bm{\theta}}}}

Refer to caption
Figure 2: Sketched Isotropic Gaussian Regularization (SIGReg): Given some arbitrary input data with density pxp_{x} with support that may or may not lie on a manifold (left), a Deep network (DN) encoder (f𝜽f_{{\bm{\theta}}}) produces embeddings 𝒛=f𝜽​(𝒙){\bm{z}}=f_{{\bm{\theta}}}({\bm{x}}) with some distribution 𝒛∼pz{\bm{z}}\sim p_{z} (middle). Our proposed Backward Cramér-Wold Statistics (Section 4) objective pushes pzp_{z} to match a target distribution ptp_{t} by projecting the embeddings along 1​d1d directions (middle, arrows) and enforcing that the univariate densities (right, colored lines) match the distribution of ptp_{t}, projected along the same directions. Any popular statistical test (provided in Section 4.2) can assess the goodness-of-fit–in practice we argue for characteristic function tests (Section 4.2). By using SIGReg with ptp_{t} isotropic Gaussian (right, black lines), we introduce a lean and provably optimal (Section 3) JEPA, coined LeJEPA, free of numerous heuristics and able to produce competitive performances (Sections 5 and 6).

1 Introduction

The method separates two design decisions: choose related views whose shared content should be retained, and prevent the representation from collapsing while those views are aligned. LeJEPA supplies an explicit regularizer for the second decision. It removes several teacher–student training mechanisms in the tested recipe, while retaining ordinary choices such as architecture, augmentations, optimizer and training budget.

2 Background and Notations

The notation distinguishes independent training examples, related views of each example and the coordinates of an embedding. This separation is essential: similarity within a view group and diversity across examples play different roles.

2.1 Notations and Definitions

Inputs are organized as examples × views × input coordinates. A neural network maps each view to an embedding. Independence assumptions concern different examples, not necessarily views of the same example. The downstream probe is a separately trained prediction rule, so a statement about its statistical risk is not automatically a statement about arbitrary end-to-end tasks.

Definition 1: JEPA

Equation 1 JEPA⁡(𝒙)⇔\displaystyle{\rm JEPA}({\bm{x}})\iff Enc(𝒙n,t+1,.) is predictable from Enc(𝒙n,t,.),∀n,t and Enc(𝒙.,.,.) is not degenerate.\displaystyle{\rm Enc}\left({\bm{x}}_{n,t+1,.}\right)\text{ is predictable from }{\rm Enc}\left({\bm{x}}_{n,t,.}\right),\forall n,t\text{ and }{\rm Enc}\left({\bm{x}}_{.,.,.}\right)\text{ is not degenerate}. (1)
[Uncaptioned image]
[Uncaptioned image]
[Uncaptioned image]

2.2 The Need for Reliable Pretraining

A low agreement loss does not rule out shortcut representations. Whitening, contrastive negatives and asymmetric moving-teacher designs address this in different ways. LeJEPA asks for a direct, scalable distribution constraint instead of treating a successful teacher recipe as the only available collapse-prevention mechanism.

2.3 The Need for Actionable Theory

Theory is used to motivate an implementable objective rather than only explain a completed training run. The paper studies particular probe families and statistical assumptions. Its design principle is most useful when those assumptions and the chosen views are kept visible alongside the empirical results.

3 Latent Euclidean: Embeddings Should be Isotropic Gaussian

The theoretical target is an isotropic Gaussian: a centered Gaussian distribution with equal variance in every direction. This is a geometric preference under the paper’s fixed-energy and downstream-risk formulations, not a proof that all useful data must be Gaussian or that arbitrary labels become predictable after Gaussianization.

3.1 Linear Probing

For linear probes, compare embeddings spanning the same features and having the same total variance, but distributing that variance differently across directions. Uneven covariance can amplify estimation variance and regularization bias for unfavorable downstream target directions. Isotropy improves these worst-case geometric properties. This comparison controls the available features; it does not show that replacing informative features with arbitrary Gaussian noise is beneficial.

Equation OLS
β^=arg⁡minβ∈ℝK​‖𝒚−𝒁​β‖22+λ​‖β‖22,\hat{\beta}=\underset{\beta\in\mathbb{R}^{K}}{\arg\min}\|{\bm{y}}-{\bm{Z}}\beta\|_{2}^{2}+\lambda\|\beta\|_{2}^{2}, (OLS)

Lemma 1: Anisotropy amplifies bias

Whenever λK>λ1\lambda_{K}>\lambda_{1}, there always exists a downstream task (𝒚{\bm{y}}) for which 𝒁aniso{\bm{Z}}_{\rm aniso} produces a higher bias estimator than 𝒁iso{\bm{Z}}_{\rm iso} for λ>0\lambda>0. (Proof in Section B.1.)

Lemma 2: Anisotropy amplifies variance

With λ=0\lambda=0, the total variance of β^\hat{\beta} (OLS) is minimized for 𝒁iso{\bm{Z}}_{\rm iso} with tr​(Var​(𝜷^aniso))>tr​(Var​(𝜷^iso))\text{tr}(\text{Var}(\hat{\bm{\beta}}_{\text{aniso}}))>\text{tr}(\text{Var}(\hat{\bm{\beta}}_{\text{iso}})). (Proof in Section B.2.)

3.2 Nonlinear Probing

The nonlinear analysis considers radius-based neighbors and kernel regression, with smooth densities, smooth target functions and specified variance or geometric constraints. Under the paper’s risk formulation, a Gaussian shape emerges as optimal. These regularity and target-class assumptions are part of the claim; they cannot be dropped when applying the conclusion to a new task.

Equations kNN–Kernel
𝒚^​(𝒒):=1|𝒩r0​(𝒒)|​∑n∈𝒩r0​(𝒒)𝒚n,\displaystyle\widehat{{\bm{y}}}({\bm{q}}):=\frac{1}{|\mathcal{N}_{r_{0}}({\bm{q}})|}\sum_{n\in\mathcal{N}_{r_{0}}({\bm{q}})}{\bm{y}}_{n}, (kNN)
𝒚^​(𝒒)​\triangleq​∑n=1NKh​(𝒒−𝒛n)​𝒚n∑n=1NKh​(𝒒−𝒛n).\displaystyle\widehat{\bm{y}}({\bm{q}})\triangleq\frac{\sum_{n=1}^{N}K_{h}({\bm{q}}-{\bm{z}}_{n}){\bm{y}}_{n}}{\sum_{n=1}^{N}K_{h}({\bm{q}}-{\bm{z}}_{n})}. (Kernel)

Theorem 1: isotropic Gaussian Optimality

The integrated square bias (ISB) over query points is given by

Equation ISBk​-NN=r04(K+2)2​τg2​J​(p)+O⁡(r04),\displaystyle\text{ISB}_{k\text{-NN}}=\frac{r_{0}^{4}}{(K+2)^{2}}\tau_{g}^{2}J(p)+O(r_{0}^{4}), (k-NN) ISBkernel≤(h2​μ2​(K)2)2​(2​B2+8​L2​J​(p))+o⁡(h4),\displaystyle\text{ISB}_{\text{kernel}}\leq\Big(\frac{h^{2}\mu_{2}(K)}{2}\Big)^{2}\Big(2B^{2}+8L^{2}J(p)\Big)+o(h^{4}), (kernel)

and among distributions with a scalar-based covariance constraint, the isotropic Gaussian is the unique minimizer of the integrated square bias. (Proof in Sections B.4 and B.7.)

Original figure
Figure 3: Illustration of Section 3.1 showcasing how anisotropic (right) embeddings lead to higher variance estimator compared to isotropic embeddings (left). We sample 100100 training points for the 22-class classification task and fit a logistic regression–repeating the process over numerous training set sample. Each sampling results in a decision boundary (purple).

3.3 Geometric and Practical Insights

Synthetic examples illustrate why poorly scaled directions make probe fitting less stable. Equalized directions reduce this geometric disadvantage. For a reader, the useful distinction is between making existing information easier to read and learning the right information in the first place.

4 SIGReg: Reliable Isotropic Gaussian Regularization in High-Dimension

SIGReg turns the Gaussian target into a differentiable training penalty. It compares projected embedding samples with the desired one-dimensional Gaussian and aggregates across projection directions.

4.1 Hypothesis Testing as a Judge

A distribution test asks whether observed samples provide evidence against the Gaussian target. Failure to reject is not proof of equality. At the population level, matching every one-dimensional projection identifies the full distribution; finite samples and finitely many directions need additional coverage and regularity arguments.

Equations 2–4
H0:P𝜽=Qvs.H1:P𝜽≠Q,\displaystyle H_{0}:P_{{\bm{\theta}}}=Q\quad\text{vs.}\quad H_{1}:P_{{\bm{\theta}}}\neq Q, (2)
H0​(𝒂):P𝜽(𝒂)=Q(𝒂)​vs.​H1​(𝒂):P𝜽(𝒂)≠Q(𝒂),\displaystyle H_{0}({\bm{a}}):P_{\bm{\theta}}^{({\bm{a}})}=Q^{({\bm{a}})}\;\;\text{vs.}\;\;H_{1}({\bm{a}}):P_{\bm{\theta}}^{({\bm{a}})}\not=Q^{({\bm{a}})}, (3)
T𝔸​({f𝜽​(𝒙n)}n=1N)​\triangleq​max𝒂∈𝔸⁡T⁡({𝒂⊤​f𝜽​(𝒙n)}n=1N).\displaystyle T_{{\mathbb{A}}}(\{f_{{\bm{\theta}}}({\bm{x}}_{n})\}_{n=1}^{N})\triangleq\max_{{\bm{a}}\in{\mathbb{A}}}T(\{{\bm{a}}^{\top}f_{{\bm{\theta}}}({\bm{x}}_{n})\}_{n=1}^{N}). (4)

Lemma 3: Hyperspherical Cramér-Wold

Let X,YX,Y be ℝd\mathbb{R}^{d}-valued random vectors, then

Equation⟨𝒖,X⟩=d⟨𝒖,Y⟩,∀𝒖∈𝕊d−1⇔X=dY.\langle{\bm{u}},X\rangle\stackrel{{\scriptstyle d}}{{=}}\langle{\bm{u}},Y\rangle,\forall{\bm{u}}\in\mathbb{S}^{d-1}\iff X\stackrel{{\scriptstyle d}}{{=}}Y.

Convergence in distribution also holds. (Proof in Section B.8.)

Theorem 2: Sufficiency of directional tests

Equation 4 is a valid statistical test for Equation 3 as

Equation P=Q\displaystyle P=Q ⟹lim supn→∞Pr⁡(T𝔸​({f𝜽​(𝒙n)}n=1N)≥τα)≤α,(level)\displaystyle\implies\limsup_{n\to\infty}\Pr\left(T_{{\mathbb{A}}}(\{f_{{\bm{\theta}}}({\bm{x}}_{n})\}_{n=1}^{N})\geq\tau_{\alpha}\right)\leq\alpha,\text{\bf(level)} P≠Q\displaystyle P\neq Q ⟹lim supn→∞Pr⁡(T𝔸​({f𝜽​(𝒙n)}n=1N)≥τα)=1,(power)\displaystyle\implies\limsup_{n\to\infty}\Pr\left(T_{{\mathbb{A}}}(\{f_{{\bm{\theta}}}({\bm{x}}_{n})\}_{n=1}^{N})\geq\tau_{\alpha}\right)=1,\text{\bf(power)}

(Proof in Section B.9.)

Refer to caption
Refer to caption
Figure 4: Examples of distributions living on the surface of the sphere with varying Sobolev smoothness coefficients α\alpha. As per Section 4.3, the greater α\alpha is, the more global will be the impact of SIGReg for a given number of directions MM. Practically, this represents the distribution of the encoder’s output. Because the target density (isotropic Gaussian) is smooth, the α\alpha coeffcients of the embedding will quickly grow hereby making SIGReg (Section 4.2) immune to the curse of dimensionality.
Original figure
Figure 5: Constructed data density with “X” distribution whose marginals are standard Gaussian and whose covariance is identity (left densities). Applying M=10M=10 projections on the half circle directions produces 1010 univariate distributions that can be compared against a standard Gaussian (left) using any preferred statistic from Section 4.2. The appropriate direction is able to capture the degenerate distribution of the data hereby creating a spike in the statistic value.

4.2 SIGReg: Sketching the Epps-Pulley Test is Stable and Scalable

The method averages projection-wise penalties to supply learning signal across directions. The choice of one-dimensional statistic matters for both what departures it detects and how stable its gradients are.

Definition 2: SIGReg (PyTorch code in )

SIGReg sketches a statistical test TT towards isotropic Gaussian

EquationSIGRegT​(𝔸,{f𝜽​(𝒙n)}n=1N)​\triangleq​1|𝔸|​∑𝒂∈𝔸T⁡({𝒂⊤​f𝜽​(𝒙n)}n=1N),{\rm SIGReg}_{T}({\mathbb{A}},\{f_{{\bm{\theta}}}({\bm{x}}_{n})\}_{n=1}^{N})\triangleq\frac{1}{|{\mathbb{A}}|}\sum_{{\bm{a}}\in{\mathbb{A}}}T(\{{\bm{a}}^{\top}f_{{\bm{\theta}}}({\bm{x}}_{n})\}_{n=1}^{N}),
(5)

where we recommend the Epps-Pulley test (Section 4.2.3) for TT.

4.2.1 Moments are Unstable and Insufficient

Matching only a finite number of moments leaves different distributions indistinguishable. High powers of large samples can also create unstable gradients. The paper uses these two issues to motivate a statistic that describes more of the distribution without relying on high-order polynomial moments.

Equations Jarque-Bera–Extended Jarque-Bera
JB⁡(𝒖)​\triangleq\displaystyle{\rm JB}({\bm{u}})\triangleq N6​(skew^​(𝒖)2+(kurt^​(𝒖)−32)2),\displaystyle\frac{N}{6}\left(\widehat{\rm skew}({\bm{u}})^{2}+\left(\frac{\widehat{\rm kurt}({\bm{u}})-3}{2}\right)^{2}\right), (Jarque-Bera)
EJB⁡(𝒖)​\triangleq\displaystyle{\rm EJB}({\bm{u}})\triangleq N​μ^​(𝒖)2σ^​(𝒖)2+(N−1)​(σ^​(𝒖)2−1)22+JB⁡(𝒖).\displaystyle\frac{N\hat{\mu}({\bm{u}})^{2}}{\hat{\sigma}({\bm{u}})^{2}}+\frac{(N-1)\left(\hat{\sigma}({\bm{u}})^{2}-1\right)^{2}}{2}+{\rm JB}({\bm{u}}). (Extended Jarque-Bera)

Theorem 3: Insufficiency of K Moments

Minimizing the following objective with ck>0,∀kc_{k}>0,\forall k

Equation∑k=1Kck​(mk​(P𝜽(𝒂))−mk​(Q(𝒂)))2,\sum_{k=1}^{K}c_{k}\left(m_{k}\left(P_{{\bm{\theta}}}^{({\bm{a}})}\right)-m_{k}\left(Q^{({\bm{a}})}\right)\right)^{2},

for finite KK does not imply P𝜽(𝒂)=Q(𝒂)P_{{\bm{\theta}}}^{({\bm{a}})}=Q^{({\bm{a}})}. (Proof in Section B.11.)

Original figure
Figure 6: N=100N=100 samples are drawn from a 10241024-dimensional standard Gaussian, and the first 22 coordinates are altered to produce the “X” distribution from Figure 5 (left-most column). For each statistic (all other columns), we perform gradient descent on the samples to minimize their value, at each iteration step with sample M=10M=10 random directions to evaluate SIGReg (recall Section 4.2). We obtain that albeit this is a high-dimensional distribution with limited number of samples, SIGReg is able to capture the degenerate subspace and adapt the data accordingly to match an isotropic Gaussian distribution. Additional figures with varying dimensions and number of 1d projections are provided in Figure 16.

4.2.2 Cumulative Density Functions are Impractical

The discussed cumulative-distribution tests require sorting or related order statistics, which complicate efficient distributed differentiation. This is an implementation motivation for the chosen alternative, rather than a claim that such tests have no statistical value.

Equations Cramér-von Mises–Watson
Tw\displaystyle T_{w} =N​∫−∞∞(FN​(x)−F⁡(x))2​w​(x)​𝑑F​(x)\displaystyle=N\int_{-\infty}^{\infty}\left(F_{N}(x)-F(x)\right)^{2}w(x)dF(x)
w⁡(x)\displaystyle w(x) =1,\displaystyle=1, (Cramér-von Mises)
w⁡(x)\displaystyle w(x) =[F⁡(x)​(1−F⁡(x))]−1,\displaystyle=[F(x)(1-F(x))]^{-1}, (Anderson-Darling)
U2=Tw−N​(F¯−12)2.U^{2}=T_{w}-N\left(\bar{F}-\frac{1}{2}\right)^{2}. (Watson)

4.2.3 Characteristic Functions are Stable, Scalable and Identifiable

A characteristic function summarizes a distribution through averages of complex exponentials, a Fourier-style description. The Epps–Pulley penalty compares this summary with that of a Gaussian over a weighted frequency range. Its bounded trigonometric components support the stated gradient and curvature bounds, and sample averages can be combined efficiently across devices.

Equation Epps–Pulley
E​P=N​∫−∞∞|ϕ^X​(t)−ϕ⁡(t)|2​w​(t)​𝑑t.EP=N\int_{-\infty}^{\infty}\left|\hat{\phi}_{X}(t)-\phi(t)\right|^{2}w(t)dt. (Epps–Pulley)

Theorem 4: Stability of Epps-Pulley Test

(Epps–Pulley) satisfies for samples z1,…,zNz_{1},\dots,z_{N}

Equation|∂E​P​(𝐚)∂zi|≤4​σ2N,|∂2E​P​(𝐚)∂zi2|≤C​π​σ32​N,\left|\frac{\partial EP(\mathbf{a})}{\partial z_{i}}\right|\leq\frac{4\sigma^{2}}{N},\hskip 9.24994pt\left|\frac{\partial^{2}EP(\mathbf{a})}{\partial z_{i}^{2}}\right|\leq\frac{C\sqrt{\pi}\sigma^{3}}{2N},

with constant CC, and bandwidth σ\sigma. (Proof in Section B.12.)

SIGReg with Epps-Pulley statistic with DDP support and 𝒪⁡(N)\mathcal{O}(N) time and memory complexity. x is a (N, K) tensor, num_slices is |𝔸||{\mathbb{A}}| in Section 4.2, ‘global_step‘ is used for sync. sampling across GPUs and can be omited for single-GPU training. An optimized implementation with caching is also provided in our official codebase, computation times provided in Table 6.

def SIGReg(x, global_step, num_slices=256):
# slice sampling – synced across devices –
dev = dict(device=x.device)
g = torch.Generator(**dev)
g.manual_seed(global_step)
proj_shape = (x.size(1), num_slices)
A = torch.randn(proj_shape, generator=g, **dev)
A /= A.norm(p=2, dim=0)
# – Epps-Pulley stat. see Sec. 4.3 for alt. –
# integration points
t = torch.linspace(-5, 5, 17, **dev)
# theoretical CF for N(0, 1) and Gauss. window
exp_f = torch.exp(-0.5 * t**2)
# empirical CF – gathered across devices –
x_t = (x @ A).unsqueeze(2) * t # (N, M, T)
ecf = (1j * x_t).exp().mean(0)
ecf = all_reduce(ecf, op="AVG")
# weighted L2 distance
err = (ecf - exp_f).abs().square().mul(exp_f)
N = x.size(0) * world_size
T = torch.trapz(err, t, dim=1) * N
return T

4.3 How SIGReg Beats the Curse of Dimensionality

Finite projection coverage is the practical issue: how many directions are enough to detect a departure from Gaussianity? The following arguments combine smoothness assumptions, resampling during training and controlled experiments.

Smoothness Beats the Curse of Dimensionality

Smoothness can relate a discrepancy on untested directions to discrepancies on tested directions. The rate depends on embedding dimension, smoothness and how directions cover the sphere. The resulting bounds are conditional; a few projections are not an unconditional certificate of an arbitrary high-dimensional distribution.

Original figure
Figure 7: Expected directional statistic at the end of training (y-axis) for varying MM (number of directions used at each training step, x-axis). The MM directions are either resampled (green) or kept fixed (blue) at each training step. While for fixed directions we benefit from Section 4.3 bound where increasing MM reduces the overall expected loss, being able to resample at every step provides significant coverage boost for free.

Theorem 5: Unified Error Bounds

Let p𝜽∈Hα​(ℝK)p_{{\bm{\theta}}}\in H^{\alpha}(\mathbb{R}^{K}), 𝒂∼𝒰⁡(𝒮K−1){\bm{a}}\sim\mathcal{U}(\mathcal{S}^{K-1}), and (Epps–Pulley)=0=0, i.e., Pθ(𝐚)=Q(𝐚),∀𝒂∈𝔸P_{\theta}^{(\mathbf{a})}=Q^{(\mathbf{a})},\forall{\bm{a}}\in{\mathbb{A}}, then

Equation𝔼𝒂[∫ℝ|φa(t)−φ𝒩(t)|2dt]≤C(K,α)|𝔸|−2α/(K−1)×∫0∞‖φ⋅​(r)−φ𝒩​(r)‖Hα​(𝒮K−1)2​dr,\mathbb{E}_{{\bm{a}}}\left[\int_{\mathbb{R}}\left|\varphi_{a}(t)-\varphi_{\mathcal{N}}(t)\right|^{2}dt\right]\leq C(K,\alpha)|{\mathbb{A}}|^{-2\alpha/(K-1)}\\ \times\int_{0}^{\infty}\left\|\varphi_{\cdot}(r)-\varphi_{\mathcal{N}}(r)\right\|_{H^{\alpha}(\mathcal{S}^{K-1})}^{2}dr,

(Proof in Section B.10.)

SGD Beats the Curse of Dimensionality

Drawing new directions on successive minibatches covers more possibilities than repeatedly testing one small fixed set. The paper’s experiments compare these choices. The accumulated training signal should still be distinguished from a finite-sample hypothesis-test guarantee at one fixed checkpoint.

Empirical Validation on Synthetic Data

The synthetic experiment inserts a difficult two-dimensional non-Gaussian pattern into an otherwise Gaussian high-dimensional cloud. Optimizing samples directly tests whether projection penalties can find and remove that defect, separately from encoder architecture and semantic learning.

5 LeJEPA: Stable and Scalable Implementation

The implemented objective combines view prediction/agreement and SIGReg. A mixing coefficient balances the two. Computational approximations make the distribution penalty practical without constructing all pairwise sample distances.

5.1 LeJEPA: SIGReg + Prediction Loss

The characteristic-function integral is approximated with a short deterministic quadrature grid. Minibatch estimation introduces a bias of order inverse batch size in the stated calculation; numerical quadrature adds a separate approximation. Global and local views supply prediction pairs. The core loss can be short, but the complete training system still requires data, augmentation, architecture and optimization settings.

LeJEPA implementation–works out-of-the-box on any dataset, with DDP, with any backbone, e.g., torchvision or timm. For non-ViT architectures (e.g., ResNet), set global_views = all_views. We use bs for the minibatch size, SIGReg is from .

def LeJEPA(global_views, all_views, lambd):
"""global_views and all_views are lists of tensors, lambd is a scalar"""
# embedding of global views
g_emb = forward(torch.cat(glob_views))
# embedding of local views
# if resnet: skip with a_emb=g_emb
a_emb = forward(torch.cat(all_views))
# LeJEPA loss
centers = g_emb.view(-1, bs, K).mean(0)
a_emb = a_emb.view(-1, bs, K)
sim = (centers - a_emb).square().mean()
sigreg = mean(SIGReg(emb, global_step) for emb in a_emb)
return (1-lambd)*sim + lambd*sigreg

Theorem 6: Vanishing gradient bias

The expectation of (Epps–Pulley) satisfies

Equation𝔼⁡[L^n​(θ)]=L⁡(θ)+1N​∫ℝws​(t)​(1−|φP​(t)|2)​𝑑t,\mathbb{E}\left[\widehat{L}_{n}(\theta)\right]=L(\theta)+\frac{1}{N}\int_{\mathbb{R}}w_{s}(t)\big(1-|\varphi_{P}(t)|^{2}\big)dt,

therefore both the loss and its derivative have a bias of order O⁡(1/n)O(1/n). (Proof in Section B.13.)

Equations 6–9
ℒpred​({𝒛n,v}v=1V)=\displaystyle\mathcal{L}_{\rm pred}(\{{\bm{z}}_{n,v}\}_{v=1}^{V})= 1Vg​∑v=1Vg1V​∑v′=1V‖𝒛n,v−𝒛n,v′‖22\displaystyle\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|{\bm{z}}_{n,v}-{\bm{z}}_{n,v^{\prime}}\right\|_{2}^{2} (6)
=\displaystyle= 1V​∑v′=1V‖1Vg​∑v=1Vg𝒛n,v−𝒛n,v′‖22\displaystyle\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|\frac{1}{V_{\rm g}}\sum_{v=1}^{V_{\rm g}}{\bm{z}}_{n,v}-{\bm{z}}_{n,v^{\prime}}\right\|_{2}^{2} (7)
1V​∑v′=1V‖𝝁n−𝒛n,v′‖22,\displaystyle\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|\bm{\mu}_{n}-{\bm{z}}_{n,v^{\prime}}\right\|_{2}^{2}, (8)
ℒLeJEPA​({𝒙n,v}n,v=1B,V)=λV​∑v=1VSIGReg⁡({{𝒛n,v}n=1B})+1−λB∑n=1Bℒ(Vg)pred({𝒛n,v}v=1V).\mathcal{L}_{\rm LeJEPA}(\{{\bm{x}}_{n,v}\}_{n,v=1}^{B,V})=\frac{\lambda}{V}\sum_{v=1}^{V}{\rm SIGReg}(\{\{{\bm{z}}_{n,v}\}_{n=1}^{B}\})\\ +\frac{1-\lambda}{B}\sum_{n=1}^{B}\mathcal{L}^{(V_{\rm g})}_{\rm pred}(\{{\bm{z}}_{n,v}\}_{v=1}^{V}). (9)

5.2 Relation to Prior Work

Random-projection distribution matching connects to sliced methods and kernel discrepancies. In a limiting moment-matching choice, the framework recovers mean and covariance conditions related to VICReg. The Gaussian-testing version constrains more than those first two moments.

6 LeJEPA: Empirical Validation

The experiments separately examine training stability, loss-based model selection, domain-specific pretraining, larger backbones and qualitative semantic features. Each supplies evidence for a different part of the proposed recipe.

6.1 LeJEPA’s Stability Across Hyper-Parameters and Architectures

Ablations vary loss weight, number of views, minibatch size, projection count, integration settings and backbone. The paper recommends a practical starting recipe and reports broad stability in those ranges. SIGReg is not literally parameter-free: its numerical settings and the surrounding training choices still exist even when defaults transfer well.

Original figure
Figure 8: Inet100 with 400400 pretraining epochs and resnet50 backbone. We depict linear probe performances as a function of λ\lambda and the number of views VV (recall (9)). We observe that performances are stable over λ\lambda–with peak performance obtain by slightly adjust λ\lambda proportionally to the number of views. The corresponding performance values are provided in Table 7.
Table 1: ViT/Large-14, on inet1k pretraining for 100 epochs and evaluated with frozen backbone linear probing (top1 accuracy, %).LeJEPA’s performance is stable across all its hyperparameters and while some may slightly improve performance, e.g., the number of slices |𝔸||{\mathbb{A}}| and the projector sizes, none of the choices lead to a catastrophic collapse.

(a) (Epps–Pulley) parameters

integration num_slices config/bstat_n_points
5 17 41
[−1,1][-1,1] 512 71.82 72.13 72.04
2048 72.88 72.30 72.69
[−3,3][-3,3] 512 73.95 74.16 74.04
2048 75.02 74.68 74.77
[−5,5][-5,5] 512 73.71 74.21 74.15
2048 74.50 74.80 74.77

(b) Number of local/global views

# global_views (VgV_{\rm g}) 1 2 4
# views (V=Vg+VlV=V_{\rm g}+V_{\rm l})
4 53.06 72.26 –
6 58.65 73.07 73.68
8 64.46 74.24 73.94
10 68.97 74.06 75.08

(c) Mini-batch size

batch_size 128 256 512 1024
72.20 74.15 74.72 74.07

(d) Embedding/Projector dim.

num_slices 1024 4096
emb. dim. 512 2048 512 2048
proj. dim.
64 75.29 75.32 75.50 75.65
128 74.77 75.09 75.26 75.47
256 74.56 74.66 75.08 75.02
512 73.94 74.11 74.81 74.65
1024 73.65 73.94 74.71 74.79

(e) Register tokens

reg_tokens 0 1 2 4 8
num_slices
1024 75.14 75.18 75.08 75.34 75.23
4096 75.61 75.58 75.67 75.63 75.84
Original figure
Figure 9: INet10 pretraining and frozen backbone linear evaluation across 5050 timm models using LeJEPA out of the box. We cross-validate the learning rate and weight-decay. While there is a small variation between the best and worst performing model, we clearly see that across 5050 models spanning 88 families, LeJEPA is able to produce non-trivial representations able to solve the downstream task at SOTA levels.

Experiment Details 1

We strive for simplicity and thus adopt a unified pretraining pipeline. The following parameters apply to all experiments and figures unless stated otherwise in the corresponding caption and come from Section 6.1:

•

LeJEPA’s implementation is given in with hyperparameter λ\lambda

•

All backbones are from timm{\rm timm} and all optimizers/schedulers are from PyTorch{\rm PyTorch} without modifications

•

We employ eight views (V=8V=8) containing two global views (Vg=2V_{\rm g}=2) with resolution 224x224 and 96x96 for the local views

•

AdamW optimizer with lr∈{5​e−3,5​e−4}{\rm lr}\in\{5e-3,5e-4\} and wd∈{1​e−1,1​e−2,1​e−5}{\rm wd}\in\{1e-1,1e-2,1e-5\}–no scheduler on weight-decay, standard linear warm-up cosine-annealing for lr{\rm lr}

6.2 LeJEPA’s Training Loss is Informative of Downstream Performance

Across the tested runs, lower combined training loss correlates strongly with higher downstream accuracy. This makes it a potentially useful monitoring signal. Correlation in these experiments is not a guaranteed quality ordering for every future dataset, label set or checkpoint.

Refer to caption
Refer to caption
Refer to caption
Figure 10: (SIGReg, prediction loss) 2​d2d-plane with downstream task accuracy shown with colors from blue (low) to red (high). We clearly observe that within this plane, there exists trade-off fronts between the two terms of LeJEPA producing similar downstream performance corresponding to different values of λ\lambda. Yet, those fronts are linear and pointed towards the lower left corner, i.e., LeJEPA’s training loss informs of downstream test performance across models and datasets (columns). Additional models and datasets provided in Figure 21.
Original figure
Figure 11: Spearman correlation (y-axis) between LeJEPA’s training loss and downstream accuracy on the dataset’s classification task with a frozen backbone and linear evaluation. The x-axis varies α\alpha in Equation 10 following our scaling law of the loss w.r.t. λ\lambda. Using α=0\alpha=0 recovers the plain training loss. We clearly observe a very high correlation already for α=0\alpha=0, which further increases up to 99%99\% for α=0.4\alpha=0.4. The entire set of points is obtained across numerous hyper-parameters such as learning rate, weight decay, number of epochs, λ\lambda–demonstrating how LeJEPA’s training loss is strongly predictive of downstream performance which can be used for label-free cross-validation.
Equation 10
C(α)=ρs​(train_lossλα,test_accuracy).\displaystyle C^{(\alpha)}=\rho_{s}\left(\frac{\text{train\_loss}}{\lambda^{\alpha}},\text{test\_accuracy}\right). (10)

6.3 In-Domain LeJEPA Outperforms Frontier Model Transfer Learning

On Galaxy10, pretraining directly on the target visual domain with the proposed recipe can outperform transferring the selected natural-image foundation models. The comparison supports the value of in-domain pretraining under these budgets; it does not isolate the objective from every difference in data and model history.

Original figure
Figure 12: Small architecture in-domain (Galaxy10) LeJEPA pretraining with linear probe evaluation using frozen backbone or full finetuning (columns) and with varying number of samples per class (x-axis). We compare against state-of-the-art foundation models (DINOv2/v3, IJEPA) over 33 different random seeds. We observe that LeJEPA enables in-domain pretraining out of the box across architectures and able to outperform frontier foundation models. Corresponding numbers are provided in Table 3.
Table 2: Few-shot classification accuracy (percentages) on 8 datasets spanning textures, objects, and fine-grained categories. Our LeJEPA achieves superior performance on fine-grained tasks (DTD, flowers102, food101) while requiring only 100 pretraining epochs compared to I-JEPA’s 300 epochs—a 3× reduction in training time and computational resources without sacrificing downstream task performance. This efficiency gain is particularly valuable for practical applications where training budget is limited. Bold indicates best performance within the IN-1K comparison group, all numbers are percentages.
Dataset
shots model params pretrain epochs DTD aircr. cars cifar10 cifar100 flowers102 food pets avg.
1 LeJEPA ViT-L 304M IN-1K 100 33.21 9.37 3.40 51.65 27.01 48.53 17.14 46.11 29.55
LeJEPA ConvNeXtV2-H 660M IN-1K 100 32.15 8.07 4.28 50.95 31.48 48.74 17.95 58.98 31.58
I-JEPA ViT-H 632M IN-1K 300 27.71 9.86 4.33 56.52 30.58 44.69 14.53 53.38 30.20
I-JEPA ViT-H + STOP 632M IN-1K 300 26.60 11.18 4.75 56.27 35.20 47.17 15.75 59.47 32.05
I-JEPA ViT-H (22K) 632M IN-22K 900 27.98 13.00 3.45 61.84 34.70 89.72 19.62 30.86 35.15
10 LeJEPA ViT-L 304M IN-1K 100 64.72 35.25 22.25 85.15 59.77 92.53 50.90 77.00 60.95
LeJEPA ConvNeXtV2-H 660M IN-1K 100 61.84 30.67 24.46 85.74 63.29 91.78 49.32 78.53 60.70
I-JEPA ViT-H 632M IN-1K 300 57.68 33.82 21.96 88.77 66.42 88.24 43.97 83.23 60.51
I-JEPA ViT-H + STOP 632M IN-1K 300 57.00 39.77 25.21 90.09 70.32 90.16 45.68 85.13 62.92
I-JEPA ViT-H (22K) 632M IN-22K 900 58.74 43.52 18.27 94.83 75.23 98.94 49.06 67.66 63.28
all LeJEPA ViT-L 304M IN-1K 100 78.30 57.01 57.28 96.50 83.71 91.21 82.05 89.74 79.48
LeJEPA ConvNeXtV2-H 660M IN-1K 100 76.60 52.99 54.88 96.15 81.34 91.11 77.64 89.76 77.56
I-JEPA ViT-H 632M IN-1K 300 73.32 56.61 54.47 97.54 86.42 86.47 81.02 92.11 78.50
I-JEPA ViT-H + STOP 632M IN-1K 300 73.87 61.95 61.27 98.02 87.78 88.08 81.72 92.88 80.70
I-JEPA ViT-H (22K) 632M IN-22K 900 75.67 65.39 49.79 98.46 89.95 98.54 81.58 87.19 80.82
Refer to caption
Refer to caption
Refer to caption
Figure 13: Emergent Object Segmentation via Last Layer Thresholding. LeJEPA naturally learns to segment and track salient objects (shown in attention maps on the right of each video) without explicit supervision. The results display impressive visual quality and strong temporal consistency across video frames (videos provided on our project page). This emergent capability demonstrates the rich semantic representations learned through our self-supervised approach.
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 14: LeJEPA learns rich semantic representations through self-supervised learning. PCA visualization of last-layer features from LeJEPA (ViT-Large, 100 epochs on ImageNet-1K). For each image, features are independently projected to RGB using the first 3 principal components. Without any supervision, LeJEPA spontaneously develops semantically meaningful representations: notice how warm colors (red/magenta/pink) consistently capture foreground objects (parrot bodies, dog face), while cool colors (cyan/green/yellow) represent backgrounds and foliage. This emergent object-background separation and perceptual grouping discovered the visual structure of the world purely from unlabeled data.

6.4 LeJEPA Scales Across Data and Models

ImageNet experiments scale encoder families and compare transfer with I-JEPA variants. Results show that the method can train larger backbones under the stated schedule. Parameter count, training duration, data and evaluation protocol should remain attached to any claimed advantage.

6.5 Emergent Semantic Structure in LeJEPA Representations

Attention and principal-component visualizations show foreground and semantic structure. Thresholded attention also produces temporally coherent masks in selected video examples without segmentation supervision. These qualitative demonstrations are not a complete quantitative video-segmentation benchmark.

7 Conclusion

LeJEPA offers an explicit, scalable alternative for collapse prevention and a theory-informed implementation. Useful semantics still depend on the views, data and predictive task. This makes the slow-feature counterexample and the later world-model-identifiability paper helpful companion readings.

Acknowledgments

The authors acknowledge feedback from Mike Rabbat and Lucas Maes.

Appendix A Additional Details on Nonlinear Probing

A.1 kNN Probing

The neighbor-probe analysis averages labels in a radius around a query. Density smoothness, bounded derivatives and a smooth target function permit local expansions of bias. These conditions explain why the argument is more specific than a claim about arbitrary language or vision labels.

Equation kNN
𝒚^(𝒒):=1𝒚⁡(𝒒)∑n:‖𝒛n−𝒒‖≤r0𝒚n,\displaystyle\widehat{{\bm{y}}}({\bm{q}}):=\frac{1}{{\bm{y}}({\bm{q}})}\sum_{n:\left\lVert{\bm{z}}_{n}-{\bm{q}}\right\rVert\leq r_{0}}{\bm{y}}_{n}, (kNN)

Lemma 4: k-NN Pointwise Bias

The (kNN) estimator has bias at query 𝒒{\bm{q}} given by

EquationBias(𝒒)=r02d+2(∇η(𝒒)⊤∇logpz(𝒒)+12Δη(𝒛))+o⁡(r02),\mathrm{Bias}({\bm{q}})=\frac{r_{0}^{2}}{d+2}\Big(\nabla\eta({\bm{q}})^{\top}\nabla\log p_{z}({\bm{q}})+\tfrac{1}{2}\Delta\eta({\bm{z}})\Big)\\ +o(r_{0}^{2}),

where the remainder o⁡(r02)o(r_{0}^{2}) is uniform in 𝒒{\bm{q}}. (Proof in Section B.3.)

Theorem 7: k-NN isotropic Gaussian Optimality

The integrated squared bias of (kNN) satisfies

Equation𝔼𝒛​[Bias​(𝒛)2]=r04(K+2)2​τg2​J​(p)+O⁡(r04),\mathbb{E}_{{\bm{z}}}\big[\mathrm{Bias}({\bm{z}})^{2}\big]=\frac{r_{0}^{4}}{(K+2)^{2}}\tau_{g}^{2}J(p)\\ +O(r_{0}^{4}),

and the isotropic Gaussian is the unique minimizer of the integrated square bias. (Proof in Section B.4.)

A.2 Kernel Probing

Kernel regression uses distance-weighted label averages instead of a hard neighborhood. The derivation adds tail and regularity assumptions and constrains embedding spread before comparing density shapes. The retained statements specify the optimization problem to which Gaussian optimality applies.

Equations normalized–NW
∫ℝdK⁡(u)​𝑑u=1,\displaystyle\int_{\mathbb{R}^{d}}K(u)du=1, (normalized)
∫ℝdu​K​(u)​𝑑u=0,\displaystyle\int_{\mathbb{R}^{d}}uK(u)du=0, (symmetric)
∫ℝdu​u⊤​K​(u)​𝑑u=μ2​(K)​Id,\displaystyle\int_{\mathbb{R}^{d}}uu^{\top}K(u)du=\mu_{2}(K)I_{d}, (isotropic)
R⁡(K)​\triangleq​∫ℝdK​(u)2​𝑑u<∞,\displaystyle R(K)\triangleq\int_{\mathbb{R}^{d}}K(u)^{2}du<\infty, (finite roughness)
𝒚^​(𝒒)​\triangleq​∑n=1NKh​(𝒒−𝒙n)​𝒚n∑n=1NKh​(𝒒−𝒙n).\displaystyle\widehat{\bm{y}}({\bm{q}})\triangleq\frac{\sum_{n=1}^{N}K_{h}({\bm{q}}-{\bm{x}}_{n}){\bm{y}}_{n}}{\sum_{n=1}^{N}K_{h}({\bm{q}}-{\bm{x}}_{n})}. (NW)

Lemma 5: Kernel Bias and Variance

For any fixed 𝒒∈ℝd{\bm{q}}\in\mathbb{R}^{d} with p⁡(𝒒)>0p({\bm{q}})>0, as h→0h\to 0 and n​hd→∞nh^{d}\to\infty,

Equation Bias​[𝒚^​(𝒒)]\displaystyle\mathrm{Bias}\big[\widehat{\bm{y}}({\bm{q}})\big] =h2​μ2​(K)2(Δ𝒚(𝒒)+2∇𝒚(𝒒)⊤∇logp(𝒒))+o(h2),\displaystyle=\frac{h^{2}\mu_{2}(K)}{2}\Big(\Delta{\bm{y}}({\bm{q}})+2\nabla{\bm{y}}({\bm{q}})^{\top}\nabla\log p({\bm{q}})\Big)+o(h^{2}), Var​[𝒚^​(𝒒)]\displaystyle\mathrm{Var}\big[\widehat{\bm{y}}({\bm{q}})\big] =R⁡(K)n​hd​v⁡(𝒒)p⁡(𝒒)+o⁡((n​hd)−1).\displaystyle=\frac{R(K)}{nh^{d}}\frac{v({\bm{q}})}{p({\bm{q}})}+o\big((nh^{d})^{-1}\big).

The o⁡(⋅)o(\cdot) terms are uniform over compact sets where pp is bounded away from zero. (Proof in Section B.5.)

Equation
ℳ(L,B)\triangleq{m∈C2(ℝd):∥∇𝒚(𝒒)∥≤L,|Δ𝒚(𝒒)|≤B,∀𝒒∈ℝd},\mathcal{M}(L,B)\triangleq\Big\{m\in C^{2}(\mathbb{R}^{d}):\|\nabla{\bm{y}}({\bm{q}})\|\leq L,\\ |\Delta{\bm{y}}({\bm{q}})|\leq B,\forall{\bm{q}}\in\mathbb{R}^{d}\Big\},

Theorem 8: Kernel isotropic Gaussian Optimality

The integrated squared bias of (NW) satisfies

Equationsupm∈ℳ⁡(L,B)𝔼z​[Bias⁡[𝒚^​(𝒛)]]≤(h2​μ2​(K)2)2​(2​B2+8​L2​J​(p))+o⁡(h4),\sup_{m\in\mathcal{M}(L,B)}\mathbb{E}_{z}\left[\mathrm{Bias}\big[\widehat{\bm{y}}({\bm{z}})\big]\right]\leq\Big(\frac{h^{2}\mu_{2}(K)}{2}\Big)^{2}\Big(2B^{2}+8L^{2}J(p)\Big)+o(h^{4}),

and the integrated variance is independent of pp. Among all densities pp on ℝd\mathbb{R}^{d} with total-variance constrained, e.g., Tr⁡(Σ)=c\Tr(\Sigma)=c, the isotropic Gaussian is the unique minimizer. (Proof in Section B.7.)

Appendix B Proofs

B.1 Proof of Section 3.1

The ridge-regression proof identifies a target direction aligned with a small covariance eigenvalue. Under equal total variance, anisotropy can increase shrinkage bias in that unfavorable direction. The argument is about a controlled worst-case comparison.

Equations 11–13
𝜷^=(𝐗T​𝐗+λwd​𝐈)−1​𝐗T​𝐘.\hat{\bm{\beta}}=(\mathbf{X}^{T}\mathbf{X}+\lambda_{\rm wd}\mathbf{I})^{-1}\mathbf{X}^{T}\mathbf{Y}. (11)
Bias​(𝜷^)\displaystyle\text{Bias}(\hat{\bm{\beta}}) =𝔼⁡[𝜷^]−𝜷true\displaystyle=\mathbb{E}[\hat{\bm{\beta}}]-\bm{\beta}_{\text{true}}
=(𝐗T​𝐗+λwd​𝐈)−1​𝐗T​𝐗​𝜷true−𝜷true\displaystyle=(\mathbf{X}^{T}\mathbf{X}+\lambda_{\rm wd}\mathbf{I})^{-1}\mathbf{X}^{T}\mathbf{X}\bm{\beta}_{\text{true}}-\bm{\beta}_{\text{true}}
=−λwd​(𝐗T​𝐗+λwd​𝐈)−1​𝜷true\displaystyle=-\lambda_{\rm wd}(\mathbf{X}^{T}\mathbf{X}+\lambda_{\rm wd}\mathbf{I})^{-1}\bm{\beta}_{\text{true}}
=−λwd​𝐐​(𝚲+λ​𝐈)−1​𝐐T​𝜷true\displaystyle=-\lambda_{\rm wd}\mathbf{Q}(\bm{\Lambda}+\lambda\mathbf{I})^{-1}\mathbf{Q}^{T}\bm{\beta}_{\text{true}}
λ1+λ2+⋯+λpp=λ¯.\frac{\lambda_{1}+\lambda_{2}+\cdots+\lambda_{p}}{p}=\bar{\lambda}. (12)
𝜷true=κ⋅𝐪p,\bm{\beta}_{\text{true}}=\kappa\cdot\mathbf{q}_{p}, (13)
‖Bias​(𝜷^)‖isotropic=λwdλ¯+λwd​‖𝜷true‖,\displaystyle\|\text{Bias}(\hat{\bm{\beta}})\|_{\text{isotropic}}=\frac{\lambda_{\rm wd}}{\bar{\lambda}+\lambda_{\rm wd}}\|\bm{\beta}_{\text{true}}\|,
‖Bias​(𝜷^)‖non-isotropic=λwdλp+λwd​‖𝜷true‖\displaystyle\|\text{Bias}(\hat{\bm{\beta}})\|_{\text{non-isotropic}}=\frac{\lambda_{\rm wd}}{\lambda_{p}+\lambda_{\rm wd}}\|\bm{\beta}_{\text{true}}\|
λwdλp+λwd>λwdλ¯+λwd\frac{\lambda_{\rm wd}}{\lambda_{p}+\lambda_{\rm wd}}>\frac{\lambda_{\rm wd}}{\bar{\lambda}+\lambda_{\rm wd}}
‖Bias​(𝜷^)‖non-isotropic>‖Bias​(𝜷^)‖isotropic\|\text{Bias}(\hat{\bm{\beta}})\|_{\text{non-isotropic}}>\|\text{Bias}(\hat{\bm{\beta}})\|_{\text{isotropic}}

B.2 Proof of Section 3.1

For the unregularized estimator, the proof expresses total parameter variance through inverse covariance eigenvalues. Convexity shows why an equal spread minimizes this quantity under the stated fixed-sum constraint.

Equations
Var​(𝜷^|𝐗)\displaystyle\text{Var}(\hat{\bm{\beta}}|\mathbf{X}) =𝔼⁡[(𝜷^−𝜷)​(𝜷^−𝜷)T|𝐗]\displaystyle=\mathbb{E}[(\hat{\bm{\beta}}-\bm{\beta})(\hat{\bm{\beta}}-\bm{\beta})^{T}|\mathbf{X}]
=𝔼⁡[(𝐗T​𝐗)−1​𝐗T​𝜺​𝜺T​𝐗​(𝐗T​𝐗)−1|𝐗]\displaystyle=\mathbb{E}[(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\bm{\varepsilon}\bm{\varepsilon}^{T}\mathbf{X}(\mathbf{X}^{T}\mathbf{X})^{-1}|\mathbf{X}]
=(𝐗T​𝐗)−1​𝐗T​𝔼​[𝜺​𝜺T|𝐗]​𝐗​(𝐗T​𝐗)−1\displaystyle=(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbb{E}[\bm{\varepsilon}\bm{\varepsilon}^{T}|\mathbf{X}]\mathbf{X}(\mathbf{X}^{T}\mathbf{X})^{-1}
=(𝐗T​𝐗)−1​𝐗T​(σ2​𝐈n)​𝐗​(𝐗T​𝐗)−1\displaystyle=(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}(\sigma^{2}\mathbf{I}_{n})\mathbf{X}(\mathbf{X}^{T}\mathbf{X})^{-1}
=σ2​(𝐗T​𝐗)−1\displaystyle=\sigma^{2}(\mathbf{X}^{T}\mathbf{X})^{-1}
tr​(Var​(𝜷^))=σ2​tr​(𝐆−1)=σ2​∑j=1p1λj\text{tr}(\text{Var}(\hat{\bm{\beta}}))=\sigma^{2}\text{tr}(\mathbf{G}^{-1})=\sigma^{2}\sum_{j=1}^{p}\frac{1}{\lambda_{j}}
𝐆=𝐐​𝚲​𝐐T\mathbf{G}=\mathbf{Q}\bm{\Lambda}\mathbf{Q}^{T}
1K​∑k=1K1λk>11K​∑j=1Kλk\displaystyle\frac{1}{K}\sum_{k=1}^{K}\frac{1}{\lambda_{k}}>\frac{1}{\frac{1}{K}\sum_{j=1}^{K}\lambda_{k}}
⇔1K​∑k=1K1λk>1K​∑k=1K11K​∑j=1Kλk\displaystyle\iff\frac{1}{K}\sum_{k=1}^{K}\frac{1}{\lambda_{k}}>\frac{1}{K}\sum_{k=1}^{K}\frac{1}{\frac{1}{K}\sum_{j=1}^{K}\lambda_{k}}
⇔∑k=1K1λk>∑k=1K11K​∑j=1Kλk\displaystyle\iff\sum_{k=1}^{K}\frac{1}{\lambda_{k}}>\sum_{k=1}^{K}\frac{1}{\frac{1}{K}\sum_{j=1}^{K}\lambda_{k}}
⇔tr​(Var​(𝜷^))aniso>tr​(Var​(𝜷^))iso\displaystyle\iff\text{tr}(\text{Var}(\hat{\bm{\beta}}))_{\text{aniso}}>\text{tr}(\text{Var}(\hat{\bm{\beta}}))_{\text{iso}}

B.3 Proof of Section A.1

Local Taylor expansions of target and embedding density inside a small ball produce the neighbor estimator’s leading bias terms. Neighborhood symmetry removes odd-order contributions; the small-radius assumptions control the remainder.

Equations
𝔼⁡[η^​(x)]=∫B⁡(0,r0)η⁡(x+z)​p​(x+z)​𝑑z∫B⁡(0,r0)p⁡(x+z)​𝑑zto second order in ​r0,\mathbb{E}\big[\widehat{\eta}(x)\big]\;=\;\frac{\int_{\mathrm{B}(0,r_{0})}\eta(x+z)p(x+z)dz}{\int_{\mathrm{B}(0,r_{0})}p(x+z)dz}\quad\text{to second order in }r_{0},
∫B⁡(0,r)z​𝑑z=0,∫B⁡(0,r)z​z⊤​𝑑z=Vold+2d+2​Id,∫B⁡(0,r)‖z‖2​𝑑z=d​Vold+2d+2.\int_{\mathrm{B}(0,r)}zdz=0,\qquad\int_{\mathrm{B}(0,r)}zz^{\top}dz=\frac{{\rm Vol}^{d+2}}{d+2}I_{d},\qquad\int_{\mathrm{B}(0,r)}\left\lVert z\right\rVert^{2}dz=\frac{d{\rm Vol}^{d+2}}{d+2}.
p⁡(x+z)\displaystyle p(x+z) =p(x)+∇p(x)⊤z+12z⊤Hp(x)z+O(∥z∥3),\displaystyle=p(x)+\nabla p(x)^{\top}z+\tfrac{1}{2}z^{\top}Hp(x)z+O(\|z\|^{3}),
η⁡(x+z)\displaystyle\eta(x+z) =η(x)+∇η(x)⊤z+12z⊤Hη(x)z+O(∥z∥3),\displaystyle=\eta(x)+\nabla\eta(x)^{\top}z+\tfrac{1}{2}z^{\top}H\eta(x)z+O(\|z\|^{3}),
𝒟⁡(x)\displaystyle\mathcal{D}(x) \triangleq​∫B⁡(0,r0)p⁡(x+z)​𝑑z\displaystyle\triangleq\int_{\mathrm{B}(0,r_{0})}p(x+z)dz
=∫B⁡(0,r0)[p(x)+∇p(x)⊤z+12z⊤Hp(x)z+Rp(x;z)]dz\displaystyle=\int_{\mathrm{B}(0,r_{0})}\Big[p(x)+\nabla p(x)^{\top}z+\tfrac{1}{2}z^{\top}Hp(x)z+R_{p}(x;z)\Big]dz
=Vol0d​p​(x)+Vol0d+22​(d+2)​tr​(H​p​(x))+O⁡(r0d+3),\displaystyle={\rm Vol}_{0}^{d}p(x)\;+\;\frac{{\rm Vol}_{0}^{d+2}}{2(d+2)}\mathrm{tr}\big(Hp(x)\big)\;+\;O(r_{0}^{d+3}),
𝒩⁡(x)\displaystyle\mathcal{N}(x) \triangleq​∫B⁡(0,r0)η⁡(x+z)​p​(x+z)​𝑑z\displaystyle\triangleq\int_{\mathrm{B}(0,r_{0})}\eta(x+z)p(x+z)dz
=∫[η(x)+∇η(x)⊤z+12z⊤Hη(x)z][p(x)+∇p(x)⊤z+12z⊤Hp(x)z]dz+O(r0d+3)\displaystyle=\int\Big[\eta(x)+\nabla\eta(x)^{\top}z+\tfrac{1}{2}z^{\top}H\eta(x)z\Big]\Big[p(x)+\nabla p(x)^{\top}z+\tfrac{1}{2}z^{\top}Hp(x)z\Big]dz+O(r_{0}^{d+3})
=η(x)p(x)vdr0d+η(x)vd​r0d+22​(d+2)tr(Hp(x))+vd​r0d+2d+2∇η(x)⋅∇p(x)+vd​r0d+22​(d+2)p(x)tr(Hη(x))+O(r0d+3).\displaystyle=\eta(x)p(x)v_{d}r_{0}^{d}+\eta(x)\frac{v_{d}r_{0}^{d+2}}{2(d+2)}\mathrm{tr}\big(Hp(x)\big)+\frac{v_{d}r_{0}^{d+2}}{d+2}\nabla\eta(x)\cdot\nabla p(x)+\frac{v_{d}r_{0}^{d+2}}{2(d+2)}p(x)\mathrm{tr}\big(H\eta(x)\big)+O(r_{0}^{d+3}).
𝒩⁡(x)−η⁡(x)​𝒟​(x)=vd​r0d+2d+2​(∇η​(x)⋅∇p​(x)+12​p​(x)​Δ​η​(x))+O⁡(r0d+3).\mathcal{N}(x)-\eta(x)\mathcal{D}(x)=\frac{v_{d}r_{0}^{d+2}}{d+2}\Big(\nabla\eta(x)\cdot\nabla p(x)+\tfrac{1}{2}p(x)\Delta\eta(x)\Big)+O(r_{0}^{d+3}).
𝒩⁡(x)𝒟⁡(x)−η⁡(x)\displaystyle\frac{\mathcal{N}(x)}{\mathcal{D}(x)}-\eta(x) =vd​r0d+2d+2​(∇η⋅∇p+12​p​Δ​η)+O⁡(r0d+3)vd​r0d​p​(1+α​r02+O⁡(r03))\displaystyle=\frac{\frac{v_{d}r_{0}^{d+2}}{d+2}\left(\nabla\eta\cdot\nabla p+\frac{1}{2}p\Delta\eta\right)+O(r_{0}^{d+3})}{v_{d}r_{0}^{d}p\left(1+\alpha r_{0}^{2}+O(r_{0}^{3})\right)}
=r02d+2​(∇η⋅∇pp+12​Δ​η)​(1−α​r02+O⁡(r03))+O⁡(r03)\displaystyle=\frac{r_{0}^{2}}{d+2}\left(\frac{\nabla\eta\cdot\nabla p}{p}+\frac{1}{2}\Delta\eta\right)\Big(1-\alpha r_{0}^{2}+O(r_{0}^{3})\Big)\ +\ O(r_{0}^{3})
=r02d+2​(∇η​(x)⋅∇log⁡p​(x)+12​Δ​η​(x))+o⁡(r02),\displaystyle=\frac{r_{0}^{2}}{d+2}\Big(\nabla\eta(x)\cdot\nabla\log p(x)+\tfrac{1}{2}\Delta\eta(x)\Big)\ +\ o(r_{0}^{2}),
𝔼⁡[η^​(x)]−η⁡(x)=r02d+2​(∇η​(x)⋅∇log⁡p​(x)+12​Δ​η​(x))+o⁡(r02),\mathbb{E}\big[\widehat{\eta}(x)\big]-\eta(x)=\frac{r_{0}^{2}}{d+2}\Big(\nabla\eta(x)\cdot\nabla\log p(x)+\tfrac{1}{2}\Delta\eta(x)\Big)\ +\ o(r_{0}^{2}),

B.4 Proof of Section A.1

The squared-bias calculation separates the density score, target gradient and target curvature. It averages over a specified isotropic prior on target gradients, an important assumption behind the resulting distribution preference.

Equations 14–15
Bias⁡(𝒙)=\displaystyle\mathrm{Bias}({\bm{x}})= r02d+2​(∇η​(x)⋅∇log⁡p​(x))+r022​(d+2)​Δ​η​(x)+o⁡(r02)\displaystyle\frac{r_{0}^{2}}{d+2}\Big(\nabla\eta(x)\cdot\nabla\log p(x)\Big)\;+\;\frac{r_{0}^{2}}{2(d+2)}\Delta\eta(x)\;+\;o(r_{0}^{2})
=\displaystyle= r02d+2​(A⁡(x)+C⁡(x))+o⁡(r02),\displaystyle\frac{r_{0}^{2}}{d+2}\big(A(x)+C(x)\big)+o(r_{0}^{2}),
𝔼⁡[Bias​(X)2]\displaystyle\mathbb{E}\big[\mathrm{Bias}(X)^{2}\big] =𝔼⁡[(r02d+2)2​(A​(x)2+2​A​(x)​C​(x)+C​(x)2)+o⁡(r04)]\displaystyle=\mathbb{E}\big[\left(\frac{r_{0}^{2}}{d+2}\right)^{2}\big(A(x)^{2}+2A(x)C(x)+C(x)^{2}\big)+o(r_{0}^{4})\big] (14)
=(r02d+2)2​{𝔼⁡[A​(X)2]⏟score-gradient term+2​𝔼​[A⁡(X)​C​(X)]⏟cross term+𝔼⁡[C​(X)2]⏟curvature term}+o⁡(r04).\displaystyle=\left(\frac{r_{0}^{2}}{d+2}\right)^{2}\Big\{\underbrace{\mathbb{E}\big[A(X)^{2}\big]}_{\text{score-gradient term}}+\underbrace{2\mathbb{E}\big[A(X)C(X)\big]}_{\text{cross term}}+\underbrace{\mathbb{E}\big[C(X)^{2}\big]}_{\text{curvature term}}\Big\}+o(r_{0}^{4}). (15)

1) The score-gradient term 𝔼 ⁡ [ A ​ ( X ) 2 ] .

The density-score contribution becomes a Fisher-information functional after averaging over the isotropic target-gradient prior.

Equation
𝔼⁡[A​(X)2]=\displaystyle\mathbb{E}\big[A(X)^{2}\big]= 𝔼X​[𝔼η​[A​(X)2]]\displaystyle\mathbb{E}_{X}\big[\mathbb{E}_{\eta}[A(X)^{2}]\big]
=\displaystyle= 𝔼X[𝔼η[(∇η(x)⊤v(x))2]]\displaystyle\mathbb{E}_{X}\big[\mathbb{E}_{\eta}[\big(\nabla\eta(x)^{\top}v(x)\big)^{2}]\big]
=\displaystyle= 𝔼X[𝔼η[∇η(x)⊤(v(x)v(x)⊤)∇η(x)]]\displaystyle\mathbb{E}_{X}\big[\mathbb{E}_{\eta}[\nabla\eta(x)^{\top}\Big(v(x)v(x)^{\top}\Big)\nabla\eta(x)]\big]
=\displaystyle= 𝔼X[𝔼η[tr(v(x)v(x)⊤∇η(x)∇η(x)⊤)]]\displaystyle\mathbb{E}_{X}\big[\mathbb{E}_{\eta}[\mathrm{tr}\Big(v(x)v(x)^{\top}\nabla\eta(x)\nabla\eta(x)^{\top}\Big)]\big]
=\displaystyle= 𝔼X[tr(v(x)v(x)⊤𝔼η[∇η(x)∇η(x)⊤])]\displaystyle\mathbb{E}_{X}\big[\mathrm{tr}\Big(v(x)v(x)^{\top}\mathbb{E}_{\eta}[\nabla\eta(x)\nabla\eta(x)^{\top}]\Big)\big]
=\displaystyle= 𝔼X​[τg2​‖v⁡(x)‖2]\displaystyle\mathbb{E}_{X}\big[\tau_{g}^{2}\|v(x)\|^{2}\big]
=\displaystyle= τg2​𝔼X​[‖v⁡(X)‖2]\displaystyle\tau_{g}^{2}\mathbb{E}_{X}\big[\|v(X)\|^{2}\big]
=\displaystyle= τg2​∫ℝd‖∇log⁡p​(x)‖2​p​(x)​𝑑x\displaystyle\tau_{g}^{2}\int_{\mathbb{R}^{d}}\|\nabla\log p(x)\|^{2}p(x)dx

2) The cross term 2 ​ 𝔼 ​ [ A ⁡ ( X ) ​ C ​ ( X ) ] .

The cross term is controlled using the stated symmetry and target-function assumptions. Its treatment cannot be assumed unchanged for an arbitrary downstream target family.

Equations
A(x)C(x)=12(∇η(x)⊤v(x))Δη(x).A(x)C(x)=\frac{1}{2}\big(\nabla\eta(x)^{\top}v(x)\big)\Delta\eta(x).
(r02d+2)2⋅2​𝔼​[A⁡(X)​C​(X)]=O⁡(r04),\left(\frac{r_{0}^{2}}{d+2}\right)^{2}\cdot 2\mathbb{E}[A(X)C(X)]=\ O(r_{0}^{4}),

3) The curvature term 𝔼 ⁡ [ C ​ ( X ) 2 ] .

The remaining term measures target curvature. It separates a target-dependent contribution from the density-dependent part being optimized.

Equation
𝔼⁡[C​(X)2]=\displaystyle\mathbb{E}\big[C(X)^{2}\big]= 𝔼X​[𝔼η​[C​(X)2]]\displaystyle\mathbb{E}_{X}\big[\mathbb{E}_{\eta}[C(X)^{2}]\big]
=\displaystyle= 14𝔼X[𝔼η[(Δη(X))2]\displaystyle\frac{1}{4}\mathbb{E}_{X}\big[\mathbb{E}_{\eta}[(\Delta\eta(X))^{2}\big]

Putting it together.

Combining the terms shows which part of the approximation error can be improved by changing the embedding density under this model.

Equations
𝔼⁡[Bias​(X)2]\displaystyle\mathbb{E}\big[\mathrm{Bias}(X)^{2}\big] =(r02d+2)2​{τg2​J​(p)+O⁡(1)}+o⁡(r04)\displaystyle=\left(\frac{r_{0}^{2}}{d+2}\right)^{2}\Big\{\tau_{g}^{2}J(p)+O(1)\Big\}+o(r_{0}^{4})
=r04(d+2)2​τg2​J​(p)+O⁡(r04),\displaystyle=\frac{r_{0}^{4}}{(d+2)^{2}}\tau_{g}^{2}J(p)\;+\;O(r_{0}^{4}),
J⁡(p):=∫ℝd‖∇log⁡p​(x)‖2​p​(x)​𝑑xJ(p)\;:=\;\int_{\mathbb{R}^{d}}\|\nabla\log p(x)\|^{2}p(x)dx

Lemma 6: Special case: Recovery of VCReg

Let pp be a mean-zero probability density on ℝd\mathbb{R}^{d} with covariance Σ=𝔼⁡[X​X⊤]≻0\Sigma=\mathbb{E}[XX^{\top}]\succ 0. Then

EquationJ⁡(p)≥tr⁡(Σ−1),J(p)\;\geq\;\mathrm{tr}(\Sigma^{-1}),

with equality if and only if p=𝒩⁡(0,Σ)p=\mathcal{N}(0,\Sigma).

Equation
ℐ(θ)=𝔼[∇θlogpθ(X)∇θlogpθ(X)⊤]=𝔼[∇logp(X)∇logp(X)⊤],\mathcal{I}(\theta)\;=\;\mathbb{E}\big[\nabla_{\theta}\log p_{\theta}(X)\nabla_{\theta}\log p_{\theta}(X)^{\top}\big]\;=\;\mathbb{E}\big[\nabla\log p(X)\nabla\log p(X)^{\top}\big],

Step 2: Optimizing over covariance shapes under scalar constraints

The next step compares allowable covariance shapes under different scalar constraints. The constraint determines what it means to compare distributions with the same available spread.

Equation
tr⁡(Σ−1)=∑i=1d1λi.\mathrm{tr}(\Sigma^{-1})=\sum_{i=1}^{d}\frac{1}{\lambda_{i}}.

(a) Trace constraint.

With fixed covariance trace, the argument favors equal variance across coordinates through a convexity bound.

Equations
(∑i=1d1λi)​(∑i=1dλi)≥(∑i=1d1)2=d2,\left(\sum_{i=1}^{d}\frac{1}{\lambda_{i}}\right)\left(\sum_{i=1}^{d}\lambda_{i}\right)\;\geq\;\left(\sum_{i=1}^{d}1\right)^{2}=d^{2},
minΣ≻0:tr⁡(Σ)=ttr(Σ−1)=d2t,attained atΣ=tdId.\min_{\Sigma\succ 0:\ \mathrm{tr}(\Sigma)=t}\ \mathrm{tr}(\Sigma^{-1})\;=\;\frac{d^{2}}{t},\quad\text{attained at}\quad\Sigma=\frac{t}{d}I_{d}.

(b) Determinant constraint.

The determinant-constrained case fixes a volume-related quantity and analyzes the corresponding covariance optimum.

Equations
1d∑i=1dμi≥(∏i=1dμi)1/d=δ−1/d,\frac{1}{d}\sum_{i=1}^{d}\mu_{i}\;\geq\;\left(\prod_{i=1}^{d}\mu_{i}\right)^{1/d}=\delta^{-1/d},
minΣ≻0:det(Σ)=δtr(Σ−1)=dδ−1/d,attained atΣ=δ1/dId.\min_{\Sigma\succ 0:\ \det(\Sigma)=\delta}\ \mathrm{tr}(\Sigma^{-1})\;=\;d\delta^{-1/d},\quad\text{attained at}\quad\Sigma=\delta^{1/d}I_{d}.

(c) Frobenius-norm constraint.

The Frobenius-norm case constrains the squared size of the covariance matrix before comparing its eigenvalues.

Equations
ℒ⁡(λ,ν)=∑i=1d1λi+ν⁡(∑i=1dλi2−c2)\mathcal{L}(\lambda,\nu)=\sum_{i=1}^{d}\frac{1}{\lambda_{i}}+\nu\left(\sum_{i=1}^{d}\lambda_{i}^{2}-c^{2}\right)
minΣ≻0:‖Σ‖F=ctr(Σ−1)=∑i=1d1λi=d3/2c,attained atΣ=cdId.\min_{\Sigma\succ 0:\ \|\Sigma\|_{F}=c}\ \mathrm{tr}(\Sigma^{-1})\;=\;\sum_{i=1}^{d}\frac{1}{\lambda_{i}}\;=\;\frac{d^{3/2}}{c},\quad\text{attained at}\quad\Sigma=\frac{c}{\sqrt{d}}I_{d}.

(d) Spectral-radius constraint.

The spectral-radius case restricts the largest covariance eigenvalue. It is a different feasible set from fixing total variance.

Equations
∑i=1d1λi≥∑i=1d1r=dr,\sum_{i=1}^{d}\frac{1}{\lambda_{i}}\;\geq\;\sum_{i=1}^{d}\frac{1}{r}\;=\;\frac{d}{r},
minΣ≻0:ρ⁡(Σ)≤rtr(Σ−1)=dr,attained atΣ=rId.\min_{\Sigma\succ 0:\ \rho(\Sigma)\leq r}\ \mathrm{tr}(\Sigma^{-1})\;=\;\frac{d}{r},\quad\text{attained at}\quad\Sigma=rI_{d}.

Conclusion: Isotropic Gaussian is optimal

The Gaussian and isotropic conclusions combine density and covariance optimization under the stated assumptions. They do not prove that every semantically useful representation must be identical to this optimum.

Theorem 9: Special case: Recovery of VCReg

Fix one of the following scalar covariance constraints for a mean-zero distribution pp on ℝd\mathbb{R}^{d}:

•

trace: tr⁡(Cov⁡(X))=t\mathrm{tr}(\mathrm{Cov}(X))=t,

•

determinant: det(Cov⁡(X))=δ\det(\mathrm{Cov}(X))=\delta,

•

Frobenius norm: ‖Cov⁡(X)‖F=c\|\mathrm{Cov}(X)\|_{F}=c,

•

spectral radius upper bound: ρ⁡(Cov⁡(X))≤r\rho(\mathrm{Cov}(X))\leq r.

Then the Fisher-information functional J⁡(p)J(p) is minimized over all such pp by the isotropic Gaussian pG=𝒩⁡(0,s​Id)p_{G}=\mathcal{N}(0,sI_{d}) with ss chosen to satisfy the constraint. The minimal values are:

Equationtrace ​t:Jmin=d2t,s=td,determinant ​δ:Jmin=dδ−1/d,s=δ1/d,Frobenius ​c:Jmin=d3/2c,s=cd,spectral radius ​r:Jmin=dr,s=r.\begin{array}[]{ll}\text{trace }t:&J_{\min}=\dfrac{d^{2}}{t},\hskip 9.24994pts=\dfrac{t}{d},\\[4.64996pt] \text{determinant }\delta:&J_{\min}=d\delta^{-1/d},\hskip 9.24994pts=\delta^{1/d},\\[4.64996pt] \text{Frobenius }c:&J_{\min}=\dfrac{d^{3/2}}{c},\hskip 9.24994pts=\dfrac{c}{\sqrt{d}},\\[4.64996pt] \text{spectral radius }r:&J_{\min}=\dfrac{d}{r},\hskip 9.24994pts=r.\end{array}

In each case, pGp_{G} is the unique minimizer (up to null sets).

B.5 Proof of Section A.2

The kernel-probe proof repeats the local bias/variance analysis with kernel-weighted neighborhoods and its stated smoothness and tail conditions.

Equations
Bn​(x):=∑i=1nKh​(x−Xi)​Yi,An​(x):=∑i=1nKh​(x−Xi),B_{n}(x):=\sum_{i=1}^{n}K_{h}(x-X_{i})Y_{i},\qquad A_{n}(x):=\sum_{i=1}^{n}K_{h}(x-X_{i}),
𝔼​[An​(x)]\displaystyle\mathbb{E}[A_{n}(x)] =n​𝔼​[Kh​(x−X)]\displaystyle=n\mathbb{E}\big[K_{h}(x-X)\big]
=n​∫ℝdh−d​K​(x−uh)​p​(u)​𝑑u\displaystyle=n\int_{\mathbb{R}^{d}}h^{-d}K\Big(\frac{x-u}{h}\Big)p(u)du
=n∫ℝdK(t)p(x−ht)dt(t:=(x−u)/h)\displaystyle=n\int_{\mathbb{R}^{d}}K(t)p(x-ht)dt\qquad(t:=(x-u)/h)
=n∫ℝdK(t)(p(x)−ht⊤∇p(x)+h22t⊤∇2p(x)t+o(h2))dt\displaystyle=n\int_{\mathbb{R}^{d}}K(t)\Big(p(x)-ht^{\top}\nabla p(x)+\frac{h^{2}}{2}t^{\top}\nabla^{2}p(x)t+o(h^{2})\Big)dt
=n⁡(p⁡(x)+h22​∫t⊤​∇2p​(x)​t​K​(t)​dt⏟=μ2​(K)​Δ​p​(x)+o⁡(h2)),\displaystyle=n\Big(p(x)+\frac{h^{2}}{2}\underbrace{\int t^{\top}\nabla^{2}p(x)tK(t)dt}_{=\mu_{2}(K)\Delta p(x)}+o(h^{2})\Big),
𝔼​[Bn​(x)]\displaystyle\mathbb{E}[B_{n}(x)] =n​𝔼​[Kh​(x−X)​Y]=n​∫K⁡(t)​(m​p)​(x−h​t)​𝑑t\displaystyle=n\mathbb{E}\big[K_{h}(x-X)Y\big]=n\int K(t)(mp)(x-ht)dt
=n∫K(t)((mp)(x)−ht⊤∇(mp)(x)+h22t⊤∇2(mp)(x)t+o(h2))dt\displaystyle=n\int K(t)\Big((mp)(x)-ht^{\top}\nabla(mp)(x)+\frac{h^{2}}{2}t^{\top}\nabla^{2}(mp)(x)t+o(h^{2})\Big)dt
=n⁡(m⁡(x)​p​(x)+h22​μ2​(K)​tr​(∇2(m​p)​(x))+o⁡(h2))\displaystyle=n\Big(m(x)p(x)+\frac{h^{2}}{2}\mu_{2}(K)\mathrm{tr}\big(\nabla^{2}(mp)(x)\big)+o(h^{2})\Big)
=n(m(x)p(x)+h2​μ2​(K)2(pΔm+mΔp+2∇m⊤∇p)(x)+o(h2)),\displaystyle=n\Big(m(x)p(x)+\frac{h^{2}\mu_{2}(K)}{2}\big(p\Delta m+m\Delta p+2\nabla m^{\top}\nabla p\big)(x)+o(h^{2})\Big),
a0+h2​a2+o⁡(h2)b0+h2​b2+o⁡(h2)=a0b0+h2​a2​b0−a0​b2b02+o⁡(h2),\frac{a_{0}+h^{2}a_{2}+o(h^{2})}{b_{0}+h^{2}b_{2}+o(h^{2})}=\frac{a_{0}}{b_{0}}+h^{2}\frac{a_{2}b_{0}-a_{0}b_{2}}{b_{0}^{2}}+o(h^{2}),
𝔼​[Bn​(x)]𝔼​[An​(x)]\displaystyle\frac{\mathbb{E}[B_{n}(x)]}{\mathbb{E}[A_{n}(x)]} =m⁡(x)+h2​μ2​(K)2​(pΔm+mΔp+2∇m⊤∇p)p−mpΔpp2|x+o⁡(h2)\displaystyle=m(x)+\frac{h^{2}\mu_{2}(K)}{2}\frac{\big(p\Delta m+m\Delta p+2\nabla m^{\top}\nabla p\big)p-mp\Delta p}{p^{2}}\Big|_{x}+o(h^{2})
=m(x)+h2​μ2​(K)2(Δm(x)+2∇m(x)⊤∇p​(x)p⁡(x))+o(h2),\displaystyle=m(x)+\frac{h^{2}\mu_{2}(K)}{2}\Big(\Delta m(x)+2\nabla m(x)^{\top}\frac{\nabla p(x)}{p(x)}\Big)+o(h^{2}),
Var​[m^​(x)]≈Var​[Bn​(x)](𝔼⁡[An​(x)])2.\mathrm{Var}[\widehat{m}(x)]\approx\frac{\mathrm{Var}[B_{n}(x)]}{(\mathbb{E}[A_{n}(x)])^{2}}.
Var​[Bn​(x)]\displaystyle\mathrm{Var}[B_{n}(x)] =∑i=1nVar(Kh(x−Xi)Yi)(independence)\displaystyle=\sum_{i=1}^{n}\mathrm{Var}\big(K_{h}(x-X_{i})Y_{i}\big)\quad\text{(independence)}
=n​𝔼​[Kh​(x−X)2​Var​(Y∣X)]=n​𝔼​[Kh​(x−X)2​v​(X)]\displaystyle=n\mathbb{E}\big[K_{h}(x-X)^{2}\mathrm{Var}(Y\mid X)\big]=n\mathbb{E}\big[K_{h}(x-X)^{2}v(X)\big]
=n​∫h−2​d​K​(x−uh)2​v​(u)​p​(u)​𝑑u\displaystyle=n\int h^{-2d}K\Big(\frac{x-u}{h}\Big)^{2}v(u)p(u)du
=n​h−d​∫K​(t)2​v​(x−h​t)​p​(x−h​t)​𝑑t=n​h−d​(R⁡(K)​v​(x)​p​(x)+o⁡(1)),\displaystyle=nh^{-d}\int K(t)^{2}v(x-ht)p(x-ht)dt=nh^{-d}\Big(R(K)v(x)p(x)+o(1)\Big),
𝔼⁡[An​(x)]=n⁡(p⁡(x)+o⁡(1)).\mathbb{E}[A_{n}(x)]=n\big(p(x)+o(1)\big).
Var⁡[m^​(x)]≈n​h−d​R​(K)​v​(x)​p​(x)n2​p​(x)2=R⁡(K)n​hd​v⁡(x)p⁡(x)+o⁡((n​hd)−1),\mathrm{Var}[\widehat{m}(x)]\approx\frac{nh^{-d}R(K)v(x)p(x)}{n^{2}p(x)^{2}}=\frac{R(K)}{nh^{d}}\frac{v(x)}{p(x)}+o\big((nh^{d})^{-1}\big),

B.6 Proof of Equation 6 to Equation 7

This calculation relates the characteristic-function discrepancy to the expression used by the proposed regularizer. The exact algebraic expressions are preserved.

Equations 16–26
1Vg​∑v=1Vg1V​∑v′=1V‖𝐳n,v−𝐳n,v′‖22=1V​∑v′=1V‖𝐳¯−𝐳n,v′‖22\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\frac{1}{V}\sum_{v^{\prime}=1}^{V}\|\mathbf{z}_{n,v}-\mathbf{z}_{n,v^{\prime}}\|_{2}^{2}=\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left\|\bar{\mathbf{z}}-\mathbf{z}_{n,v^{\prime}}\right\|_{2}^{2} (16)
LHS =1Vg​V​∑v=1Vg∑v′=1V‖𝐳n,v−𝐳n,v′‖22\displaystyle=\frac{1}{V_{g}V}\sum_{v=1}^{V_{g}}\sum_{v^{\prime}=1}^{V}\|\mathbf{z}_{n,v}-\mathbf{z}_{n,v^{\prime}}\|_{2}^{2} (17)
=1Vg​V​∑v=1Vg∑v′=1V(‖𝐳n,v‖22−2​𝐳n,vT​𝐳n,v′+‖𝐳n,v′‖22)\displaystyle=\frac{1}{V_{g}V}\sum_{v=1}^{V_{g}}\sum_{v^{\prime}=1}^{V}\left(\|\mathbf{z}_{n,v}\|_{2}^{2}-2\mathbf{z}_{n,v}^{T}\mathbf{z}_{n,v^{\prime}}+\|\mathbf{z}_{n,v^{\prime}}\|_{2}^{2}\right) (18)
=1Vg​∑v=1Vg‖𝐳n,v‖22−2Vg​V​∑v=1Vg∑v′=1V𝐳n,vT​𝐳n,v′+1V​∑v′=1V‖𝐳n,v′‖22\displaystyle=\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\|\mathbf{z}_{n,v}\|_{2}^{2}-\frac{2}{V_{g}V}\sum_{v=1}^{V_{g}}\sum_{v^{\prime}=1}^{V}\mathbf{z}_{n,v}^{T}\mathbf{z}_{n,v^{\prime}}+\frac{1}{V}\sum_{v^{\prime}=1}^{V}\|\mathbf{z}_{n,v^{\prime}}\|_{2}^{2} (19)
=1Vg​∑v=1Vg‖𝐳n,v‖22−2V​𝐳¯T​∑v′=1V𝐳n,v′+1V​∑v′=1V‖𝐳n,v′‖22\displaystyle=\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\|\mathbf{z}_{n,v}\|_{2}^{2}-\frac{2}{V}\bar{\mathbf{z}}^{T}\sum_{v^{\prime}=1}^{V}\mathbf{z}_{n,v^{\prime}}+\frac{1}{V}\sum_{v^{\prime}=1}^{V}\|\mathbf{z}_{n,v^{\prime}}\|_{2}^{2} (20)
RHS =1V​∑v′=1V(‖𝐳¯‖22−2​𝐳¯T​𝐳n,v′+‖𝐳n,v′‖22)\displaystyle=\frac{1}{V}\sum_{v^{\prime}=1}^{V}\left(\|\bar{\mathbf{z}}\|_{2}^{2}-2\bar{\mathbf{z}}^{T}\mathbf{z}_{n,v^{\prime}}+\|\mathbf{z}_{n,v^{\prime}}\|_{2}^{2}\right) (21)
=‖𝐳¯‖22−2V​𝐳¯T​∑v′=1V𝐳n,v′+1V​∑v′=1V‖𝐳n,v′‖22\displaystyle=\|\bar{\mathbf{z}}\|_{2}^{2}-\frac{2}{V}\bar{\mathbf{z}}^{T}\sum_{v^{\prime}=1}^{V}\mathbf{z}_{n,v^{\prime}}+\frac{1}{V}\sum_{v^{\prime}=1}^{V}\|\mathbf{z}_{n,v^{\prime}}\|_{2}^{2} (22)
1Vg​∑v=1Vg‖𝐳n,v‖22=‖𝐳¯‖22\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\|\mathbf{z}_{n,v}\|_{2}^{2}=\|\bar{\mathbf{z}}\|_{2}^{2} (23)
‖𝐳¯‖22\displaystyle\|\bar{\mathbf{z}}\|_{2}^{2} =‖1Vg​∑v=1Vg𝐳n,v‖22\displaystyle=\left\|\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\mathbf{z}_{n,v}\right\|_{2}^{2} (24)
=1Vg2​∑v=1Vg∑v′′=1Vg𝐳n,vT​𝐳n,v′′\displaystyle=\frac{1}{V_{g}^{2}}\sum_{v=1}^{V_{g}}\sum_{v^{\prime\prime}=1}^{V_{g}}\mathbf{z}_{n,v}^{T}\mathbf{z}_{n,v^{\prime\prime}} (25)
=1Vg​∑v=1Vg‖𝐳n,v‖22\displaystyle=\frac{1}{V_{g}}\sum_{v=1}^{V_{g}}\|\mathbf{z}_{n,v}\|_{2}^{2} (26)

B.7 Proof of Section A.2

The remaining kernel-probe argument compares admissible embedding densities under the specified constraints, completing the conditional optimality result.

Equations
Bias[m^(x)]=h2​μ2​(K)2(Δm(x)+2∇m(x)⊤∇logp(x))+o(h2).\mathrm{Bias}[\widehat{m}(x)]=\frac{h^{2}\mu_{2}(K)}{2}\Big(\Delta m(x)+2\nabla m(x)^{\top}\nabla\log p(x)\Big)+o(h^{2}).
ℬ2​(h,p,m)\displaystyle\mathcal{B}^{2}(h;p,m) =(h2​μ2​(K)2)2∫(Δm(x)+2∇m(x)⊤∇logp(x))2p(x)dx+o(h4)\displaystyle=\Big(\frac{h^{2}\mu_{2}(K)}{2}\Big)^{2}\int\Big(\Delta m(x)+2\nabla m(x)^{\top}\nabla\log p(x)\Big)^{2}p(x)dx+o(h^{4})
≤(h2​μ2​(K)2)2∫(2(Δm(x))2+2(2∇m(x)⊤∇logp(x))2)p(x)dx+o(h4)\displaystyle\leq\Big(\frac{h^{2}\mu_{2}(K)}{2}\Big)^{2}\int\Big(2(\Delta m(x))^{2}+2(2\nabla m(x)^{\top}\nabla\log p(x))^{2}\Big)p(x)dx+o(h^{4})
=(h2​μ2​(K)2)2(2∫(Δm(x))2p(x)dx+8∫(∇m(x)⊤∇logp(x))2p(x)dx)+o(h4),\displaystyle=\Big(\frac{h^{2}\mu_{2}(K)}{2}\Big)^{2}\Big(2\int(\Delta m(x))^{2}p(x)dx+8\int(\nabla m(x)^{\top}\nabla\log p(x))^{2}p(x)dx\Big)+o(h^{4}),
∫(Δ​m)2​p≤∫B2​p=B2.\int(\Delta m)^{2}p\leq\int B^{2}p=B^{2}.
(∇m(x)⊤∇logp(x))2≤∥∇m(x)∥2∥∇logp(x)∥2≤L2∥∇logp(x)∥2\displaystyle(\nabla m(x)^{\top}\nabla\log p(x))^{2}\leq\|\nabla m(x)\|^{2}\|\nabla\log p(x)\|^{2}\leq L^{2}\|\nabla\log p(x)\|^{2}
⟹∫(∇m(x)⊤∇logp(x))2p(x)dx≤L2∫∥∇logp(x)∥2p(x)dx=L2J(p).\displaystyle\implies\int(\nabla m(x)^{\top}\nabla\log p(x))^{2}p(x)dx\leq L^{2}\int\|\nabla\log p(x)\|^{2}p(x)dx=L^{2}J(p).
𝒱⁡(h,p)\displaystyle\mathcal{V}(h;p) =∫(R⁡(K)n​hd​v⁡(x)p⁡(x)+o⁡((n​hd)−1))​p​(x)​𝑑x=R⁡(K)n​hd​∫v⁡(x)​𝑑x+o⁡((n​hd)−1),\displaystyle=\int\Big(\frac{R(K)}{nh^{d}}\frac{v(x)}{p(x)}+o\big((nh^{d})^{-1}\big)\Big)p(x)dx=\frac{R(K)}{nh^{d}}\int v(x)dx+o\big((nh^{d})^{-1}\big),

B.8 Proof of Section 4.1

The projection argument uses uniqueness of characteristic functions: agreement of every one-dimensional projected law identifies the full joint law. It is a population statement requiring every direction, before finite testing is considered.

Theorem 10: Cramér-Wold Cramér and Wold [1936]

Let XX and YY be random vectors in ℝD\mathbb{R}^{D}:

Equation 27 X​=𝑑​Y⇔⟨X,a⟩​=𝑑​⟨Y,a⟩,∀𝒂∈ℝD.\displaystyle X\overset{d}{=}Y\iff\langle X,a\rangle\overset{d}{=}\langle Y,a\rangle,\forall{\bm{a}}\in\mathbb{R}^{D}. (27)
Equations
φX​(t)=𝔼⁡[ei​⟨t,X⟩]=𝔼⁡[ei​s​⟨u,X⟩]=𝔼⁡[ei​s​⟨u,Y⟩]=𝔼⁡[ei​⟨t,Y⟩]=φY​(t).\varphi_{X}(t)\;=\;\mathbb{E}\big[e^{i\langle t,X\rangle}\big]\;=\;\mathbb{E}\big[e^{is\langle u,X\rangle}\big]\;=\;\mathbb{E}\big[e^{is\langle u,Y\rangle}\big]\;=\;\mathbb{E}\big[e^{i\langle t,Y\rangle}\big]\;=\;\varphi_{Y}(t).
⟨t,Xn⟩=s⁡⟨u,Xn⟩→𝑑s⁡⟨u,X⟩=⟨t,X⟩.\langle t,X_{n}\rangle\;=\;s\langle u,X_{n}\rangle\xrightarrow{d}s\langle u,X\rangle\;=\;\langle t,X\rangle.
ψn,t=𝔼⁡[ei​⟨t,Xn⟩]⟶𝔼⁡[ei​⟨t,X⟩]=ψt,for all ​t∈ℝd.\psi_{n,t}\;=\;\mathbb{E}\big[e^{i\langle t,X_{n}\rangle}\big]\;\longrightarrow\;\mathbb{E}\big[e^{i\langle t,X\rangle}\big]\;=\;\psi_{t},\qquad\text{for all }t\in\mathbb{R}^{d}.

B.9 Proof of Section 4.1

The consistency argument assumes sufficiently dense direction coverage and a detectable neighborhood around a separating direction. These assumptions connect projection-wise tests to an eventual full-distribution decision.

Equations
infa∈Ulimn→∞Pr⁡(Ta,n≥un​(α))=1.\inf_{a\in U}\lim_{n\to\infty}\Pr\big(T_{a,n}\geq u_{n}(\alpha)\big)=1.
infa∈Ulimn→∞Pr⁡(Ta,n≥un​(α))= 1.\inf_{a\in U}\lim_{n\to\infty}\Pr\big(T_{a,n}\geq u_{n}(\alpha)\big)\;=\;1.
Pr⁡(Ψn=1)=Pr⁡(Mn≥un​(α))≥Pr⁡(Tan,n≥un​(α))⟶ 1,\Pr(\Psi_{n}=1)\;=\;\Pr\big(M_{n}\geq u_{n}(\alpha)\big)\;\geq\;\Pr\big(T_{a_{n},n}\geq u_{n}(\alpha)\big)\;\longrightarrow\;1,

B.10 Proof of Section 4.3

The finite-direction approximation bound uses smoothness and quasi-uniform coverage of the sphere. Its dimension and regularity dependence should be kept when interpreting the practical projection count.

Equations
𝔼b[|g(b)−g∗(b)|2]≤C(D,α)M−2α/(D−1)∥g∥Hα​(𝕊D−1)2,\mathbb{E}_{b}\left[|g(b)-g^{*}(b)|^{2}\right]\leq C(D,\alpha)M^{-2\alpha/(D-1)}\|g\|_{H^{\alpha}(\mathbb{S}^{D-1})}^{2},
∥f−PLf∥L2​(𝕊d)≤(1+L2)−α/2∥f∥Hα​(𝕊d),\|f-P_{L}f\|_{L^{2}(\mathbb{S}^{d})}\leq(1+L^{2})^{-\alpha/2}\|f\|_{H^{\alpha}(\mathbb{S}^{d})},
∥f−IMf∥L2​(𝕊d)≤C(1+L2)−α/2∥f∥Hα​(𝕊d),\|f-I_{M}f\|_{L^{2}(\mathbb{S}^{d})}\leq C(1+L^{2})^{-\alpha/2}\|f\|_{H^{\alpha}(\mathbb{S}^{d})},
𝔼ω|f(ω)−IMf(ω)|2≤C(d,α)M−2α/d∥f∥Hα​(𝕊d)2,\mathbb{E}_{\omega}|f(\omega)-I_{M}f(\omega)|^{2}\leq C(d,\alpha)M^{-2\alpha/d}\|f\|_{H^{\alpha}(\mathbb{S}^{d})}^{2},

B.11 Proof of Section 4.2.1

A linear-algebra construction makes two distinct discrete distributions agree on any prescribed finite set of moments. This explains why low-order moment matching alone cannot identify a general distribution.

B.12 Proof of Section 4.2.3

Bounded complex exponentials yield bounds on the chosen loss derivatives, while moment penalties can grow polynomially with sample magnitude. The bounds concern the statistic and its weighting scale, not every component of a full neural-network optimizer.

Equations
ws​(t)=e−s2​t2,s>0,w_{s}(t)=e^{-s^{2}t^{2}},\qquad s>0,
D⁡(P,G)=∫ℝws​(t)​|φP​(t)−φG​(t)|2​𝑑t.D(P,G)=\int_{\mathbb{R}}w_{s}(t)\big|\varphi_{P}(t)-\varphi_{G}(t)\big|^{2}dt.
φ^N​(t)=1N​∑i=1Nei​t​Xi,\widehat{\varphi}_{N}(t)=\frac{1}{N}\sum_{i=1}^{N}e^{itX_{i}},
D^V=∫ℝws​(t)​|φ^N​(t)−φG​(t)|2​𝑑t.\widehat{D}_{V}=\int_{\mathbb{R}}w_{s}(t)\big|\widehat{\varphi}_{N}(t)-\varphi_{G}(t)\big|^{2}dt.
∂D^V∂Xi=\displaystyle\frac{\partial\widehat{D}_{V}}{\partial X_{i}}= ∫ℝws​(t)​2​ℜ⁡((φ^N​(t)−φG​(t))​∂φ^N​(t)∂Xi¯)​𝑑t,\displaystyle\int_{\mathbb{R}}w_{s}(t)2\Re\!\Big(\big(\widehat{\varphi}_{N}(t)-\varphi_{G}(t)\big)\overline{\frac{\partial\widehat{\varphi}_{N}(t)}{\partial X_{i}}}\Big)dt,
∂φ^N​(t)∂Xi=\displaystyle\frac{\partial\widehat{\varphi}_{N}(t)}{\partial X_{i}}= 1N​i​t​ei​t​Xi,\displaystyle\frac{1}{N}ite^{itX_{i}},
|∂D^V∂Xi|\displaystyle\left|\frac{\partial\widehat{D}_{V}}{\partial X_{i}}\right| ≤2N​∫ws​(t)​|t|​(|φ^N​(t)|+|φG​(t)|)​𝑑t\displaystyle\leq\frac{2}{N}\int w_{s}(t)|t|\big(|\widehat{\varphi}_{N}(t)|+|\varphi_{G}(t)|\big)dt
≤4N​∫ws​(t)​|t|​𝑑t\displaystyle\leq\frac{4}{N}\int w_{s}(t)|t|dt
=4N​s2,\displaystyle=\frac{4}{Ns^{2}},
|∂D^V∂Xi|≤4N​∫ℝws​(t)​|t|​𝑑t=4N​s2.\Bigg|\frac{\partial\widehat{D}_{V}}{\partial X_{i}}\Bigg|\;\leq\;\frac{4}{N}\int_{\mathbb{R}}w_{s}(t)|t|dt\;=\;\frac{4}{Ns^{2}}.
|∂2D^V∂Xi2|≤CN​∫ℝws​(t)​t2​𝑑t=CN⋅π2​s3,\Bigg|\frac{\partial^{2}\widehat{D}_{V}}{\partial X_{i}^{2}}\Bigg|\;\leq\;\frac{C}{N}\int_{\mathbb{R}}w_{s}(t)t^{2}dt\;=\;\frac{C}{N}\cdot\frac{\sqrt{\pi}}{2s^{3}},
D^k=(ϕ¯−μ)⊤​W​(ϕ¯−μ),ϕ¯:=1N​∑i=1Nϕ⁡(Xi),ϕ⁡(x)=(x,x2,…,xk)⊤,\widehat{D}_{k}\;=\;(\bar{\phi}-\mu)^{\top}W(\bar{\phi}-\mu),\qquad\bar{\phi}:=\frac{1}{N}\sum_{i=1}^{N}\phi(X_{i}),\quad\phi(x)=(x,x^{2},\dots,x^{k})^{\top},
∂D^k∂Xi=\displaystyle\frac{\partial\widehat{D}_{k}}{\partial X_{i}}= 2N​(ϕ¯−μ)⊤​W​∂ϕ⁡(Xi)∂Xi,\displaystyle\frac{2}{N}(\bar{\phi}-\mu)^{\top}W\frac{\partial\phi(X_{i})}{\partial X_{i}},
∂ϕ⁡(X)∂X=\displaystyle\frac{\partial\phi(X)}{\partial X}= (1,2​X,3​X2,…,k​Xk−1)⊤.\displaystyle\big(1,2X,3X^{2},\dots,kX^{k-1}\big)^{\top}.
∂D^k∂Xi=2N​∑r=1kcr​r​Xir−1,\frac{\partial\widehat{D}_{k}}{\partial X_{i}}=\frac{2}{N}\sum_{r=1}^{k}c_{r}rX_{i}^{r-1},
|∂D^k∂Xi|→|Xi|→∞∞as|Xi|k−1.\left|\frac{\partial\widehat{D}_{k}}{\partial X_{i}}\right|\;\xrightarrow[|X_{i}|\to\infty]{}\;\infty\quad\text{as}\quad|X_{i}|^{k-1}.

B.13 Proof of Section 5.1

Expanding the empirical characteristic-function square isolates the minibatch bias. Sampling bias and deterministic quadrature error are separate quantities; rescaling the integration grid does not remove the former automatically.

Equations 28–33
|ϕn​(t)−ψ⁡(t)|2=ϕn​(t)​ϕn​(t)¯−ψ⁡(t)​ϕn​(t)¯−ψ⁡(t)¯​ϕn​(t)+|ψ⁡(t)|2.\big|\phi_{n}(t)-\psi(t)\big|^{2}=\phi_{n}(t)\overline{\phi_{n}(t)}-\psi(t)\overline{\phi_{n}(t)}-\overline{\psi(t)}\phi_{n}(t)+\big|\psi(t)\big|^{2}.
𝔼⁡[|ϕn−ψ|2]=\displaystyle\mathbb{E}\left[\big|\phi_{n}-\psi\big|^{2}\right]= 𝔼⁡[|ϕn|2]−ψ​𝔼​[ϕn¯]−ψ¯​𝔼​[ϕn]+|ψ|2,\displaystyle\mathbb{E}\left[|\phi_{n}|^{2}\right]-\psi\mathbb{E}\left[\overline{\phi_{n}}\right]-\overline{\psi}\mathbb{E}\left[\phi_{n}\right]+|\psi|^{2}, (28)
=\displaystyle= 𝔼⁡[|ϕn|2]−ψ​𝔼⁡[ϕn]¯−ψ¯​1n​∑j=1n𝔼⁡[Zj]+|ψ|2,\displaystyle\mathbb{E}\left[|\phi_{n}|^{2}\right]-\psi\overline{\mathbb{E}[\phi_{n}]}-\overline{\psi}\frac{1}{n}\sum_{j=1}^{n}\mathbb{E}[Z_{j}]+|\psi|^{2}, (29)
=\displaystyle= 𝔼⁡[|ϕn|2]−ψ​ϕθ¯−ψ¯​ϕθ+|ψ|2,\displaystyle\mathbb{E}\left[|\phi_{n}|^{2}\right]-\psi\overline{\phi_{\theta}}-\overline{\psi}\phi_{\theta}+|\psi|^{2}, (30)
=\displaystyle= 𝔼⁡[|ϕn|2]−2​R​e​(ψ¯​ϕθ)+|ψ|2,\displaystyle\mathbb{E}\left[|\phi_{n}|^{2}\right]-2\mathrm{Re}\big(\overline{\psi}\phi_{\theta}\big)+|\psi|^{2}, (31)
=\displaystyle= 𝔼⁡[|1n​∑j=1nZj|2]−2​R​e​(ψ¯​ϕθ)+|ψ|2,\displaystyle\mathbb{E}\left[\left|\frac{1}{n}\sum_{j=1}^{n}Z_{j}\right|^{2}\right]-2\mathrm{Re}\big(\overline{\psi}\phi_{\theta}\big)+|\psi|^{2}, (32)
=\displaystyle= 1n2​∑j=1n∑l=1n𝔼⁡[Zj​Zl¯]−2​R​e​(ψ¯​ϕθ)+|ψ|2,\displaystyle\frac{1}{n^{2}}\sum_{j=1}^{n}\sum_{l=1}^{n}\mathbb{E}\left[Z_{j}\overline{Z_{l}}\right]-2\mathrm{Re}\big(\overline{\psi}\phi_{\theta}\big)+|\psi|^{2}, (33)
𝔼⁡[Zj​Zl¯]={𝔼⁡[|Z1|2]=1,if ​j=l,𝔼⁡[Zj]​𝔼⁡[Zl]¯=ϕθ​ϕθ¯=|ϕθ|2,if ​j≠l,\mathbb{E}\left[Z_{j}\overline{Z_{l}}\right]=\begin{cases}\mathbb{E}\left[|Z_{1}|^{2}\right]=1,&\text{if }j=l,\\[4.0pt] \mathbb{E}[Z_{j}]\overline{\mathbb{E}[Z_{l}]}=\phi_{\theta}\overline{\phi_{\theta}}=|\phi_{\theta}|^{2},&\text{if }j\neq l,\end{cases}
𝔼⁡[|ϕn|2]=\displaystyle\mathbb{E}\left[|\phi_{n}|^{2}\right]= 1n2​(n+n⁡(n−1)​|ϕθ|2)\displaystyle\frac{1}{n^{2}}\Big(n+n(n-1)|\phi_{\theta}|^{2}\Big)
=\displaystyle= 1n+(1−1n)​|ϕθ|2\displaystyle\frac{1}{n}+\left(1-\frac{1}{n}\right)|\phi_{\theta}|^{2}
=\displaystyle= |ϕθ|2+1−|ϕθ|2n\displaystyle|\phi_{\theta}|^{2}+\frac{1-|\phi_{\theta}|^{2}}{n}
𝔼⁡[|ϕn−ψ|2]\displaystyle\mathbb{E}\left[\big|\phi_{n}-\psi\big|^{2}\right] =(|ϕθ|2+1−|ϕθ|2n)−2​R​e​(ψ¯​ϕθ)+|ψ|2\displaystyle=\left(|\phi_{\theta}|^{2}+\frac{1-|\phi_{\theta}|^{2}}{n}\right)-2\mathrm{Re}\big(\overline{\psi}\phi_{\theta}\big)+|\psi|^{2}
=(|ϕθ|2−2​R​e​(ψ¯​ϕθ)+|ψ|2)+1−|ϕθ|2n\displaystyle=\big(|\phi_{\theta}|^{2}-2\mathrm{Re}\big(\overline{\psi}\phi_{\theta}\big)+|\psi|^{2}\big)+\frac{1-|\phi_{\theta}|^{2}}{n}
=|ϕθ−ψ|2+1−|ϕθ|2n.\displaystyle=\big|\phi_{\theta}-\psi\big|^{2}+\frac{1-|\phi_{\theta}|^{2}}{n}.
𝔼⁡[∇θDn​(t)]=∇θ|ϕθ​(t)−ψ⁡(t)|2+∇θ1−|ϕθ​(t)|2n,\mathbb{E}\left[\nabla_{\theta}D_{n}(t)\right]=\nabla_{\theta}\big|\phi_{\theta}(t)-\psi(t)\big|^{2}+\nabla_{\theta}\frac{1-|\phi_{\theta}(t)|^{2}}{n},
L⁡(θ)≈∑kωk​|ϕθ​(tk)−ψ⁡(tk)|2,L^n​(θ)≈∑kωk​|ϕn​(tk)−ψ⁡(tk)|2,L(\theta)\approx\sum_{k}\omega_{k}\big|\phi_{\theta}(t_{k})-\psi(t_{k})\big|^{2},\quad\widehat{L}_{n}(\theta)\approx\sum_{k}\omega_{k}\big|\phi_{n}(t_{k})-\psi(t_{k})\big|^{2},
Bias(θ)=−1n∑kωk∇θ|ϕθ(tk)|2.\text{Bias}(\theta)=-\frac{1}{n}\sum_{k}\omega_{k}\nabla_{\theta}\big|\phi_{\theta}(t_{k})\big|^{2}.

B.14 Proof of VICReg’s Recovery

The argument derives the mean and covariance conditions obtained when projected first and second moments match the target in every direction.

Part I: 𝔼 ⁡ [ 𝐗 ] = 𝟎

Zero mean in every projected direction implies the full embedding mean vector is zero.

Equations 35–39
𝔼⁡[𝐚T​𝐗]=0for all ​𝐚∈ℝd​ with ​‖𝐚‖=1\mathbb{E}[\mathbf{a}^{T}\mathbf{X}]=0\quad\text{for all }\mathbf{a}\in\mathbb{R}^{d}\text{ with }\|\mathbf{a}\|=1 (35)
𝐚T​𝔼​[𝐗]=0for all unit vectors ​𝐚\mathbf{a}^{T}\mathbb{E}[\mathbf{X}]=0\quad\text{for all unit vectors }\mathbf{a} (36)
𝐚∗=𝝁‖𝝁‖2\mathbf{a}^{*}=\frac{\bm{\mu}}{\|\bm{\mu}\|_{2}} (37)
(𝐚∗)T​𝝁=0(\mathbf{a}^{*})^{T}\bm{\mu}=0 (38)
(𝐚∗)T​𝝁=(𝝁‖𝝁‖2)T​𝝁=𝝁T​𝝁‖𝝁‖2=‖𝝁‖22‖𝝁‖2=‖𝝁‖2>0(\mathbf{a}^{*})^{T}\bm{\mu}=\left(\frac{\bm{\mu}}{\|\bm{\mu}\|_{2}}\right)^{T}\bm{\mu}=\frac{\bm{\mu}^{T}\bm{\mu}}{\|\bm{\mu}\|_{2}}=\frac{\|\bm{\mu}\|_{2}^{2}}{\|\bm{\mu}\|_{2}}=\|\bm{\mu}\|_{2}>0 (39)

Part II: Cov ⁡ ( 𝐗 ) = 𝐈 d

Unit variance along coordinate axes fixes diagonal covariance entries; combined-axis directions then force off-diagonal entries to zero. This recovers identity covariance, which alone does not identify a Gaussian density.

Equations 40–50
Var⁡(⟨𝐗,𝐚⟩)=𝔼⁡[(⟨𝐗,𝐚⟩)2]=𝔼⁡[(𝐚T​𝐗)2]\mathrm{Var}(\langle\mathbf{X},\mathbf{a}\rangle)=\mathbb{E}[(\langle\mathbf{X},\mathbf{a}\rangle)^{2}]=\mathbb{E}[(\mathbf{a}^{T}\mathbf{X})^{2}] (40)
𝔼⁡[(𝐚T​𝐗)2]=𝔼⁡[𝐚T​𝐗𝐗T​𝐚]=𝐚T​𝔼​[𝐗𝐗T]​𝐚\mathbb{E}[(\mathbf{a}^{T}\mathbf{X})^{2}]=\mathbb{E}[\mathbf{a}^{T}\mathbf{X}\mathbf{X}^{T}\mathbf{a}]=\mathbf{a}^{T}\mathbb{E}[\mathbf{X}\mathbf{X}^{T}]\mathbf{a} (41)
𝐚T​𝚺​𝐚=1for all unit vectors ​𝐚\mathbf{a}^{T}\bm{\Sigma}\mathbf{a}=1\quad\text{for all unit vectors }\mathbf{a} (42)
𝐞iT​𝚺​𝐞i=Σi​i=1\mathbf{e}_{i}^{T}\bm{\Sigma}\mathbf{e}_{i}=\Sigma_{ii}=1 (43)
𝐚=𝐞i+𝐞j‖𝐞i+𝐞j‖2=𝐞i+𝐞j2\mathbf{a}=\frac{\mathbf{e}_{i}+\mathbf{e}_{j}}{\|\mathbf{e}_{i}+\mathbf{e}_{j}\|_{2}}=\frac{\mathbf{e}_{i}+\mathbf{e}_{j}}{\sqrt{2}} (44)
𝐚T​𝚺​𝐚=12​(𝐞i+𝐞j)T​𝚺​(𝐞i+𝐞j)=1\mathbf{a}^{T}\bm{\Sigma}\mathbf{a}=\frac{1}{2}(\mathbf{e}_{i}+\mathbf{e}_{j})^{T}\bm{\Sigma}(\mathbf{e}_{i}+\mathbf{e}_{j})=1 (45)
12​(𝐞iT​𝚺​𝐞i+2​𝐞iT​𝚺​𝐞j+𝐞jT​𝚺​𝐞j)\displaystyle\frac{1}{2}(\mathbf{e}_{i}^{T}\bm{\Sigma}\mathbf{e}_{i}+2\mathbf{e}_{i}^{T}\bm{\Sigma}\mathbf{e}_{j}+\mathbf{e}_{j}^{T}\bm{\Sigma}\mathbf{e}_{j}) =1\displaystyle=1 (46)
12​(Σi​i+2​Σi​j+Σj​j)\displaystyle\frac{1}{2}(\Sigma_{ii}+2\Sigma_{ij}+\Sigma_{jj}) =1\displaystyle=1 (47)
12​(1+2​Σi​j+1)\displaystyle\frac{1}{2}(1+2\Sigma_{ij}+1) =1\displaystyle=1 (48)
1+Σi​j\displaystyle 1+\Sigma_{ij} =1\displaystyle=1 (49)
Σi​j\displaystyle\Sigma_{ij} =0\displaystyle=0 (50)

Appendix C Background

The background appendix derives ordinary least squares from a linear model and its normal equations, making explicit the invertibility and noise assumptions used in the probe discussion.

Equations 51
𝐲=𝐗​𝜷+𝜺\mathbf{y}=\mathbf{X}\bm{\beta}+\bm{\varepsilon}
𝔼⁡[εi]=0,Var​(εi)=σ2,Cov​(εi,εj)=0​ for ​i≠j\mathbb{E}[\varepsilon_{i}]=0,\quad\text{Var}(\varepsilon_{i})=\sigma^{2},\quad\text{Cov}(\varepsilon_{i},\varepsilon_{j})=0\text{ for }i\neq j
SSR​(𝜷)=∑i=1n(yi−𝐱iT​𝜷)2=(𝐲−𝐗​𝜷)T​(𝐲−𝐗​𝜷)\text{SSR}(\bm{\beta})=\sum_{i=1}^{n}(y_{i}-\mathbf{x}_{i}^{T}\bm{\beta})^{2}=(\mathbf{y}-\mathbf{X}\bm{\beta})^{T}(\mathbf{y}-\mathbf{X}\bm{\beta})
SSR​(𝜷)\displaystyle\text{SSR}(\bm{\beta}) =𝐲T​𝐲−2​𝜷T​𝐗T​𝐲+𝜷T​𝐗T​𝐗​𝜷\displaystyle=\mathbf{y}^{T}\mathbf{y}-2\bm{\beta}^{T}\mathbf{X}^{T}\mathbf{y}+\bm{\beta}^{T}\mathbf{X}^{T}\mathbf{X}\bm{\beta} (51)
∂SSR∂𝜷=−2​𝐗T​𝐲+2​𝐗T​𝐗​𝜷\frac{\partial\text{SSR}}{\partial\bm{\beta}}=-2\mathbf{X}^{T}\mathbf{y}+2\mathbf{X}^{T}\mathbf{X}\bm{\beta}
−2​𝐗T​𝐲+2​𝐗T​𝐗​𝜷=𝟎-2\mathbf{X}^{T}\mathbf{y}+2\mathbf{X}^{T}\mathbf{X}\bm{\beta}=\mathbf{0}
𝐗T​𝐗​𝜷=𝐗T​𝐲\mathbf{X}^{T}\mathbf{X}\bm{\beta}=\mathbf{X}^{T}\mathbf{y}
𝜷^=(𝐗T​𝐗)−1​𝐗T​𝐲\boxed{\hat{\bm{\beta}}=(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}}
Original figure
Figure 15: Depiction of the expected BCS loss upper bound (Section 4.3) for various smoothness values α\alpha. We clearly see that as the smoothness increases (blue to red), as the upper bound decreases more and more rapidly with MM.

dimension=128, slices=10
Original figure dimension=128, slices=100
Original figure dimension=1024, slices=100
Original figure

Figure 16: Reprise of Figure 6 for additional dimensions and number of 1d projections.
Original figure
Original figure
Original figure
Figure 17: Depiction of the distribution of optimized β\beta values from OLS when comparing 𝒁iso{\bm{Z}}_{\rm iso} and 𝒁aniso{\bm{Z}}_{\rm aniso} from Sections 3.1 and 3.1. We clearly observe that the anisotropic version (blue) provides much lower variance compared to the isotropic case (red). We consider a binary classification (linear separable class) (top row), a linear regression task (middle row), and a nonlinear regression task with smooth targets (bottom row). For each case, we resample the training samples numerous times and produce an estimate for β\beta each time. Because the data is 22-dimensional, we can visualize the β\beta distribution directly.
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 18: Depiction of accuracy (top) and cosine similarity between estimated and true estimator (bottom) for the OLS setting with varying strength of Tikhonov regularization (x-axis) comparing isotropic and anisotropic embeddings. As per Section 5.1, the anisotropic distribution creates a bias in the OLS estimation for nonzero regularization.
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 19: Additional figures provides in Figure 19
Table 3: Performance metrics across different sample sizes from Figure 12
Freeze Backbone Model Name Samples per Class
All 1 2 5 10 100 1000
No LeJEPA (Ours)
ConvNeXt-V2 Nano 82.72 29.42 36.65 50.94 59.85 75.34 81.97
LeViT-128 79.41 18.45 24.08 33.11 41.76 64.59 77.59
ResNet-18 82.15 23.34 31.56 43.82 54.64 73.53 81.41
ResNet-34 83.28 24.27 31.51 44.23 53.95 74.93 82.32
Baselines
DINOv2 Small 78.34 21.05 21.71 30.33 36.23 60.81 75.55
DINOv3 ViT-S/16 81.60 24.71 29.43 37.71 44.71 69.87 80.54
Yes LeJEPA (Ours)
ConvNeXt-V2 Nano 76.52 28.74 36.65 50.60 59.50 72.62 77.24
LeViT-128 69.00 25.85 33.30 45.52 52.43 64.37 69.39
ResNet-18 75.95 30.48 38.22 50.85 58.86 72.70 76.39
ResNet-34 78.17 31.08 38.33 52.26 60.63 74.77 78.62
Baselines
DINOv2 Small 67.62 27.68 32.22 40.72 47.72 62.49 67.89
DINOv3 ViT-S/16 71.38 30.17 36.65 45.74 51.51 65.90 71.35
Original figure
Figure 20: Proposed trapezoid quadrature for the Epps-Pulley statistic as implemented in . We depict the approximation error of the integral for various distributions, demonstrate rapid convergence (faster than quadratic show in grey line) across possible embedding distributions.
Refer to caption
Refer to caption
Refer to caption
Figure 21: Additional figures for Figure 10.
Table 4: Top 1 accuracy (in %) with LeJEPA pretraining on Imagenet-100 for 400 epochs (All values are percentages)
backbone resnet50 vit_small_patch8_224 vit_tiny_patch8_224
Projector 1-layer 2-layer 3-layer 1-layer 2-layer 3-layer 1-layer 2-layer 3-layer
w/ predictor w/ SWA
False False 79.71 82.44 83.93 76.59 80.77 81.07 71.79 76.87 80.37
True 79.79 82.69 83.50 79.96 83.63 84.12 75.86 82.36 80.50
True False 79.41 82.44 83.57 77.58 79.41 81.91 67.74 77.64 80.73
True 78.87 82.04 82.82 77.11 81.77 82.58 69.53 78.27 79.77
Table 5: Small architecture in-domain LeJEPA pretraining from random initialization across datasets and architectures, with frozen backbone linear evaluation. First, LeJEPA is able to produce near state-of-the-art performances on tiny dataset with only a thousand samples, e.g., flowers102. Second, on non-natural image data, LeJEPA clearly outperforms the latest frontier vision models, e.g., Galaxy10. See Figure 12 for additional experiments with varying number of training samples and with full finetuning.
Pretraining flowers102 cifar100 food101 inet10 cifar10 galaxy10
# train. samples 1020 50000 75750 13000 50000 11008
LeJEPA (convnextv2_nano) 14M in-domain 64.34 69.26 69.59 90.81 92.22 76.05
LeJEPA (resnet18) 11M in-domain 74.57 69.94 73.57 92.36 92.51 75.32
LeJEPA (resnet34) 21M in-domain 71.85 70.44 74.95 92.80 93.16 77.29
LeJEPA (resnext26ts) 8M in-domain 82.19 69.10 76.77 92.82 91.59 73.78
LeJEPA (swin_tiny) 27M in-domain 63.94 65.08 78.40 92.87 92.67 74.89
IJEPA-inet22k (ViT-H/14) 630M inet1k 85.76 86.93 81.06 98.65 97.77 62.93
Table 6: Time (in millisecond) to compute the proposed SIGReg loss from on a Tesla V100-SXM2-16GB for varying mini-batch size (NN), number of slices (MM), integration points. Results are computed over 1010 runs.
N M
# integration
points
mean (ms) std (ms)
512 512 16 0.465236 0.011642
512 512 64 0.461317 0.003894
512 512 256 0.627644 0.003337
2048 512 16 1.406441 0.002415
8192 512 16 6.188304 0.007226
8192 8192 16 8.685009 0.038829
32768 512 16 26.373118 0.012732
512 2048 16 0.465614 0.005274
512 8192 16 0.670379 0.006854
Table 7: Number of Figure 8.
resnet50
λ\lambda 0.001 0.005 0.010 0.020 0.025 0.050 0.100 0.150 0.200 0.300 0.400 0.500
#views
2 81.41 82.73 83.49 82.99 82.23 - - - - - - -
4 79.88 83.04 84.36 84.68 84.33 83.00 82.91 81.05 78.58 - - -
8 76.67 81.58 83.59 83.49 83.76 84.32 83.66 83.07 82.16 81.00 79.25 77.72

Appendix D Details on Low-Discrepancy Sequences

A Sobol sequence is transformed into approximate sphere directions through Gaussian quantiles and normalization. This aims for more even directional coverage; the nonlinear mapping need not preserve the original discrepancy guarantee exactly.

Appendix E Shapiro-Wilk Test

This appendix reviews Shapiro–Wilk-style normality statistics and approximations. It supplies context for the paper’s decision not to use an order-statistic-based training penalty.

Equations 52–58
E⁡(Xi)=mi and cov⁡(Xi,Xj)=vi​j,i,j=1,2,…,nE\left(X_{i}\right)=m_{i}\quad\text{ and }\quad\operatorname{cov}\left(X_{i},X_{j}\right)=v_{ij},\quad i,j=1,2,\ldots,n (52)
W=(∑i=1nai​Yi)∑i=1n(Yi−Y¯)2=(𝐚𝐘)S2𝐚′=(a1,a2,…,an)=𝐦𝐕−1(𝐦𝐕−1𝐕−1𝐦)−1/2S2=∑i=1n(Yi−Y¯)2\begin{array}[]{l}W=\frac{\left(\sum_{i=1}^{n}a_{i}Y_{i}\right)}{\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}}=\frac{(\mathbf{a}\mathbf{Y})}{S^{2}}\\ \mathbf{a}^{\prime}=\left(a_{1},a_{2},\ldots,a_{n}\right)=\mathbf{m}\mathbf{V}^{-1}\left(\mathbf{m}\mathbf{V}^{-1}\mathbf{V}^{-1}\mathbf{m}\right)^{-1/2}\\ \mathrm{S}^{2}=\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}\end{array} (53)
E⁡(Xi)=mi≈Φ−1​(i−38n+14)i=1,2,…,nE\left(X_{i}\right)=m_{i}\approx\Phi^{-1}\left(\frac{i-\frac{3}{8}}{n+\frac{1}{4}}\right)\quad i=1,2,\ldots,n (54)
E(Xi)=mi≈Φ−1(iN+1),i=1,2,…,nE\left(X_{i}\right)=m_{i}\approx\Phi^{-1}\left(\frac{i}{N+1}\right),\quad i=1,2,\ldots,n (55)
cov⁡(Xi,Xj)=vi​j≈pi​pj(n+2)​f​(mi)​f​(mj),i,j=1,2,…,n\operatorname{cov}\left(X_{i},X_{j}\right)=v_{ij}\approx\frac{p_{i}p_{j}}{(n+2)f\left(m_{i}\right)f\left(m_{j}\right)},\quad i,j=1,2,\ldots,n (56)
pi=in+1p_{i}=\frac{i}{n+1} (57)
𝐕−1=(n+1)​(n+2)×(2​ϕ2​(m1)−ϕ⁡(m1)​ϕ​(m2)00…0−ϕ⁡(m1)​ϕ​(m2)2​ϕ2​(m2)−ϕ⁡(m2)​ϕ​(m3)0…00−ϕ⁡(m2)​ϕ​(m3)2​ϕ2​(m3)−ϕ⁡(m3)​ϕ​(m4)…0⋮0000…2​ϕ2​(mn))\begin{array}[]{l}\mathbf{V}^{-1}=(n+1)(n+2)\\ \times\left(\begin{array}[]{cccccc}2\phi^{2}\left(m_{1}\right)&-\phi\left(m_{1}\right)\phi\left(m_{2}\right)&0&0&\ldots&0\\ -\phi\left(m_{1}\right)\phi\left(m_{2}\right)&2\phi^{2}\left(m_{2}\right)&-\phi\left(m_{2}\right)\phi\left(m_{3}\right)&0&\ldots&0\\ 0&-\phi\left(m_{2}\right)\phi\left(m_{3}\right)&2\phi^{2}\left(m_{3}\right)&-\phi\left(m_{3}\right)\phi\left(m_{4}\right)&\ldots&0\\ \vdots&&&&&\\ 0&0&0&0&\ldots&2\phi^{2}\left(m_{n}\right)\end{array}\right)\end{array} (58)

Appendix F Multivariate Statistics

The final appendix lists multivariate characteristic-function, moment-generating-function and moment tests. Their formulas clarify the alternatives and their computational costs; they are not all implemented as the default SIGReg objective.

Equations 59–68
ψ^N​(𝒕)=1N​∑n=1Ne−i​𝒕⊤​𝒚n.\displaystyle\hat{\psi}_{N}({\bm{t}})=\frac{1}{N}\sum_{n=1}^{N}e^{-i{\bm{t}}^{\top}{\bm{y}}_{n}}.
N∫|ψ^N(𝒕)−ψ0(𝒕)|2ω(𝒕)dt=N∫|ψ^N(𝒕)−e−∥𝒕∥2/2|2ω(𝒕)dt,\displaystyle N\int|\hat{\psi}_{N}({\bm{t}})-\psi_{0}({\bm{t}})|^{2}\omega({\bm{t}})dt=N\int|\hat{\psi}_{N}({\bm{t}})-e^{-\|{\bm{t}}\|_{2}/2}|^{2}\omega({\bm{t}})dt,
BHEPn,β=\displaystyle\mathrm{BHEP}_{n,\beta}= 1n​∑j,k=1nexp⁡(−β2​‖Yn,j−Yn,k‖22)\displaystyle\frac{1}{n}\sum_{j,k=1}^{n}\exp\left(-\frac{\beta^{2}\left\|Y_{n,j}-Y_{n,k}\right\|^{2}}{2}\right)
−2(1+β2)d/2∑j=1nexp(−β2​‖Yn,j‖22​(1+β2))+n(1+2​β2)d/2.\displaystyle-\frac{2}{\left(1+\beta^{2}\right)^{d/2}}\sum_{j=1}^{n}\exp\left(-\frac{\beta^{2}\left\|Y_{n,j}\right\|^{2}}{2\left(1+\beta^{2}\right)}\right)+\frac{n}{\left(1+2\beta^{2}\right)^{d/2}}.
βn=2−1/2((2d+1)n/4)1/(d+4)\beta_{n}=2^{-1/2}((2d+1)n/4)^{1/(d+4)} (59)
Tn,β=πd/2​(1n​∑i,j=1n1βd/2​exp⁡(‖Yn,i+Yn,j‖24​β)+n(β−1)d/2CLOSE\displaystyle T_{n,\beta}=\pi^{d/2}\left(\frac{1}{n}\sum_{i,j=1}^{n}\frac{1}{\beta^{d/2}}\exp\left(\frac{\left\|Y_{n,i}+Y_{n,j}\right\|^{2}}{4\beta}\right)+\frac{n}{(\beta-1)^{d/2}}\right.
−2∑j=1n1(β−1/2)d/2exp(‖Yn,j‖24​β−2)),\displaystyle\left.-2\sum_{j=1}^{n}\frac{1}{(\beta-1/2)^{d/2}}\exp\left(\frac{\left\|Y_{n,j}\right\|^{2}}{4\beta-2}\right)\right),
Tn,γ:=∫ℝdUn2​(t)​wγ​(t)​𝑑tUn​(t):=n​(Rn​(t)​Mn​(t)−1)\begin{array}[]{l}T_{n,\gamma}:=\int_{\mathbb{R}^{d}}U_{n}^{2}(t)w_{\gamma}(t)\mathrm{d}t\\ U_{n}(t):=\sqrt{n}\left(R_{n}(t)M_{n}(t)-1\right)\end{array} (60)
Tn,γ=\displaystyle T_{n,\gamma}= (πγ)d/2{12​n3∑j,k,l,m=1n[exp(‖Yj​k+‖2−‖Yℓ​m−‖24​γ)cos(Yj​k+⁣⊤​Yℓ​m−2​γ)\displaystyle\left(\frac{\pi}{\gamma}\right)^{d/2}\left\{\frac{1}{2n^{3}}\sum_{j,k,l,m=1}^{n}\left[\exp\left(\frac{\left\|Y_{jk}^{+}\right\|^{2}-\left\|Y_{\ell m}^{-}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{jk}^{+\top}Y_{\ell m}^{-}}{2\gamma}\right)\right.\right. (61)
+\displaystyle+ exp(‖Yj​k+‖2−‖Yℓ​m+‖24​γ)cos(Yj​k+⁣⊤​Yℓ​m+2​γ)]\displaystyle\left.\exp\left(\frac{\left\|Y_{jk}^{+}\right\|^{2}-\left\|Y_{\ell m}^{+}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{jk}^{+\top}Y_{\ell m}^{+}}{2\gamma}\right)\right]
−2n∑j,k=1nexp(‖Yn,j‖2−‖Yn,k‖24​γ)cos(Yn,j⊤​Yn,k2​γ)+n},\displaystyle\left.-\frac{2}{n}\sum_{j,k=1}^{n}\exp\left(\frac{\left\|Y_{n,j}\right\|^{2}-\left\|Y_{n,k}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{n,j}^{\top}Y_{n,k}}{2\gamma}\right)+n\right\},
T~n,γ:=∫ℝdUn​(t)​wγ​(t)​𝑑t.\widetilde{T}_{n,\gamma}:=\int_{\mathbb{R}^{d}}U_{n}(t)w_{\gamma}(t)\mathrm{d}t. (62)
T~n,γ=(πγ)d/2​n​(1n2​∑j,k=1nexp⁡(‖Yn,j‖2−‖Yn,k‖24​γ)​cos⁡(Yn,j⊤​Yn,k2​γ)−1)\widetilde{T}_{n,\gamma}=\left(\frac{\pi}{\gamma}\right)^{d/2}\sqrt{n}\left(\frac{1}{n^{2}}\sum_{j,k=1}^{n}\exp\left(\frac{\left\|Y_{n,j}\right\|^{2}-\left\|Y_{n,k}\right\|^{2}}{4\gamma}\right)\cos\left(\frac{Y_{n,j}^{\top}Y_{n,k}}{2\gamma}\right)-1\right) (63)
HVn,γ:=n​∫‖∇Mn​(t)−t​Mn​(t)‖2​w~γ​(t)​𝑑t\mathrm{HV}_{n,\gamma}:=n\int\left\|\nabla M_{n}(t)-tM_{n}(t)\right\|^{2}\widetilde{w}_{\gamma}(t)\mathrm{d}t (64)
HVn,γ=1n​(πγ)d/2​∑j,k=1nexp⁡(‖Yn,j,k+‖24​γ)​(Yn,j⊤​Yn,k−‖Yn,j,k+‖22​γ+d2​γ+‖Yn,j,k+‖24​γ2).\mathrm{HV}_{n,\gamma}=\frac{1}{n}\left(\frac{\pi}{\gamma}\right)^{d/2}\sum_{j,k=1}^{n}\exp\left(\frac{\left\|Y_{n,j,k}^{+}\right\|^{2}}{4\gamma}\right)\left(Y_{n,j}^{\top}Y_{n,k}-\frac{\left\|Y_{n,j,k}^{+}\right\|^{2}}{2\gamma}+\frac{d}{2\gamma}+\frac{\left\|Y_{n,j,k}^{+}\right\|^{2}}{4\gamma^{2}}\right). (65)
b1,d=1n2​∑j,k=1n(Yn,j⊤​Yn,k)3b_{1,d}=\frac{1}{n^{2}}\sum_{j,k=1}^{n}\left(Y_{n,j}^{\top}Y_{n,k}\right)^{3} (66)
b~1,d=1n2​∑j,k=1nYn,j⊤​Yn,k​‖Yn,j‖2​‖Yn,k‖2\widetilde{b}_{1,d}=\frac{1}{n^{2}}\sum_{j,k=1}^{n}Y_{n,j}^{\top}Y_{n,k}\left\|Y_{n,j}\right\|^{2}\left\|Y_{n,k}\right\|^{2} (67)
b2,d=1n​∑j=1n‖Yn,j‖4b_{2,d}=\frac{1}{n}\sum_{j=1}^{n}\left\|Y_{n,j}\right\|^{4} (68)
Concept demonstration

Open image ↗