The opening diagram compares dense Gaussian regularization with a sparse, rectified target. Applying ReLU creates exact zeros. After that change, a random linear projection no longer has the same simple Gaussian target law, which motivates matching projected samples from both the learned and target distributions.
1 Introduction
The paper changes JEPA’s anti-collapse target rather than only adding a penalty to a pretrained dense representation. Its aim is to retain task information while controlling the fraction of zero coordinates. Distribution matching connects the desired sparse statistics to a trainable objective; empirical accuracy and sparsity are measured separately.
Figure 2: Rectified Laplace () and Rectified Gaussian () as special cases of Rectified Generalized Gaussian distributions. Assume and . For any , the Truncated Generalized Gaussian over the product support is the maximum differential entropy distribution under a fixed expected -norm constraint. For , further admits a radial–angular decomposition with , where is uniformly distributed with respect to the surface measure on the unit -sphere confined to the positive orthant and follows a Gamma distribution. Rectified Laplace and Rectified Gaussian arise via coordinate-wise mixing of the corresponding truncated distributions with a Dirac measure at zero, yielding a distribution with expected -norm guarantees, where and denote the cumulative distribution functions of the standard Laplace and standard Gaussian distributions respectively.
2 Background
Two sparsity notions are distinguished: exact zeros, counted by the zero-norm notation, and concentration of magnitude, estimated from one- and two-norm ratios. ReLU maps negative activations to zero, creating a point mass at zero. Nonnegative or sparse features can be useful, but the measured sparsity does not itself establish semantic interpretability.
3 Sparse and Maximum-Entropy Distributions
The target family combines continuous positive values with a controllable probability of exact zero. Maximum-entropy claims are made under specified support and expected-moment constraints; changing those constraints changes the optimum.
3.1 Generalized Gaussian Distributions
The generalized Gaussian family varies shape through a positive parameter p. It includes a Laplace distribution at p=1 and an ordinary Gaussian at p=2, with location and scale controlling the distribution’s placement and spread.
Definition 3.1(Generalized Gaussian Distribution ).
The Generalized Gaussian distribution over the support has the probability density function
Equation 1
(1)
where is the gamma function.
3.2 Truncated Generalized Gaussian Distributions
Truncation keeps only values in an allowed region and renormalizes their probabilities. A positive truncated distribution has no negative values, but unlike rectification it does not transfer all negative probability to an exact zero.
Let be a subset of with positive Lebesgue measure. The Truncated Generalized Gaussian distribution is the restriction of the Generalized Gaussian distribution to the support . The probability density function of is given by
Equation 2
(2)
where is the indicator function that evaluates to if and otherwise. The partition function is
Equation 3
(3)
3.3 Maximum Entropy under ℓ p Constraints
Under a fixed expected p-th-power magnitude and specified support, the derivation identifies the corresponding maximum-differential-entropy distribution. This is a constrained statistical statement, not a claim that maximum entropy guarantees the best representation for arbitrary labels.
Proposition 3.3(Maximum Entropy Characterizations of Multivariate Truncated Generalized Gaussian Distributions).
The maximum entropy distribution over , where is a subset of with positive Lebesgue measure, under the constraints
Equation 4
(4)
is the Multivariate Truncated Generalized Gaussian distribution with probability density function
Equation 5
(5)
where and is the partition function.
3.4 Rectified Generalized Gaussian Distributions
Rectification samples the underlying generalized Gaussian, keeps positive values and maps nonpositive values to zero. The result mixes a discrete spike at zero with a continuous positive distribution. That mixed nature distinguishes it from a merely truncated density.
Definition 3.4(Rectified Generalized Gaussian).
The Rectified Generalized Gaussian distribution is a mixture between a discrete Dirac measure (Definition B.4) and a Truncated Generalized Gaussian distribution with probability density function
Equations 6–7
(6)
(7)
where is the cumulative distribution function for the standard Generalized Gaussian distribution .
(a)Necessity of Rectification.
(b)Controllable Sparsity.
(c)Sparsity and Performance Tradeoff.
Figure 3: Rectified LpJEPA achieves controllable sparsity and favorable sparsity-performance tradeoffs under proper parameterizations. (a) We report CIFAR-100 validation accuracy and the sparsity metric for four settings where we match non-rectified features or rectified features to either Rectified Generalized Gaussian or conventional Generalized Gaussian . Rectified LpJEPA achieves the best sparsity-performance tradeoffs compared to other settings. (b) We compare the normalized norm of pretrained Rectified LpJEPA features against the theoretical predictions of Proposition 3.5 as varies. Empirical sparsity closely follows the predicted behavior across different values of and . (c) We plot the Pareto frontier of sparsity versus accuracy across varying values of and . Performance drops sharply only when more than of entries are zero.
3.5 Sparsity and Entropy
For a symmetric zero-centered target, roughly half the coordinates are zero after rectification. Shifting the underlying distribution toward negative values increases that fraction. Because the target has both discrete and continuous parts, ordinary continuous differential entropy is not the right single measure; the paper uses Rényi information dimension and a related entropy characterization.
Proposition 3.5(Sparsity).
Let in dimension. Then
Equations 8–9
(8)
(9)
where is the sign function and is the lower regularized gamma function.
Theorem 3.6(Rényi Information Dimension Characterizations of Multivariate Rectified Generalized Gaussian Distributions).
Let be a Rectified Generalized Gaussian random vector. The Rényi information dimension of is , and the -dimensional entropy of is given by
Equations 10–12
(10)
(11)
(12)
where is the discrete Shannon entropy, denotes the differential entropy, and is a Bernoulli random variable that equals with probability and with probability .
Rectified LpJEPA aligns related rectified embeddings while matching their distribution to the selected sparse target. RDMReg implements the matching; mean, scale and shape control the target’s sparsity and spread.
4.1 Cramér–Wold Based Distribution Matching
At the population level, two distributions agree exactly when all one-dimensional linear projections agree. In practice, a finite collection of projected sample comparisons approximates this criterion; it is not a proof of full joint-distribution equality.
The encoder/projector outputs pass through ReLU. Training balances agreement between paired views with a distance between learned features and samples drawn from the target distribution. This directly teaches the rectified representation, rather than hoping that sparsifying a separately trained dense representation preserves its information.
Equations 14–15
(14)
(15)
4.3 The Necessity of Two-Sample Hypothesis Testing
Unlike a Gaussian, a rectified target is not closed under arbitrary linear projections. The implementation therefore projects both learned and target samples and compares the two empirical sets. Its sliced Wasserstein objective sorts each projected sample set and matches corresponding values. The reported finite projection count works empirically under the tested settings.
The connection to nonnegative variance–covariance regularization is conditional: matching projected marginals along covariance eigenvectors yields an isotropic centered covariance. This does not mean arbitrary finitely sampled directions exactly recover every covariance condition. Eigenvector-based projections are tested as an acceleration.
4.5 Hyperparameters of the Target Distributions
Location, scale and shape jointly control sparsity. Scale can normalize either the pre-rectification or post-rectification variance; these are different choices. A closed-form moment expression and bisection procedure support the latter. The default recipe normalizes the unrectified variance.
Table 1: Linear Probe Results on ImageNet-100.
Acc1 (%) is higher-is-better (); sparsity is lower-is-better ().
Bold denotes best and underline denotes second-best in each column (ties allowed).
The experiments measure both task performance and representation statistics. A method with more zeros is not automatically better: useful sparsity must be evaluated together with accuracy, dependence and transfer.
5.1 Experimental Settings
Baselines include dense contrastive and variance-regularized methods and their nonnegative counterparts. Linear probes read either encoder features or rectified projector features, which are different representations. Both exact-zero and magnitude-based sparsity are reported to avoid confusing many small values with actual zeros.
Figure 4: Rectified LpJEPA empirically achieves higher-entropy, more independent features with dataset-adaptive sparsity. (a) The averaged univariate -dimensional entropy of the Rectified LpJEPA features are computed against the sparsity metric across varying and . Overall, we observe the expected behavior of sparsity-entropy tradeoff (b) We evaluate the normalized Hilbert-Schmidt independence Criterion (nHSIC) for LpJEPA, Rectified LpJEPA, and other baselines. Rectified LpJEPA achieves smaller nHSIC values compared to VICReg or NVICReg that only penalizes second-order statistics. (c) The relative mean absolute deviations (MAD) away from the median of the and sparsity metrics are computed over different methods. Rectified LpJEPA exhibits the highest variations of sparsity for different downstream dataset. Additional visualizations can be found in Figure12.
5.2 Necessity of Rectifications
Rectifying both features and the target gives the best tested sparsity–accuracy tradeoff. Asking nonnegative features to follow an unrectified isotropic Gaussian creates an incompatible target and harms performance. Non-rectified variants can classify well while remaining dense.
5.3 Anti-Collapse via Continuous Mapping Theorems
A continuous-mapping argument explains how ReLU transforms a distribution. It does not imply that applying ReLU after dense pretraining preserves useful labels. The post-hoc probe ablation tests that difference and favors training directly for the rectified target.
5.4 Controllable Sparsity
Observed nonzero fractions track the target-family prediction across tested means, shapes and architectures. This supports controllability of the statistic in these experiments, while label information still needs its own evaluation.
5.5 Sparsity and Performance Tradeoff
Accuracy declines gradually over part of the tested sparsity range, then drops sharply at very high sparsity, around95% zeros in the illustrated setting. That threshold belongs to this task and recipe; it is not a universal safe compression ratio.
The paper uses pairwise Hilbert–Schmidt independence measurements as a tractable dependence diagnostic. Lower values indicate less measured pairwise dependence, beyond covariance alone. They do not by themselves prove that every coordinate is jointly independent or that a semantic factorization has been recovered.
5.7 Rényi Information Dimension and Entropy
The entropy analysis accounts for a mixture of zero and positive values. Summed marginal entropies upper-bound joint entropy, with dependencies determining the gap. Sparsity and entropy occupy a tradeoff curve; pairwise dependence diagnostics are evidence about that gap, not an exact high-dimensional joint-entropy estimate.
5.8 Pretraining and Transfer Evaluations
ImageNet pretraining and few/full-shot transfer compare task accuracy with sparsity across datasets. Out-of-distribution inputs can have different activation-sparsity patterns, suggesting a diagnostic. These examples do not establish a calibrated or universally reliable out-of-distribution detector.
6 Conclusion
The contribution is a target-distribution approach to learning sparse JEPA representations. Its practical value depends on retaining task performance while controlling zeros; a desired marginal distribution alone cannot determine which information a model should keep.
Acknowledgements
The authors acknowledge research funding, discussions and NYU computing support.
Impact Statement
The impact statement describes general representation-learning research and identifies no paper-specific societal consequence. It is not an application-level deployment evaluation.
Appendix
Appendix A Additional Backgrounds
The background compares contrastive, variance-based and distribution-matching objectives, plus post-hoc sparse autoencoders. This paper learns sparsity during pretraining instead of treating all sparsification procedures as equivalent.
Appendix B Properties of Univariate Generalized Gaussian, Truncated Generalized Gaussian, and Rectified Generalized Gaussian Distributions
B.1 Univariate Case - Generalized Gaussian
The appendix fixes the generalized Gaussian parameterization, moments and special cases. Its scale differs from some conventions, so formulas should not be copied without matching the parameterization.
Equations B.1–B.3
(B.1)
(B.2)
(B.3)
Remark B.1.
The probability density function of the Generalized Gaussian distribution can also be written as
Equation B.4
(B.4)
where . We choose the particular presentation in Definition 3.1 for its connection to the family of -norm spherical distributions (Gupta and Song, 1997).
Remark B.2.
When , the Generalized Gaussian distribution reduces to the Laplace distribution with probability density function
Equation B.5
(B.5)
If , we recover the Gaussian distribution
Equation B.6
(B.6)
B.2 Univariate Case - Truncated Generalized Gaussian
A truncated law is the conditional distribution restricted to positive values. The normalization depends on how much probability the original law assigned to that region.
Definition B.3(Truncated Generalized Gaussian Probability Measure).
Let be a Generalized Gaussian random variable with parameter , location , and scale .
The Truncated Generalized Gaussian probability measure
on the measurable space
is defined as the conditional distribution of given , i.e.,
Equation
for any , where
denotes the cumulative distribution function
of the standardized Generalized Gaussian distribution.
B.3 Univariate Case - Rectified Generalized Gaussian
The rectified law combines a Dirac mass at zero and a continuous positive component. A mixed reference measure is needed to write its density precisely; ordinary Lebesgue-density language alone misses the atom at zero.
(a)
(b)
(c)
Figure 5: The Probability Density Function of Generalized Gaussian , Truncated Generalized Gaussian , and Rectified Generalized Gaussian across varying with fixed and . is the CDF of the Generalized Gaussian . (a) The case when . (b) When , we obtain Laplace, Truncated Laplace, and Rectified Laplace. (c) For , we have Gaussian, Truncated Gaussian, and Rectified Gaussian.
Definition B.4(Dirac Measure).
The Dirac measure over a measurable space for a given is defined as
Equation B.7
(B.7)
for any measurable set .
Definition B.5(Measure-Theoretical Definition of the Rectified Generalized Gaussian).
Fix parameters , , and . We denote as the real line equipped with Borel -algebra. Let be the Lebesgue measure on and let be the Dirac measure at presented in Definition B.4. The probability measure of the Rectified Generalized Gaussian random variable is given by the mixture
Equation B.8
(B.8)
where is the Truncated Generalized Gaussian probability measure in Definition B.3 and is the CDF of the standard Generalized Gaussian . Define the mixed measure . By Lemma B.7, the Radon-Nikodym derivative of with respect to exists and is given by
Equation B.9
(B.9)
Lemma B.6(Absolute Continuity).
The Rectified Generalized Gaussian probability measure in Definition B.5 is absolutely continuous with respect to the mixed measure , i.e. .
Proof.
According to Folland (1999), if is a signed measure and is a positive measure on the same measurable space , then if for every implies .
Let’s consider the case of . By definition, . Since both and are non-negative measures, . We observe that implies by the definition of the Dirac measure. Thus
Equations B.10–B.11
(B.10)
(B.11)
where the first term vanishes because . It’s trivial that is absolutely continuous with respect to the Lebesgue measure. Since , we have . Thus
Equation B.12
(B.12)
and we have proven the absolutely continuity result .
∎
Lemma B.7(Radon–Nikodym Derivative).
The Radon–Nikodym derivative of the Rectified Generalized Gaussian probability measure with respect to the mixed measure exists and is given by
Equation B.13
(B.13)
Proof.
By Lemma B.6, so the Radon–Nikodym derivative exists and it suffices to show that for any we have
Equation B.14
(B.14)
We start by expanding the integral with respect to a sum of measures
Equation B.15
(B.15)
By the property of the Dirac measure, we have
Equation B.16
(B.16)
We observe that and . So we have
Equation B.17
(B.17)
Now the second term can be expanded as
Equations B.18–B.19
(B.18)
(B.19)
where the term
Equation B.20
(B.20)
simply vanishes. Thus we are left with
Equations B.21–B.24
(B.21)
(B.22)
(B.23)
(B.24)
By Definition B.3, the Truncated Generalized Gaussian probability measure is given by
Equations B.25–B.26
(B.25)
(B.26)
Thus we have the identity
Equation B.28
(B.28)
Putting everything together, we arrive at
Equations B.29–B.30
(B.29)
(B.30)
Thus we have proven the form of the Radon–Nikodym Derivative.
∎
Equations B.31–B.33
(B.31)
(B.32)
(B.33)
Corollary B.8(Equivalent Definition of Rectified Generalized Gaussian).
The probability density function of the Rectified Generalized Gaussian distribution can also be written as
Equations B.34–B.35
(B.34)
(B.35)
where is the cumulative distribution function for the standard Generalized Gaussian distribution .
Proof.
We can simplify the expression as
Equation B.36
(B.36)
So we have
Equation B.37
(B.37)
where the extra terms cancel out due to symmetry around . Thus we have recovered the forms in Definition 3.4.
∎
B.4 Expectation and Variance of the Rectified Generalized Gaussian Distribution
Closed-form first and second moments are derived with gamma-function integrals. They permit variance normalization after rectification, which is different from normalizing the underlying unrectified variable.
Proposition B.9.
Let and be the sign function. Let be the lower incomplete gamma function, be the upper incomplete gamma function, be the gamma function, and be the lower regularized gamma function. Then
Equations B.38–B.41
(B.38)
(B.39)
(B.40)
(B.41)
Equations B.42–B.65
(B.42)
(B.43)
(B.44)
(B.45)
(B.46)
(B.47)
(B.48)
(B.49)
(B.50)
(B.51)
(B.52)
(B.53)
(B.54)
(B.55)
(B.56)
(B.57)
(B.58)
(B.59)
(B.60)
(B.61)
(B.62)
(B.63)
(B.64)
(B.65)
Definition B.10(Gamma Functions).
If and , then
Equations B.67–B.68
(B.67)
(B.68)
where and are the lower and upper incomplete gamma functions. By definition, we also have
B.5 Simulation Techniques for Rectified Generalized Gaussian
The retained sampling algorithm draws from the underlying generalized Gaussian and rectifies the result. It preserves the intended probability of an exact zero.
Algorithm 1 Simulation of the Rectified Generalized Gaussian Random Variables
Input: parameter , location , scale
Output: sample
Sample
Sample
Set
Set
return
Algorithm 2 Bisection Search for the Scale Parameter for Rectified Generalized Gaussian with Unit Variance
Input: parameter , location , tolerance
Output: scale such that { is defined in Proposition B.9.}
Define
Define
Choose initial bounds and such that
repeat
ifthen
else
endif
until
return
Appendix C Properties of Multivariate Generalized Gaussian, Truncated Generalized Gaussian, and Rectified Generalized Gaussian Distributions
C.1 Multivariate Case - Multivariate Generalized Gaussian
The multivariate target here is built from independent coordinate distributions. This product construction is an assumption of the target family, not a result automatically true of any trained feature matrix.
Equation C.87
(C.87)
C.2 Multivariate Case - Multivariate Truncated Generalized Gaussian
Restricting the product law to the positive support produces the corresponding multivariate truncated target, with the appropriate normalization.
C.3 Multivariate Case - Multivariate Rectified Generalized Gaussian
Independent rectified coordinates produce mass on multiple lower-dimensional faces according to which coordinates are zero. This geometry explains the need for mixed discrete/continuous entropy tools.
C.4 Proof of Proposition 3.5 (Theoretical Expected ℓ 0 Sparsity)
The expected number of active coordinates follows by summing their nonzero probabilities. This gives a target statistic for the empirical sparsity comparisons.
Equations C.88–C.91
(C.88)
(C.89)
(C.90)
(C.91)
(a) Empirical Variance of Generalized Gaussian.
(b) Theoretical Variance of Generalized Gaussian.
(c) Empirical Variance of Rectified Generalized Gaussian.
(d) Theoretical Variance of Rectified Generalized Gaussian.
Figure 6: Variance of Generalized Gaussian Distribution and Rectified Generalized Gaussian distributions under the choice of .Top row: Variance of . (a) The empirical variance of . (b) The theoretical variance of by evaluating EquationB.2. Bottom row: Variance of . (c) The empirical variance of . (d) The theoretical variance of by evaluating EquationB.41. The empirical variance in (a) and (c) are computed by i.i.d sampling samples from 32 dimensions from either or . The per-dimension variance is estimated and we report the average variance across dimensions as a function of the mean shift value and the parameter .
(a) Empirical Variance of Generalized Gaussian.
(b) Theoretical Variance of Generalized Gaussian.
(c) Empirical Variance of Rectified Generalized Gaussian.
(d) Theoretical Variance of Rectified Generalized Gaussian.
Figure 7: Variance of Generalized Gaussian Distribution and Rectified Generalized Gaussian distributions under the choice of .Top row: Variance of . (a) The empirical variance of . (b) The theoretical variance of by evaluating EquationB.2. Bottom row: Variance of . (c) The empirical variance of . (d) The theoretical variance of by evaluating EquationB.41. The empirical variance in (a) and (c) are computed by i.i.d sampling samples from 32 dimensions from either or . The per-dimension variance is estimated and we report the average variance across dimensions as a function of the mean shift value and the parameter .
(a) Values of across varying and .
(b) Values of across varying and .
Figure 8: Values of and under Different Choices of and . (a) The values of are invariant to the mean shift value . (b) changes as a function of both and .
Appendix D Choice of σ for Rectified Generalized Gaussian
D.1 How does σ affect the variance?
Changing scale changes how much variance remains after rectification. The appendix compares the chosen normalization rules using the derived moment formulas.
D.2 How does σ affect the sparsity?
Scale interacts with location in determining the zero probability. Keeping one parameter fixed while changing another need not preserve the same sparsity.
D.3 How does σ affect performance?
The normalization comparison measures downstream accuracy as well as the desired feature statistics. Its favored choice is an empirical recipe for the reported experiments.
(a) Theoretical Norms Under Different Choices of
(b) Theoretical and Empirical of Norms under the Choice of
(c) Theoretical and Empirical of Norms under the Choice of
Figure 9: The theoretical and empirical normalized norms under Different Choices of . (a) We report the theoretical norms based on Proposition 3.5 for for varying and . (b) The empirical norms of pretrained Rectified LpJEPA features are measured against the theoretical norms of the target Rectified Generalized Gaussian distribution for varying and . (c) We plot the empirical norms of pretrained Rectified LpJEPA features against the theoretical norms of the target Rectified Generalized Gaussian distribution for varying and .
(a) Accuracy versus Mean Shift Value
(b) Accuracy versus Sparsity Metric
(c) Accuracy versus Sparsity Metric
Figure 10: The Sparsity-Performance Tradeoffs under Different Chocies of . (a) We report CIFAR-100 validation accuracy for pretrained Rectified LpJEPA projector representations under varying . Under the same mean shift value , choosing leads to better performance compared to if is more negative. (b) Projector accuracy is plotted against the sparsity metrics measured over the pretrained Rectified LpJEPA projector representations. The gaps between and are negligible. (c) Switching from to sparsity metrics, we observe the same behavior. In fact, attains minor advantages in the sparsity-performance tradeoffs, especially when or .
(a) Bisection Convergence of for .
(b) Continuous Mapping Theorem
(c) Necessity of Rectifications.
Figure 11: Additional Results on the Choices of , the Location of , and the Ablations of for Rectified LpJEPA. (a) We report the bisection convergence error as a function of optimization iterations for finding the optimal (see AppendixD). (b) We compared Rectified LpJEPA versus a version of distribution matching towards the Rectified Generalizd Gaussian distribution via the continuous mapping theorem (see Section5.3). Rectified LpJEPA is the better design. (c) We show that Rectified LpJEPA attains the best sparsity-performance tradeoffs across various ablations of under the sparsity metrics . See Section5.2 for details.
(a) Sparsity Across Transfer Tasks
(b) Sparsity Across Transfer Tasks
Figure 12: Pretrained dense and sparse representations exhibits varying level of sparsity across different downstream tasks. We compare the and sparsity metrics for Rectified LpJEPA versus other baselines (see AppendixH) pretrained over ImageNet-100 across a variety of downstream tasks. (a) Rectified LpJEPA has varying sparsity over different datasets as we vary the mean shift value . CL and VICReg always have all entries being non-zero due to the lack of explicit rectifications. (b) Under sparsity metric , we observe varying sparsity for Rectified LpJEPA over different datasets for mean shift values sparsity . We observe that NCL in fact achieves the lowest metric, but as we show in Figure3(c) the most amount of variations is attained by Rectified LpJEPA.
Appendix E Maximum Differential Entropy Distributions
E.1 Derivation of Maximum Entropy Continuous Multivariate Probability Distributions under Support Constraints
A constrained maximum-entropy derivation uses normalization, support and expected magnitude. The Lagrange multipliers encode these constraints; without them there is no identical optimization problem.
Lemma E.1(Maximum Entropy Continuous Multivaraite Probability Distributions).
Let be a measurable set with positive Lebesgue measure. We define as measurable functions and let . Consider the optimization problem
Equations E.92–E.95
(E.92)
s.t.
(E.93)
(E.94)
(E.95)
We denote the set of functions that satisfies the given constraints as
Equation E.96
(E.96)
Assume the set is nonempty and that there exists
such that
Equation E.97
(E.97)
Then any maximizer of the optimization problem has the form
Equation E.98
(E.98)
where are chosen to satisfy the constraints .
Proof.
We can form the Lagrangian functional of the constrained optimization problem as
Equation E.99
(E.99)
where are Lagrange multipliers. Let be a maximizer that is strictly positive almost everywhere (a.e.) over . We denote as an arbitrary integrable perturbation supported on such that for sufficiently small . Thus the Gateaux derivative of in the direction of is given by
Equations E.100–E.104
(E.100)
(E.101)
(E.102)
(E.103)
(E.104)
Thus the functional derivative is
Equation E.106
(E.106)
Since this expression must vanish for all admissible perturbations , we get almost everywhere on . Solving for yields
Equation E.107
(E.107)
Absorbing the constant terms into , we end up with
Equation E.108
(E.108)
∎
E.2 Proof of Proposition 3.3 (Maximum Entropy Distribution Under the ℓ p Norm and Support Constraint)
The proof specializes the constrained calculation to the expected p-th-power magnitude and stated support, producing the generalized Gaussian form.
Equations E.109–E.115
(E.109)
(E.110)
(E.111)
(E.112)
(E.113)
(E.114)
(E.115)
E.3 Maximum Entropy Distribution Under the ℓ p Norm with Full Support
Removing the support restriction recovers the full-support generalized Gaussian case and its moment relationship.
and we recover the Generalized Gaussian distribution with zero mean and scale parameter .
Equation E.117
(E.117)
Equations E.118–E.120
(E.118)
(E.119)
(E.120)
E.4 Maximum Entropy Distribution Under the ℓ 1 Norm Constraint
The first-power constraint yields the Laplace-family special case under the specified support.
Corollary E.3.
The maximum entropy distribution over under the constraints
Equation E.121
(E.121)
is the product of independent univariate symmetric Laplace distributions with zero mean and scale parameter
Equation E.122
(E.122)
Equations E.123–E.125
(E.123)
(E.124)
(E.125)
E.5 Maximum Entropy Distribution under the ℓ 2 Norm Constraint
The second-power constraint yields the Gaussian-family special case under the specified support.
Corollary E.4.
The maximum entropy distribution over under the constraints
Equation E.126
(E.126)
is the multivariate Gaussian distribution with mean and covariance
Equation E.127
(E.127)
When and , the density function takes the form of
Equation E.128
(E.128)
Proof.
Notice that the vector-valued mean constraint and matrix-valued covariance constraint can be factorized as a collection of scalar-valued constraints
Equation E.129
(E.129)
By Lemma E.1, the maximum entropy distribution has the form
Equations E.130–E.132
(E.130)
(E.131)
(E.132)
for and , which is the multivariate Gaussian distribution up to normalizations. When and , the density function trivially evaluates to
Equation E.133
(E.133)
which is the maximum-entropy distribution under the expected norm constraints based on Lemma E.1.
∎
Appendix F Rényi Information Dimension and Entropy
F.1 Definition of the d ( ξ ) -dimensional Entropy
Rényi information dimension describes how quantized entropy grows as the resolution becomes finer. It distinguishes discrete, continuous and mixed distributions.
Consider a real-valued random variable and the discretization , where preserves only the integral part of . For example, . Under suitable conditions, the information dimension exists and is given by
Equation F.135
(F.135)
where
Equation F.136
(F.136)
is the Shannon entropy for a discrete random variable with probabilities for .
If the information dimension exists, the -dimensional entropy is defined as
Equation F.137
(F.137)
Definition F.3(-dimensional Entropy for Mixed Measures (Rényi, 1959)).
Let be a random variable with probability measure
Equation F.138
(F.138)
where is discrete and is absolutely continuous with respect to the Lebesgue measure. Then the information dimension , and the -dimensional entropy is given by
Equation F.139
(F.139)
where is the Lebesgue measure.
F.2 Proof of Theorem 3.6 (The d ( ξ ) -dimensional Entropy of the Rectified Generalized Gaussian Distribution)
The theorem splits the rectified law into zero/nonzero indicators and continuous active coordinates. Their contributions characterize its information dimension and associated entropy.
Equations F.140–F.147
(F.140)
(F.141)
(F.142)
(F.143)
(F.144)
(F.145)
(F.146)
(F.147)
F.3 Empirical Estimators of the d ( ξ ) -dimensional entropy
The appendix specifies empirical estimators rather than assuming ordinary differential-entropy estimators apply unchanged to data with exact zeros.
Lemma F.4(Probability Measure Under Rectification).
Let be a real-valued random variable where is absolutely continuous with respect to the Lebesgue measure , i.e. . Then the probability measure of over is
Equation F.148
(F.148)
where is the Dirac measure and and
Equation F.149
(F.149)
for any Borel .
Equations F.150–F.156
(F.150)
(F.151)
(F.152)
(F.153)
(F.154)
(F.155)
(F.156)
Lemma F.5(Radon–Nikodym Derivative Under Rectifications).
Let be a real-valued random variable where is absolutely continuous with respect to the Lebesgue measure , i.e. . Consider and let be the Dirac measure in Definition B.4. Then the Radon–Nikodym derivative of with respect to is given by
Equation F.157
(F.157)
Equations F.158–F.171
(F.158)
(F.159)
(F.160)
(F.161)
(F.162)
(F.163)
(F.164)
(F.165)
(F.166)
(F.167)
(F.168)
(F.169)
(F.170)
(F.171)
F.4 Alternative Interpretation of the d ( ξ ) -dimensional Entropy
An alternative interpretation relates the mixed entropy to uncertainty about activation patterns and uncertainty in positive magnitudes.
Equations F.172–F.174
(F.172)
(F.173)
(F.174)
Lemma F.6.
The entropy is equivalent to the -dimensional entropy
Equation F.175
(F.175)
Equations F.176–F.186
(F.176)
(F.177)
(F.178)
(F.179)
(F.180)
(F.181)
(F.182)
(F.183)
(F.184)
(F.185)
(F.186)
F.5 Generalization of the Entropy Decomposition of Total Correlation
The total-correlation decomposition is extended to the stated mixed-entropy framework. Marginal entropies alone still do not reveal all joint dependencies.
Equations F.187–F.190
(F.187)
(F.188)
(F.189)
(F.190)
Appendix G Hilbert-Schmidt Independence Criterion
The dependence statistic uses kernel comparisons of coordinate pairs. Its normalization and finite-sample implementation determine the reported values; pairwise tests are not a universal certificate of full factorial independence.
Equations G.191–G.192
(G.191)
(G.192)
Appendix H Baseline Designs
The baseline definitions fix which features are rectified and which invariance, contrastive or covariance terms are optimized. These choices are necessary to interpret the sparse-versus-dense comparison.
The covariance connection requires matching target projections along empirical covariance eigenvectors. It is more specific than asserting exact covariance recovery from any small random set.
Proposition I.1(Implicit Regularization of Second-Order Statistics).
Let be a neural network feature random vector with covariance matrix . We denote the eigendecomposition as with the set of eigenvectors being . Let be the Rectified Generalized Gaussian random vector and define . If for all , then .
Additional CIFAR-100 probes retain separate measurements for representation quality and zero fraction.
J.2 Ablations on Projector Dimensions
Projector-width ablations show how the dimension of the regularized representation affects the result; the encoder representation is evaluated separately.
Transformer experiments test whether the recipe transfers beyond convolutional backbones under the stated training settings.
J.4 ImageNet-1K Experiments
ImageNet-1K results extend the scale comparison. Data, architecture and schedule remain part of the performance claim.
Table 2: ImageNet-1K Linear Probe Results after 100 Epochs.
We report encoder and projector top-1 accuracy (%). Rectified LpJEPA uses as the target distribution.
The asymptotic analysis separates projection computation, sorting and representation dimensions. Big-O notation does not by itself determine the practical crossover in runtime.
Equations J.199–J.201
(J.199)
(J.200)
(J.201)
Table 5: Linear Probe Results on CIFAR-100. Acc1 (%) is higher-is-better (); sparsity is lower-is-better (). Bold denotes best and underline denotes second-best in each column (ties allowed).
(a) The Pareto frontier of performance against sparsity
(b) Correlation between and Sparsity Metrics
(c) The Effect of Number of Random Projections
Figure 13: Additional results on the sparsity-performance tradeoffs, the correlation between different sparsity metrics, and the effect of numbers of random projections on performance. (a) We present another version of Figure2(c) where the sparsity metric is switched from to . Again, we observe the same Pareto frontier with a sharp drop in performance under extreme sparsity. (b) Across different backbones, Rectified LpJEPAs with the target distribution being the Rectified Laplace distributions learn sparser representations as we decrease the mean shift value . Specifically, we observe that the and metrics are quite correlated. Thus both metrics are effective in measuring sparsity. (c) We test Rectified LpJEPA models with batch size and varying feature dimension as long as numbers of projections. As we increase the dimensions , we observe that the number of random projections required for good performance do not grow and are small relative to . Hence Rectified LpJEPA maintains stable sampling efficiency as the feature dimension grows in these experiments.
J.8 Additional Results on Eigenvectors
Eigenvector projection experiments investigate faster covariance alignment. They supplement rather than replace the assumptions of the theoretical connection.
J.9 Additional Results on Transfer Sparsity
Transfer datasets produce different sparsity profiles. These are exploratory signals for distribution shift, not a complete detection evaluation.
(a) Variance Loss when
(b) Covariance Loss when
(c) Accuracy when
(d) Variance Loss when
(e) Covariance Loss when
(f) Accuracy when
(g) Variance Loss when
(h) Covariance Loss when
(i) Accuracy when
Figure 14: Incorporating eigenvectors into random projections accelerates implicit VCReg loss minimization and speed-up convergence. We pretrain Rectified LpJEPA over CIFAR-100 with target distributions where the mean shift value . We consider three settings of selecting the projection vectors where we set . We denote the setting as ”Rand” if all are uniformly sampled from the unit sphere. Since in our setting the batch size is always smaller than the feature dimension, i.e. , we call the setting ”Rand + Full Eig” if we select the top- eigenvectors and mix them with random projection vectors. We also consider ”Rand + Bottom Eig” by sampling the bottom half eigenvectors and mixing with random projections. (a) (d) (g) We evaluate the variance loss between the three settings across varying . Incorporating all eigenvectors puts overly strong constraints, whereas penalizing the bottom half eigenvectors gives us good performances. (b) (e) (h) the covariance losses are evaluated during training and plotted for different projection vector settings across . Regularizing all top- eigenvector projections leads to significant implicit covariance loss minimization, where the covariance loss is the average of the off-diagonal of the empirical covariance matrix. (c) (f) (i) We report the projector accuracy against epoch for different projection settings. Using eigenvectors leads to both faster convergence and ultimately better performance.
Table 6: 1-shot linear probe accuracy (%) using encoder features. All results are in the 1-shot setting. Avg. denotes the mean across datasets.
Table 12: Projector Dimension Ablation on ImageNet-100.
For each method and projector dimension, we report the best linear probe accuracy (%) over learning rates .
Encoder and projector accuracies are measured using frozen features. Bold indicates the best value within each projector-dimension block.
Appendix K Qualitative Analysis of Rectified LpJEPA
K.1 k -Nearest Neighbors Visualizations
Nearest-neighbor examples inspect which images share learned sparse features. They are qualitative checks of representation behavior.
K.2 Visual Attribution Maps
Attribution visualizations show which image regions influence selected responses. Such maps suggest useful structure without uniquely identifying a causal semantic feature.
Appendix L Implementation Details
L.1 Pretraining data and setup
The implementation appendix records pretraining datasets and the training/evaluation split used by each experiment.
L.2 Architecture
Architectural settings identify encoder, projection head and rectification placement, which determine which representation receives the distribution loss.
L.3 Data augmentation
The augmentation recipe defines the related views and therefore the information encouraged to remain consistent.
L.4 Optimization
Optimization settings fix the learning rate, schedule and related training choices for reproducing the results.
L.5 Distribution matching objective
The distribution-matching recipe specifies target sampling, projection count and empirical comparison, making finite-sample approximations explicit.
L.6 Compute and runtime
Compute and runtime are reported for the stated experiment sizes and hardware; they should accompany efficiency comparisons.
L.7 Transfer evaluation protocol
Few-shot and full-shot transfer protocols define how labels and frozen or adapted features are used.
L.8 Reproducibility.
The reproducibility notes point to the code and settings needed to reconstruct the reported experiments.
The post-hoc ReLU protocol tests a different question from training with rectification. It keeps that distinction explicit in the comparisons.
L.10 Vision Transformer (ViT) experiments
The final settings specify the Transformer-backbone experiments, preventing an architecture change from being mistaken for an objective-only ablation.
Table 13: Mean-Shift Sweep (ViT) with Baselines.
We report encoder Acc1 (val_acc1), projector Acc1 (val_proj_acc1), and sparsity metrics.
Bold indicates best in a column; underline indicates second best.
For sparsity columns, lower is better (more sparse).
Figure 15: Nearest neighbors in feature space (ImageNet synset; unambiguous class).
Top- cosine nearest neighbors in the projector space for a query labeled as pirate ship. Both dense and sparse methods retrieve pirate ships consistently, illustrating that even at high sparsity our models can preserve semantic consistency when the query is unambiguous.Figure 16: Nearest neighbors in feature space (ImageNet synset; full scene).
Top- cosine nearest neighbors in the projector space for a query labeled as tabby cat (n02123045) that contains both the cat and a salient laptop/desk context. Dense methods (e.g., SimCLR) can return a mixture of cat and laptop/desk neighbors. In contrast, highly sparse variants tend to commit to a single factor: at MSV neighbors are predominantly tabby cats, while at MSV neighbors flip to predominantly laptop/desk images.Figure 17: Nearest neighbors in feature space (probe crop).
Top- cosine nearest neighbors in the projector space for a zoomed-in query that isolates the cat from Figure16 by removing the competing laptop/desk cues. In this less ambiguous setting, neighbors remain tabby cats across both dense and highly sparse methods, including the most sparse variants.Figure 18: Representation-focused attribution across methods.
Grad-CAM-style attribution maps computed on the projector representation for two views of the same scene (a tabby cat lying on a laptop). Rows compare dense baselines (SimCLR, VICReg, LeJEPA), sparse baselines (RepReLU variants), and our family (where corresponds to Laplace and is the mean-shift value, MSV) at increasing mean-shift values, which induce increasing sparsity.Figure 19: Distribution of representation sparsity in the transfer setting.
Violin plots showing the distribution of output-feature sparsity for correctly (green) and incorrectly (red) classified samples across datasets and methods. All models are pretrained on ImageNet-100 and evaluated using a full-shot linear-probe transfer setup, where a linear classifier is trained on frozen representations using of each downstream dataset. Sparsity is computed per sample from the frozen output features.
At a glance
Intuition. A dense Gaussian feature cloud almost never contains exact zeros. Rectifying a feature, with ReLU, maps negative values to zero and makes a sparse code possible. For a symmetric zero-centered target, about half the coordinates are active; shifting it toward negative values makes fewer active. The question is how to learn those statistics without losing information needed by a task.
Rectified LpJEPA aligns paired views and matches their rectified embeddings to a controllable sparse target. Its main distinction from LeJEPA is the target distribution and the resulting two-sample projection test. Experiments evaluate sparsity together with accuracy, dependence and transfer.
75 references
Alonso-Gutierrez et al. (2018)D. Alonso-Gutierrez, J. Prochno, and C. ThaeleGaussian fluctuations for high-dimensional random projections of -balls.
External Links: 1710.10130,
Link
Anderson et al. (1997)J. Anderson, H. B. Barlow, R. L. Gregory, G. E. Hinton, and Z. GhahramaniGenerative models for discovering sparse distributed representations.
Philosophical Transactions of the Royal Society B: Biological Sciences352 (1358), pp. 1177–1190.
External Links: ISSN 0962-8436,
Document,
Link,
https://royalsocietypublishing.org/rstb/article-pdf/352/1358/1177/84454/rstb.1997.0101.pdf
Assran et al. (2023)M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. BallasSelf-supervised learning from images with a joint-embedding predictive architecture.
External Links: 2301.08243,
Link
Attwell and Laughlin (2001)D. Attwell and S. B. LaughlinAn energy budget for signaling in the grey matter of the brain.
Journal of Cerebral Blood Flow & Metabolism21 (10), pp. 1133–1145.
Balestriero et al. (2023)R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. GoldblumA cookbook of self-supervised learning.
External Links: 2304.12210,
Link
Balestriero and LeCun (2025)R. Balestriero and Y. LeCunLeJEPA: provable and scalable self-supervised learning without the heuristics.
External Links: 2511.08544,
Link
Bardes et al. (2022)A. Bardes, J. Ponce, and Y. LeCunVICReg: variance-invariance-covariance regularization for self-supervised learning.
External Links: 2105.04906,
Link
Barlow et al. (1961)H. B. Barlow et al.Possible principles underlying the transformation of sensory messages.
Sensory communication1 (01), pp. 217–233.
Barthe et al. (2005)F. Barthe, O. Guédon, S. Mendelson, and A. NaorA probabilistic approach to the geometry of the lp-ball.
The Annals of Probability33 (2).
External Links: ISSN 0091-1798,
Link,
Document
Bonneel et al. (2015)N. Bonneel, J. Rabin, G. Peyré, and H. PfisterSliced and radon wasserstein barycenters of measures.
Journal of Mathematical Imaging and Vision51 (1), pp. 22–45.
Caron et al. (2021)M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. JoulinEmerging properties in self-supervised vision transformers.
External Links: 2104.14294,
Link
Chakraborty et al. (2025)D. Chakraborty, Y. LeCun, T. G. J. Rudner, and E. Learned-MillerImproving pre-trained self-supervised embeddings through effective entropy maximization.
External Links: 2411.15931,
Link
Chartrand and Yin (2008)R. Chartrand and W. YinIteratively reweighted algorithms for compressive sensing.
In 2008 IEEE international conference on acoustics, speech and signal processing,
pp. 3869–3872.
Chen et al. (2020)T. Chen, S. Kornblith, M. Norouzi, and G. HintonA simple framework for contrastive learning of visual representations.
External Links: 2002.05709,
Link
Cover and Thomas (2006)T. M. Cover and J. A. ThomasElements of information theory (wiley series in telecommunications and signal processing).
Wiley-Interscience, USA.
External Links: ISBN 0471241954
da Costa et al. (2022)V. G. T. da Costa, E. Fini, M. Nabi, N. Sebe, and E. RicciSolo-learn: a library of self-supervised methods for visual representation learning.
Journal of Machine Learning Research23 (56), pp. 1–6.
External Links: Link
Dytso et al. (2018)A. Dytso, R. Bustin, H. V. Poor, and S. ShamaiAnalytical properties of generalized gaussian distributions.
Journal of Statistical Distributions and Applications5 (1), pp. 6.
External Links: Document,
Link,
ISSN 2195-5832
Ermolov et al. (2021)A. Ermolov, A. Siarohin, E. Sangineto, and N. SebeWhitening for self-supervised representation learning.
External Links: 2007.06346,
Link
Fang et al. (1990)K. Fang, S. Kotz, and K. W. NgSymmetric multivariate and related distributions.
1st edition, Chapman and Hall/CRC.
External Links: Document
Gao et al. (2024)L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. WuScaling and evaluating sparse autoencoders.
External Links: 2406.04093,
Link
Garrido et al. (2022)Q. Garrido, Y. Chen, A. Bardes, L. Najman, and Y. LecunOn the duality between contrastive and non-contrastive self-supervised learning.
External Links: Document,
Link
Glorot et al. (2011)X. Glorot, A. Bordes, and Y. BengioDeep sparse rectifier neural networks.
In Proceedings of the fourteenth international conference on artificial intelligence and statistics,
pp. 315–323.
Gretton et al. (2012)A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. J. SmolaA kernel two-sample test.
Journal of Machine Learning Research.
Gretton et al. (2005)A. Gretton, O. Bousquet, A. Smola, and B. SchölkopfMeasuring statistical dependence with hilbert-schmidt norms.
In International conference on algorithmic learning theory,
pp. 63–77.
Grill et al. (2020)J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. ValkoBootstrap your own latent: a new approach to self-supervised learning.
External Links: 2006.07733,
Link
Gupta and Song (1997)A.K. Gupta and D. SongLp-norm spherical distribution.
Journal of Statistical Planning and Inference60 (2), pp. 241–260.
External Links: ISSN 0378-3758,
Document,
Link
Halko et al. (2011)N. Halko, P. Martinsson, and J. A. TroppFinding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions.
SIAM Review53 (2), pp. 217–288.
He et al. (2020)K. He, H. Fan, Y. Wu, S. Xie, and R. GirshickMomentum contrast for unsupervised visual representation learning.
External Links: 1911.05722,
Link
He et al. (2016)K. He, X. Zhang, S. Ren, and J. SunDeep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 770–778.
Jing et al. (2022)L. Jing, P. Vincent, Y. LeCun, and Y. TianUnderstanding dimensional collapse in contrastive self-supervised learning.
External Links: 2110.09348,
Link
Kim et al. (2019)I. Kim, S. Balakrishnan, and L. WassermanRobust multivariate nonparametric tests via projection averaging.
The Annals of Statistics47 (6), pp. 3417–3441.
Kolouri et al. (2018)S. Kolouri, G. K. Rohde, and H. HoffmannSliced-wasserstein autoencoder.
In International Conference on Learning Representations (Workshop),
External Links: Link
Kotz et al. (2012)S. Kotz, T. Kozubowski, and K. PodgorskiThe laplace distribution and generalizations: a revisit with applications to communications, economics, engineering, and finance.
Springer Science & Business Media.
Kuang et al. (2025)Y. Kuang, Y. Dagade, D. Chakraborty, E. Learned-Miller, R. Balestriero, T. G. J. Rudner, and Y. LeCunRadial-VCReg: more informative representation learning through radial gaussianization.
In UniReps: 3rd Edition of the Workshop on Unifying Representations in Neural Models,
External Links: Link
Learned-Miller et al. (2003)E. G. Learned-Miller et al.ICA using spacings estimates of entropy.
Journal of machine learning research4 (Dec), pp. 1271–1295.
Lee and Seung (2000)D. Lee and H. S. SeungAlgorithms for non-negative matrix factorization.
In Advances in Neural Information Processing Systems, T. Leen, T. Dietterich, and V. Tresp (Eds.),
Vol. 13, pp. .
External Links: Link
Liu et al. (2022)Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. XieA convnet for the 2020s.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pp. 11976–11986.
Mialon et al. (2022)G. Mialon, R. Balestriero, and Y. LeCunVariance covariance regularization enforces pairwise independence in self-supervised representations.
arXiv preprint arXiv:2209.14905.
Nadjahi et al. (2020)K. Nadjahi, V. De Bortoli, J. Delon, and A. GenevayStatistical and topological properties of sliced probability divergences.
In Advances in Neural Information Processing Systems,
Vol. 33.
Nair and Hinton (2010)V. Nair and G. E. HintonRectified linear units improve restricted boltzmann machines.
In Proceedings of the 27th international conference on machine learning (ICML-10),
pp. 807–814.
Nardon and Pianca (2009)M. Nardon and P. PiancaSimulation techniques for generalized gaussian densities.
Journal of Statistical Computation and Simulation79 (11), pp. 1317–1329.
External Links: Document,
Link,
https://doi.org/10.1080/00949650802290912
Olshausen and Field (1996)B. A. Olshausen and D. J. FieldEmergence of simple-cell receptive field properties by learning a sparse code for natural images.
Nature381 (6583), pp. 607–609.
Selvaraju et al. (2019)R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. BatraGrad-cam: visual explanations from deep networks via gradient-based localization.
International Journal of Computer Vision128 (2), pp. 336–359.
External Links: ISSN 1573-1405,
Link,
Document
Socci et al. (1997)N. Socci, D. Lee, and H. S. SeungThe rectified gaussian distribution.
In Advances in Neural Information Processing Systems, M. Jordan, M. Kearns, and S. Solla (Eds.),
Vol. 10, pp. .
External Links: Link
Tibshirani (1996)R. TibshiraniRegression shrinkage and selection via the lasso.
Journal of the Royal Statistical Society Series B: Statistical Methodology58 (1), pp. 267–288.
Vasicek (1976)O. VasicekA test for normality based on sample entropy.
Journal of the Royal Statistical Society Series B: Statistical Methodology38 (1), pp. 54–59.
Wen et al. (2025)T. Wen, Y. Wang, Z. Zeng, Z. Peng, Y. Su, X. Liu, B. Chen, H. Liu, S. Jegelka, and C. YouBeyond matryoshka: revisiting sparse coding for adaptive representation.
External Links: 2503.01776,
Link
Yerxa et al. (2023)T. Yerxa, Y. Kuang, E. Simoncelli, and S. ChungLearning efficient coding of natural images with maximum manifold capacity representations.
External Links: 2303.03307,
Link
Yu et al. (2020)Y. Yu, K. H. R. Chan, C. You, C. Song, and Y. MaLearning diverse and discriminative representations via the principle of maximal coding rate reduction.
External Links: 2006.08558,
Link
Zbontar et al. (2021)J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. DenyBarlow twins: self-supervised learning via redundancy reduction.
External Links: 2103.03230,
Link