Literature
Display
← All papers

Generative Representation Learning for Scalable Latent Tokenization

Alan Baade · Kyle Sargent · Vincent Sitzmann · Eric Ryan Chan · Justin Johnson · Li Fei-Fei · Gordon Wetzstein

Preprint posted · arXiv (preprint) ↗

Computer vision10 min text · CondensedOriginal paper ↗
Contents

Overview

RePR learns image representations by fitting embedding similarities to the incremental denoising benefit provided by another image view. It uses generative prediction to reduce shortcut-driven feature suppression and tests the resulting features as conditioning for image generation.

At a glance

Relative Prediction Representations estimates a weighted pointwise-mutual-information target from a conditional generative model and trains embeddings to approximate it. Recursive feature prediction improves representations beyond pixel prediction alone. ImageNet and text-to-image experiments show useful downstream generative features, including lower generation error under selected training-compute budgets. The headline seven-to-eight-fold encoder-efficiency comparison is extrapolated from fitted scaling curves, not directly achieved by the largest trained model. Practical noise weighting, finite-model estimation, and inference architecture depart from the idealized information-theoretic objective.

1 Introduction

Discriminative self-supervision can solve its matching task using a salient shortcut while discarding other predictable features. Generative models instead need to account for detailed observations, but their internal features are not automatically convenient representations. RePR connects the two through relative prediction: how much does an additional view improve prediction of a target patch? The representation is trained to reproduce that improvement and then evaluated primarily through downstream image generation.

2 Related Work

Contrastive and noncontrastive methods learn invariances or predict features, while masked autoencoders and diffusion models predict observations. RePR follows the information-theoretic interpretation of contrastive predictive coding and implicit matrix factorization, but derives supervised embedding targets from generative prediction differences rather than a classification loss over negatives. Downstream generation uses Latent Forcing, which generates representation features before pixels.

3 Background

Pointwise mutual information measures the change in log likelihood of an observation after conditioning on another view. A contrastive optimum recovers this quantity only up to a conditioning-view-dependent offset; finite-dimensional embeddings introduce further restrictions. RePR aims to estimate useful relative prediction quantities directly. Mutual information, pointwise mutual information, and an embedding dot product are distinct objects, so the theoretical interpretation depends on the stated modeling and factorization assumptions.

Equations 1–3
PMI⁡(𝐱,𝐲)=PMI⁡(𝐲,𝐱)=log⁡pX|Y​(𝐱∣𝐲)pX​(𝐱)=log⁡pY|X​(𝐲∣𝐱)pY​(𝐲)=log⁡pX,Y​(𝐱,𝐲)pX​(𝐱)​pY​(𝐲).\displaystyle\operatorname{PMI}({\mathbf{x}},{\mathbf{y}})=\operatorname{PMI}({\mathbf{y}},{\mathbf{x}})=\log\frac{p_{X\mid Y}({\mathbf{x}}\mid{\mathbf{y}})}{p_{X}({\mathbf{x}})}=\log\frac{p_{Y\mid X}({\mathbf{y}}\mid{\mathbf{x}})}{p_{Y}({\mathbf{y}})}=\log\frac{p_{X,Y}({\mathbf{x}},{\mathbf{y}})}{p_{X}({\mathbf{x}})\,p_{Y}({\mathbf{y}})}. (1)
⟨Rθ​(𝐱),Rθ​(𝐲)⟩≈PMI⁡(𝐱,𝐲)+a⁡(𝐱).\langle R_{\theta}({\mathbf{x}}),R_{\theta}({\mathbf{y}})\rangle\approx\operatorname{PMI}({\mathbf{x}},{\mathbf{y}})+a({\mathbf{x}}). (2)
PMI⁡(𝐱,𝐲)=log⁡PGθ​(𝐲|𝐱)−log⁡PGθ​(𝐲).\operatorname{PMI({\mathbf{x}},{\mathbf{y}})}=\log P_{G_{\theta}}({\mathbf{y}}|{\mathbf{x}})-\log P_{G_{\theta}}({\mathbf{y}}).\vskip-16.00008pt (3)

4 Method

The method converts differences between conditional denoising errors into targets for representation similarities. It uses a practical noise-weighting schedule, predicts incremental information about one patch, and fits this target with a stop-gradient regression objective.

Equations 4–5
ℒPMI\displaystyle\mathcal{L}_{\mathrm{PMI}} =‖⟨Rθ​(𝐱),Rθ​(𝐲)⟩−PMI⁡(𝐱,𝐲)‖22\displaystyle=\Bigl\|\langle R_{\theta}({\mathbf{x}}),R_{\theta}({\mathbf{y}})\rangle-\operatorname{PMI}({\mathbf{x}},{\mathbf{y}})\Bigr\|_{2}^{2} (4)
=‖⟨Rθ​(𝐱),Rθ​(𝐲)⟩−stopgrad⁡(log⁡PGθ​(𝐲∣𝐱)−log⁡PGθ​(𝐲))‖22.\displaystyle=\Bigl\|\langle R_{\theta}({\mathbf{x}}),R_{\theta}({\mathbf{y}})\rangle-\operatorname{stopgrad}\!\Bigl(\log P_{G_{\theta}}({\mathbf{y}}\mid{\mathbf{x}})-\log P_{G_{\theta}}({\mathbf{y}})\Bigr)\Bigr\|_{2}^{2}. (5)

4.1 PMI Estimation from Generative Models

The diffusion variational bound relates a log probability to integrated denoising error, following Kingma and colleagues. Subtracting conditional and unconditional errors estimates how informative a context is about a target. Equality to the true likelihood requires an optimal denoiser and appropriate endpoint/discrete-data assumptions; approximation errors need not cancel for finite trained models. The computation is therefore an estimated prediction difference, rather than an exact density-ratio oracle.

Equations 6–8
−log⁡p⁡(𝐱)≤−VLB​(𝐱)=DK​L(q(𝐳1|𝐱)||p(𝐳1))⏟Prior loss+𝔼q⁡(𝐳0|𝐱)​[−log⁡p⁡(𝐱|𝐳0)]⏟Reconstruction loss+ℒ∞​(𝐱).⏟Diffusion loss\displaystyle-\log p({\mathbf{x}})\leq-\text{VLB}({\mathbf{x}})=\underbrace{D_{KL}(q({\mathbf{z}}_{1}|{\mathbf{x}})||p({\mathbf{z}}_{1}))}_{\text{Prior loss}}+\underbrace{\mathbb{E}_{q({\mathbf{z}}_{0}|{\mathbf{x}})}\left[-\log p({\mathbf{x}}|{\mathbf{z}}_{0})\right]}_{\text{Reconstruction loss}}+\underbrace{\mathcal{L}_{\infty}({\mathbf{x}}).}_{\text{Diffusion loss}} (6)
log⁡p⁡(𝐱)\displaystyle\log p({\mathbf{x}}) =VLB(𝐱)=−ℒ∞(𝐱)=−𝔼t∼𝐔⁡[0,1]​ϵ∼𝒩⁡(0,𝐈)[1−tt3‖𝐱−𝔼(𝐱∣𝐱t,t)‖22].\displaystyle=\text{VLB}({\mathbf{x}})=-\mathcal{L}_{\infty}({\mathbf{x}})=-\mathbb{E}_{t\sim\mathbf{U}[0,1]{\bm{\epsilon}}\sim\mathcal{N}(0,\mathbf{I})}\left[\frac{1-t}{t^{3}}\left\rVert{\mathbf{x}}-\mathbb{E}({\mathbf{x}}\mid{\mathbf{x}}_{t},t)\right\lVert_{2}^{2}\right]. (7)
PMI(𝐱,𝐲)=𝔼t∼𝐔⁡[0,1],ϵ∼𝒩⁡(0,𝐈)[1−tt3(‖𝐲−𝔼(𝐲∣𝐲t;t)‖22−‖𝐲−𝔼(𝐲∣𝐱,𝐲t,t)‖22)].\displaystyle\operatorname{PMI}({\mathbf{x}},{\mathbf{y}})=\mathbb{E}_{t\sim\mathbf{U}[0,1],{\bm{\epsilon}}\sim\mathcal{N}(0,\mathbf{I})}\left[\frac{1-t}{t^{3}}\left(\left\rVert{\mathbf{y}}-\mathbb{E}({\mathbf{y}}\mid{\mathbf{y}}_{t};t)\right\lVert_{2}^{2}-\left\rVert{\mathbf{y}}-\mathbb{E}({\mathbf{y}}\mid{\mathbf{x}},{\mathbf{y}}_{t},t)\right\lVert_{2}^{2}\right)\right]. (8)

4.2 Timestep Weighting

The likelihood-derived weighting strongly emphasizes fine detail at small noise levels. RePR replaces it with a chosen weighting function to emphasize useful prediction scales and improve numerical behavior. This changes the target to weighted mutual-information-like prediction, not exact pointwise mutual information. The ablation using the unmodified singular weighting produces NaNs, making this practical modification central to the working method.

Equations 9–12
𝐲s\displaystyle\mathbf{y}_{s} =(1−s)​𝐲+s​ϵ,ϵ∼𝒩⁡(0,I).\displaystyle=(1-s)\mathbf{y}+s\bm{\epsilon},\qquad\bm{\epsilon}\sim\mathcal{N}(0,I). (9)
PMI⁡(𝐱,𝐲s)\displaystyle\operatorname{PMI}(\mathbf{x},\mathbf{y}_{s}) =log⁡ps​(𝐲s∣𝐱)ps​(𝐲s).\displaystyle=\log\frac{p_{s}(\mathbf{y}_{s}\mid\mathbf{x})}{p_{s}(\mathbf{y}_{s})}. (10)
PMI∗⁡(𝐱,𝐲)\displaystyle\operatorname{PMI}^{*}(\mathbf{x},\mathbf{y}) =𝔼s∼U⁡[0,1],ϵ​[q⁡(s)​PMI⁡(𝐱,𝐲s)],\displaystyle=\mathbb{E}_{s\sim U[0,1],\,\bm{\epsilon}}\left[q(s)\operatorname{PMI}(\mathbf{x},\mathbf{y}_{s})\right], (11)
PMI∗⁡(𝐱,𝐲)\displaystyle\operatorname{PMI}^{*}(\mathbf{x},\mathbf{y}) =𝔼t∼f⁡(t),ϵ​[g⁡(t)​(‖𝐲−𝔼⁡(𝐲∣𝐲t,t)‖22−‖𝐲−𝔼⁡(𝐲∣𝐲t,t,𝐱)‖22)].\displaystyle=\mathbb{E}_{t\sim f(t),\,\bm{\epsilon}}\left[g(t)\left(\left\|\mathbf{y}-\mathbb{E}({\mathbf{y}}\mid\mathbf{y}_{t},t)\right\|_{2}^{2}-\left\|\mathbf{y}-\mathbb{E}({\mathbf{y}}\mid\mathbf{y}_{t},t,\mathbf{x})\right\|_{2}^{2}\right)\right]. (12)

4.3 Bootstrapped PMI Estimation

Instead of estimating a full similarity in one expensive step, training samples a context and another patch and regresses the incremental information supplied by that patch. Exact consistent increments can telescope to a full information quantity. This motivates bootstrapping, but does not prove that finite, biased, noise-weighted estimates retain every property of exact mutual information.

Equations 13–14
⟨Rθ​(𝐱),Rθ​(y≤k)−Rθ​(y≤k−1)⟩=PMI⁡(𝐱,y≤k)−PMI⁡(𝐱,y≤k−1)\displaystyle\left\langle R_{\theta}({\mathbf{x}}),R_{\theta}(y{\leq k})-R_{\theta}(y_{\leq k-1})\right\rangle=\operatorname{PMI}({\mathbf{x}},y_{\leq k})-\operatorname{PMI}({\mathbf{x}},y_{\leq k-1})
=∑i=1klog⁡p⁡(yi∣y<i,𝐱)p⁡(yi∣y<i)−∑i=1k−1log⁡p⁡(yi∣y<i,𝐱)p⁡(yi∣y<i)=log⁡p⁡(yk∣y<k,𝐱)p⁡(yk∣y<k)=PMI⁡(yk,𝐱∣y<k).\displaystyle\qquad=\sum_{i=1}^{k}\log\frac{p(y_{i}\mid y_{<i},{\mathbf{x}})}{p(y_{i}\mid y_{<i})}-\sum_{i=1}^{k-1}\log\frac{p(y_{i}\mid y_{<i},{\mathbf{x}})}{p(y_{i}\mid y_{<i})}=\log\frac{p(y_{k}\mid y_{<k},{\mathbf{x}})}{p(y_{k}\mid y_{<k})}=\operatorname{PMI}(y_{k},{\mathbf{x}}\mid y_{<k}). (13)
∑k=1K[PMI⁡(𝐱,y≤k)−PMI⁡(𝐱,y≤k−1)]=PMI⁡(𝐱,y≤K)−PMI⁡(𝐱,∅)=PMI⁡(𝐱,𝐲).\sum_{k=1}^{K}\left[\operatorname{PMI}({\mathbf{x}},y_{\leq k})-\operatorname{PMI}({\mathbf{x}},y_{\leq k-1})\right]=\operatorname{PMI}({\mathbf{x}},y_{\leq K})-\operatorname{PMI}({\mathbf{x}},\varnothing)=\operatorname{PMI}({\mathbf{x}},{\mathbf{y}}). (14)

4.4 Representation Objective

The predicted increment is a dot product between the first view’s representation and the difference between second-view representations with and without the target patch. The generative-model target is detached while the representation learns by squared error. Since mutual-information matrices need not be positive semidefinite and weighted targets may be asymmetric, separate input roles matter: the implementation marks the target patch and permits asymmetric representations. A single shared symmetric Gram matrix could not in general represent every such target.

Equation 15
ℒRePR\displaystyle\mathcal{L}_{\mathrm{RePR}} =‖⟨Rθ​(𝐱),Rθ​(𝐲,𝐳)−Rθ​(𝐲)⟩−PMI∗⁡(𝐱,𝐳∣𝐲)‖22\displaystyle=\Bigl\|\left\langle R_{\theta}({\mathbf{x}}),R_{\theta}({\mathbf{y}},{\mathbf{z}})-R_{\theta}({\mathbf{y}})\right\rangle-\operatorname{PMI}^{*}({\mathbf{x}},{\mathbf{z}}\mid{\mathbf{y}})\Bigr\|_{2}^{2} (15)
=|⟨Rθ​(𝐱),Rθ​(𝐲,𝐳)−Rθ​(𝐲)⟩\displaystyle=\Bigl\|\left\langle R_{\theta}({\mathbf{x}}),R_{\theta}({\mathbf{y}},{\mathbf{z}})-R_{\theta}({\mathbf{y}})\right\rangle
−𝔼t∼f⁡(t),ϵ[g(t)stopgrad(‖𝐳−𝔼(𝐳∣𝐳t,t,𝐲)‖22−‖𝐳−𝔼(𝐳∣𝐳t,t,𝐱,𝐲)‖22)]∥22.\displaystyle-\mathbb{E}_{t\sim f(t),\,\bm{\epsilon}}\Bigl[g(t)\operatorname{stopgrad}\Bigl(\left\|\mathbf{z}-\mathbb{E}({\mathbf{z}}\mid\mathbf{z}_{t},t,{\mathbf{y}})\right\|_{2}^{2}-\left\|\mathbf{z}-\mathbb{E}({\mathbf{z}}\mid\mathbf{z}_{t},t,\mathbf{x},{\mathbf{y}})\right\|_{2}^{2}\Bigr)\Bigr]\Bigr\|_{2}^{2}.

5 Architecture

A masked autoregressive prediction branch and a representation branch share an image encoder. Prediction supplies supervision during training; the final representation branch is used downstream. Its output is a spatial feature map rather than solely a global classification token.

Original figure
Figure 1: Architecture. Left: 1) Our Prediction Model (bottom) estimates weighted data likelihoods for a target patch pp using an MAR architecture, which combines an MAE Encoder EθE_{\theta} and Decoder DθD_{\theta} with a diffusion prediction MLP head MθM_{\theta}. 2) The Representation Model RθR_{\theta} (top) outputs vector representations supervised to factorize a weighted PMI. Right: We enforce spatial equivariance in RθR_{\theta} by hiding absolute position information but outputting a feature for every patch.

5.1 Prediction Model

An eighteen-layer encoder feeds a six-layer prediction decoder and a small diffusion multilayer perceptron that denoises patches. This follows masked autoregressive modeling and pixel-space denoising designs, including MAR and JiT. Conditional predictions with and without a view use the same sampled noise, reducing variance in their difference.

Equation 16
ℒMAR=w⁡(t)​‖Mθ​(zp,pt,t)−p‖22.\mathcal{L}_{\text{MAR}}=w(t)||M_{\theta}(z_{p},p_{t},t)-p||_{2}^{2}. (16)

5.2 Representing Patch Features

The representation branch adds three self-attention and three cross-attention layers to the shared eighteen-layer encoder. Relative positional structure and queries centered on patch locations produce a full patch map; an extra row and column of queries provide padding features. An additional classification-token representation objective supports linear probing. In the largest configuration, inference has twenty-four layers and width 1,024, rather than including the complete training-time prediction decoder.

5.3 Masking Strategy

Two block-masked views may overlap, while the target patch is outside both. The reported sampling masks 75–100% of view A and 93.75–100% of view B. Repeating sampled patches and views amortizes encoding. These choices control which conditional prediction problems supervise the representation; they are not a claim that all possible view/patch relationships are exhaustively observed.

5.4 Iterative Feature Prediction

After an initial pixel-prediction stage, later iterations train a new model to predict the preceding model’s features. Each stage uses the earlier representation as its target space. The reported compute includes preceding stages, so an iterative model’s final-stage epoch count alone understates its total training. Improvements are empirical and do not establish that arbitrarily many iterations monotonically improve a representation.

5.5 Joint Optimization

Training combines the masked generative loss with the representation regression loss, with a reported one-to-three update schedule between the joint and prediction-only components. Thirty-two Monte Carlo noise samples estimate each incremental target and add only 1.3% compute in the reported accounting. Shared noise reduces variance, while stopping gradients through the target avoids directly optimizing the predictor to make its own information estimate easier to fit.

6 Experiments

Experiments use ImageNet and GPIC, examining shortcut suppression, model/compute scaling, and image-generation quality. The primary generation comparison predicts both learned features and pixels without a separately trained perceptual/GAN image decoder. Text-to-image experiments nevertheless use an external FLAN-T5-large text encoder, and the later representation-autoencoder experiment adds separate perceptual/GAN components.

6.1 Feature Suppression

A synthetic watermark supplies each training image’s identity in every 16-by-16 patch. It makes view matching easy without understanding image content: DINOv3 and LeJEPA lose classification-probe quality under this manipulation, whereas RePR remains useful. This is evidence against one constructed feature-suppression failure mode, not a universal robustness guarantee for real-world shortcuts, distribution shifts, or all contrastive architectures.

Original figure
Figure 2: Discriminative representation learning can suffer from feature suppression when a small subset of features, such as a watermark, is sufficient to distinguish examples. This degrades representation quality, as reflected by collapsing ImageNet class-probing performance. RePR remains robust to feature suppression.

6.2 Scaling Laws

At fixed depth, encoder widths span roughly twenty to 306 million inference parameters, with training budgets varying by about two orders of magnitude. Features condition a fixed downstream generator: ImageNet evaluation uses JiT-B trained for eighty epochs and FID on ten thousand samples; GPIC uses MiniT2I and a DINOv2-feature Fréchet distance after one GPIC epoch. These metrics and datasets are not directly interchangeable.

The best measured RePR ImageNet value in the scaling table is FID 24.73, versus 17.95 for the referenced DINO feature baseline; the best listed GPIC distance is 325 versus 277.1 for the web-scale reference. The claimed seven-to-eight-fold encoder compute reduction extrapolates fitted curves to those reference scores. It is not an experimentally reached equality. The largest ImageNet models overfit after about two hundred epochs, and some individual longer runs regress slightly, so the observed scaling is not uniformly monotonic.

Original figure
Figure 3: Tokenizer Encoder Scaling Laws: RePR predictably scales as a tokenizer encoder. When holding size and training duration of a generative model constant, greater RePR size and FLOPs improve generation performance. Iterative feature prediction (Sec. 5.4) yields further improvement.

6.3 Compute Optimal Generation

At a fixed total budget, spending some training compute on learned representation features improves a Latent Forcing generator over directly generating pixels. The medium-budget experiment allocates about twenty or thirty-five percent to two-stage forty-four- or seventy-eight-million-parameter encoders. At the larger guided setting, a seventy-eight-million-parameter, three-stage RePR encoder plus a JiT-L generator gives FID 2.18 and Inception Score 338.4 at approximately 3.49×10²⁰ training FLOPs, compared with pixel JiT-L at FID 2.36 and Inception Score 298.5 for 3.57×10²⁰.

The three encoder stages total 830 effective epochs, and guidance and the small autoguidance model are included in the reported accounting. Compute is approximated as six times parameter count times training tokens, excluding self-attention operations. It is not measured wall-clock cost or a complete inference-cost comparison, and “compute optimal” applies to the tested configurations and budget range.

Original figure
Figure 4: Total (Encoder + Diffusion) training compute versus FID-10K for JiT and JiT with Latent Forcing using RePR, showing that representation learning is compute-optimal for generation.

6.4 Ablation Studies

Mask rates, block masking, representation-loss weight, recursive stages, and noise weighting all affect performance. Some ablations use linear-probe accuracy as a proxy rather than downstream generation. At six hundred total epochs, one, two, and three representation stages yield generation FIDs 30.68, 24.73, and 23.81. Excessive representation-loss weights substantially reduce probe accuracy. The exact likelihood-style singular noise weighting fails numerically; clamped or smoothly reweighted alternatives train, with nontrivial sensitivity to the sampling schedule. The tables retain every setting rather than presenting a single successful recipe as inevitable.

Table 1: Ablation study for RePR Training. Default configuration in gray, best results bolded. Ablations (a,b,c) train RePR-78M for 100 epochs and (d,f) for 200 epochs. Due to the cost of training generative models, we use ImageNet (Deng et al., 2009) Top-1 Accuracy as a proxy metric.
Complete original table 1 ablation including all rows, columns, and original emphasis.

6.5 GAN+LPIPS Decoding

A separate experiment uses representation autoencoders with perceptual and adversarial reconstruction losses. After twenty epochs of DiT-XL generation training, the listed RePR encoder gives FID 11.13, versus 12.46 for I-JEPA-H and 13.84 for MoCo v3-B. This compares roughly matched encoder-training budgets, not a setting free of external perceptual-model costs. The cited perceptual/GAN component alone has substantial external training compute. Appendix reconstruction results also favor RePR on rFID and PSNR, while using separate decoder training and different final-stage/total-epoch conventions.

Table 3: RAE comparisons at 20ep of DiT-XL training for ImageNet-only encoders. IJEPA, MocoV3 results from RAEv2 (Singh et al., 2026)
Complete original table 3 rae including all rows, columns, and original emphasis.
Table 4: Representation learning is compute optimal. Under a total FLOPs budget, RePR training with Latent Forcing (Baade et al., 2026) outperforms direct pixel diffusion using JiT (Li and He, 2026) with guidance, 50K generations.
Complete original table 4 compute including all rows, columns, and original emphasis.

7 Conclusion

Relative generative prediction is a useful supervision signal for representations, and the experiments support recursive feature learning and allocating some generation-training compute to representations. Limits include approximate finite-model density estimates, manually chosen noise/masking schedules, idealized factorization arguments, extrapolated scaling claims, and overfitting at larger ImageNet budgets. The work chiefly tests generative utility and a synthetic shortcut stress test; it does not establish optimal features for every downstream perception or control task.

Acknowledgments

The acknowledgments identify NSF graduate fellowship support and computing support from Rhoda AI, together with feedback from colleagues.

AI use statement

The authors disclose AI assistance for coding, visualization, boilerplate, proof checking, readability, and search. They state that authors reviewed the outputs and developed the central algorithmic ideas themselves.

Appendix A Appendix

The appendix retains the exact training pseudocode, reconstruction experiments, implementation settings, theoretical qualifications, qualitative generations, watermark construction, and complete raw scaling tables.

A.1 Algorithm

The original algorithm is preserved as an indented code block. It encodes the two views and their union, predicts masked content, estimates the incremental denoising benefit under a no-gradient block with shared noise, and fits the representation increment. The final loss adds the generative term and a weighted representation-regression term. All variable names, indexing, and line breaks remain available for checking the operational meaning of the equations.

Algorithm 1 RePR: Training Pseudocode
Minimal Implementation, see Sec. 5 for details. mA​Bm_{AB}: mask for A​BAB. pip_{i}: index of pp. NN: Number of Monte Carlo integration steps. Efficiency-only parameters ARepeatA_{\text{Repeat}} and pRepeatp_{\text{Repeat}} omitted for brevity. For feature prediction, the MAR target is replaced with Rθold​(Eθold​(x))R_{\theta_{\text{old}}}(E_{\theta_{\text{old}}}(x)) and pp is replaced with Rθold​(Eθold​(x))​[pi]R_{\theta_{\text{old}}}(E_{\theta_{\text{old}}}(x))[p_{i}]
A, B, AB, p, m_AB, p_i = get_masks(X)
a, b, ab, bp = E(A), E(B), E(AB), E(B, p)
d_ab, d_b = D(ab), D(b)

x = X[m_AB]
t, eps = sample_t_eps(x)
x_t = (1 - t) * x + t * eps
x_pred = M(x_t, t, d_ab[m_AB])
mar_loss = mean(w(t) * (x_pred - x)**2)

with no_grad():
  t, eps = sample_t_eps(p, N)
  p_t = (1 - t) * repeat(p, N) + t * eps
  p_ab = M(p_t, t, d_ab[p_i])
  p_b = M(p_t, t, d_b[p_i])
  pmi = mean(g(t)*((p_b-p)**2-(p_ab-p)**2))

pred = dot(R(a), R(bp) - R(b))
repr_loss = mean((pred - pmi)**2)

loss = mar_loss + loss_weight * repr_loss

A.2 Additional Experiments

The additional reconstruction comparison reports RePR-B third-stage rFID 0.47 and PSNR 21.72, versus DINO’s 0.49 and 18.09 with the specified decoder setup. Reconstruction fidelity and downstream sample generation are different measurements. The table’s six-hundred-epoch label refers to the last encoder stage; the corresponding cumulative three-stage training can total 1,030 epochs, which matters when comparing encoder budgets.

Original figure
Figure 5: Monte Carlo PMI convergence versus number of Monte Carlo Integration Steps, measured as the MSE between the PMI when estimated with NN steps and the PMI when estimated with 4096 steps. Pixel space prediction, default configuration RePR-173M. Furthermore, sharing the noise among different likelihood calculations strongly reduces variance. We empirically choose N=32N=32 steps at which diffusion integration uses 1.3%1.3\% of total flops during training.
Table 5: Reconstruction performance for RePR-B Iter 3, 600ep, using the decoder configuration from RAE (Zheng et al., 2026).
Model rFID Reconstruction PSNR
DINOv2-B (Oquab et al., 2024) 0.49 18.09 dB
RePR-B Iter. 3 600ep (Ours) 0.47 21.72 dB

A.3 Implementation Details

Reported training uses batch size 2,048, AdamW, base learning rate 0.00024 with width-dependent parameterization, no weight decay or dropout, and gradient clipping at 0.2. The representation-loss coefficient is 0.005. The prediction target standard deviation is scaled to 0.5 for feature prediction; pixel and feature stages use different logit-normal time means. The practical weighting is proportional to (1−t)/max(0.33,t)², with thirty-two noise samples. Effective epochs count image views relative to ImageNet size, not simply optimizer steps.

All model, masking, and generation settings are retained in the original tables. Guidance uses the specified time interval, classifier-free scale, and autoguidance settings; these are part of the reported best generation configuration. The text-to-image setup includes a pretrained FLAN-T5-large text encoder. Source HTML misrenders several configuration and numerical tables as positioned spans, so native PDF vector table bodies replace those broken conversions while retaining captions and anchors.

A.4 Properties of PMI

The appendix discusses idealized reconstruction from prediction-based information. Its argument assumes uncorrupted conditioning observations, support for the relevant pixel values, arbitrarily small noise, and access to an idealized search over possible target values. It motivates preserving predictable information but does not prove lossless finite-dimensional practical encodings under masking and imperfect optimization. Pointwise-information matrices can be indefinite, and weighted prediction information can be asymmetric; role-dependent factors address this expressivity issue rather than making a shared symmetric dot-product matrix universally sufficient.

Table 6: Iteration 1 (Pixel) RePR configurations on ImageNet 256×\times256. Joint MAR and RePR steps (Alg. 1) and MAR-only steps (Eq 16) are interleaved with a 1:3 ratio, respectively.
Complete original table 6 config including all rows, columns, and original emphasis.
Table 7: Latent Forcing (Baade et al., 2026) decoder training at 256×\times256 resolution with 16×\times16 image patches. JiT-B used for scaling laws, JiT-L used for baseline comparisons.
config ImageNet ImageNet GPIC
decoder JiT-B (Li and He, 2026) JiT-L (Li and He, 2026) MiniT2I (Wang et al., 2026)
hidden dim 768 1024 768
backbone depth 12 20 17
output experts Baade et al. (2026) 2 ×\times 1024 dim 2 ×\times 1024 dim -
text encoder - - FLAN-T5-large
max text tokens - - 128
epochs 80
global batch size 1024
optimizer AdamW
peak lr 2×10−42\times 10^{-4}
lr schedule constant after warmup
warmup epochs 5
weight decay 0.0
pixel time μ\mu 0.40.4
pixel time σ\sigma 0.8
latent time μ\mu 1.61.6
latent time σ\sigma 1.0
latent selection probability 0.5
latent loss weight 1.0
tmint_{\min} (Li and He, 2026) 0.05
latent normalization μ=0,σ2=0.52\mu=0,\sigma^{2}=0.5^{2}

A.5 Qualitative Results

The retained image grids show generated samples from the reported representation-conditioned models. Latent Forcing jointly models feature and pixel trajectories, avoiding a fixed lossy image decoder in this evaluation. Its pixel-space generation path does not imply that the representation alone exactly reconstructs every input image.

Refer to caption
Figure 6: Uncurated class-conditional Imagenet Generations, from our compute optimal JiT-L.
Original figure
Figure 7: Feature cosine similarity between patch features and a center patch feature shown in red, with and without spatial equivariance (Sec 5.2).

A.6 Watermarking

Each image receives a unique synthetic watermark repeated within every image patch, creating an easy identity signal across masked views. The figure preserves the construction and examples. This deliberately strong shortcut makes the suppression experiment interpretable but differs from uncontrolled naturally occurring correlations.

Original figure
Figure 8: Example Clean and Watermarked images from Imagenet, used in Sec. 6.1. We follow the watermarking strategy from Enigma (2026) exactly, except for that we shrink watermarks from 32×3232\times 32 to 16×1616\times 16 pixels, so that every individual patch provides sufficient signal for suppression.

A.7 Raw Numbers for Scaling Plots

The complete raw ImageNet and GPIC tables retain every listed width, parameter count, stage length, compute value, and quality score. They expose saturation, overfitting, and small nonmonotonic changes that fitted scaling curves can hide. The comparison to larger pretrained encoders uses extrapolation to their scores; it should be read alongside the measured points and the approximate FLOP accounting.

Table 8: ImageNet scaling result tables, evaluated using FID-10K after 80 epochs of generative model training. Iteration 2 epoch counts include 210 epochs of Iteration 1 training (200 epochs + 10 epochs of learning rate decay). The DINOv2 baseline obtains 17.95 FID at 1.1685129×10221.1685129\times 10^{22} encoder-training FLOPs.
Iteration 1: Pixel Iteration 2: Latent
Encoder (width) Epochs FID Epochs FID
20M (256) 110 41.61 210+110210+110 36.24
210 40.19 210+210210+210 34.89
410 37.01 – –
610 37.21 – –
44M (384) 110 40.93 210+110210+110 31.64
210 38.20 210+210210+210 29.48
410 36.07 210+410210+410 29.22
610 34.82 – –
78M (512) 110 38.26 210+110210+110 29.81
210 35.72 210+210210+210 27.80
410 34.31 210+410210+410 27.36
610 34.00 – –
173M (768) 110 37.22 210+210210+210 25.82
210 33.73 210+410210+410 24.73
410 31.27 – –
610 30.68 – –
306M (1024) 110 36.14 – –
211 31.23 – –
Table 9: GPIC scaling result tables, evaluated using FD-DINOv2-10K after 1 GPIC epoch of generative model training. For consistency between approaches, epochs are measured as effective epochs on ImageNet. For reference, 80 ImageNet epochs is roughly equal to 1 GPIC epoch. The WebSSL-DINO-1B reference obtains an FD-10K of 277.1 at 9.78×10219.78\times 10^{21} encoder-training FLOPs. MiniT2I obtains an FD-10K of 546.3.
Iteration 1: Pixel Iteration 2: Latent
Encoder (width) ImageNet Epochs-Worth FD ImageNet Epochs-Worth FD
44M (384) 110 514 210+110210+110 416
210 479 210+210210+210 393
410 456 – –
78M (512) 110 498 210+110210+110 384
210 463 210+210210+210 365
410 443 – –
173M (768) 110 450 210+110210+110 359
210 423 210+210210+210 354
410 409 – –
306M (1024) 110 451 210+110210+110 347
210 404 210+210210+210 325
410 381 – –
Concept demonstration

Open image ↗