Literature
Display
← All papers

Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation

Alan Baade · Eric Ryan Chan · Kyle Sargent · Changan Chen · Justin Johnson · Ehsan Adeli · Li Fei-Fei

Preprint posted · arXiv (preprint) ↗

Computer vision8 min text · CondensedOriginal paper ↗
Contents

At a glance

Latent Forcing jointly denoises raw image pixels and pretrained visual features, giving each stream its own noise level. Generating the latent structure first provides an intermediate representation for subsequent pixel synthesis, while the final image comes directly from the pixel stream. On ImageNet at 256 × 256, this improves the tested pixel-transformer baselines at comparable model scale and training duration. It removes a separately trained image decoder but still relies on a pretrained feature encoder for training targets and, in guided sampling, an additional guidance model.

1 Introduction

Latent diffusion makes image generation easier by modeling a compressed representation, but decoding can lose fine detail. Pure pixel diffusion avoids that reconstruction bottleneck yet follows the information ordering induced by noisy pixels. Latent Forcing separates these concerns: learn semantic structure early, then generate pixels without discarding original image information. Its claim is about useful generation order at the tested scale, rather than universal superiority to all latent or pixel architectures.

Original figure
Figure 1: Conceptual Diagram. We diffuse both latents and pixels together, each with their own time variable. This allows us to denoise an easier trajectory than pure pixel diffusion by diffusing latents first, leading to improved performance.

2 Related Work

The method builds on flow-based diffusion, latent-image models, and recent pixel transformers. JiT supplies direct clean-image prediction; REPA is the comparison for aligning diffusion representations with pretrained features. Unlike a separate latent decoder, the proposed model jointly predicts latent features and pixels. Unlike temporal diffusion forcing, its two streams describe the same image rather than an autoregressive sequence.

Related work also motivates noise scheduling and representation-conditioned generation. These connections frame the central experiment: whether ordering pretrained structure before pixels helps beyond merely adding a representation-alignment loss.

3 Ordering the Diffusion Process

Each modality receives its own time variable. Training can cover different pairs of noise levels, and inference follows a chosen monotone trajectory through that joint space.

3.1 Flow-Based Diffusion Review

The flow convention moves from Gaussian noise at time zero to clean data at time one. The model predicts the clean input and is trained with velocity-loss weighting, following JiT, rather than directly predicting a high-dimensional noisy target.

3.2 Flow on Multiple Tokenizers

A shared model predicts outputs for each noisy modality, with separate loss weights and possibly correlated noise times. At inference, modality-specific schedules determine how much each stream advances during a numerical integration step. Their relative signal-to-noise ratios describe the order in which information becomes available. Since the latent is a deterministic function of the image, keeping raw pixels in the modeled pair avoids a lossy image-reconstruction bottleneck; the paper expresses this through joint-distribution factorization, not an experimentally measured exact image likelihood.

Equations 1–3
ℒ=∑i=1kλi​𝔼​‖𝐯θ,i​(𝐳1,t1,…,𝐳k,tk,t1,…,tk)−(𝐱i−ϵi)‖2\mathcal{L}=\sum_{i=1}^{k}\lambda_{i}\mathbb{E}\|\mathbf{v}_{\theta,i}(\mathbf{z}_{1,t_{1}},\dots,\mathbf{z}_{k,t_{k}},t_{1},\dots,t_{k})-(\mathbf{x}_{i}-\boldsymbol{\epsilon}_{i})\|^{2} (1)
𝐳i,fi​(s)=𝐳i,fi​(t)+(fi​(s)−fi​(t))⋅𝐯θ,i​(⋅)\mathbf{z}_{i,f_{i}(s)}=\mathbf{z}_{i,f_{i}(t)}+(f_{i}(s)-f_{i}(t))\cdot\mathbf{v}_{\theta,i}(\cdot) (2)
𝒪⁡(tglobal)=(fi​(tglobal)2​𝕍​[𝐱i](1−fi​(tglobal))2)i=1k\mathcal{O}(t_{\text{global}})=\left(\frac{f_{i}(t_{\text{global}})^{2}\mathbb{V}[\mathbf{x}_{i}]}{(1-f_{i}(t_{\text{global}}))^{2}}\right)_{i=1}^{k} (3)

3.3 Scaling as Time Scheduling

Scaling a modality changes its signal-to-noise ratio, which is equivalent to shifting its noise schedule. Combining differently scaled token spaces therefore chooses an implicit generation order even with a shared time variable. The retained time-shift equation makes this relation explicit.

Equation 4
fα​-shift​(t)=t​α1+(α−1)​tf_{\alpha\text{-shift}}(t)=\frac{t\alpha}{1+(\alpha-1)t} (4)

4 Latent Forcing

Experiments instantiate the method as an ImageNet pixel diffusion transformer, principally using DINOv2 latents and 256 × 256 images.

4.1 Tokenization

Pixels become 256 patches of dimension 768. Spatially aligned DINOv2 features, Data2Vec2 features, or lower-resolution pixels form the second stream. Pixel values are normalized to −1 through 1, and latent variance is matched to them. Sampling discards the generated features and rescales the predicted pixels, requiring no learned image decoder.

The pretrained feature extractor is a source of training targets and prior visual knowledge. Calling inference end to end does not mean the feature representations were learned from scratch jointly with this generator.

4.2 Architecture

Separate time embeddings and input/output projections accommodate the two streams. Their projected patch embeddings are added, keeping the transformer token count unchanged. An optional split of the last four layers into two two-layer output branches lets latent and pixel predictions specialize without adding layers or total stated operations. The standard experimental model is a large vision transformer trained for eighty epochs; the system comparison also uses two hundred epochs.

Original figure
Figure 2: Model Architecture. Our approach makes minimal changes to diffusion transformers. Left: We add the per-patch embeddings of latents and pixels together, keeping the same number of tokens. Middle: We use two time variables instead of one for adaLN. Right: Optionally, we take the last M=4M=4 transformer layers and split them into two M/2M/2-layer output experts.

4.3 Prediction Target

The network predicts clean pixels and latents, then converts these predictions into velocity terms for the loss. Clipping the denominator near clean timesteps prevents numerical instability. The usual clipping constant is 0.05, with a different setting for the exploratory multi-schedule training.

Equation 5
ℒ=‖(𝐱pred−𝐳t)max⁡(1−t,tclip)−(𝐱−𝐳t)max⁡(1−t,tclip)‖2\mathcal{L}=\left\|\frac{(\mathbf{x}_{\text{pred}}-\mathbf{z}_{t})}{\max(1-t,t_{\text{clip}})}-\frac{(\mathbf{x}-\mathbf{z}_{t})}{\max(1-t,t_{\text{clip}})}\right\|^{2} (5)

4.4 Training Time vs Inference Time Ordering

The multi-schedule model independently samples each stream’s noise time, allowing one trained network to test different inference orders. The single-schedule model trains only around its selected trajectory, concentrating limited optimization on the intended sampling distribution. The former isolates ordering effects; the latter supplies the strongest baseline comparisons.

4.5 Multi-Schedule Diffusion

Exploratory training uses shifted uniform time sampling, a clipping constant of one third, and balanced latent/pixel loss magnitudes. Across DINOv2, Data2Vec2, and downsampled-pixel representations, denoising latent structure earlier is better than denoising full-resolution pixels earlier. For DINOv2, the unguided ten-thousand-sample distance drops from 55.35 with a strongly pixel-first shift to 18.65 with the tested latent-first shift of sixteen.

Matched-reconstruction-error examples show that latent-conditioned predictions preserve more recognizable spatial structure at early noise levels. The heat maps indicate that pixels help infer latents over much of the trajectory, while latent features add little to already mostly denoised pixels. These are diagnostic associations supporting the ordering explanation, not proof that one schedule is optimal for every dataset or model.

Original figure
Figure 3: FID-10K values for different diffusion trajectories through a joint DINOv2 and Pixel Space.
Table 1: FID scores for Multi-Schedule Models across different schedules. Schedules follow tlatent=fα​(tglobal),tpixel=tglobalt_{\text{latent}}=f_{\alpha}(t_{\text{global}}),t_{\text{pixel}}=t_{\text{global}} from Eq. 4
FID-10K w/o Guidance, per α\alpha Shift
Latent Model Pixel Earlier Eq. SNR Latent Earlier
1/64 1/16 1/4 1 4 16 64
64×6464\times 64 Pixels 44.51 44.45 44.35 44.57 44.20 42.35 42.31
Data2Vec2 55.19 50.24 38.24 27.69 24.26 23.61 24.44
DINOv2-B+Reg 55.35 50.64 37.63 24.39 18.99 18.65 18.90
Original figure
Figure 4: Output single-step 𝐱\mathbf{x}-predictions in the pixel space from the Multi-Schedule Model, where each column reconstructs with the same PSNR. Top: Predictions when pixel features are partially denoised, and DINOv2 features are fully noised. Bottom: Predictions when latent DINOv2 features are partially denoised and pixels are fully noised. At low PSNR levels, DINOv2 features preserve significantly more spatial information.
Original figure
Figure 5: PSNRs on DINOv2 and Pixel features at different timestep combinations in the Multi-Schedule Model.

5 Single-Schedule Modeling

The remaining experiments commit training and inference to a particular order, primarily a cascade that generates latent features before pixels.

5.1 Time Sampling

Training first selects a latent or pixel phase. During a latent phase, pixels remain fully noisy and only latent loss is active; during a pixel phase, the latent is clean by default and only pixel loss is active. Separate logit-normal schedules choose the within-phase noise level. Balanced loss magnitudes and the probability of selecting each phase control the effective optimization allocation.

Table 2: Ablation for the probability of sampling a latent timestep. FID-10K without guidance.
PLatentP_{\text{Latent}} 0.3 0.4 0.5 0.6 0.7 0.8
FID ↓\downarrow 12.54 12.42 12.77 13.17 13.48 16.13
Table 3: Ablation: Logit schedule mean for latent timesteps, using DINOv2. FID-10K.
μLatent\mu_{\text{Latent}} FID Unguided↓\downarrow
1.0 14.11
1.2 13.97
1.4 14.41
Table 4: Ablation: Logit schedule mean for pixel timesteps. FID-10K.
μPixel\mu_{\text{Pixel}} FID Unguided↓\downarrow
0.0 18.28
0.4 16.13
0.8 16.38

5.2 Improving Cascaded Generation

An additional early-pixel-time sample on 10% of updates improves the cascade modestly, as do separate output branches. Small noise added to latent conditioning during training addresses degradation seen with longer training: at two hundred epochs, the listed unguided score is 16.47 without this augmentation versus 10.93 with a maximum latent-noise amount of 25%. Adding noise during inference instead hurts. The interpretation is robustness to imperfect generated latents, rather than a rule to keep conditioning noisy during sampling.

Table 5: Ablations for the two modeling choices in Latent Forcing that deviate from the standard DiT pipeline. FID-10K Unguided.
Model FID Unguided ↓\downarrow
LF-DiT-L 12.42
−-Output Expert Layers 12.93
−-10% Early Timesteps 12.98
Table 6: Max noise applied to latent features during pixel timesteps vs FID-10K w/o guidance. Top: Without noise FID increases when training longer. Bottom: Adding small noise fixes this issue.
Noise-β\beta Epochs FID↓\downarrow
0% 80 13.97
0% 200 16.47
25% 80 13.48
25% 200 10.93
50% 80 14.64
Table 7: Inference noise at 200EP, when training with max 25% noise (t>0.75t>0.75). Despite noise improving cascaded generation at training, it’s harmful at inference.
Noise FID
0% 10.93
5% 11.07
10% 11.46
15% 11.81
25% 12.55

5.3 Time Schedules

The tested cascade obtains ten-thousand-sample distances of 12.42 unguided and 6.60 guided, compared with 13.48/8.16 for a variance-shift trajectory and 20.98/10.73 for the selected linear offset. All use fifty Heun integration steps; the cascade allocates twenty-five to each phase. These numbers use a different sample count from the main fifty-thousand-sample comparisons and should not be mixed directly.

Table 8: FID-10K scores for different Single-Schedule Model time schedules. A cascaded schedule performs best, and joint denoising according to a variance shift (Equation. 4) works well.
Time Schedule FID Unguided↓\downarrow FID Guided
Cascaded 12.42 6.60
Lin. Offset o=0.1o=0.1 20.98 10.73
Var. Shift α=9\alpha=9 13.48 8.16

5.4 Guidance

The paper evaluates classifier-free guidance and AutoGuidance, which uses a smaller, weaker separately trained generator. AutoGuidance is strongest in most reported experiments. For the system-level comparison only, interval-restricted classifier-free guidance is used for latent steps and AutoGuidance for pixel steps. Guided scores therefore depend on extra model training and sampling configuration, not just the principal generator.

5.5 Distillation vs Ordering

At eighty epochs, conditional fifty-thousand-sample distance is 9.76 unguided for DINOv2 Latent Forcing, versus 18.60 for JiT with representation alignment and 25.18 for JiT alone. Guided values are 4.18, 4.57, and 5.64, a smaller separation. The controlled comparison supports an ordering benefit beyond representation distillation in this setup; it does not show that every use of distillation becomes ineffective.

Table 9: FID-50K scores for conditional generation at 80 epochs.
Model FID Unguided↓\downarrow FID Guided↓\downarrow
JiT 25.18 5.64
JiT+REPA 18.60 4.57
LF-DiT DINOv2 9.76 4.18
LF-DiT D2V2 12.46 5.45
Original figure
Figure 6: Curated Results from LF-DiT-L at 200 epochs, using AutoGuidance with ω=1.5\omega=1.5.
Table 10: FID-50K for unconditional generation at 80 epochs.
Model FID Unguided↓\downarrow FID Guided↓\downarrow
JiT 53.26 44.80
JiT+REPA 35.04 24.40
LF-DiT DINOv2 20.44 13.36
LF-DiT D2V2 20.99 15.56
Table 11: System-level comparison of pixel and latent diffusion models, with latent diffusion approaches sorted by PSNR to show the information lost during encoding. FID-50K Results on ImageNet 256×\times256, (U) Unguided, (G) Guided. Citations: RAE (Zheng et al., 2025), DiT (Peebles and Xie, 2023), SD-VAE (Rombach et al., 2021b), SiT (Ma et al., 2024), REPA (Yu et al., 2025), MAR (Li et al., 2024), Detok (Yang et al., 2025), LightningDiT (Yao et al., 2025), REPA-E (Leng et al., 2025), Uniflow (Yue et al., 2025), ADM (Dhariwal and Nichol, 2021), SiD (Hoogeboom et al., 2023), SiD2 (Hoogeboom et al., 2024), JiT (Li and He, 2025).
Model Params Dec. Params Epochs PSNR↑\uparrow FID(U)↓\downarrow FID(G)↓\downarrow
Latent Diffusion
RAE 839M 415M 800 18.09 1.51 1.13
DiT-XL/2+SD-VAE 675M 49M 1400 23.40 9.62 3.04
SiT-XL/2+SD-VAE 675M 49M 1400 23.40 8.3 2.62
REPA+SD-VAE 675M 49M 800 23.40 5.9 1.42
MAR + Detok 479M 86M 800 24.06 1.86 1.35
LightningDiT 675M 41M 800 25.29 2.17 1.35
REPA-E 675M 41M 1480 26.25 1.69 1.12
MAR + UniFlow 479M 300M 400 32.48 2.45 1.85
Pixel Diffusion
ADM 554M 0 400 ∞\infty 10.94 3.94
SiD UViT/2 2B 0 - ∞\infty 2.77 2.44
SiD2 UViT/2 - 0 - ∞\infty - 1.73
ViT Pixel Diffusion
JiT-L 459M 0 200 ∞\infty 16.21 2.79
LF-DiT-L 465M 0 200 ∞\infty 7.2 2.48

5.6 Conditioning as Ordering

Unconditional models always receive the no-class label during training. DINOv2 Latent Forcing obtains 20.44 unguided and 13.36 guided, versus 35.04/24.40 for JiT with representation alignment and 53.26/44.80 for JiT. Data2Vec2 Latent Forcing also improves over the alignment baseline, despite a less extensively trained encoder. This shows the effect is not confined to one pretrained representation.

5.7 Discussion: Towards Rethinking Compression

At two hundred epochs, the 465-million-parameter model records 7.20 unguided and 2.48 guided, compared with 16.21/2.79 for 459-million-parameter JiT. Several larger, longer-trained latent models and other pixel architectures have better absolute guided scores. The supported result is improved performance among the compared plain pixel transformers at this compute scale. Infinite reconstruction signal-to-noise ratio refers to retaining the original pixels as input data, not perfect generated images. The retained system table separates decoder size, training duration, reconstruction quality, and generation distance.

6 Conclusion

Making generation order explicit improves the tested pixel-transformer models while retaining an uncompressed pixel output. The study is confined to ImageNet-256 and the reported architectures, schedules, and budgets; scaling to broader image distributions, text conditioning, or higher resolutions is not demonstrated. The main practical tradeoff is a simpler image-output path with additional latent targets, schedule choices, and guidance-model tuning.

7 Acknowledgments

The authors acknowledge National Science Foundation graduate fellowship support and discussion with Yue Zhao.

Appendix A Appendix

The appendix records architecture, optimizer and sampling settings, then details how guidance models and intervals were selected.

A.1 Implementation Details

The full configuration table is retained. The large model has twenty-four layers and hidden dimension 1,024; training uses Adam, batch size 1,024, learning rate 0.0002, five warmup epochs, and eighty or two hundred total epochs. Pixel and latent bottlenecks are 128-dimensional. The table separately specifies feature normalization, source encoders, loss weights, time samplers, and the fifty-step Heun sampler.

Table 12: Configuration of Latent Forcing.
ViT-S ViT-B ViT-L
Architecture
Depth 12 12 24
Hidden Dim 384 768 1024
Heads 6 12 16
Patch Size 16
Pixel Bottleneck 128
Latent Bottleneck 128
Dropout 0
In-context CLS (Li and He, 2025) 32
In-context Start Block 4
Training
Epochs 80 (ablation), 200
Warmup Epochs 5
Optimizer Adam (Kingma and Ba, 2014)
Adam Betas β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
Batch Size 1024
Learning Rate 2e-4
Learning Rate Schedule constant
Weight Decay 0
EMA Decay 0.9999
Latent Sampler μ=−1.2,σ=1.0\mu=-1.2,\sigma=1.0
Pixel Sampler μ=−0.8,σ=0.8\mu=-0.8,\sigma=0.8
Class Token Drop 0.1
Sampling
ODE Solver Heun
ODE Steps Total 50
ODE Steps Pixel (Cascaded) 25
ODE Steps Latent (Cascaded) 25
Global Time Steps linear in [0.0, 1.0]
Pixels
Normalization [−1,1][-1,1] Linear Min-Max Rescaling
Standard Deviation (For Rescaling Sec 3.2) 0.485
Loss Weight λPixel\lambda_{\text{Pixel}} (Eq. 1) 1.01.0
DINOv2 Latents
Normalization Global norm to μ=0,σ2=1\mu=0,\sigma^{2}=1 per each 16×16×76816\times 16\times 768 float, Following RAE (Zheng et al., 2025)
Model DinoV2-Base + Registers (Oquab et al., 2023; Darcet et al., 2023)
Loss Weight λLatent\lambda_{\text{Latent}} (Eq. 1) 0.3330.333
Layer 12, Pre Norm, Following RAE (Zheng et al., 2025)
Data2Vec2 Latents
Normalization Global norm to μ=0,σ2=1\mu=0,\sigma^{2}=1 per each channel
Model Data2Vec2-Large, 150ep (Baevski et al., 2023)
Loss Weight λLatent\lambda_{\text{Latent}} (Eq. 1) 0.250.25
Layer 12

A.2 AutoGuidance

Four smaller guidance architectures are trained and checkpoint-swept. A small transformer with large-model output branches works best in the reported setting; its selected checkpoint is at epoch forty with exponential-moving-average decay 0.9995. Candidate checkpoints use two-thousand-sample evaluation and guidance schedules use eight-thousand-sample sweeps. AutoGuidance is also tested for JiT baselines but does not outperform their classifier-free guidance. This selection cost is additional to training the main model.

A.3 CFG on a Limited Interval

Only the system-level comparison uses interval guidance. Latent steps use classifier-free guidance strength 3 over the interval 0.06–1 and a time-shift factor 0.575; pixel steps use AutoGuidance strength 1.5 without an interval. Other reported experiments use AutoGuidance without interval restriction. Distinguishing these settings is necessary when comparing tables.

Concept demonstration

Open image ↗