Literature
Display
← All papers

A Language-Informed Loss for Visual Generation

Suchir Agarwal · Keshigeyan Chandrasegaran · Ziyu Chen · Kyle Sargent · Juan Carlos Niebles · Li Fei-Fei · Jiajun Wu

Preprint posted · arXiv (preprint) ↗

Computer vision12 min text · CondensedOriginal paper ↗
Contents

Overview

LILIE learns an invertible image transform by captioning noise-corrupted features, then freezes the transform to provide an additional loss for pixel-space image generation.

Refer to caption
Figure 1: Generation quality with LILIE. Upper left: Pixel and LILIE supervision at two generator sizes evaluated on GPIC Test-50K with 50 Euler steps. CFG scale is selected using GPIC Validation-50K. All models are trained for one epoch on GPIC Full (Chandrasegaran et al., 2026). Mosaic: selected ×256256\!\times\!256 samples from the 1.1B diffusion model with LILIE supervision. Four samples include prompt excerpts. We include randomly selected samples from GPIC Test-50K in Appendix G.

At a glance

Language-Informed and Lossless Image Encoder learns a bijection with the same dimensionality as its image input. A vision–language model supplies caption-prediction supervision during encoder training; the frozen encoder then measures errors between a generator’s image prediction and its target. On GPIC at 256-by-256 resolution, the 1.1-billion-parameter generator improves its DINOv2-feature Fréchet distance from 66.84 to 53.01 when adding LILIE to pixel loss. These results concern one dataset, resolution, and generator family; an invertible representation is not itself a guarantee of semantic correctness or a lossless image-compression system.

1 Introduction

Pixelwise squared error assigns equal cost to perturbations with different semantic effects. LILIE learns a different geometry: caption-relevant information must remain recoverable after noise is added to an invertible image transform. The encoder keeps all image dimensions, while caption prediction shapes which errors become costly. The downstream generator still predicts pixels; LILIE is a training loss and does not become a mandatory decoding stage at sampling time.

Refer to caption
Figure 2: Two perturbed image patches (additive Gaussian noise) with equal pixel Euclidean distance of 1000 from the original patches. Caption: “Three airplanes fly in the sky against a cloudy backdrop.”
Refer to caption
Figure 3: Training and using LILIE. (a) We jointly train LILIE and VLM LoRA parameters on noise-corrupted features with next token prediction. (b) The diffusion model trains on Pixel and LILIE L2L^{2} losses, comparing its prediction with the target image in pixel space and through the learned encoder.

2 Related Work

The work connects pixel-space diffusion, neural perceptual losses, invertible architectures, and caption-supervised representation learning.

Pixel-space diffusion.

Pixel-space generators avoid a separate lossy image tokenizer. LILIE builds on JiT and keeps the generator architecture fixed while changing its training signal, rather than attributing improvements to a new sampling or denoising architecture.

Feature-space losses for generation.

Perceptual losses compare predicted and target images through pretrained features. PixelGen combines such features with pixel-space supervision, while representation-alignment methods supervise hidden states. LILIE learns an invertible feature space specifically for the output-image loss; it is distinct from aligning the denoiser’s internal activations.

Invertible networks.

The encoder uses the invertible autoregressive transformer design from BiFlow. Its architecture is a normalizing-flow-style bijection, but its objective is caption cross-entropy, not a likelihood or density-modeling objective.

Language-supervised visual representations.

Language-supervised vision methods use contrastive alignment, caption prediction, or both. LILIE uses caption cross-entropy as its only encoder-training loss. Its supervision still depends on a pretrained vision–language model; “language-only” describes the new training objective, not the absence of externally pretrained components.

3 Methodology

Training has two stages. First, jointly update an invertible image encoder and adapters in a captioning model using noisy encoded images. Then discard the captioning model, freeze the encoder, and use feature-space squared error alone or alongside pixel loss to train a text-conditioned generator.

3.1 LILIE: A Language-Informed and Lossless Image Encoder

LILIE maps an image to an equally shaped tensor using invertible affine blocks. Corruption encourages caption-relevant content to remain recoverable; invertibility prevents the underlying encoder from discarding dimensions of the image.

Learning an invertible map.

Each encoded training image is normalized to root-mean-square magnitude one-half, corrupted by Gaussian noise, and captioned by a LoRA-adapted SmolVLM. Caption cross-entropy updates the encoder and LoRA weights. Per-image normalization prevents the encoder from defeating noise by simply scaling up all outputs. A moving estimate of the raw features’ second moment is recorded for later use. The noisy normalized captioning input is not itself invertible; the underlying encoder is, and downstream scaling uses a fixed corpus statistic to preserve that property.

Refer to caption
Figure 4: LILIE training diagram. We expand on Figure 3(a) to include TARFlow internals. We provide additional details on affine coupling and transformer internals in Appendix A.1.
Algorithm 1 LILIE training.

Input: Image-caption batches data, initial invertible encode, pretrained vlm with LoRA, noise level sigma, EMA decay beta.

Output: Trained encode and second moment m2.

Original PDF algorithm, preserving indentation and colored syntax.
Algorithm 2 JiT training loss.

Input: Frozen encode and m2 from Algorithm 1; target x, prediction x_pred, time weight w(t), loss weight lambda.

Output: Generator training loss.

Original PDF algorithm, preserving indentation and colored syntax.

3.2 Training a T2I Generative Model with LILIE

The frozen encoder supplies an image comparison loss for a generator that continues to predict clean pixels.

Base objective.

JiT-T2I conditions on a frozen Qwen3-1.7B text encoder and trains with its clean-image form of the flow-matching objective. Captions are dropped for ten percent of training examples to support classifier-free guidance. LILIE does not remove the generator’s language-conditioning model.

Equation
ℒPixel=𝔼⁡[w⁡(t)​‖x−x^‖22],w⁡(t)=1max⁡(1−t,tclip)2,\mathcal{L}_{\text{Pixel}}=\mathbb{E}[w(t)\|x-\hat{x}\|_{2}^{2}],\quad w(t)=\frac{1}{\max(1-t,t_{\text{clip}})^{2}},

Augmenting with LILIE.

For downstream loss, encoder outputs are divided by twice the square root of the stored corpus second moment, a fixed nonzero rescaling. The target and predicted image are compared in that space with the same time-dependent weighting as the base objective. The paper tests a LILIE-only objective and pixel loss plus a weighted LILIE term. Exact feature equality implies exact input equality under the bijection, but finite training errors and numerical conditioning still matter.

Equations 1
ℰ~​(x)≜ℰθ​(x)2​m2,\tilde{\mathcal{E}}(x)\triangleq\frac{\mathcal{E}_{\theta}(x)}{2\sqrt{m_{2}}}, (1)
ℒLILIE=𝔼⁡[w⁡(t)​‖ℰ~​(x)−ℰ~​(x^)‖22],ℒPixel+LILIE=ℒPixel+λ​ℒLILIE,\mathcal{L}_{\text{LILIE{}}}=\mathbb{E}[w(t)\|\tilde{\mathcal{E}}(x)-\tilde{\mathcal{E}}(\hat{x})\|_{2}^{2}],\qquad\mathcal{L}_{\text{Pixel+LILIE{}}}=\mathcal{L}_{\text{Pixel}}+\lambda\mathcal{L}_{\text{LILIE{}}},

4 Experiments

Experiments compare loss choices while holding each generator’s architecture and training schedule fixed. Encoder pretraining adds a separate cost beyond the generator training budget.

Table 1: Generation results on GPIC. We compare generation quality achieved by JiT-T2I-XL (700M) and JiT-T2I-g (1.1B) models trained with different loss functions. Models train for 390,600 steps, and we report results at 781,200 steps for selected configurations. We compare Pixel, LILIE-only, and Pixel+LILIE losses at the 700M generator size. At 1.1B, we compare (i) Pixel loss; (ii) standalone feature-space losses with DINOv2, SigLIP 2, LPIPS (VGG16), and LILIE backbones; (iii) feature-space losses complementary to Pixel loss. Lower is better for gFD, higher is better for precision, recall, coverage, and values near 1 are best for density. We bold rows achieving lowest gFD within each generator size and training budget. “GPIC compliant” indicates compliance with the GPIC evaluation protocol. We note that the DINOv2 and SigLIP 2 models used for feature-space losses are both distilled from larger models, while VGG16 and LILIE are not distilled. DINOv2, SigLIP 2, and LPIPS losses result in degenerate generation quality when used without a Pixel loss term. Key result: LILIE loss yields competitive quality for end-to-end visual generation, both when used as a standalone loss and as a complement to Pixel loss. Table B.1 includes selected CFG values and gFD decomposition.
Original PDF numerical table, preserving every row and column.

4.1 Experimental Setup

The setup controls the downstream dataset and generator schedule, tunes loss/noise settings on validation data, and reports test metrics with independently selected guidance scales.

Data.

Encoder and generator training use GPIC image–caption pairs at 256-by-256 resolution. A full epoch has one hundred million examples and about 390,600 steps at batch 256. Caption lengths fall into tag, short, medium, and long categories, each with a corresponding encoder-training prompt.

Encoders.

The default 32-million-parameter LILIE-S has four affine blocks with four transformer layers each, width 384, and patch size sixteen. SmolVLM-256M uses rank-sixty-four LoRA adapters. Encoder training reportedly takes about forty hours on TPU v5e-32; this cost is separate from training the generator.

Generative models.

The two JiT-T2I generators have approximately 700 million and 1.1 billion parameters and share a frozen Qwen3-1.7B text encoder. Main results use one GPIC epoch, selected longer runs use two, and ablations use two hundred thousand steps. Reported generator times are twenty-seven and forty hours on TPU v6e-16, respectively.

Perceptual loss baselines.

Baselines use one feature loss at a time from VGG16 LPIPS, DINOv2, or SigLIP 2, with representation alignment disabled. Combined objectives use the paper’s specified perceptual metrics and time gate; feature-only experiments remove pixel loss and the gate and use squared feature distance for DINOv2/SigLIP 2. Thus the feature-only failures should be interpreted under these particular implementations.

Evaluation.

Quality, precision, recall, density, coverage, and squared maximum mean discrepancy are measured in DINOv2 space. Guidance is chosen separately for each model on Validation-50K, then evaluated on Test-50K with fifty Euler steps unless varied. The Fréchet metric measures distributional feature similarity and is not a direct test of prompt correctness, human preference, or every visual detail.

Hyperparameter selection.

A validation sweep tests five encoder-noise levels and three mixed-loss weights, plus feature-only loss, using the smaller generator at two hundred thousand steps. Noise standard deviation 0.25 and LILIE weight two are selected for the main runs. The main conclusions depend on this tuning rather than on an arbitrary noise scale.

4.2 Generation Results

LILIE improves distributional generation metrics over pixel-only training, while separate alignment benchmarks and perceptual-loss comparisons provide a more qualified picture.

Original figure
Figure 5: gFD over prompt length. We take three sets of 50K prompts from GPIC Test, each containing only short, medium, or long captions. We then sample from the Pixel, LILIE, and Pixel+LILIE JiT-T2I-g models at their respective optimal CFG scales (Table B.1). We report gFD over the GPIC Test set. Our optimal Pixel+LILIE configuration outperforms the Pixel baseline at every prompt length.

LILIE improves generation quality.

Adding LILIE improves test Fréchet distance from 80.24 to 65.86 for the 700-million-parameter generator and from 66.84 to 53.01 for the larger model, alongside improved precision and recall. For these one-epoch pixel and mixed configurations, the appendix reports three trials: the larger-model values are 66.84±1.72 versus 53.01±0.27, using sample standard deviations. Most other comparisons are single runs.

At two epochs, the larger model improves from 49.20 to 40.18. LILIE is competitive with, but does not dominate, all perceptual losses: pixel+LPIPS reaches 48.65 at one epoch and 39.35 at two, while pixel+DINOv2 reaches 47.88 and 38.29 under different pretraining/protocol conditions.

LILIE loss can be used on its own.

LILIE alone reaches 68.88 and 55.78 at the two model sizes, improving over pixel-only training. The tested DINOv2- and SigLIP-only losses give severely degenerate distributions with zero recall, while LPIPS alone is also much worse at 158.45. The invertible feature map gives a uniquely identifying target when matched exactly, although that mathematical property alone does not establish why every finite optimization run will succeed.

Original figure
Figure 6: Effect of informative captions. gFD on GPIC Test-50K for JiT-T2I-XL with Pixel or Pixel+LILIE loss. “Random” indicates replacing ground-truth captions with random token sequences during LILIE training. We observe that this prevents LILIE from learning a representation that improves generation quality.

Language supervision matters.

Randomizing encoder-training caption tokens while preserving lengths and prompts gives downstream Fréchet distance 82.06, versus 80.24 for pixels and 65.86 for real-caption LILIE. The generator still trains on genuine captions. This supports the role of informative language in learning the loss rather than an effect from simply adding an invertible network.

4.3 Sampling Steps

At the tested sampling budgets, pixel+LILIE has lower Fréchet distance than pixels. Twenty-five mixed-loss steps reach 58.23, below the pixel model’s fifty-step 68.67 in this sweep. These sweep values differ from the main table’s three-trial means. Gains beyond fifty steps are small; LILIE-only actually worsens from 55.78 to 56.63 at seventy-five steps. Guidance chosen at fifty steps is held fixed, so the sweep does not retune every method at every budget.

Refer to caption
Figure 7: Effect of Euler sampling steps. Left: JiT-T2I-g gFD on GPIC Test-50K at 390,600 training steps, with each model’s CFG scale chosen at 50 steps and held constant. Right: three selected qualitative examples with Euler steps increasing across columns and Pixel, LILIE, and Pixel+LILIE down rows. Prompts and initial noise are shared across methods and step counts.

4.4 Encoder Size and Data Scaling

The size/data ablations keep generator training fixed at two hundred thousand steps and vary the separately trained encoder. More encoder capacity or training data generally improves Fréchet distance within the tested range, with smaller metric-specific exceptions.

Refer to caption
Figure 8: Qualitative samples over LILIE model size and training data. Selected qualitative JiT-T2I-g samples from GPIC use the same prompt and initial noise in each column. The highlighted row is LILIE-S/100M, our main configuration. Model size decreases above it, and training data decreases below it. All generators are trained for 200K steps.
Table 2: Model size ablation. gFD ↓\downarrow at 200K generator steps. All LILIE encoders train on 100M images. The default LILIE-S encoder row is highlighted.
Encoder Params JiT-T2I-XL JiT-T2I-g
Pixel – 114.59 101.66
LILIE-μ\mu 1.5M 96.11 84.66
LILIE-T 9.1M 92.66 80.87
LILIE-S 32M 91.79 76.59
Table 3: Training data ablation. gFD ↓\downarrow at 200K generator steps. All encoders are LILIE-S. The default 100M-image training budget row is highlighted.
# Training Images JiT-T2I-XL JiT-T2I-g
Pixel 114.59 101.66
10M 97.79 83.01
30M 93.52 81.18
100M 91.79 76.59

Model size.

Micro, tiny, and small encoders have 1.5, 9.1, and 32 million parameters. With the larger generator, their Fréchet distances are 84.66, 80.87, and 76.59, compared with pixel-only 101.66. Even the micro encoder helps. These values come from shorter generator training and should not be compared directly with the main full-epoch scores.

Training data.

Training LILIE-S on ten, thirty, and one hundred million images reduces the larger generator’s Fréchet distance from 83.01 to 81.18 to 76.59; the smaller generator improves from 97.79 to 93.52 to 91.79. Increased encoder training therefore helps at a fixed generator budget, but increases total system training compute.

5 Discussion and Conclusion

The learned feature loss amplifies some small near-data perturbations: at interpolation time 0.99, the reported residual standard deviation is about 4.9 times that in pixel space despite similar overall feature scale. This describes a measured sensitivity of the learned representation, not a universal guarantee that every amplified error is semantically important.

Original figure
Figure 9: LILIE provides a stronger training signal closer to the data distribution. Average standard deviation ratios for LILIE versus pixel values xtx_{t}, and residuals from clean images xx, using the LILIE-S encoder on GPIC Nano (1M train images). xtx_{t} denotes linear interpolation between pure noise (t=0t=0) and real data (t=1t=1).

Comparisons with pretrained perceptual losses.

Combined perceptual losses are strong competitors, and LILIE is not uniformly the best metric score. DINOv2 and SigLIP 2 have different, much larger pretraining distributions that may overlap GPIC test data; DINOv2 is also the evaluation feature extractor. The authors label these useful representation comparisons but not controlled GPIC-protocol-compliant experiments. LPIPS remains a protocol-compliant strong baseline. Combining LILIE with multiple perceptual losses is left untested.

Table 4: Comparison of feature-space losses. “GPIC compliant” indicates compliance with the GPIC evaluation protocol. “SSL” refers to self-supervised learning, and “Language” refers to language-supervised learning. The VGG16 backbone is label-supervised, trained for ImageNet classification. SigLIP 2 is trained with both language-supervised and SSL objectives. Our method is only supervised by language.
Loss Supervised SSL Language Training images Not distilled GPIC compliant
DINOv2 - ✓ - 142M ✗ ✗
SigLIP 2 - ✓ ✓ 10B ✗ ✗
LPIPS (VGG16) ✓ - - 1.28M ✓ ✓
LILIE - - ✓ 100M ✓ ✓

A simple language-informed loss for visual generation.

A compact invertible encoder trained through language can provide useful supervision for pixel-space generation, either alone or with pixel loss. The reported gain does not require the encoder or captioning model during generator sampling. Released code, encoders, and generator weights are referenced in the original paper.

Acknowledgements.

The authors acknowledge feedback from Stanford Vision Lab and the CogAI group.

Appendix

The appendix retains all training configurations, supplementary metrics, noise/loss sweeps, limitations, exact captioning prompts, and complete qualitative sample grids.

Appendix A Additional Training Details

Implementation details explain preprocessing, noisy vision–language inputs, and the invertible transformer architecture.

preprocess in Algorithm 1 .

Images are resized with antialiased bicubic interpolation so the shorter side is 256 pixels, center-cropped to 256-by-256, and mapped from byte intensities to the interval minus one to one. These preprocessing operations define the image domain being modeled; the claim of invertibility concerns the subsequent encoder, not recovery of discarded crop or resize information.

SmolVLM input.

Noise-corrupted features replace image input to SmolVLM and are not clipped to the usual image-value interval. LoRA adaptation lets the captioning model handle these inputs. Clipping would discard information and change the training procedure.

A.1 Encoder Architecture

Images are patchified and passed through a stack of autoregressive affine blocks with alternating patch orders, then reassembled to the original tensor shape. The original architectural diagram retains all scale/shift operations and transformer internals.

Block parameters.

For each patch, the affine log-scale and shift depend only on preceding patches in the block’s order and are clipped to minus one through one. The first patch passes through unchanged. This causal structure supports inversion; it does not make the transform a dimensionality-reducing tokenizer.

Transformer internals.

Each block projects patches to the bottleneck width, adds positional embeddings, and applies causal transformer layers with width divided by sixty-four attention heads and a fourfold feedforward expansion. A zero-initialized final head produces scale and shift values. Dropping its last position and prepending zeros ensures that each patch’s transformation uses only earlier patches.

Refer to caption
Figure A.1: Encoder implementation details. (a) The block stack and autoregressive affine map. The tuple γt=(αt,μt)\gamma_{t}=(\alpha_{t},\mu_{t}) contains the log-scale and shift of patch tt. (b) The causal transformer within one block, with bottleneck dimension HH and LL layers. LN denotes layer normalization. The lower row expands one transformer layer.

Appendix B Experimental Details

The appendix retains full generation metrics and guidance choices, encoder/generator configurations, and training curves. Three-trial means and sample standard deviations apply to the one-epoch pixel and pixel+LILIE settings at both sizes, rather than every table entry. Main runs use 390,600 steps; selected two-epoch runs use 781,200.

Refer to caption
Figure B.1: Left: Training loss for LILIE-S over one epoch on GPIC Full. Right: Training loss for JiT-T2I-XL and -g trained with Pixel+LILIE loss (λ=2\lambda=2) over one epoch on GPIC Full.
Table B.1: Full generation results on GPIC Test-50K. Models train for one epoch on GPIC-Full (390,600 steps), and we train selected configurations for two epochs (781,200 steps). All results use 50 Euler steps. CFG is selected on GPIC Validation-50K at 0.05 granularity. For JiT-T2I-g, LPIPS, DINOv2, and SigLIP 2 denote perceptual losses in the corresponding feature space, used alone and combined with the pixel loss. Despite the large compute cost associated with each pretraining run (Section 4), we perform three one-epoch pretraining trials for Pixel and Pixel+LILIE at both generator sizes (LILIE seeds 100,101,102, JiT-T2I seeds 0,1,2). For these one-epoch configurations, we report mean ±\pm one sample standard deviation for each metric over the three trials. gFD is decomposed into mean (gFDμ\mathrm{gFD}_{\mu}) and covariance (gFDΣ\mathrm{gFD}_{\Sigma}) components. Precision, recall, density, and coverage measure sample fidelity and diversity. MMD is reported in raw squared units.
Original PDF numerical table, preserving every row and column.
Table B.2: LILIE configurations.
LILIE-μ\mu LILIE-T LILIE-S
architecture
flow backbone Improved TARFlow (Zhai et al., 2025; Lu et al., 2026b)
blocks KK 2 4 4
layers per block LL 2 4 4
bottleneck dim HH 128 192 384
heads 2 3 6
parameters 1.5M 9.1M 32M
image size 256
patch size 16
coupling clip range 1.0
VLM SmolVLM-256M (Marafioti et al., 2025)
LoRA (Hu et al., 2022) rank 64
LoRA α\alpha 32
LoRA targets all linear layers except the output head
max caption tokens 256
training
training steps 390,600 (1 epoch on GPIC Full)
warmup steps 20,000
optimizer AdamW (Loshchilov and Hutter, 2019), β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
batch size 256
flow learning rate 1.2e-3 8e-4 4e-4
VLM learning rate 4e-4
learning rate schedule linear warmup, cosine decay to 1e-6
weight decay 1e-4
gradient clipping AutoClip (Seetharaman et al., 2020), 10th percentile
noise level σ\sigma 0.25
feature RMS target 1/2
ema decay for m2m_{2} 0.99
Refer to caption
Figure B.2: gFD over CFG on GPIC Validation-50K. We show gFD on the GPIC Validation-50K set against CFG for JiT-T2I-XL and JiT-T2I-g models for Pixel, LILIE, and Pixel+LILIE. Selected CFG scales in Table B.1 are indicated with dashed vertical lines.
Table B.3: Generative model configurations.
JiT-T2I-XL JiT-T2I-g
architecture
depth 24 16
hidden dim 1024 1536
heads 16 24
bottleneck 128 256
parameters 700M 1.1B
image size 256
patch size 16
text encoder Qwen3-1.7B (Yang et al., 2025), frozen
max caption tokens 300
training
training steps 200K (ablations), 390,600, 781,200
warmup steps 4,000
optimizer AdamW (Loshchilov and Hutter, 2019), β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
batch size 256
learning rate 1e-4
learning rate schedule constant
weight decay 0
gradient clipping 1.0
ema decay 0.9999
time sampler logit​(t)∼𝒩⁡(μ,σ2)\text{logit}(t)\sim\mathcal{N}(\mu,\sigma^{2}), μ=−0.8\mu=-0.8, σ=0.8\sigma=0.8
noise scale 1.0
clip of (1−t)(1-t) in division 0.05
caption drop (for CFG) 0.1
sampling
ODE solver Euler
ODE steps 50
time steps linear in [0.0, 1.0]
CFG scale (Ho and Salimans, 2022) selected on GPIC Validation-50K
weights EMA

B.1 Additional Benchmarks

On GenEval, mixed loss gives 36.41±1.32 overall versus pixel-only 33.57±1.38, with standard errors over prompts. LILIE-only gives 32.49±1.32, and mixed loss lowers the two-object score from 28.28 to 24.24 despite improving some other categories. DPG-Bench overall changes modestly from 73.00±0.63 to 73.63±0.61; LILIE-only is 72.86±0.61. These results do not establish uniform compositional or language-alignment improvements. GenEval removes the leading “a photo of” phrase for all three configurations.

Table B.4: GenEval scores (%, higher is better) using the original prompts with the leading a photo of prefix removed, which improves scores for all 3 configurations. Errors are ±\pm one standard error over prompts. Bold indicates the best result per column.
Loss Overall Single Two Counting Colors Position Color attr.
Pixel 33.57±\,\pm\,1.38 67.50±\,\pm\,4.34 28.28±\,\pm\,3.26 14.69±\,\pm\,2.42 53.72±\,\pm\,4.21 18.00±\,\pm\,2.78 19.25±\,\pm\,2.79
LILIE 32.49±\,\pm\,1.32 62.50±\,\pm\,4.00 26.52±\,\pm\,3.17 14.06±\,\pm\,2.63 56.12±\,\pm\,4.03 13.00±\,\pm\,2.32 22.75±\,\pm\,2.80
Pixel+LILIE 36.41±\,\pm\,1.32 77.50±\,\pm\,3.57 24.24±\,\pm\,3.16 15.62±\,\pm\,2.68 56.12±\,\pm\,3.89 23.75±\,\pm\,3.02 21.25±\,\pm\,2.92
Table B.5: DPG-Bench scores (%, higher is better). Errors are ±\pm one standard error over prompts. Bold indicates the best result per column.
Loss Overall Global Entity Attribute Relation Other
Pixel 73.00±\,\pm\,0.63 76.60±\,\pm\,2.31 82.21±\,\pm\,0.56 86.59±\,\pm\,0.53 92.30±\,\pm\,0.58 70.00±\,\pm\,2.91
LILIE 72.86±\,\pm\,0.61 76.90±\,\pm\,2.29 82.13±\,\pm\,0.58 86.25±\,\pm\,0.54 91.68±\,\pm\,0.60 64.40±\,\pm\,3.07
Pixel+LILIE 73.63±\,\pm\,0.61 79.33±\,\pm\,2.15 82.49±\,\pm\,0.57 86.15±\,\pm\,0.53 92.50±\,\pm\,0.57 68.80±\,\pm\,2.89

B.2 Implementation of Perceptual Loss Baselines

Perceptual baselines train generators from scratch with one auxiliary feature model and no hidden-state representation alignment. Their loss definitions and time gating are preserved for reproducibility.

LPIPS.

The LPIPS baseline uses the standard pretrained VGG16 implementation.

DINOv2 and SigLIP 2.

DINOv2 and SigLIP 2 provide 256 patch tokens of width 768, with SigLIP features taken before pooling. Combined pixel objectives use average patch cosine dissimilarity; feature-only objectives use normalized squared feature distance. The paper retains both formulas.

Equation
dcos​(x^,x)\displaystyle d_{\cos}(\hat{x},x) =1P​∑p=1P(1−fp​(x^)⊤​fp​(x)‖fp​(x^)‖2​‖fp​(x)‖2),\displaystyle=\frac{1}{P}\sum_{p=1}^{P}\left(1-\frac{f_{p}(\hat{x})^{\top}f_{p}(x)}{\left\lVert f_{p}(\hat{x})\right\rVert_{2}\left\lVert f_{p}(x)\right\rVert_{2}}\right),
d2​(x^,x)\displaystyle d_{2}(\hat{x},x) =1P​D​∑p=1P‖fp​(x^)−fp​(x)‖22.\displaystyle=\frac{1}{PD}\sum_{p=1}^{P}\left\|f_{p}(\hat{x})-f_{p}(x)\right\|_{2}^{2}.

Aggregate form and time gating.

Mixed perceptual loss is enabled only after interpolation time 0.3, with weights 0.1 for LPIPS and 0.01 for DINOv2/SigLIP 2. Feature-only runs use unit weight and remove both pixel loss and the time gate. These baseline settings differ from LILIE’s default weight two and should be included when interpreting the comparison.

Equation
ℒ=ℒPixel+𝟏t≥τ​λ​ℒPerceptual,\mathcal{L}=\mathcal{L}_{\text{Pixel}}+\bm{1}_{t\geq\tau}\lambda\mathcal{L}_{\text{Perceptual}},

Appendix C Noise Level and Loss Weight

The full validation sweep preserves all twenty encoder-noise/objective combinations and their selected guidance scales. Zero-noise encoders produce very poor generation, with distances around 1,500–2,165, despite their invertible architecture. The best mixed setting is noise 0.25 and weight two, at validation distance 92.07; raising noise does not improve that best result. Guidance in this ablation uses a coarse five-value grid, whereas the final full-result table reports finer 0.05-scale selection.

Table C.1: Noise level and loss weight. gFD on GPIC Validation-50K with JiT-T2I-XL at 200K training steps. The selected CFG scale is in parentheses. The Pixel baseline scores 114.90 at CFG 7.75. Bold marks the lowest gFD in the sweep.
Original PDF numerical table, preserving every row and column.

Appendix D Model Size and Data Ablations

Full size/data tables include distributional-distance components, fidelity/diversity metrics, and guidance choices. Fréchet distance improves with the tested encoder scaling, but individual precision, recall, and coverage entries need not improve monotonically. All generators here have two hundred thousand steps, so these tables isolate encoder changes within that budget rather than total-compute matching.

Table D.1: Full model-size ablation results. The LILIE encoders are trained on 100M images. TARFlow configuration is hidden dimension/blocks/layers (H/K/L) per block. Our primary setting is highlighted.
Original PDF numerical table, preserving every row and column.
Table D.2: Full training-data ablation results. We ablate the number of images used to train the LILIE-S encoder (32M parameters). The 100M rows are the same LILIE-S configurations as in Table D.1 and are highlighted as our primary setting.
Original PDF numerical table, preserving every row and column.

Appendix E Limitations

All experiments use GPIC, 256-by-256 images, JiT-T2I generators, and SmolVLM-256M encoder supervision. The largest encoder is only 32 million parameters; larger captioning models, larger encoders, other domains/resolutions, and an optimal encoder-training-budget rule remain unexplored. Training requires captions and a pretrained vision–language model. The paper does not test mixtures of LILIE and other perceptual losses.

Appendix F Captioning Prompts

All four original system/user prompt pairs are preserved verbatim with indentation. They request unordered tags, one-to-two-sentence short captions, two-to-three-sentence medium captions, or five-to-seven-sentence detailed captions. These prompts are matched to GPIC’s caption-length labels during encoder training.

[tag]
system:
  Your goal is to write a compressed keyword-style caption (unordered tags)
  for the given image. The output must be a short, unordered list of tags
  describing the main visible content.
user:
  Write a keyword-style caption (tag-style) for the image shown.
[short]
system:
  Your goal is to write a short caption (1-2 sentences) that describes the
  given image in simple language that a middle school student can
  understand. The caption must be enough to identify the main subject and
  the basic setting/action.
user:
  Write a short caption (1-2 sentences) for the image shown.
[medium]
system:
  Your goal is to write a medium-length caption (2-3 sentences) that
  describes the given image in simple language that a middle school student
  can understand. The caption must be enough to identify the main subject
  and the basic setting/action.
user:
  Please write a caption (2-3 sentences) for the image shown.
[long]
system:
  Your goal is to write a long, highly detailed caption (5-7 sentences)
  that describes the given image in simple language that a middle school
  student can understand. The caption should include enough detail for them
  to understand what appears in the foreground and background and picture
  how the scene is arranged.
user:
  Please write a caption (5-7 sentences) for the image shown.
Figure F.1: Prompts given to SmolVLM-256M during LILIE training for the four GPIC caption-length categories: tag, short, medium, and long.

Appendix G Samples

The full appendix retains intermediate clean-image predictions along fifty-step trajectories, five grids of randomly selected held-out examples comparing real images and all three generation losses, and the selected lily mosaic. Each model uses its chosen guidance scale. Selected illustrations and random sample grids serve different evidential roles; neither replaces the quantitative test metrics.

Refer to caption
Figure G.1: Selected sampling trajectories over 50 steps. We fix a 50 step sampling budget and show clean image predictions at intermediate Euler steps 0, 5, 10, and 25, followed by the final image at step 50. Each example uses the same prompt and initial noise across methods.
Refer to caption
Figure G.2: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Refer to caption
Figure G.3: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Refer to caption
Figure G.4: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Refer to caption
Figure G.5: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Refer to caption
Figure G.6: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Refer to caption
Figure G.7: A mosaic of lilies and water lilies sampled from JiT-T2I-g trained with Pixel+LILIE.
Concept demonstration

Open image ↗