Literature
Display
← All papers

VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compression

Kyle Sargent · Ruiqi Gao · Philipp Henzler · Charles Herrmann · Aleksander Hołyński · Li Fei-Fei · Jiajun Wu · Jason Zhang

· CVPR 2026 ↗

Computer vision10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition reviews the complete fourteen-page arXiv v1 dated 17 December 2025, including the supplementary material. The title and all eight authors match; the existing CVPR 2026 publication link is retained.

Complete original Figure 1.
Figure 1: Left. We propose a new post-training technique for diffusion autoencoders which uses Vision-Language Models to judge different decodings of the same image and leverage these judgements to improve the autoencoder through Diffusion DPO. Right. Our method, VLIC, demonstrates substantial improvements in the overall reconstruction quality, as well as better alignment to human perception.

At a glance

VLIC trains a diffusion image compressor using pairwise reconstruction preferences from a vision-language model. A discrete encoder and stochastic decoder first learn reconstruction; Diffusion DPO then favors samples judged closer to the original. The main reward combines Gemini 2.5-Flash judgments with LPIPS agreement.

Human studies and image metrics show gains over selected baselines, especially on MS-COCO, while PO-ELIC remains stronger on CLIC 2022. Preference agreement, perceptual quality, pixel accuracy and compressed size are separate measurements; no universal improvement across all of them is established.

1 Introduction

Compression trades transmitted bits against reconstruction quality. Pixel errors need not reflect the importance of readable text, recognizable faces or other semantic details. Learned perceptual losses help, but optimizing their scores can exploit weaknesses that do not improve what people see.

The paper asks whether a pretrained VLM can judge two reconstructions directly, avoiding a separately distilled differentiable reward network. It extends FlowMo with scalar quantization, entropy coding and Diffusion DPO. Its main system still uses LPIPS during pretraining and preference filtering, so it does not eliminate learned perceptual metrics entirely.

The contribution is a codec/post-training design plus evidence on two preference benchmarks, three compression datasets and reward ablations. Figure 2 preserves qualitative comparisons at their displayed, sometimes different, bitrates; these images are examples rather than a matched-rate statistical test.

Complete original Figure 2.
Figure 2: Qualitative results on standard image compression datasets. Top: We compare VLIC  with HiFiC [34] and PO-ELIC [18] on a CLIC 2022 image [46] at various bits per pixel (bpp). Bottom: We compare our approach with HiFiC and PerCo on MS-COCO [28]. We find that our approach represents perceptually relevant fine details, faces, and textures more faithfully

2 Related Work

The relevant alternatives differ in how they decode images, approximate human visual similarity and optimize a preference signal.

Perceptually-oriented Image Compression.

GAN codecs such as HiFiC and PO-ELIC often decode quickly. Diffusion methods either reconstruct from a compressed bottleneck or use the diffusion reverse process itself as an entropy code. VLIC follows the bottleneck-autoencoder family, using FlowMo’s architecture with a separate autoregressive entropy model.

Approximations to Human Perception of Visual Similarity.

LPIPS, E-LPIPS, DreamSim and DISTS approximate aspects of human image similarity. Text-guided codecs and pretrained visual models also target semantic information. VLIC instead obtains direct judgments from a VLM; its final training pipeline retains LPIPS as a complementary constraint.

Aligning Diffusion Models to Preferences.

Differentiable reward methods backpropagate a score, whereas preference optimization can use discrete rankings. VLIC uses the diffusion adaptation of DPO, inspired by earlier VLM rewards for image generation. The paper’s DDPO shorthand refers here to Diffusion Direct Preference Optimization, which must be distinguished from the separately cited Denoising Diffusion Policy Optimization method.

3 Method

An encoder deterministically maps an image to a discrete latent sequence. An autoregressive entropy model compresses that sequence; a diffusion decoder samples a reconstruction. During post-training, two samples from the same code become a preference pair.

Complete original Figure 3.
Figure 3: Method. An original image is encoded to a one-dimensional discrete latent code via an encoder. The discrete code is entropy coded by an auto-regressive language model. The diffusion decoder samples two reconstructions conditioned on the latent code, which are ranked via a VLM. The resulting preference is used to train the full diffusion autoencoder via Diffusion DPO [47].

Architecture and training.

The transformer codec replaces FlowMo’s lookup-free quantizer with finite scalar quantization, removing its commitment and entropy losses. Pretraining combines rectified-flow reconstruction with LPIPS on the one-step denoised prediction.

Diffusion DPO compares the current decoder’s denoising error against a fixed reference for winning and losing samples. Its logistic objective favors the winner relative to the loser; β controls departure from the reference and the timestep weight depends on signal-to-noise ratio. Both samples share an original image and latent conditioning, reducing variation from differing inputs.

The original flow loss is retained with weight λFlow to stabilize longer post-training. The encoder remains trainable so its representation can adapt to the new reward. Afterwards, the encoder and decoder are frozen while a separate autoregressive transformer learns token probabilities for arithmetic coding.

All five numbered objectives remain below. The source defines velocity as v=ε+x, but does not specify the interpolation convention needed to interpret that sign; this original expression is retained. Numerical β and λFlow settings, decoder step count and guidance scale are not supplied in the paper.

Equations 1–5
LDDPO​(θ)=−𝔼​log⁡σ⁡(−β​ω​(λt)​(Δw−Δl)),L_{\textrm{DDPO}}(\theta)=-\mathbb{E}\log\sigma\left(-\beta\omega(\lambda_{t})(\Delta^{w}-\Delta^{l})\right), (1)
Δw=‖ϵw−ϵθ​(𝐱^tw,𝐱,t)‖22−‖ϵw−ϵref​(𝐱^tw,𝐱,t)‖22,\small{\Delta^{w}=\left\lVert\boldsymbol{\epsilon}^{w}-\boldsymbol{\epsilon}_{\theta}(\mathbf{\hat{x}}_{t}^{w},\mathbf{x},t)\right\rVert_{2}^{2}-\left\lVert\boldsymbol{\epsilon}^{w}-\boldsymbol{\epsilon}_{\text{ref}}(\mathbf{\hat{x}}_{t}^{w},\mathbf{x},t)\right\rVert_{2}^{2},} (2)
Δl=‖ϵl−ϵθ​(𝐱^tl,𝐱,t)‖22−‖ϵl−ϵref​(𝐱^tl,𝐱,t)‖22.\small{\Delta^{l}=\left\lVert\boldsymbol{\epsilon}^{l}-\boldsymbol{\epsilon}_{\theta}(\mathbf{\hat{x}}_{t}^{l},\mathbf{x},t)\right\rVert_{2}^{2}-\left\lVert\boldsymbol{\epsilon}^{l}-\boldsymbol{\epsilon}_{\text{ref}}(\mathbf{\hat{x}}_{t}^{l},\mathbf{x},t)\right\rVert_{2}^{2}.} (3)
LFlow​(θ)=𝔼ϵ,x,t​(‖𝐯−𝐯θ​(𝐱,𝐱t,t)‖22),L_{\mathrm{Flow}}(\theta)=\mathbb{E}_{\epsilon,x,t}(\|\mathbf{v}-\mathbf{v}_{\theta}(\mathbf{x},\mathbf{x}_{t},t)\|_{2}^{2}), (4)
L⁡(θ)=LDDPO​(θ)+λFlow​LFlow​(θ)L(\theta)=L_{\mathrm{DDPO}}(\theta)+\lambda_{\mathrm{Flow}}L_{\mathrm{Flow}}(\theta) (5)

𝒗=ϵ+𝒙\boldsymbol{v}=\boldsymbol{\epsilon}+\boldsymbol{x}

The VLIC reward function.

Gemini 2.5-Flash receives the original and two reconstructions, explains visible differences and rates relative similarity from −5 to +5. Negative favors A; positive favors B. The full prompt prioritizes human-relevant semantic changes over texture changes.

Each seed evaluates both reconstruction orders, reverses the corresponding signs and takes the sign of the summed ratings. Votes are then summed over three seeds. The final pair is used only when the VLM ranking agrees with LPIPS; disagreement is discarded. Thus ordinary accepted-pair scoring uses six VLM calls before filtering. Exact handling of tied votes or equal LPIPS scores is not specified.

Order reversal reduces positional bias, repeated seeds reduce noise, and LPIPS agreement constrains the reward. These measures improve selected outcomes but do not guarantee a correct judgment; the filtering also changes which training pairs survive.

Equations
rB,0i=−rA,0i=VLM⁡(𝒙,𝒙^𝟎𝑨,𝒙^𝟎𝑩,i)r_{B,0}^{i}=-r_{A,0}^{i}=\mathrm{VLM}(\boldsymbol{x},\boldsymbol{\hat{x}_{0}^{A}},\boldsymbol{\hat{x}_{0}^{B}},i)
rA,1i=−rB,1i=VLM⁡(𝒙,𝒙^𝟎𝑩,𝒙^𝟎𝑨,i).r_{A,1}^{i}=-r_{B,1}^{i}=\mathrm{VLM}(\boldsymbol{x},\boldsymbol{\hat{x}_{0}^{B}},\boldsymbol{\hat{x}_{0}^{A}},i).
rAi=sign⁡(rA,0i+rA,1i),rBi=sign⁡(rB,0i+rB,1i).r_{A}^{i}=\mathrm{sign}(r_{A,0}^{i}+r_{A,1}^{i}),~~r_{B}^{i}=\mathrm{sign}(r_{B,0}^{i}+r_{B,1}^{i}).
rA=∑i=1nrAi,rB=∑i=1nrBir_{A}=\sum_{i=1}^{n}r_{A}^{i},~~r_{B}=\sum_{i=1}^{n}r_{B}^{i}
Complete original Figure 4.
Figure 4: Quantitative Evaluation on Image Compression Datasets. Overall, VLIC achieves competitive or state-of-the-art performance. VLIC performs particularly well on perceptual metrics and particularly well on MS-COCO, which contains a high percentage of images with human-relevant characteristics such as text and faces.

4 Experiments

The experiments evaluate image quality and human preferences, then separate the effects of preference post-training, the VLM, repeated judgments and LPIPS filtering.

Our setup.

Two nominal pre-entropy rates, 0.07 and 0.21 bits per pixel, are trained on 256×256 ImageNet images. Pretraining uses one million Adam steps at 10⁻⁴; DPO uses 8,000 steps at 5×10⁻⁷. Both use batch 256, bfloat16 JAX and 256 TPUv4 accelerators.

Online preference buffers contain about 2,560 candidates and refresh every 250 steps; LPIPS/VLM disagreement can reduce their usable size. VLM queries run asynchronously while optimization uses a slightly older buffer. The authors report online buffers work better than an offline set, without a numerical offline comparison here.

At inference, overlapping tiles extend the 256-pixel model to larger images. A shifted diffusion schedule and classifier-free guidance are used; discrete conditioning is dropped in 10% of training cases. Nominal rates before entropy coding should not be substituted for the plotted realized bitrates.

4.1 Main results.

The main comparison uses dataset-specific baseline availability and evaluates both perceptual fidelity and conventional pixel distortion.

Baselines.

HiFiC is run across all datasets. PerCo uses an open-source replication, and is evaluated only on low-resolution MS-COCO. HFD’s CLIC 2020 scores come from its paper after reproducing its evaluation pipeline. PO-ELIC uses released CLIC 2022 reconstructions. Missing entries reflect unavailable implementations or unsupported resolution; this is not one fully rerun common baseline suite.

Datasets.

MS-COCO uses 30,000 validation images cropped to 256×256 with the DALL-E protocol, compared with HiFiC and PerCo. CLIC 2020 uses 428 high-resolution training-set images, up to four megapixels, compared with HiFiC and HFD. CLIC 2022 uses thirty high-resolution test images, compared with HiFiC and PO-ELIC.

MS-COCO Human Elo ↑\uparrow FD-DINO ↓\downarrow FID ↓\downarrow LPIPS ↓\downarrow PSNR ↑\uparrow
Ours (0.07bpp) 858 62.25 2.20 0.274 21.78
−- LPIPS post-training only 838 63.63 2.33 0.274 21.77
Ours (0.21bpp) 1112 16.83 1.35 0.168 26.50
−- LPIPS post-training only 1103 16.96 1.30 0.169 26.54
Table 1: Importance of VLM. At multiple BPP (prior to entropy coding), post-training with the VLM + LPIPS objective provides gains over post-training the compression model with LPIPS alone.
Accuracy BAPPS-Val Compressed Images
Human 73.99 72.15
LPIPS 69.56 92.32
DreamSim 68.13†\dagger -
VLM 69.44 83.80
Table 2: Human 2AFC benchmarks. Gemini 2.5-Flash can replicate human judgments on 2AFC datasets zero-shot. †Number taken from paper.

Metrics.

LPIPS and PSNR measure paired-image discrepancy; FID and FD-DINO measure distributional differences. The authors give greatest weight to human Elo and then FD-DINO, while acknowledging that perceptual quality and pixel accuracy can conflict at a fixed rate.

Human studies collect 1,812 CLIC 2020, 2,523 CLIC 2022 and 15,705 MS-COCO pairwise ratings. Elo is averaged over 10,000 random orderings to reduce update-order dependence. Elo values are relative to the compared pool and study design, not percentages or directly comparable absolute quality across datasets.

High-resolution distributional evaluation uses random 256×256 crops. The prose specifies one hundred crops per CLIC 2022 image and ten per MS-COCO image, despite discussing CLIC 2020 in that passage; a CLIC 2020 crop count is not explicitly given. Human raters also see crops for the high-resolution datasets. The paper does not report rater counts/demographics, confidence intervals or repeated-training-seed uncertainty.

Results.

Figure 4 shows particularly strong human/perceptual performance on MS-COCO and generally favorable comparisons with HiFiC, PerCo and HFD. PO-ELIC outperforms VLIC on CLIC 2022, whose test set contains thirty images; absent PO-ELIC results elsewhere do not establish VLIC’s superiority there.

Improved perceptual scores can accompany worse PSNR, notably against HFD and HiFiC. This supports a different rate/perception tradeoff rather than lossless recovery or a claim that all reconstruction details are more accurate.

4.2 Analysis

The analysis tests whether VLM judgments agree with people, whether they add value beyond LPIPS, and how noise affects the post-training signal.

Replicating human judgments.

The new compressed-image preference set has 5,401 tuples, each rated at least three times. The human baseline holds out one rater and compares that answer with the remaining raters’ average; it is not a human correctness ceiling.

On BAPPS, VLM agreement is 69.44%, versus LPIPS 69.56% and held-out humans 73.99%. On compressed images, VLM reaches 83.80%, below LPIPS 92.32% but above the single-human agreement of 72.15%. These results support useful zero-shot correlation, not superior judgment to LPIPS or humans in every sense. The authors attribute noisy human agreement to highly similar reconstructions.

Importance of the VLM.

Adding VLM agreement to LPIPS-only preference training raises human Elo from 838 to 858 at nominal 0.07 bpp and from 1103 to 1112 at 0.21 bpp. FD-DINO also improves. At the higher rate, however, FID worsens from 1.30 to 1.35 and PSNR falls from 26.54 to 26.50. At the lower rate LPIPS ties at 0.274. The advantage is metric-dependent, and the table provides no uncertainty on those differences.

Mitigating noise.

Figure 6 shows the same two reconstructions ranked inconsistently after swapping their order. Repeated seeds reduce this failure mode. Figure 5 rises from roughly 67% agreement with one seed toward 71% with seven or eight, then falls slightly; more calls are not strictly monotonically better in the plotted run. Main training uses three seeds as a compute/quality compromise.

Complete original Figure 5.
Figure 5: Scaling self-ensembling. The VLM becomes more predictive of human judgment on BAPPS [23] as test-time compute (number of VLM seeds) is scaled.
MS-COCO FD-DINO ↓\downarrow FID ↓\downarrow LPIPS ↓\downarrow PSNR ↑\uparrow
Ours (VLIC) 67.83 2.31 0.278 21.68
−- No ensemble w/ LPIPS 67.68 2.10 0.280 21.29
−- No post-training 82.31 2.40 0.300 21.27
−- No self-ensembling 68.36 2.15 0.280 21.53
Table 3: Ablation study. Various components of reward design are necessary for best performance.

Ablation study.

Removing DPO worsens all four Table 3 metrics. Removing LPIPS filtering improves distributional FD-DINO/FID (67.83/2.31 to 67.68/2.10), but worsens LPIPS/PSNR (0.278/21.68 to 0.280/21.29). Removing self-ensembling similarly improves FID to 2.15 while degrading the other three metrics.

Consequently the full reward is not best on every measure, and this ablation does not include human Elo. Its full-model numbers differ from Table 1’s nominal 0.07-bpp configuration, so the two tables should not be treated as identical runs.

Complete original Figure 6.
Figure 6: Failure modes. VLMs can hallucinate an incorrect ranking when the images are highly similar, such as in this case when the VLM fails to be self-consistent when the order of reconstructed images is reversed.

Limitations.

Diffusion decoding is slower than typical GAN decoding, and multi-call VLM rewards cost more than a small perceptual network. The paper does not quantify decoding latency, VLM query expense or preference-pair acceptance rate. A stochastic perceptual reconstruction can preserve plausible appearance while changing details; the method is lossy.

5 Conclusion

Pairwise VLM judgments can help train a perceptual compressor when combined with diffusion preference optimization and noise mitigation. The evidence is strongest for the tested datasets, model and reward design; broader improvements from future VLMs remain a hypothesis.

Acknowledgments.

The authors thank Ben Poole, David Minnen and Dina Bashkirova for discussions.

Appendix A Models and training

All three components are transformers. Low/high-rate models each total 1.01B parameters. The encoder uses width 768 and eight layers; decoder width 1152 and sixteen layers; entropy coder width 768 and sixteen layers. Both codec transformers use patch size four, sequence length 4352 and eight scalar-quantizer levels. Token sizes are six versus eighteen; entropy sequences have length 384 versus 1152. Tables 4–5 retain the exact configuration.

Training uses ImageNet at 256-pixel resolution, random horizontal flips and center crops. Pretraining has no dropout or weight decay. DPO refreshes its 2,560-sample buffer every 250 steps. Finally, the NanoGPT-based entropy model trains for 200k AdamW steps at 10⁻⁴ with dropout 0.1 and decay 0.025, while codec weights stay frozen.

Low BPP Encoder Decoder Entropy coder
Hidden dim. 768768 11521152 768768
Number of layers 88 1616 1616
Patch size 44 44 -
1D Token size 66 66 -
FSQ levels 88 88 -
Sequence length 43524352 43524352 384384
Table 4: Model hyperparameters for low BPP configuration. Total parameter count is 1.01B
High BPP Encoder Decoder Entropy coder
Hidden dim. 768768 11521152 768768
Number of layers 88 1616 1616
Patch size 44 44 -
1D Token size 1818 1818 -
FSQ levels 88 88 -
Sequence length 43524352 43524352 11521152
Table 5: Model hyperparameters for high BPP configuration. Total parameter count is 1.01B

Appendix B VLM prompt

The exact Gemini 2.5-Flash prompt is retained below. It establishes the A/B score sign, favors semantic fidelity, discourages extreme scores without clear evidence and requires a final RATING field. Post-training uses the parsed numerical rating rather than directly optimizing the explanation text.

“In this task, you’ll be asked to compare an original image with two AI reconstructions of that image, Reconstruction A followed by Reconstruction B. You’ll see triplets of images like (original, A, B). You’ll give a relative rating of the two images. This is between -5 and 5, inclusive. Higher scores mean that Reconstruction B is relatively better. So if you give a score of -2, you think A is kind of better, and if you give a score of +5, you think B is obviously significantly better. Be sparing with the higher magnitude scores - you should expect that most triplets you see won’t have an obviously better reconstruction. In your ratings, make sure to prioritize differences that are semantically important to a human observer. If a distortion changes the meaning of an image to a human observer, then it’s more significant than if a distortion changes the texture of an image. Your response should conclude with “RATING: X”, where X is your rating, i.e. -2 or 3. Now here is the image triplet that you need to rate. Make sure to provide a lot more reasoning and make sure to carefully look at each of the three images and provide meticulous visual justifications based on evidence from each image! Remember, negative scores mean that A is better, and positive scores mean that B is better.”
Complete original Figure 7.
Figure 7: Tiled inference for arbitrary resolutions. From top to bottom: Original image, tiling strategy, reconstructed image. The margin size (we use 8 pixels in this work) must be large enough to communicate information between patches during diffusion to avoid unsightly border artifacts, but not so large as to waste BPP.

Appendix C Inference

Choose the smallest nonnegative tile counts whose coverage reaches the image height and width, using native resolution r=256 and margin p=8. Resize to those coverage dimensions, encode each overlapping tile independently, then jointly diffuse the tiles using MultiDiffusion. Overlap communicates context across borders but costs extra coded area. This is a resolution-extension procedure, not preservation of the exact original pixel grid.

Equation

kh,kwk_{h},k_{w}

r+kh​(r−p)≥hr+k_{h}(r-p)\geq h

r+kw​(r−p)≥wr+k_{w}(r-p)\geq w

r+kh​(r−p),r+kw​(r−p)r+k_{h}(r-p),r+k_{w}(r-p)

Appendix D User studies

Raters see A, the original and B in that order and select one of five responses: A much/slightly closer, about the same, or B slightly/much closer. Images are displayed at width 480 pixels; CLIC 2020/2022 use random crops. The exact interface is not disclosed for the authors’ stated policy reasons. The mapping from these five responses to Elo updates, including ties, is not fully specified.

Appendix E Additional capabilities

An additional experiment asks the VLM to transcribe text and rewards small edit distance from the original transcription. The codec learns to censor readable text, bounding the empty-output penalty by the original text length instead of preserving characters. Figure 8 documents reward exploitation; alternative edit-distance formulations give only small, generally imperceptible improvements.

Complete original Figure 8.
Figure 8: Censoring readable text. A failure case of an edit-distance based reward on readable text determined by the VLM causes the model to degenerate to censoring all readable text in the images.

Appendix F Disclaimer

Some benchmark preparation flips images horizontally. For visual presentation the authors may flip both VLIC and baseline reconstructions back, especially when text makes orientation noticeable. This display adjustment is disclosed and should not be mistaken for a new reconstruction result.

Appendix G Image attributions

The full original image-source, creator, identifier and license attribution list is preserved below, including credits for the external supplementary gallery.

The authors’ interactive gallery provides additional scene and method comparisons beyond the manuscript figures.

Figure 1, Truck

Figure 1, Street Sign

Figure 1, Bar

Figure 2, Chicago Skyline

  • Sourced From: CLIC 2022
  • Image ID: 07113e38700d3f0dab7a9f34d4512 98a54de3cef3bc4e03945d5fead4f513ecd
  • License: Unsplash

Figure 2, Street Sign

Figure 2, Women on Phone

Figure 2, Soldiers

Figure 3, Gazelles

Figure 6, Mouse

Gallery, Bridge

  • Sourced From: CLIC 2022
  • Image ID: 732bf474788c19c0c1fae6dd7689d4c da2f4e0632a1c7725e970b69d44d08f3e
  • License: Unsplash

Gallery, Tennis player

Gallery, Woman

Gallery, Hunter

Gallery, Dinner

Gallery, Lady in dress

Gallery, Cat

Gallery, Firehouse

Concept demonstration

Open image ↗