Literature
Display
← All papers

V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think

Bingda Tang · Yuhui Zhang · Xiaohan Wang · Jiayuan Mao · Ludwig Schmidt · Serena Yeung-Levy

Preprint posted · CVPR 2026 Findings (accepted) ↗

Efficient Learning13 min text · CondensedOriginal paper ↗
Contents

Overview

At a glance

Variational GRPO replaces intractable image log-likelihoods with denoising-loss surrogates and stabilizes their optimization using shared noise/timesteps, stratified sampling, adaptive weighting and scenario-dependent update controls. It improves all four reported FLUX reward metrics and reaches a similar aggregate SD 3.5 Medium score with 580 rather than about 1,700 gradient updates. These are promising benchmark and computational-count results; the surrogate is not an exact likelihood ratio, and the stated speedups are not measured wall-clock guarantees.

1 Introduction

Text-to-image post-training needs rewards to influence a multi-step denoiser whose final-image likelihood is generally unavailable. Trajectory-based methods instead optimize tractable stochastic transitions, tying training to a particular rollout formulation. V-GRPO treats the whole generated image as one action and reuses an objective related to denoising pretraining.

The practical claim is that this simpler surrogate can work well when its stochastic variation and update size are controlled. It can roll out with a higher-order deterministic sampler without differentiating or assigning policy ratios to every transition. This edition covers all 14 pages of v1, dated 25 April 2026, with eight figures including all six ablation panels, six tables, Algorithm 1, 17 numbered equations and 47 references.

Complete original PDF S1.T1
Table 1: V-GRPO delivers state-of-the-art performance with substantial speedup over baselines. Results are averaged across all evaluated reward functions and reported on the test sets. Full results are presented in Tab. 2 and Tab. 3, and the experimental setup is described in Sec. 5.1 and Appendix A.
Original selectable table
Method #Steps NFEπ𝜽old\text{NFE}_{\pi^{\boldsymbol{\theta}_{\text{old}}}} NFEπ𝜽\text{NFE}_{\pi^{\boldsymbol{\theta}}} Reward
FLUX.1-dev — — — 1.25
+ BranchGRPO 300 13.68 13.68 1.40
+ MixGRPO 300 25 4 1.41
+ V-GRPO 150 16 + 4 4 1.42
+ V-GRPO 300 16 + 4 4 1.45
SD 3.5 M (w/o CFG) — — — 0.95
+ DiffusionNFT 1.7K 40 + 40 40 1.71
+ V-GRPO 580 40 + 6.9 6.9 1.71

2 Related Work

Offline reward-weighted regression and direct preference optimization use fixed preference data, while online approaches refresh outputs from the current model. MixGRPO and BranchGRPO improve trajectory-based optimization with specialized sampling. Earlier denoising-loss surrogates performed poorly on visual generation; this work revisits their stability rather than introducing an exact tractable image likelihood.

DiffusionNFT also uses pretraining-style objectives but contrasts policies rather than using the same GRPO surrogate. The comparisons are with particular reported implementations and settings. The paper’s broad superiority language is stronger than what two model families and these reward benchmarks establish.

3 Preliminaries

3.1 Denoising Generative Models

A forward process mixes clean data with noise through schedule coefficients. The network regresses a chosen target, such as clean data, noise or velocity, using timestep-dependent weighting. Reverse sampling combines the current state, network output and optional stochastic noise; setting the stochastic term to zero yields an ODE-style update.

Compatible parameterizations can be related under a fixed forward process, motivating reuse of denoising losses across diffusion and flow models. This does not mean arbitrary trained networks, discretizations or weighting choices define identical final-image distributions. The experiments use rectified-flow models.

Equations 1–4
𝐳t=at​𝐱+bt​ϵ,\mathbf{z}_{t}=a_{t}\mathbf{x}+b_{t}\boldsymbol{\epsilon}, (1)
ℒw​(𝜽)=𝔼t,π0​(𝐱),π1​(ϵ)​[wt​‖NNt𝜽​(𝐳t)−𝐫t​(𝐱,ϵ)‖22],\mathcal{L}_{w}(\boldsymbol{\theta})=\mathbb{E}_{t,\pi_{0}(\mathbf{x}),\pi_{1}(\boldsymbol{\epsilon})}\Big[w_{t}\Big\|\texttt{NN}^{\boldsymbol{\theta}}_{t}(\mathbf{z}_{t})-\mathbf{r}_{t}(\mathbf{x},\boldsymbol{\epsilon})\Big\|_{2}^{2}\Big], (2)
𝐫t=ct​𝐱+dt​ϵ,\mathbf{r}_{t}=c_{t}\mathbf{x}+d_{t}\boldsymbol{\epsilon}, (3)
𝐱ti−1=αti​𝐱ti+βti​NNti𝜽​(𝐱ti)+σti​ϵ~ti,\mathbf{x}_{t_{i-1}}=\alpha_{t_{i}}\mathbf{x}_{t_{i}}+\beta_{t_{i}}\,\texttt{NN}^{\boldsymbol{\theta}}_{t_{i}}(\mathbf{x}_{t_{i}})+\sigma_{t_{i}}\,\boldsymbol{\tilde{\epsilon}}_{t_{i}}, (4)

3.2 Group Relative Policy Optimization

GRPO normalizes each reward by the mean and standard deviation of outputs for the same prompt, avoiding a learned value critic. A PPO-style clipped objective limits the incentive for large likelihood-ratio changes. V-GRPO applies that template to an image-level surrogate rather than averaging transition-level ratios.

The printed normalization omits a zero-variance safeguard, relevant when all outputs receive the same coarse reward; implementation handling is not specified here. The method’s ratio approximation remains distinct from the exact ratio in the introductory GRPO formulation.

Equations 5–6
Ai=Ri−mean​({Ri}i=1G)std​({Ri}i=1G).A_{i}=\frac{R_{i}-\text{mean}(\{R_{i}\}_{i=1}^{G})}{\text{std}(\{R_{i}\}_{i=1}^{G})}. (5)
𝒥GRPO​(𝜽)=\displaystyle\mathcal{J}^{\text{GRPO}}(\boldsymbol{\theta})= 𝔼{𝐨i}i=1G∼π𝜽old[1G∑i=1Gmin(ρ𝜽iAi,\displaystyle\mathbb{E}_{\{\mathbf{o}_{i}\}_{i=1}^{G}\sim\pi^{\boldsymbol{\theta}_{\text{old}}}}\bigg[\frac{1}{G}\sum_{i=1}^{G}\min\big(\mathbf{\rho}^{\boldsymbol{\theta}}_{i}A_{i}, (6)
clip(ρ𝜽i,1−ϵ,1+ϵ)Ai)].\displaystyle\text{clip}(\mathbf{\rho}^{\boldsymbol{\theta}}_{i},1-\epsilon,1+\epsilon)A_{i}\big)\bigg].
Complete original PDF alg1
Algorithm 1 V-GRPO
Original selectable algorithm
0:  Initial policy 𝜽\boldsymbol{\theta}, reward functions {Rk}k=1K{\{R^{k}\}_{k=1}^{K}}, prompt dataset 𝒟\mathcal{D}, hyperparameters β,ϵ,η,NMC,…\beta,\epsilon,\eta,N_{\text{MC}},\dots
1:  for iteration=1,…,M\text{iteration}=1,\dots,M do
2:   Sample NN batches of prompts 𝒟bN⊂𝒟\mathcal{D}_{\text{bN}}\subset\mathcal{D}.
3:   Update the behavior policy 𝜽old←𝜽\boldsymbol{\theta}_{\text{old}}\leftarrow\boldsymbol{\theta}.
4:   Generate GG outputs {𝐨i}i=1G∼π𝜽old(⋅|c)\{\mathbf{o}_{i}\}_{i=1}^{G}\sim\pi^{\boldsymbol{\theta}_{\text{old}}}(\cdot|c) for each prompt c∈𝒟bNc\in\mathcal{D}_{\text{bN}}.
5:   Store a set of timestep-noise pairs {(tj,ϵj)}j=1NMC\{(t_{j},\boldsymbol{\epsilon}_{j})\}_{j=1}^{N_{\text{MC}}}, where timesteps are drawn via stratified sampling rather than uniform sampling. This set is shared across all outputs for each prompt cc, rather than sampled independently per output 𝐨i\mathbf{o}_{i}.
6:   for each output 𝐨i\mathbf{o}_{i} generated from each prompt cc do
7:    Compute rewards from each function {Rik}k=1K\{R_{i}^{k}\}_{k=1}^{K}.
8:    Aggregate advantages Ai=Aggregate​({Rik})A_{i}=\text{Aggregate}(\{R_{i}^{k}\}), and apply soft-clipping Aisoft=η⋅tanh(1ηAi)A^{\text{soft}}_{{}_{i}}=\eta\cdot\tanh(\frac{1}{\eta}A_{i}).
9:    Compute ELBO-based surrogates with adaptive loss weighting ℒ^adaptive​(𝜽old∣𝐨i,c)=1NMC​∑j=1NMCℓi𝜽old​(tj,ϵj)\hat{\mathcal{L}}^{\text{adaptive}}(\boldsymbol{\theta_{\text{old}}}\mid\mathbf{o}_{i},c)=\frac{1}{N_{\text{MC}}}\sum_{j=1}^{N_{\text{MC}}}\ell^{\boldsymbol{\theta}_{\text{old}}}_{i}(t_{j},\boldsymbol{\epsilon}_{j}).
10:   end for
11:   for gradient step=1,…,N\text{gradient step}=1,\dots,N do
12:    Sample a single batch 𝒟b⊂𝒟bN\mathcal{D}_{\text{b}}\subset\mathcal{D}_{\text{bN}}.
13:    for each sample 𝐨i\mathbf{o}_{i} in the batch 𝒟b\mathcal{D}_{\text{b}} do
14:     Compute ℒ^adaptive​(𝜽∣𝐨i,c)\hat{\mathcal{L}}^{\text{adaptive}}(\boldsymbol{\theta}\mid\mathbf{o}_{i},c) with adaptive loss weighting using the current policy 𝜽\boldsymbol{\theta} and stored {(tj,ϵj)}j=1NMC\{(t_{j},\boldsymbol{\epsilon}_{j})\}_{j=1}^{N_{\text{MC}}}.
15:     Calculate the importance ratio ρi𝜽=exp⁡(−ℒ^adaptive​(𝜽∣𝐨i,c)+ℒ^adaptive​(𝜽old∣𝐨i,c))\rho^{\boldsymbol{\theta}}_{i}=\exp(-\hat{\mathcal{L}}^{\text{adaptive}}(\boldsymbol{\theta}\mid\mathbf{o}_{i},c)+\hat{\mathcal{L}}^{\text{adaptive}}(\boldsymbol{\theta}_{\text{old}}\mid\mathbf{o}_{i},c)).
16:     Compute GRPO objective with importance ratio clipping 𝒥iGRPO​(𝜽)=min⁡(ρi𝜽​Aisoft,clip​(ρi𝜽,1−ϵ,1+ϵ)​Aisoft)\mathcal{J}^{\text{GRPO}}_{i}(\boldsymbol{\theta})=\min(\rho^{\boldsymbol{\theta}}_{i}A^{\text{soft}}_{i},\text{clip}(\rho^{\boldsymbol{\theta}}_{i},1-\epsilon,1+\epsilon)A^{\text{soft}}_{i}).
17:     Compute the KL penalty 𝔻isimple(π𝜽∥π𝜽old)\mathbb{D}^{\text{simple}}_{i}(\pi^{\boldsymbol{\theta}}\|\pi^{\boldsymbol{\theta}_{\text{old}}}).
18:    end for
19:    Update the policy 𝜽←Optimizer(𝜽,∇𝜽∑(𝒥iGRPO(𝜽)−β⋅𝔻isimple(π𝜽∥π𝜽old)))\boldsymbol{\theta}\leftarrow\text{Optimizer}\big(\boldsymbol{\theta},\nabla_{\boldsymbol{\theta}}\sum(\mathcal{J}^{\text{GRPO}}_{i}\big(\boldsymbol{\theta}\big)-\beta\cdot\mathbb{D}^{\text{simple}}_{i}(\pi^{\boldsymbol{\theta}}\|\pi^{\boldsymbol{\theta}_{\text{old}}}))\big).
20:   end for
21:  end for

4 Approach

4.1 Overview

For each generated image, the algorithm estimates a negative denoising loss using a small number of sampled timestep/noise pairs. The exponential difference between current and behavior-policy losses becomes the GRPO ratio surrogate. Stored pairs are reused for the old and new evaluations, reducing irrelevant randomness and separating optimization from the rollout sampler.

Algorithm 1 refreshes the behavior policy, generates groups, computes rewards and advantages, stores old surrogate values, then updates the current policy over batches. Clipping and the old-policy penalty are enabled selectively. Exponentiating a noisy loss estimate does not make an unbiased likelihood-ratio estimate; nor does replacing two log-likelihoods with lower-bound-related objectives generally cancel their variational gaps.

Equations 7–10
log⁡π𝜽​(𝐨i∣c)←\displaystyle\log\pi^{\boldsymbol{\theta}}(\mathbf{o}_{i}\mid c)\leftarrow −ℒw​(𝜽∣𝐨i,c)\displaystyle-\mathcal{L}_{w}(\boldsymbol{\theta}\mid\mathbf{o}_{i},c) (7)
=\displaystyle= −𝔼t,π1​(ϵ)​[wt​‖NNt𝜽​(𝐳t,c)−𝐫t​(𝐨i,ϵ)‖22].\displaystyle-\mathbb{E}_{t,\pi_{1}(\boldsymbol{\epsilon})}\Big[w_{t}\Big\|\texttt{NN}^{\boldsymbol{\theta}}_{t}(\mathbf{z}_{t},c)-\mathbf{r}_{t}(\mathbf{o}_{i},\boldsymbol{\epsilon})\Big\|_{2}^{2}\Big].
ρi𝜽=π𝜽​(𝐨i|c)π𝜽old​(𝐨i|c)=\displaystyle\rho_{i}^{\boldsymbol{\theta}}=\frac{\pi^{\boldsymbol{\theta}}(\mathbf{o}_{i}|c)}{\pi^{\boldsymbol{\theta}_{\text{old}}}(\mathbf{o}_{i}|c)}= exp⁡(log⁡π𝜽​(𝐨i∣c)−log⁡π𝜽old​(𝐨i∣c))\displaystyle\exp(\log\pi^{\boldsymbol{\theta}}(\mathbf{o}_{i}\mid c)-\log\pi^{\boldsymbol{\theta}_{\text{old}}}(\mathbf{o}_{i}\mid c)) (8)
=\displaystyle= exp⁡(−ℒw​(𝜽∣𝐨i,c)+ℒw​(𝜽old∣𝐨i,c)).\displaystyle\exp(-\mathcal{L}_{w}(\boldsymbol{\theta}\mid\mathbf{o}_{i},c)+\mathcal{L}_{w}(\boldsymbol{\theta}_{\text{old}}\mid\mathbf{o}_{i},c)).
ℒ^w​(𝜽∣𝐨i,c)=1NMC​∑j=1NMCℓi𝜽​(tj,ϵj),\hat{\mathcal{L}}_{w}(\boldsymbol{\theta}\mid\mathbf{o}_{i},c)=\frac{1}{N_{\text{MC}}}\sum_{j=1}^{N_{\text{MC}}}\ell^{\boldsymbol{\theta}}_{i}(t_{j},\boldsymbol{\epsilon}_{j}), (9)
ℓi𝜽​(tj,ϵj)=wtj​‖NNtj𝜽​(𝐳tj,c)−𝐫tj​(𝐨i,ϵj)‖22.\ell^{\boldsymbol{\theta}}_{i}(t_{j},\boldsymbol{\epsilon}_{j})=w_{t_{j}}\left\|\texttt{NN}^{\boldsymbol{\theta}}_{t_{j}}(\mathbf{z}_{t_{j}},c)-\mathbf{r}_{t_{j}}(\mathbf{o}_{i},\boldsymbol{\epsilon}_{j})\right\|_{2}^{2}. (10)

4.2 Motivation

Variational diffusion connects log-likelihood lower bounds with noise-prediction error weighted by log-SNR change. Reparameterization and additional weights connect this to common pretraining objectives, providing a useful training surrogate.

The motivation establishes a relationship to variational learning, not equality between every weighted or adaptively normalized loss and the final-image log-likelihood. No theorem here bounds the error of the resulting importance ratios or guarantees policy-gradient improvement for the practical objective. Empirical stabilization is the main evidence.

Equations 11–13
ELBO𝜽​(𝐱)≤log⁡π𝜽​(𝐱).\text{ELBO}^{\boldsymbol{\theta}}(\mathbf{x})\leq\log\pi^{\boldsymbol{\theta}}(\mathbf{x}). (11)
ELBO𝜽​(𝐱)=−12​𝔼t,π1​(ϵ)​[−d​λtd​t​‖ϵ𝜽​(𝐳t)−ϵ‖22]+C,\text{ELBO}^{\boldsymbol{\theta}}(\mathbf{x})=-\frac{1}{2}\mathbb{E}_{t,\pi_{1}(\boldsymbol{\epsilon})}\Big[-\frac{d\lambda_{t}}{dt}\Big\|\boldsymbol{\epsilon}^{\boldsymbol{\theta}}(\mathbf{z}_{t})-\boldsymbol{\epsilon}\Big\|_{2}^{2}\Big]+C, (12)
ℒw′(𝜽)=12𝔼t,π0​(𝐱),π1​(ϵ)[wt′⋅−d​λtd​t∥ϵ𝜽(𝐳t)−ϵ∥22].\mathcal{L}_{w^{\prime}}(\boldsymbol{\theta})=\frac{1}{2}\mathbb{E}_{t,\pi_{0}(\mathbf{x}),\pi_{1}(\boldsymbol{\epsilon})}\Big[w^{\prime}_{t}\cdot-\frac{d\lambda_{t}}{dt}\Big\|\boldsymbol{\epsilon}^{\boldsymbol{\theta}}(\mathbf{z}_{t})-\boldsymbol{\epsilon}\Big\|_{2}^{2}\Big]. (13)

4.3 Diagnosing Instability

Loss magnitude varies strongly across timesteps in 400 sampled images. A separate analysis of roughly 20,000 FLUX samples associates larger surrogate values with larger gradient norms. Those gradients are measured through the ratio before advantage scaling or clipping, so they do not directly measure the complete training update.

The authors hypothesize that stochastic surrogate scale can overwhelm reward information. These correlations motivate the following controls; the ablations, rather than the correlations alone, test their training consequences.

Complete original PDF S4.F1
Figure 1: Per-sample loss varies substantially across timesteps. Statistics are computed over 400 samples using Eq. 10. Shaded regions indicate ±1\pm 1 standard deviation.
Complete original PDF S4.F2
Figure 2: Gradient norms scale with surrogate magnitude. Statistics are computed over ∼\sim20K samples using FLUX.1-dev. Gradient norms are computed by backpropagating through the importance ratios, without applying clipping or scaling by the advantages. Mean curves are truncated to the 1st-99th percentile range.

4.4 Reducing Surrogate Variance

Motivated by the preceding analysis, we identify three surrogate variance reduction techniques that prove effective.

Group-shared timestep-noise pairs.

All images in a prompt group use the same sampled timestep/noise pairs. This common-randomness construction removes a source of within-group differences unrelated to reward and makes output comparisons less noisy. It does not eliminate all image-dependent surrogate error or independently establish an exact likelihood ranking.

Stratified timestep sampling.

The schedule is divided into as many equal-length strata as there are Monte Carlo pairs, with one timestep drawn per stratum. Each estimate therefore covers the schedule instead of accidentally concentrating in one region. This controls timestep sampling variability; the exact schedule and finite-sample estimator still affect the objective.

Adaptive loss weighting.

Predictions are converted to clean-image coordinates, then squared error is divided by the stop-gradient mean absolute error. For rectified flow, clean-image error scales with timestep, changing the effective emphasis across noise levels. Normalization approximately equalizes gradient scales across samples.

The denominator depends on the current prediction but is held constant during differentiation. This is a practical adaptive surrogate, not a fixed exact likelihood objective. The displayed denominator has no explicit numerical floor, so zero-residual handling is an implementation detail not documented by the formula.

Equation 14
ℒadaptive​(𝜽∣𝐨i,c)=𝔼t,π1​(ϵ)​[‖𝐱𝜽​(𝐳t)−𝐨i‖22sg⁡(1d​‖𝐱𝜽​(𝐳t)−𝐨i‖1)],\mathcal{L}^{\text{adaptive}}(\boldsymbol{\theta}\mid\mathbf{o}_{i},c)=\mathbb{E}_{t,\pi_{1}(\boldsymbol{\epsilon})}\left[\frac{\left\|\mathbf{x}^{\boldsymbol{\theta}}(\mathbf{z}_{t})-\mathbf{o}_{i}\right\|_{2}^{2}}{\operatorname{sg}(\frac{1}{d}\left\|\mathbf{x}^{\boldsymbol{\theta}}(\mathbf{z}_{t})-\mathbf{o}_{i}\right\|_{1})}\right], (14)

Together the controls reduce surrogate coefficient of variation from 0.230 to 0.128 and mean within-group variation from 0.170 to 0.038. The quadratic fit between surrogate magnitude and gradient norm has R² 0.406 before and 0.328 after. The latter is a measure of association, not itself a variance or a causal sensitivity coefficient.

4.5 Controlling Gradient Steps

Variance reduction is applied throughout, while update regularization depends on the training regime. The tested choices are clipped ratios, a penalty toward the immediately preceding behavior policy, and smooth advantage bounding. No single combination is best for every reward and update schedule.

Importance ratio clipping.

Ratio clipping is the default for batches updated after the policy has moved away from the rollout policy. It limits the surrogate incentive for sufficiently large changes; it is not a hard constraint on the complete learned distribution or every parameter update. FLUX’s tested clipping ranges are small and depend on rollout length.

KL penalty.

The behavior-policy penalty discourages changes in denoised predictions on stored noisy samples. This supplies an inexpensive consistency regularizer without necessarily keeping a separate long-lived reference model. Its practical squared-prediction form should not be read as a directly measured KL between final-image distributions.

The text says only old scalar losses need be cached, but Equation 16 additionally requires old predictions or an equivalent recomputation path. That cache detail is omitted; the paper alone does not establish an implementation defect. The penalty helps retain earlier GenEval capability during OCR training but does not prevent FLUX loss spikes by itself.

Equations 15–16
𝔻(π𝜽∥π𝜽old)\displaystyle\mathbb{D}(\pi^{\boldsymbol{\theta}}\|\pi^{\boldsymbol{\theta}_{\text{old}}}) =𝔼π𝜽​(𝐨i|c)​[log⁡π𝜽​(𝐨i|c)π𝜽old​(𝐨i|c)]\displaystyle=\mathbb{E}_{\pi^{\boldsymbol{\theta}}(\mathbf{o}_{i}|c)}\Big[\log\frac{\pi^{\boldsymbol{\theta}}(\mathbf{o}_{i}|c)}{\pi^{\boldsymbol{\theta}_{\text{old}}}(\mathbf{o}_{i}|c)}\Big] (15)
=12​𝔼t,π𝜽​(𝐨i|c),π1​(ϵ)​[−d​λtd​t​‖ϵ𝜽​(𝐳t)−ϵ𝜽old​(𝐳t)‖22]+C.\displaystyle=\frac{1}{2}\mathbb{E}_{\begin{subarray}{c}t,\pi^{\boldsymbol{\theta}}(\mathbf{o}_{i}|c),\\ \pi_{1}(\boldsymbol{\epsilon})\end{subarray}}\Big[\!-\!\frac{d\lambda_{t}}{dt}\Big\|\boldsymbol{\epsilon}^{\boldsymbol{\theta}}(\mathbf{z}_{t})\!-\!\boldsymbol{\epsilon}^{\boldsymbol{\theta}_{\text{old}}}(\mathbf{z}_{t})\Big\|_{2}^{2}\Big]\!+\!C.
𝔻isimple(π𝜽∥π𝜽old)=𝔼t,π1​(ϵ)[∥𝐱𝜽(𝐳t)−𝐱𝜽old(𝐳t)∥22].\mathbb{D}^{\text{simple}}_{i}(\pi^{\boldsymbol{\theta}}\|\pi^{\boldsymbol{\theta}_{\text{old}}})=\mathbb{E}_{t,\pi_{1}(\boldsymbol{\epsilon})}\Big[\Big\|\mathbf{x}^{\boldsymbol{\theta}}(\mathbf{z}_{t})-\mathbf{x}^{\boldsymbol{\theta}_{\text{old}}}(\mathbf{z}_{t})\Big\|_{2}^{2}\Big]. (16)
Complete original PDF S4.T2
Table 2: Evaluation results for multi-reward experiments on FLUX.1-dev. All methods are trained for 300 iterations. Baseline results are sourced from their original papers [19, 21]. NFEπ𝜽old\text{NFE}_{\pi^{\boldsymbol{\theta}_{\text{old}}}} reports the sum of sampling steps and NMCN_{\text{MC}} (only for V-GRPO). Bold: best; Underline: second best; * marks the frozen strategy proposed in MixGRPO [19].
Original selectable table
Method NFEπ𝜽old\text{NFE}_{\pi^{\boldsymbol{\theta}_{\text{old}}}} NFEπ𝜽\text{NFE}_{\pi^{\boldsymbol{\theta}}} HPS-v2.1 PickScore ImageReward UnifiedReward
FLUX.1-dev — — 0.313 0.227 1.088 3.370
+ DanceGRPO 25 4 0.334 0.225 1.335 3.374
25 4∗ 0.333 0.229 1.235 3.325
25 14 0.356 0.233 1.436 3.397
+ BranchGRPO-WidPru 13.68 8.625 0.364 0.231 1.609 3.383
+ BranchGRPO-DepPru 13.68 8.625 0.369 0.235 1.625 3.404
+ BranchGRPO-Mix 13.68 4.25 0.363 0.230 1.598 3.384
+ BranchGRPO 13.68 13.68 0.363 0.229 1.603 3.386
+ MixGRPO-Flash 8 4∗ 0.357 0.232 1.624 3.402
16 4 0.358 0.236 1.528 3.407
+ MixGRPO 25 4 0.367 0.237 1.629 3.418
+ V-GRPO 16 + 4 4 0.372 0.241 1.731 3.437
25 + 4 4 0.372 0.241 1.749 3.436
Complete original PDF montage, lossless render at native image density
Figure 3: Qualitative comparison from the FLUX.1-dev main experiments. V-GRPO achieves superior performance in alignment, coherence, and style. In the second example, it shows strong text rendering capabilities without leveraging task-specific rewards or datasets.

Advantage soft-clipping.

Advantages are transformed by η tanh(A/η), preserving small values while smoothly bounding extremes. In a single-update iteration the current and old policies initially coincide: ratios are one and ratio clipping is locally inactive, while the prediction-consistency penalty has zero gradient. The policy gradient itself can still be nonzero, so advantage bounding remains useful.

It stabilizes the tested on-policy SD preference stage and shorter FLUX rollouts, but can suppress useful signal for coarse GenEval rewards. Its benefit is therefore conditional on reward and update structure.

Equation 17
Asoft=η⋅tanh⁡(1η​A),A^{\text{soft}}=\eta\cdot\tanh\left(\frac{1}{\eta}A\right), (17)

The final recipe uses all three variance controls, ratio clipping in ordinary multi-update settings, a behavior-policy penalty when retaining prior skills matters, and advantage soft-clipping for on-policy or short-rollout cases. Algorithm 1 shows all options together; the experiment configurations specify which are actually enabled.

5 Experiments

5.1 Experiment Setup

Complete original PDF S5.T3
Table 3: Evaluation results for multi-stage, multi-reward experiments on SD 3.5 M. Baseline results are sourced from DiffusionNFT [46]. NFEπ𝜽old\text{NFE}_{\pi^{\boldsymbol{\theta}_{\text{old}}}} reports the sum of sampling steps and NMCN_{\text{MC}} (for V-GRPO) or the “number of timesteps optimized” (for baselines). Gray-colored: In-domain reward. †Evaluated under 1024×\times1024 resolution. Bold: best; Underline: second best.
Original selectable table
Method #Steps NFEπ𝜽old\text{NFE}_{\pi^{\boldsymbol{\theta}_{\text{old}}}} NFEπ𝜽\text{NFE}_{\pi^{\boldsymbol{\theta}}} Rule-Based Model-Based
GenEval OCR PickScore CLIPScore HPSv2.1 Aesthetics ImgRwd UniRwd
SD XL† — — — 0.55 0.14 0.2242 0.287 0.280 5.60 0.76 2.93
SD 3.5 L† — — — 0.71 0.68 0.2291 0.289 0.288 5.50 0.96 3.25
SD 3.5 M (w/o CFG) — — — 0.24 0.12 0.2051 0.237 0.204 5.13 -0.58 2.02
+ CFG — — — 0.63 0.59 0.2234 0.285 0.279 5.36 0.85 3.03
+ FlowGRPO >>5K 40 40 0.95 0.66 0.2251 0.293 0.274 5.32 1.06 3.18
2K 40 40 0.66 0.92 0.2241 0.290 0.280 5.32 0.95 3.15
4K 40 40 0.54 0.68 0.2350 0.280 0.316 5.90 1.29 3.37
+ DiffusionNFT 1.7K 40 + 40 40 0.94 0.91 0.2380 0.293 0.331 6.01 1.49 3.49
+ V-GRPO 580 40 + 6.9 6.9 0.91 0.91 0.2350 0.298 0.341 6.02 1.52 3.43

Base models.

FLUX.1-dev is already guidance-distilled and runs without explicit classifier-free guidance. SD 3.5 Medium is normally guidance-dependent, but the main multi-stage experiments disable CFG during both training and evaluation. Reward training improves that unguided model; this is tested on the named models rather than a general guarantee that RL always replaces CFG.

Reward functions.

Rule-based rewards use GenEval for compositional alignment and OCR for text rendering. Learned rewards include HPSv2.1, PickScore, CLIPScore, ImageReward, UnifiedReward and Aesthetics. These are benchmark proxies, not a new blinded human preference study; optimizing them can introduce reward-specific biases.

Prompt datasets.

FLUX uses HPDv2 prompts. SD’s GenEval and OCR stages use Flow-GRPO prompt sets; preference stages train on Pick-a-Pic and evaluate on DrawBench. Metric and prompt choices differ by stage, so evaluation should be read with the curriculum and in-domain shading in the tables.

Training and evaluation.

Rollouts use second-order DPM-Solver++, while evaluation returns to first-order Euler to match baselines. Extra old-policy evaluations are needed because rollout predictions cannot simply supply the chosen denoising-loss samples. Rollouts use BF16, but master weights, old/current surrogate calculations and backward passes use FP32.

NFE counts model evaluations; it is not a complete wall-clock or memory measurement. Baseline results are imported from their original papers. The source reports restricted hyperparameter tuning but no repeated-seed uncertainty or common-hardware timing study.

5.2 FLUX.1-dev Main Results

After 300 FLUX training iterations, the 16-step V-GRPO configuration scores HPSv2.1 0.372, PickScore 0.241, ImageReward 1.731 and UnifiedReward 3.437, above the listed baselines in all four columns. The 25-step configuration gives 0.372, 0.241, 1.749 and 3.436, so longer rollout improves one metric slightly while reducing another slightly.

The reported 2× comparison is 150 versus 300 iterations to a similar mean reward, with different evaluation counts. Each V-GRPO iteration has four gradient updates; the table’s “steps” for FLUX are iterations, unlike the SD gradient-update counts. Table 1 averages heterogeneous raw reward scales, so its scalar score can conceal per-metric tradeoffs.

Evaluation mixes the trained model for the first 80% of denoising with the original base model for the rest, following MixGRPO. The result is therefore for this hybrid sampler, not an unmixed fine-tuned checkpoint. Selected examples demonstrate useful text and composition behavior, but do not quantify general world knowledge or human preference.

5.3 SD 3.5 M Main Results

The five-stage SD curriculum uses 580 gradient updates, versus roughly 1,700 for DiffusionNFT. It matches the rounded raw-score mean of 1.71, while trading metrics: GenEval is 0.91 versus 0.94, PickScore 0.2350 versus 0.2380 and UnifiedReward 3.43 versus 3.49; OCR ties at 0.91, while CLIPScore, HPSv2.1, Aesthetics and ImageReward are slightly higher.

Thus “matches performance” refers to a heterogeneous aggregate, not equivalence on each task. The comparison supports fewer updates and lower listed NFE; it does not establish a measured 3× time reduction. Main SD models evaluate at 512×512, while some reference models in Table 3 are marked as using 1024×1024.

Complete original PDF S5.F4
Figure 4: Results for ablation studies.
Original selectable panel captions
(a) Reducing surrogate variance is essential for training FLUX.1-dev.
(b) Importance ratio clipping, but not KL penalty, can resist loss spikes in FLUX.1-dev training.
(c) Advantage soft-clipping can stabilize the fully on-policy Stage-1 training of SD 3.5 M.
(d) Advantage soft-clipping can further stabilize FLUX.1-dev training with 16 sampling steps.
(e) Advantage soft-clipping is suboptimal for Stage-2 training of SD 3.5 M that targets GenEval.
(f) NMCN_{\text{MC}} saturates: too few samples hurt convergence, too many offer marginal gain.

5.4 Ablation Studies

Reducing surrogate variance.

On FLUX, removing shared pairs or stratification destabilizes training; removing adaptive weighting causes a smaller performance loss. On SD preference training, no single component is critical in isolation, though removing all is worse. The effects depend on the base model and setup, rather than proving each component is always necessary.

Complete original PDF S5.T4
Table 4: KL penalty preserves prior capabilities.
Original selectable table
Method GenEval OCR
Importance ratio clipping ϵ=3×10−2\epsilon=3\times 10^{-2} 0.87 0.93
KL penalty β=0.3\beta=0.3 0.91 0.91

Controlling gradient steps.

During the final SD OCR stage, a behavior-policy penalty preserves GenEval at 0.91 with OCR 0.91; ratio clipping gives GenEval 0.87 and OCR 0.93. This is retention versus specialization, not a uniform win. The prose cites a preceding GenEval value of 0.92, but Table 4 shows only the two final configurations.

FLUX still spikes under penalty-only updates. Soft-clipped advantages help SD’s single-update preference stage and 16-step FLUX rollout, yet underperform ratio clipping in the coarse-reward GenEval stage. All six Figure 4 panels are retained.

Effect of N MC .

On FLUX, two Monte Carlo pairs fail to converge well, four work substantially better, and eight add little. This is a tested saturation pattern rather than a universal optimum; SD uses ten pairs during several task-focused stages.

6 Conclusion

The study shows that a stabilized denoising-loss surrogate can be a strong, flexible online post-training method on these image benchmarks. Its value is practical compatibility with existing pretraining losses and decoupled sampling. Remaining limits include surrogate bias, reward gaming, model-specific tuning, limited uncertainty reporting and the lack of wall-clock or broader-domain evidence behind the strongest general claims.

Acknowledgments.

The authors credit feedback from Xinghan Li and partial computational support from Lambda; Serena Yeung-Levy is identified as a Chan Zuckerberg Biohub investigator.

Appendix A Additional Implementation Details

Our implementation adheres closely to the baseline methods, with deviations limited to the key techniques described in Sec. 4.4 and  Sec. 4.5.

A.1 FLUX.1-dev Experiments

Main experiments.

FLUX averages the separately normalized advantages from four rewards. AdamW uses learning rate 10⁻⁵ and weight decay 10⁻⁴, with 300 iterations, four gradient updates per iteration, global batch size eight per update and group size twelve. There is no weight EMA. Four timestep/noise pairs are used throughout.

With 25 rollout steps, ratio clipping is 0.006. With 16 steps, it is 0.01 plus advantage bound two. Neither uses the old-policy penalty. Rollouts are 720×720; evaluation is 50 steps at 1024×1024, using the trained model for the first 40 steps and the base model for the final ten. The latter is explicitly intended to mitigate reward hacking.

Ablation studies.

The appendix gives 25 rollout steps as the default FLUX ablation setting, with main-experiment settings otherwise unchanged. Figure 4(d) explicitly studies 16 steps and is the stated exception. Ablation plots show training reward, not an independent held-out human assessment.

A.2 SD 3.5 M Experiments

Main experiments.

SD first averages rewards and then normalizes group advantages, unlike FLUX’s average of per-reward advantages. LoRA uses rank 32 and scale 64; AdamW uses learning rate 3×10⁻⁴ and weight decay 10⁻⁴. Global batch size is 48 per gradient step and group size 24. Optimizer state is reset between stages, and no EMA is kept.

Stages 1 and 3 each use 150 single-update iterations on preference rewards, four Monte Carlo pairs and advantage bound three. Stage 2 adds GenEval for 100 two-update iterations with ten pairs and ratio bound 0.002. Stage 4 repeats this for 25 two-update iterations with ratio bound 0.004. Stage 5 replaces GenEval with OCR for 15 two-update iterations, ten pairs and penalty coefficient 0.3. The total is 440 iterations and 580 gradient updates.

Training and evaluation use 40 sampling steps at 512×512. The gradient-step-weighted mean pair count is about 6.9, and Table 5’s mean NFE is about 53.8 versus 120 for DiffusionNFT. Baseline stage counts are approximate because of early stopping. CLIPScore is called out-of-domain in one appendix sentence despite being used in the preference ensemble; that label is inconsistent with the stated curriculum.

Complete original PDF A1.T5
Table 5: Comparison of gradient step counts and NFEs across training stages. Our method delivers a 3×3\times speedup over DiffusionNFT in gradient steps, while also requiring fewer function evaluations (NFE) per step on average. DiffusionNFT reports per-stage step counts as approximate values due to early stopping, whereas our counts are exact.
Original selectable table
Stage DiffusionNFT V-GRPO
#Steps NFE #Steps NFE
1 (Human Preferences) 800 120 150 48
2 (GenEval) 300 120 200 60
3 (Human Preferences) 200 120 150 48
4 (GenEval) 200 120 50 60
5 (OCR) 100 120 30 60
Total 1700 120 580 53.8

Single-reward experiments.

Single-reward GenEval training runs 300 two-update iterations, or 600 updates, using Stage 4 settings. OCR runs 25 single-update iterations with ten pairs and advantage bound four. PickScore runs 300 single-update iterations with four pairs and advantage bound three.

For OCR-trained models, CFG is re-enabled when evaluating non-OCR rewards, following the baseline protocol; the blanket no-CFG description has this explicit exception. GenEval reaches 0.97 versus DiffusionNFT’s 0.98; OCR reaches 0.98 versus 0.97; PickScore ties at 0.2403. Other metrics vary, so fast optimization of a target reward is not uniform preservation of all capabilities.

Ablation studies.

SD ablations otherwise use the corresponding main-stage configuration. Figure 5’s horizontal axis extends to 200 iterations although the main preference stages are described as 150; the source does not detail that duration difference. It still shows that removing all controls is worse while individual removals have modest effects.

Appendix B Additional Results

Additional examples retain every model row, prompt and image panel. Figure 6 compares adaptive weighting under different prediction coordinates: noise prediction collapses in the shown FLUX run, velocity prediction remains stable, and clean-image prediction converges somewhat faster. These results concern that weighted training setup, not intrinsic failure of noise prediction in all diffusion training.

The single-reward table exposes cross-metric tradeoffs: OCR-focused V-GRPO reaches 0.98 OCR but only 0.47 GenEval under the stated evaluation protocol, and GenEval-focused V-GRPO has lower several learned-reward scores than DiffusionNFT. The qualitative montages are selected examples without a measured selection rate or blinded human evaluation.

Complete original PDF A2.T6
Table 6: Evaluation results for single-reward experiments on SD 3.5 M. All methods disable CFG during both training and evaluation. For models trained with the OCR reward, CFG is re-enabled when evaluating non-OCR rewards, following DiffusionNFT. Baseline results are sourced from DiffusionNFT. Gray-colored: In-domain reward. Bold: best; Underline: second best.
Original selectable table
Method #Steps Rule-Based Model-Based
GenEval OCR PickScore CLIPScore HPSv2.1 Aesthetic ImgRwd UniRwd
SD 3.5 M (w/o CFG) — 0.24 0.12 0.2051 0.237 0.204 5.13 -0.58 2.02
+ CFG — 0.63 0.59 0.2234 0.285 0.279 5.36 0.85 3.03
+ FlowGRPO 4K 0.97 0.30 0.2178 0.277 0.248 5.15 0.74 2.87
+ DiffusionNFT 1K 0.98 0.36 0.2192 0.271 0.251 5.33 0.68 2.91
+ V-GRPO 600 0.97 0.34 0.2150 0.280 0.225 5.20 0.35 2.80
+ FlowGRPO 1K 0.66 0.96 0.2194 0.280 0.257 5.18 0.31 2.86
+ DiffusionNFT 150 0.54 0.97 0.2163 0.281 0.246 5.19 0.37 2.81
+ V-GRPO 25 0.47 0.98 0.2170 0.277 0.243 5.21 0.28 2.83
+ FlowGRPO 4K 0.54 0.60 0.2362 0.257 0.295 6.42 1.17 3.17
+ DiffusionNFT 2K 0.53 0.64 0.2403 0.270 0.315 6.17 1.29 3.40
+ V-GRPO 300 0.66 0.62 0.2403 0.267 0.308 6.42 1.30 3.26
Complete original PDF A2.F5
Figure 5: Ablation studies of surrogate variance reduction techniques. Implementation details follow those of Stage-1 training in the SD 3.5 main experiments.
Complete original PDF A2.F6
Figure 6: Ablation studies of alternative reparameterizations of model predictions. Implementation details follow those used in the FLUX.1-dev experiments.
Complete original PDF montage, lossless render at native image density
Figure 7: Qualitative comparison from the FLUX.1-dev main experiments. V-GRPO achieves superior performance in alignment, coherence, and style. In the fourth example, it demonstrates strong world knowledge.
Complete original PDF montage, lossless render at native image density
Figure 8: Qualitative comparison from the SD 3.5 M main experiments. V-GRPO achieves superior performance in alignment, coherence, and style.
Concept demonstration

Open image ↗