Literature
Display
← All papers

Flow Matching for Generative Modeling

Yaron Lipman · Ricky T. Q. Chen · Heli Ben-Hamu · Maximilian Nickel · Matt Le

· ICLR 2023 ↗

Contents

Overview

At a glance

Intuition. Imagine animating a cloud of random points into realistic data. The model learns a local arrow at each position and time, rather than simulating every complete animation during training. A point x and time t go into the network; a velocity vector comes out.

Reading the math. Training matches the predicted velocity vθ(t, x) to a constructed target ut using squared error. For an ideal straight path from noise z to data a, xt = (1−t)z + ta has velocity a−z. This conditional example explains the regression target; it does not guarantee globally shortest transport. Generation still follows the learned arrows with a differential-equation solver.

Flow Matching trains a continuous normalizing flow by regressing a neural vector field against analytically constructed conditional targets. Training samples a time, a data point, and a point on a conditional probability path, without simulating the model ODE. Conditional Flow Matching has the same expected parameter gradient as the generally intractable marginal objective. Gaussian paths include diffusion-style paths and a linear mean/scale interpolation that is optimal transport between each pair of conditional Gaussian endpoints. Image experiments favor the latter in the tested likelihood, sample-quality, and sampling-cost comparisons. Sampling and likelihood evaluation still require numerical ODE integration.

1 Introduction

Likelihood-based training of continuous normalizing flows can require costly forward and backward ODE solves. Flow Matching instead specifies a path of distributions and learns a field that generates it. The key construction uses simple per-data-point paths and a conditional regression objective whose expected gradient matches the marginal objective. This extends simulation-free training beyond the particular paths of standard diffusion processes. The report evaluates these ideas on image generation and super-resolution; the opening sample grid is selected, while later appendix grids are labeled non-curated.

Complete original S1.F1 from the exact-version PDF, including every panel, label, and original color.
Figure 1: Unconditional ImageNet-128 samples of a CNF trained using Flow Matching with Optimal Transport probability paths.

2 Preliminaries: Continuous Normalizing Flows

A time-dependent vector field defines a flow through an ODE and transports a simple initial density into a more complex distribution. A neural parameterization gives a continuous normalizing flow. Under the required existence, uniqueness, and invertibility conditions, the change-of-variables formula defines its density; the continuity equation relates that density path to the field. These mathematical conditions matter: an arbitrary neural field is not automatically a globally well-behaved diffeomorphism. The construction follows the neural-ODE framework [Chen et al., 2018].

Equations 1–4
dd​t​ϕt​(x)\displaystyle\frac{d}{dt}\phi_{t}(x) =vt​(ϕt​(x))\displaystyle=v_{t}(\phi_{t}(x)) (1)
ϕ0​(x)\displaystyle\phi_{0}(x) =x\displaystyle=x (2)
pt=[ϕt]∗​p0p_{t}=[\phi_{t}]_{*}p_{0} (3)
[ϕt]∗​p0​(x)=p0​(ϕt−1​(x))​det[∂ϕt−1∂x​(x)].[\phi_{t}]_{*}p_{0}(x)=p_{0}(\phi_{t}^{-1}(x))\det\left[\frac{\partial\phi_{t}^{-1}}{\partial x}(x)\right]. (4)

3 Flow Matching

The ideal objective is expected squared error between the learned field and a target field along a prescribed density path. Zero population loss would recover that field on the relevant distribution and generate the target path under the stated regularity conditions. In practice, neither the marginal density nor its field is directly available. The next sections construct them through conditional mixtures and replace the intractable regression target. The final target is a smoothed approximation to the data distribution when the endpoint noise is positive.

Equation 5
ℒFM​(θ)=𝔼t,pt​(x)​‖vt​(x)−ut​(x)‖2,\mathcal{L}_{\scriptscriptstyle\text{FM}}(\theta)=\mathbb{E}_{t,p_{t}(x)}\|v_{t}(x)-u_{t}(x)\|^{2}, (5)

3.1 Constructing p t , u t from conditional probability paths and vector fields

For each data point x₁, a conditional path begins at a common noise distribution and ends in a narrow distribution around x₁. Averaging these densities over the data distribution defines the marginal path. The corresponding marginal field is a posterior-weighted average of conditional fields at the current position, not an unweighted average over data examples. Theorem 1 proves that this field satisfies the continuity equation for the mixture, assuming the differentiation and integration exchanges used in the proof are valid.

Equations 6–8
pt​(x)=∫pt​(x|x1)​q​(x1)​d​x1,p_{t}(x)=\int p_{t}(x|x_{1})q(x_{1})dx_{1}, (6)
p1​(x)=∫p1​(x|x1)​q​(x1)​d​x1≈q⁡(x).p_{1}(x)=\int p_{1}(x|x_{1})q(x_{1})dx_{1}\approx q(x). (7)
ut​(x)=∫ut​(x|x1)​pt​(x|x1)​q​(x1)pt​(x)​d​x1,u_{t}(x)=\int u_{t}(x|x_{1})\frac{p_{t}(x|x_{1})q(x_{1})}{p_{t}(x)}dx_{1}, (8)

3.2 Conditional Flow Matching

Conditional Flow Matching samples x₁ and a point from its conditional path, then regresses the network against the conditional field. Theorem 2 shows that its expected loss differs from the marginal Flow Matching loss only by a parameter-independent constant, so their gradients and optima coincide. This does not make every stochastic gradient identical, guarantee successful optimization, or make the conditional loss vanish when the marginal field is correct; conditional targets can have irreducible variance.

Equation 9
ℒCFM​(θ)=𝔼t,q⁡(x1),pt​(x|x1)​‖vt​(x)−ut​(x|x1)‖2,\mathcal{L}_{\scriptscriptstyle\text{CFM}}(\theta)=\mathbb{E}_{t,q(x_{1}),p_{t}(x|x_{1})}\big\|v_{t}(x)-u_{t}(x|x_{1})\big\|^{2}, (9)

4 Conditional Probability Paths and Vector Fields

The chosen family uses isotropic Gaussian conditional paths with time-dependent mean μₜ and positive standard deviation σₜ. An affine map ψₜ(x)=σₜx+μₜ generates each path, giving velocity (σ′ₜ/σₜ)(x−μₜ)+μ′ₜ. Reparameterization permits sampling noise x₀ directly and regressing against the time derivative of ψₜ(x₀). Theorem 3 gives the unique field for this specified invertible affine flow, not the only possible field that transports the same densities. A positive endpoint scale σ_min keeps the construction nonsingular while leaving residual smoothing.

Equations 10–15
pt​(x|x1)=𝒩⁡(x|μt​(x1),σt​(x1)2​I),p_{t}(x|x_{1})={\mathcal{N}}(x\,|\,\mu_{t}(x_{1}),\sigma_{t}(x_{1})^{2}I), (10)
ψt​(x)=σt​(x1)​x+μt​(x1).\psi_{t}(x)=\sigma_{t}(x_{1})x+\mu_{t}(x_{1}). (11)
[ψt]∗​p​(x)=pt​(x|x1).\left[\psi_{t}\right]_{*}p(x)=p_{t}(x|x_{1}). (12)
dd​t​ψt​(x)=ut​(ψt​(x)|x1).\frac{d}{dt}\psi_{t}(x)=u_{t}(\psi_{t}(x)|x_{1}). (13)
ℒCFM​(θ)=𝔼t,q⁡(x1),p⁡(x0)​‖vt​(ψt​(x0))−dd​t​ψt​(x0)‖2.{\mathcal{L}}_{\scriptscriptstyle\text{CFM}}(\theta)=\mathbb{E}_{t,q(x_{1}),p(x_{0})}\Big\|v_{t}(\psi_{t}(x_{0}))-\frac{d}{dt}\psi_{t}(x_{0})\Big\|^{2}. (14)
ut​(x|x1)=σt′​(x1)σt​(x1)​(x−μt​(x1))+μt′​(x1).u_{t}(x|x_{1})=\frac{\sigma^{\prime}_{t}(x_{1})}{\sigma_{t}(x_{1})}\left(x-\mu_{t}(x_{1})\right)+\mu^{\prime}_{t}(x_{1}). (15)

4.1 Special instances of Gaussian conditional probability paths

Differentiable mean and scale schedules satisfying the endpoint conditions yield many possible Gaussian paths. The paper compares diffusion-derived schedules with linear interpolation between conditional Gaussian endpoints.

Example I: Diffusion conditional VFs.

Variance-exploding and variance-preserving diffusion paths recover familiar probability-flow vector fields after reversing time to run from noise to data. The derivations preserve the required square roots, time reversals, and signs. Standard finite-time diffusion schedules only approximate a pure noise endpoint, whereas the proposed path design can impose its endpoint directly. Figure 2 contrasts a diffusion conditional score with an OT conditional vector field, so it compares different regression targets rather than claiming the two pictures show the same quantity. The connection to probability-flow ODEs is grounded in the diffusion formulation [Song et al., 2020b].

Equations 16–19
pt​(x)=𝒩⁡(x|x1,σ1−t2​I),p_{t}(x)={\mathcal{N}}(x|x_{1},\sigma_{1-t}^{2}I), (16)
ut​(x|x1)=−σ1−t′σ1−t​(x−x1).u_{t}(x|x_{1})=-\frac{\sigma^{\prime}_{1-t}}{\sigma_{1-t}}(x-x_{1}). (17)
pt​(x|x1)=𝒩⁡(x|α1−t​x1,(1−α1−t2)​I),where ​αt=e−12​T​(t),T⁡(t)=∫0tβ⁡(s)​𝑑s,p_{t}(x|x_{1})={\mathcal{N}}(x\,|\,\alpha_{1-t}x_{1},\left(1-\alpha_{1-t}^{2}\right)I),\text{where }\alpha_{t}=e^{-\frac{1}{2}T(t)},T(t)=\int_{0}^{t}\beta(s)ds, (18)
ut​(x|x1)=α1−t′1−α1−t2​(α1−t​x−x1)=−T′​(1−t)2​[e−T⁡(1−t)​x−e−12​T​(1−t)​x11−e−T⁡(1−t)].u_{t}(x|x_{1})=\frac{\alpha^{\prime}_{1-t}}{1-\alpha^{2}_{1-t}}\left(\alpha_{1-t}x-x_{1}\right)=-\frac{T^{\prime}(1-t)}{2}\left[\frac{e^{-T(1-t)}x-e^{-\frac{1}{2}T(1-t)}x_{1}}{1-e^{-T(1-t)}}\right]. (19)
Complete original S4.F2 from the exact-version PDF, including every panel, label, and original color.
Figure 2: Compared to the diffusion path’s conditional score function, the OT path’s conditional vector field has constant direction in time and is arguably simpler to fit with a parametric model. Note the blue color denotes larger magnitude while red color denotes smaller magnitude.

Example II: Optimal Transport conditional VFs.

The proposed mean is tx₁ and the scale is 1−(1−σ_min)t. A noise point follows a straight conditional trajectory with constant velocity x₁−(1−σ_min)x₀, yielding a particularly simple regression target. This is the Wasserstein-2 displacement interpolation between the standard Gaussian and the narrow Gaussian centered on a particular data point [McCann, 1997]. The paper explicitly warns that averaging these conditional constructions does not make the marginal field an optimal-transport solution. Consequently, straight conditional paths do not prove that every trajectory of the learned marginal flow is straight.

Equations 20–24
μt​(x)=t​x1, and ​σt​(x)=1−(1−σmin)​t.\mu_{t}(x)=tx_{1},\text{ and }\sigma_{t}(x)=1-(1-\sigma_{\text{min}})t. (20)
ut​(x|x1)=x1−(1−σmin)​x1−(1−σmin)​t,u_{t}(x|x_{1})=\frac{x_{1}-(1-\sigma_{\text{min}})x}{1-(1-\sigma_{\text{min}})t}, (21)
ψt​(x)=(1−(1−σmin)​t)​x+t​x1,\psi_{t}(x)=(1-(1-\sigma_{\text{min}})t)x+tx_{1}, (22)
ℒCFM​(θ)=𝔼t,q⁡(x1),p⁡(x0)​‖vt​(ψt​(x0))−(x1−(1−σmin)​x0)‖2.{\mathcal{L}}_{\scriptscriptstyle\text{CFM}}(\theta)=\mathbb{E}_{t,q(x_{1}),p(x_{0})}\Big\|v_{t}(\psi_{t}(x_{0}))-\Big(x_{1}-(1-\sigma_{\min})x_{0}\Big)\Big\|^{2}. (23)
pt=[(1−t)​id+t​ψ]⋆​p0p_{t}=[(1-t)\mathrm{id}+t\psi]_{\star}p_{0} (24)
Complete original S4.F3 from the exact-version PDF, including every panel, label, and original color.
Figure 3: Diffusion and OT trajectories.

5 Related Work

Earlier CNFs use likelihood-based ODE training or regularization to simplify integration. Other simulation-free approaches face difficult integrals or biased mini-batch gradients in the cited formulations. Conditional Flow Matching takes inspiration from denoising score matching while regressing vector fields directly and permitting more general paths. The paper also acknowledges finite-time diffusion-bridge approaches and concurrent conditional-objective work. The checkerboard experiments illustrate differences in learned density evolution and low-step integration; they are qualitative evidence for this setting rather than a general ranking of all flow or diffusion methods.

Complete original S5.F4 from the exact-version PDF, including every panel, label, and original color.
Figure 4: (left) Trajectories of CNFs trained with different objectives on 2D checkerboard data. The OT path introduces the checkerboard pattern much earlier, while FM results in more stable training. (right) FM with OT results in more efficient sampling, solved using the midpoint scheme.

6 Experiments

Experiments use CIFAR-10 and ImageNet at 32, 64, and 128 pixels, plus conditional super-resolution. Unless specified otherwise, evaluation uses the adaptive Dormand–Prince solver with absolute and relative tolerances 10⁻⁵. Table 1 preserves likelihood in bits per dimension, FID, and function-evaluation counts separately. Lower values favor the model for all three metrics, but they assess different properties.

CIFAR-10 ImageNet 32×\times32 ImageNet 64×\times64
Model NLL↓\downarrow FID↓\downarrow NFE↓\downarrow NLL↓\downarrow FID↓\downarrow NFE↓\downarrow NLL↓\downarrow FID↓\downarrow NFE↓\downarrow
Ablations
DDPM 3.12 7.48 274 3.54 6.99 262 3.32 17.36 264
Score Matching 3.16 19.94 242 3.56 5.68 178 3.40 19.74 441
ScoreFlow 3.09 20.78 428 3.55 14.14 195 3.36 24.95 601
Ours
FM w/ Diffusion 3.10 8.06 183 3.54 6.37 193 3.33 16.88 187
FM w/ OT 2.99 6.35 142 3.53 5.02 122 3.31 14.45 138
ImageNet 128×\times128
Model NLL↓\downarrow FID↓\downarrow
MGAN (Hoang et al., 2018) – 58.9
PacGAN2 (Lin et al., 2018) – 57.5
Logo-GAN-AE (Sage et al., 2018) – 50.9
Self-cond. GAN (Lučić et al., 2019) – 41.7
Uncond. BigGAN (Lučić et al., 2019) – 25.3
PGMGAN (Armandpour et al., 2021) – 21.7
FM w/ OT 2.90 20.9
Table 1: Likelihood (BPD), quality of generated samples (FID), and evaluation time (NFE) for the same model trained with different methods.

6.1 Density Modeling and Sample Quality on ImageNet

The ablations use a shared U-Net-style architecture and broadly matched training settings, with some baselines allowed extra iterations for convergence. FM with the OT path has the best reported likelihood, FID, and mean function-evaluation count across the three matched dataset columns. For ImageNet-64 it reports 3.31 BPD, FID 14.45, and 138 evaluations. CIFAR-10 FIDs are worse than commonly reported values elsewhere, which the authors attribute to an architecture not tuned for that dataset. The ImageNet-128 FID of 20.9 is compared with the selected unconditional baselines; a self-supervised conditioned model is excluded. Training-throughput comparisons also change batch size and model size, including a 25% larger FM model, so fewer iterations alone are not a matched-compute speedup.

Complete original S6.F5 from the exact-version PDF, including every panel, label, and original color.
Figure 5: Image quality during training, ImageNet 64×\times64.

6.2 Sampling Efficiency

Sampling starts from Gaussian noise and integrates the learned vector field. Image sequences and checkerboard examples show earlier emergence of structure under the OT-path recipe. Fixed-step tests compare numerical error with a 1,000-evaluation reference using 256 seeds; FM-OT needs roughly 60% as many evaluations to reach the illustrated error level. FID-versus-evaluation curves favor it among the tested objectives and solvers. These are empirical numerical and perceptual trade-offs, not guarantees for every solver, tolerance, image, or learned field. The appendix retains same-seed examples at multiple evaluation counts.

Complete original S6.F6 from the exact-version PDF, including every panel, label, and original color.
Figure 6: Sample paths from the same initial noise with models trained on ImageNet 64×\times64. The OT path reduces noise roughly linearly, while diffusion paths visibly remove noise only towards the end of the path. Note also the differences between the generated images.
Complete original S6.F7 from the exact-version PDF, including every panel, label, and original color.
Figure 7: Flow Matching, especially when using OT paths, allows us to use fewer evaluations for sampling while retaining similar numerical error (left) and sample quality (right). Results are shown for models trained on ImageNet 32×\times32, and numerical errors are for the midpoint scheme.

6.3 Conditional sampling from low-resolution images

A conditional model upsamples ImageNet validation images from 64×64 to 256×256. Compared with SR3, FM-OT improves FID from 5.2 to 3.4 and Inception Score from 180.1 to 200.8, while PSNR falls from 26.4 to 24.7 and SSIM from 0.762 to 0.747. Thus distributional sample-quality metrics improve while paired-image fidelity metrics worsen; describing the result requires that trade-off. All example pairs and their low-resolution insets are preserved.

Model FID↓\downarrow IS↑\uparrow PSNR↑\uparrow SSIM↑\uparrow
Reference 1.9 240.8 – –
Regression 15.2 121.1 27.9 0.801
SR3 (Saharia et al., 2022) 5.2 180.1 26.4 0.762
FM w/ OT 3.4 200.8 24.7 0.747
Table 2: Image super-resolution on the ImageNet validation set.

7 Conclusion

Conditional vector-field regression provides scalable training without trajectory simulation and allows paths beyond conventional diffusion schedules. The strongest empirical results here use the conditional Gaussian OT construction. More general kernels and non-isotropic paths are suggested as future work. Sampling cost, approximation error, endpoint smoothing, and the limited scope of the image benchmarks remain relevant when extending the result.

8 Social responsibility

The authors acknowledge harmful uses of generated images and suggest controlled training content and image validation as mitigations. They also motivate more efficient training as a possible way to reduce time and energy use. The paper does not provide a complete misuse-prevention method or a lifecycle energy measurement.

Appendix A Theorem Proofs

Theorem 1 averages the conditional continuity equations and exchanges integration with time differentiation and divergence under Leibniz-rule regularity. Theorem 2 expands both squared losses: the network-norm and network/target cross terms agree under the mixture identity, and the remaining target-only term is independent of the parameters. The proof assumes sufficient decay, integrability, and boundedness to justify Fubini and differentiation. Theorem 3 differentiates the affine conditional flow and substitutes its inverse, requiring σₜ>0. These proofs establish the identities under their assumptions; they do not remove finite-model, finite-data, optimization, or numerical-solver error.

Equations 25
dd​t​pt​(x)\displaystyle\frac{d}{dt}p_{t}(x) =∫(dd​tpt(x|x1))q(x1)dx1=−∫div(ut(x|x1)pt(x|x1))q(x1)dx1\displaystyle=\int\Big(\frac{d}{dt}p_{t}(x|x_{1})\Big)q(x_{1})dx_{1}=-\int\mathrm{div}\Big(u_{t}(x|x_{1})p_{t}(x|x_{1})\Big)q(x_{1})dx_{1}
=−div⁡(∫ut​(x|x1)​pt​(x|x1)​q​(x1)​d​x1)=−div⁡(ut​(x)​pt​(x)),\displaystyle=-\mathrm{div}\Big(\int u_{t}(x|x_{1})p_{t}(x|x_{1})q(x_{1})dx_{1}\Big)=-\mathrm{div}\Big(u_{t}(x)p_{t}(x)\Big),
‖vt​(x)−ut​(x)‖2\displaystyle\|v_{t}(x)-u_{t}(x)\|^{2} =‖vt​(x)‖2−2​⟨vt​(x),ut​(x)⟩+‖ut​(x)‖2\displaystyle=\left\|v_{t}(x)\right\|^{2}-2\left\langle v_{t}(x),u_{t}(x)\right\rangle+\left\|u_{t}(x)\right\|^{2}
‖vt​(x)−ut​(x|x1)‖2\displaystyle\|v_{t}(x)-u_{t}(x|x_{1})\|^{2} =‖vt​(x)‖2−2​⟨vt​(x),ut​(x|x1)⟩+‖ut​(x|x1)‖2\displaystyle=\left\|v_{t}(x)\right\|^{2}-2\left\langle v_{t}(x),u_{t}(x|x_{1})\right\rangle+\left\|u_{t}(x|x_{1})\right\|^{2}
𝔼pt​(x)​‖vt​(x)‖2\displaystyle\mathbb{E}_{p_{t}(x)}\|v_{t}(x)\|^{2} =∫‖vt​(x)‖2​pt​(x)​𝑑x=∫‖vt​(x)‖2​pt​(x|x1)​q​(x1)​d​x1​𝑑x\displaystyle=\int\|v_{t}(x)\|^{2}p_{t}(x)dx=\int\|v_{t}(x)\|^{2}p_{t}(x|x_{1})q(x_{1})dx_{1}dx
=𝔼q⁡(x1),pt​(x|x1)​‖vt​(x)‖2,\displaystyle=\mathbb{E}_{q(x_{1}),p_{t}(x|x_{1})}\|v_{t}(x)\|^{2},
𝔼pt​(x)​⟨vt​(x),ut​(x)⟩\displaystyle\mathbb{E}_{p_{t}(x)}\left\langle v_{t}(x),u_{t}(x)\right\rangle =∫⟨vt​(x),∫ut​(x|x1)​pt​(x|x1)​q​(x1)​d​x1pt​(x)⟩​pt​(x)​𝑑x\displaystyle=\int\left\langle v_{t}(x),\frac{\int u_{t}(x|x_{1})p_{t}(x|x_{1})q(x_{1})dx_{1}}{p_{t}(x)}\right\rangle p_{t}(x)dx
=∫⟨vt​(x),∫ut​(x|x1)​pt​(x|x1)​q​(x1)​d​x1⟩​𝑑x\displaystyle=\int\left\langle v_{t}(x),\int u_{t}(x|x_{1})p_{t}(x|x_{1})q(x_{1})dx_{1}\right\rangle dx
=∫⟨vt​(x),ut​(x|x1)⟩​pt​(x|x1)​q​(x1)​d​x1​𝑑x\displaystyle=\int\left\langle v_{t}(x),u_{t}(x|x_{1})\right\rangle p_{t}(x|x_{1})q(x_{1})dx_{1}dx
=𝔼q⁡(x1),pt​(x|x1)​⟨vt​(x),ut​(x|x1)⟩,\displaystyle=\mathbb{E}_{q(x_{1}),p_{t}(x|x_{1})}\left\langle v_{t}(x),u_{t}(x|x_{1})\right\rangle,
dd​t​ψt​(x)=wt​(ψt​(x)).\frac{d}{dt}\psi_{t}(x)=w_{t}(\psi_{t}(x)).
ψt′​(ψ−1​(y))=wt​(y),\psi^{\prime}_{t}(\psi^{-1}(y))=w_{t}(y), (25)
ψt−1​(y)=y−μt​(x1)σt​(x1).\psi_{t}^{-1}(y)=\frac{y-\mu_{t}(x_{1})}{\sigma_{t}(x_{1})}.
ψt′​(x)=σt′​(x1)​x+μt′​(x1).\psi_{t}^{\prime}(x)=\sigma_{t}^{\prime}(x_{1})x+\mu_{t}^{\prime}(x_{1}).
wt​(y)=σt′​(x1)σt​(x1)​(y−μt​(x1))+μt′​(x1)w_{t}(y)=\frac{\sigma^{\prime}_{t}(x_{1})}{\sigma_{t}(x_{1})}\left(y-\mu_{t}(x_{1})\right)+\mu_{t}^{\prime}(x_{1})

Appendix B The continuity equation

The continuity equation ∂ₜp+div(pv)=0 expresses conservation of probability mass. Together with a suitable initial density and regular flow conditions, it connects a vector field to the density path it generates. The spatial divergence and its coordinate definition are preserved.

Equation 26
dd​t​pt​(x)+div⁡(pt​(x)​vt​(x))=0,\frac{d}{dt}p_{t}(x)+\mathrm{div}(p_{t}(x)v_{t}(x))=0, (26)

Appendix C Computing probabilities of the CNF model

The appendix explains likelihood evaluation by reverse integration, an efficient stochastic divergence estimator, correction for data transformations, and conversion to bits per dimension. Likelihood evaluation remains an ODE computation even though training is simulation-free.

ODE for computing p 1 ​ ( x 1 )

Along a trajectory, the log-density derivative is minus the field divergence. To evaluate a given final point, integrate the state backward while accumulating the divergence, then combine the recovered prior log density with that accumulator. The sign and initial conditions in Equations 27–33 are retained. One preceding unnumbered HTML equation contains an unmatched-delimiter conversion marker; its visible display is restored from the exact PDF, with original MathML retained in a disclosure.

Equations 27–33
Complete original A3.EGx6 from the exact-version PDF, including every panel, label, and original color.
Original source MathML (contains a conversion marker for an unmatched delimiter)
dd​t​log⁡pt​(ϕt​(x))+div⁡(vt​(ϕt​(x))=0CLOSE.\displaystyle\frac{d}{dt}\log p_{t}(\phi_{t}(x))+\mathrm{div}(v_{t}(\phi_{t}(x))=0.
logp1(ϕ1(x))−logp0(ϕ0(x))=−∫01div(vt(ϕt(x)))dt\displaystyle\log p_{1}(\phi_{1}(x))-\log p_{0}(\phi_{0}(x))=-\int_{0}^{1}\mathrm{div}(v_{t}(\phi_{t}(x)))dt (27)
dd​t​[ϕt​(x)f⁡(t)]=[vt​(ϕt​(x))−div⁡(vt​(ϕt​(x)))]\displaystyle\frac{d}{dt}\begin{bmatrix}\phi_{t}(x)\\ f(t)\end{bmatrix}=\begin{bmatrix}v_{t}(\phi_{t}(x))\\ -\mathrm{div}(v_{t}(\phi_{t}(x)))\end{bmatrix} (28)
[ϕ0​(x)f⁡(0)]=[x0c].\displaystyle\begin{bmatrix}\phi_{0}(x)\\ f(0)\end{bmatrix}=\begin{bmatrix}x_{0}\\ c\end{bmatrix}. (29)
f⁡(1)=c+log⁡p1​(x1)−log⁡p0​(x0).f(1)=c+\log p_{1}(x_{1})-\log p_{0}(x_{0}). (30)
dd​s​[ϕ1−s​(x)f⁡(1−s)]=[−v1−s​(ϕ1−s​(x))div⁡(v1−s​(ϕ1−s​(x)))]\displaystyle\frac{d}{ds}\begin{bmatrix}\phi_{1-s}(x)\\ f(1-s)\end{bmatrix}=\begin{bmatrix}-v_{1-s}(\phi_{1-s}(x))\\ \mathrm{div}(v_{1-s}(\phi_{1-s}(x)))\end{bmatrix} (31)
[ϕ1​(x)f⁡(1)]=[x10].\displaystyle\begin{bmatrix}\phi_{1}(x)\\ f(1)\end{bmatrix}=\begin{bmatrix}x_{1}\\ 0\end{bmatrix}. (32)
f⁡(0)\displaystyle f(0) =log⁡p0​(x0)−log⁡p1​(x1)\displaystyle=\log p_{0}(x_{0})-\log p_{1}(x_{1})
log⁡p1​(x1)=log⁡p0​(x0)−f⁡(0).\displaystyle\log p_{1}(x_{1})=\log p_{0}(x_{0})-f(0). (33)

Unbiased estimator to p 1 ​ ( x 1 )

Hutchinson’s estimator replaces a full Jacobian trace by zᵀDv z with E[zzᵀ]=I. Along the exact deterministic state trajectory and under the stated integrability assumptions, the resulting accumulated quantity is an unbiased estimator of log p₁. Although this subsection’s heading and final sentence refer to p₁, its proof and Equation 35 establish log-density unbiasedness. Exponentiating that estimator is generally biased, and numerical integration adds controlled approximation error. The original derivation is preserved without silently changing its claim.

Equations 34–35
dd​s​[ϕ1−s​(x)f~​(1−s)]=[−v1−s​(ϕ1−s​(x))zT​D​v1−s​(ϕ1−s​(x))​z],\displaystyle\frac{d}{ds}\begin{bmatrix}\phi_{1-s}(x)\\ \tilde{f}(1-s)\end{bmatrix}=\begin{bmatrix}-v_{1-s}(\phi_{1-s}(x))\\ z^{T}Dv_{1-s}(\phi_{1-s}(x))z\end{bmatrix}, (34)
𝔼z​[log⁡p0​(x0)−f~​(0)]\displaystyle\mathbb{E}_{z}\left[\log p_{0}(x_{0})-\tilde{f}(0)\right] =log⁡p0​(x0)−𝔼z​[f~​(0)−f~​(1)]\displaystyle=\log p_{0}(x_{0})-\mathbb{E}_{z}\left[\tilde{f}(0)-\tilde{f}(1)\right]
=log⁡p0​(x0)−𝔼z​[∫01zT​D​v1−s​(ϕ1−s​(x))​z​𝑑s]\displaystyle=\log p_{0}(x_{0})-\mathbb{E}_{z}\left[\int_{0}^{1}z^{T}Dv_{1-s}(\phi_{1-s}(x))z\,ds\right]
=log⁡p0​(x0)−∫01𝔼z​[zT​D​v1−s​(ϕ1−s​(x))​z]​𝑑s\displaystyle=\log p_{0}(x_{0})-\int_{0}^{1}\mathbb{E}_{z}\left[z^{T}Dv_{1-s}(\phi_{1-s}(x))z\right]ds
=log⁡p0​(x0)−∫01div⁡(v1−s​(ϕ1−s​(x)))​𝑑s\displaystyle=\log p_{0}(x_{0})-\int_{0}^{1}\mathrm{div}(v_{1-s}(\phi_{1-s}(x)))ds
=log⁡p0​(x0)−(f⁡(0)−f⁡(1))\displaystyle=\log p_{0}(x_{0})-\left(f(0)-f(1)\right)
=log⁡p0​(x0)−(log⁡p0​(x0)−log⁡p1​(x1))\displaystyle=\log p_{0}(x_{0})-\left(\log p_{0}(x_{0})-\log p_{1}(x_{1})\right)
=log⁡p1​(x1),\displaystyle=\log p_{1}(x_{1}),
log⁡p0​(x0)−f~​(0)\displaystyle\log p_{0}(x_{0})-\tilde{f}(0) (35)

Transformed data

Data centering and scaling require the corresponding change-of-variables determinant. Mapping model-space pixels from [−1,1] to [0,256] subtracts 7d log 2 from the continuous log density in the pixel coordinates. This correction is distinct from uniform dequantization of discrete pixel values.

Equations 36
ψ⁡(x)=φ∘ϕ⁡(x)\displaystyle\psi(x)=\varphi\circ\phi(x)
p1​(x)=ψ∗​p0​(x)\displaystyle p_{1}(x)=\psi_{*}p_{0}(x) =p0​(ϕ−1​(φ−1​(x)))​det[D​ϕ−1​(φ−1​(x))]​det[D​φ−1​(x)]\displaystyle=p_{0}(\phi^{-1}(\varphi^{-1}(x)))\det\left[D\phi^{-1}(\varphi^{-1}(x))\right]\det\left[D\varphi^{-1}(x)\right]
=(ϕ∗​p0​(φ−1​(x)))​det[D​φ−1​(x)]\displaystyle=\left(\phi_{*}p_{0}(\varphi^{-1}(x))\right)\det\left[D\varphi^{-1}(x)\right]
logp1(x)=logϕ∗p0(φ−1(x))+logdet[Dφ−1(x)].\displaystyle\log p_{1}(x)=\log\phi_{*}p_{0}(\varphi^{-1}(x))+\log\det\left[D\varphi^{-1}(x)\right].
φ⁡(y)=27​(y+1)\varphi(y)=2^{7}(y+1)
φ−1​(x)=2−7​x−1\varphi^{-1}(x)=2^{-7}x-1
log⁡p1​(x)=log⁡ϕ∗​p0​(φ−1​(x))−7​d​log⁡2.\log p_{1}(x)=\log\phi_{*}p_{0}(\varphi^{-1}(x))-7d\log 2. (36)

Bits-Per-Dimension (BPD) computation

Bits per dimension is the negative base-two log density divided by the number of scalar image coordinates. Under the stated pixel transform, the model-space expression receives an additive seven. Averaging the stochastic log-density estimator approximates the test-set BPD subject to trace-estimator variance and numerical error.

Equations 37
BPD=𝔼x1​[−log2⁡p1​(x1)d]=𝔼x1​[−log⁡p1​(x1)d​log⁡2]\mathrm{BPD}=\mathbb{E}_{x_{1}}\left[-\frac{\log_{2}p_{1}(x_{1})}{d}\right]=\mathbb{E}_{x_{1}}\left[-\frac{\log p_{1}(x_{1})}{d\log 2}\right] (37)
BPD=−log⁡ϕ∗​p0​(φ−1​(x))d​log⁡2+7\displaystyle\mathrm{BPD}=-\frac{\log\phi_{*}p_{0}(\varphi^{-1}(x))}{d\log 2}+7

Appendix D Diffusion conditional vector fields

The appendix derives conditional vector fields corresponding to diffusion probability paths. Reversing a path from data-to-noise into noise-to-data negates the time-reversed vector field; the proof checks this with the continuity equation.

Equation
dd​t​p~t​(x)=dd​t​p1−t​(x)\displaystyle\frac{d}{dt}\tilde{p}_{t}(x)=\frac{d}{dt}p_{1-t}(x) =−p1−t′​(x)\displaystyle=-p^{\prime}_{1-t}(x)
=div⁡(p1−t​(x)​u1−t​(x))\displaystyle=\mathrm{div}(p_{1-t}(x)u_{1-t}(x))
=−div⁡(p~t​(x)​(−u1−t​(x)))\displaystyle=-\mathrm{div}(\tilde{p}_{t}(x)(-u_{1-t}(x)))

Conditional VFs for Fokker-Planck probability paths

For an SDE with time-dependent scalar diffusion coefficient, the Fokker–Planck equation can be rewritten as a continuity equation with field fₜ−(gₜ²/2)∇log pₜ. This deterministic probability-flow ODE shares the SDE’s one-time density marginals, not its stochastic sample trajectories. The restriction on the diffusion form matters when generalizing the formula.

Equations 38–40
d​y=ft​d​t+gt​d​wdy=f_{t}dt+g_{t}dw (38)
d​ptd​t=−div⁡(ft​pt)+gt22​Δ​pt\frac{dp_{t}}{dt}=-\mathrm{div}(f_{t}p_{t})+\frac{g_{t}^{2}}{2}\Delta p_{t} (39)
d​ptd​t=−div(ftpt−g22∇ptptpt)=−div((ft−gt22∇logpt)pt)=−div(wtpt)\displaystyle\frac{dp_{t}}{dt}=-\mathrm{div}\Big(f_{t}p_{t}-\frac{g^{2}}{2}\frac{\nabla p_{t}}{p_{t}}p_{t}\Big)=-\mathrm{div}\Big(\big(f_{t}-\frac{g_{t}^{2}}{2}\nabla\log p_{t}\big)p_{t}\Big)=-\mathrm{div}\Big(w_{t}p_{t}\Big)
wt=ft−gt22∇logptw_{t}=f_{t}-\frac{g_{t}^{2}}{2}\nabla\log p_{t} (40)

Variance Exploding (VE) path

The variance-exploding Gaussian conditional path yields a radial field proportional to σ′ₜ/σₜ. Time reversal produces the negative time-reversed coefficient used in the main text. Endpoint limits and finite-time noise approximations remain part of the diffusion construction.

Equations
d​y=dd​t​σt2​d​w,dy=\sqrt{\frac{d}{dt}\sigma_{t}^{2}}dw,
pt​(y|y0)=𝒩⁡(y|y0,σt2​I).p_{t}(y|y_{0})={\mathcal{N}}(y|y_{0},\sigma^{2}_{t}I).
wt​(y|y0)=σt′σt​(y−y0)w_{t}(y|y_{0})=\frac{\sigma_{t}^{\prime}}{\sigma_{t}}(y-y_{0})
p~t​(y|y0)=𝒩⁡(y|y0,σ1−t2​I)\tilde{p}_{t}(y|y_{0})={\mathcal{N}}(y|y_{0},\sigma_{1-t}^{2}I)
w~t​(y|y0)=−σ1−t′σ1−t​(y−y0),\displaystyle\tilde{w}_{t}(y|y_{0})=-\frac{\sigma_{1-t}^{\prime}}{\sigma_{1-t}}(y-y_{0}),

Variance Preserving (VP) path

For the variance-preserving SDE, the integrated noise rate T determines both the decaying mean and variance. Substitution into the probability-flow formula and time reversal recover Equation 19. Figures 8–9 retain the conditional vector field and the additional ScoreFlow/DDPM checkerboard comparisons.

Equations 41
d​y=−T′​(t)2​y+T′​(t)​d​w,dy=-\frac{T^{\prime}(t)}{2}y+\sqrt{T^{\prime}(t)}dw,
fs​(y)=−T′​(s)2​y,gs=T′​(s)\displaystyle f_{s}(y)=-\frac{T^{\prime}(s)}{2}y,\quad g_{s}=\sqrt{T^{\prime}(s)}
pt​(y|y0)=𝒩⁡(y|e−12​T​(t)​y0,(1−e−T⁡(t))​I).\displaystyle p_{t}(y|y_{0})={\mathcal{N}}(y|e^{-\frac{1}{2}T(t)}y_{0},(1-e^{-T(t)})I).
wt​(y|y0)=T′​(t)2​(y−e−12​T​(t)​y01−e−T⁡(t)−y)w_{t}(y|y_{0})=\frac{T^{\prime}(t)}{2}\left(\frac{y-e^{-\frac{1}{2}T(t)}y_{0}}{1-e^{-T(t)}}-y\right) (41)
w~t​(y|y0)\displaystyle\tilde{w}_{t}(y|y_{0}) =−T′​(1−t)2​(y−e−12​T​(1−t)​y01−e−T⁡(1−t)−y)\displaystyle=-\frac{T^{\prime}(1-t)}{2}\left(\frac{y-e^{-\frac{1}{2}T(1-t)}y_{0}}{1-e^{-T(1-t)}}-y\right)
=−T′​(1−t)2​[e−T⁡(1−t)​y−e−12​T​(1−t)​y01−e−T⁡(1−t)],\displaystyle=-\frac{T^{\prime}(1-t)}{2}\left[\frac{e^{-T(1-t)}y-e^{-\frac{1}{2}T(1-t)}y_{0}}{1-e^{-T(1-t)}}\right],
Complete original A4.F8 from the exact-version PDF, including every panel, label, and original color.
Figure 8: VP Diffusion path’s conditional vector field. Compare to Figure 2.
Complete original A4.F9 from the exact-version PDF, including every panel, label, and original color.
Figure 9: Trajectories of CNFs trained with ScoreFlow (Song et al., 2021) and DDPM (Ho et al., 2020) losses on 2D checkerboard data, using the same learning rate and other hyperparameters as Figure 4.

Appendix E Implementation details

The two-dimensional experiments use a five-layer MLP with 512 neurons per layer. Image experiments use a U-Net architecture with the preprocessing and resolution-specific choices described in this appendix; the matched objective comparisons share the specified architecture and hyperparameters.

E.1 Diffusion baselines

Losses.

The appendix lists score-matching, likelihood-weighted ScoreFlow, and DDPM noise-prediction losses. A source sign-convention inconsistency needs care: Equation 42 describes s as the true Gaussian score, which is −(x−μ)/σ², whereas Equation 43 uses the positive residual (x−μ)/σ² as its target. The later sampling formula and noise conversion use that negative-score convention consistently. The displayed formulas remain unchanged; this notation issue alone does not establish an implementation error.

Equations 42–45
pt​(x|x1)=𝒩⁡(x|μt​(x1),σt2​(x1)​I).p_{t}(x|x_{1})={\mathcal{N}}(x|\mu_{t}(x_{1}),\sigma_{t}^{2}(x_{1})I).
ℒSM​(θ)\displaystyle{\mathcal{L}}_{\scriptscriptstyle\text{SM}}(\theta) =𝔼t,q⁡(x1),pt​(x|x1)​λ​(t)​‖st​(x)−∇log⁡pt​(x|x1)‖2\displaystyle=\mathbb{E}_{t,q(x_{1}),p_{t}(x|x_{1})}\lambda(t)\left\|s_{t}(x)-\nabla\log p_{t}(x|x_{1})\right\|^{2} (42)
=𝔼t,q⁡(x1),pt​(x|x1)​λ​(t)​‖st​(x)−x−μt​(x1)σt2​(x1)‖2.\displaystyle=\mathbb{E}_{t,q(x_{1}),p_{t}(x|x_{1})}\lambda(t)\left\|s_{t}(x)-\frac{x-\mu_{t}(x_{1})}{\sigma_{t}^{2}(x_{1})}\right\|^{2}. (43)
ℒNM​(θ)\displaystyle{\mathcal{L}}_{\scriptscriptstyle\text{NM}}(\theta) =𝔼t,q⁡(x1),pt​(x|x1)​‖ϵt​(x)−x−μt​(x1)σt​(x1)‖2\displaystyle=\mathbb{E}_{t,q(x_{1}),p_{t}(x|x_{1})}\left\|\epsilon_{t}(x)-\frac{x-\mu_{t}(x_{1})}{\sigma_{t}(x_{1})}\right\|^{2} (44)
=𝔼t,q⁡(x1),p0​(x0)​‖ϵt​(σt​(x1)​x0+μt​(x1))−x0‖2\displaystyle=\mathbb{E}_{t,q(x_{1}),p_{0}(x_{0})}\Big\|{\epsilon_{t}(\sigma_{t}(x_{1})x_{0}+\mu_{t}(x_{1}))-x_{0}}\Big\|^{2} (45)

Diffusion path.

The VP schedule uses β_min=0.1 and β_max=20 with a linearly increasing noise rate. Time is truncated at 1−10⁻⁵ for the stated training, likelihood, and sampling setup to avoid the singular endpoint. All schedule formulas and their square roots are retained.

Equations
μt​(x1)=α1−t​x1,σt​(x1)=1−α1−t2,where ​αt=e−12​T​(t),T⁡(t)=∫0tβ⁡(s)​𝑑s,\mu_{t}(x_{1})=\alpha_{1-t}x_{1},\quad\sigma_{t}(x_{1})=\sqrt{1-\alpha_{1-t}^{2}},\quad\text{where }\alpha_{t}=e^{-\frac{1}{2}T(t)},\quad T(t)=\int_{0}^{t}\beta(s)ds,
T⁡(s)=∫0sβ⁡(r)​𝑑r=s​βmin+12​s2​(βmax−βmin),T(s)=\int_{0}^{s}\beta(r)dr=s\beta_{\min}+\frac{1}{2}s^{2}(\beta_{\max}-\beta_{\min}),

Sampling.

Sampling uses the reverse-time probability-flow expression in Equation 46. DDPM output is divided by σₜ to obtain the positive residual convention used there. This should be read alongside the score-sign qualification above, not as a definition of the conventional positive log-density gradient.

Equation 46
ut​(x)=−T′​(1−t)2​[st​(x)−x].u_{t}(x)=-\frac{T^{\prime}(1-t)}{2}\left[s_{t}(x)-x\right]. (46)

E.2 Training & evaluation details

Table 3 preserves all architecture, batch, GPU, epoch, iteration, and learning-rate settings. Adam uses β₁=0.9, β₂=0.999, zero weight decay, and ε=10⁻⁸. Smaller resolutions use full 32-bit training and larger ones mixed precision. Likelihood estimates use uniform dequantization and log-mean density over K samples, with adaptive integration at 10⁻⁵ tolerances. Increasing K changes the reported estimate; this nonlinear averaging is separate from the unbiased log-density trace estimator. FID implementations differ for ImageNet-128 to match the cited evaluation protocol.

CIFAR10 ImageNet-32 ImageNet-64 ImageNet-128
Channels 256 256 192 256
Depth 2 3 3 3
Channels multiple 1,2,2,2 1,2,2,2 1,2,3,4 1,1,2,3,4
Heads 4 4 4 4
Heads Channels 64 64 64 64
Attention resolution 16 16,8 32,16,8 32,16,8
Dropout 0.0 0.0 0.0 0.0
Effective Batch size 256 1024 2048 1536
GPUs 2 4 16 32
Epochs 1000 200 250 571
Iterations 391k 250k 157k 500k
Learning Rate 5e-4 1e-4 1e-4 1e-4
Learning Rate Scheduler Polynomial Decay Polynomial Decay Constant Polynomial Decay
Warmup Steps 45k 20k - 20k
Table 3: Hyper-parameters used for training each model
Equation 47
log⁡1K​∑k=1Kpt​(x+uk), where ​uk∼𝒰⁡(0,1),\log\frac{1}{K}\sum_{k=1}^{K}p_{t}(x+u_{k}),\text{ where }u_{k}\sim{\mathcal{U}}(0,1), (47)

Appendix F Additional tables and figures

Table 4 reports dequantized BPD for several K values, while Figure 10 tracks sampling evaluations during training. Two ImageNet-32 entries differ slightly from main Table 1: the appendix gives 3.57 for score matching and 3.56 for FM with diffusion at its largest K, versus 3.56 and 3.54 in the main table. Both source tables are retained. Figures 11–13 contain all non-curated image grids; Figures 14–15 retain all super-resolution pairs. The final two same-noise, varying-NFE grids are numbered 16 and 17 in the exact v2 PDF but 17 and 18 in the generated HTML; visible captions follow the PDF while stable HTML anchors are retained. No sample panels are removed.

CIFAR-10 ImageNet 32×\times32 ImageNet 64×\times64
Model KK=1 KK=20 KK=50 KK=1 KK=5 KK=15 KK=1 KK=5 KK=10
Ablation
DDPM 3.24 3.14 3.12 3.62 3.57 3.54 3.36 3.33 3.32
Score Matching 3.28 3.18 3.16 3.65 3.59 3.57 3.43 3.41 3.40
ScoreFlow 3.21 3.11 3.09 3.63 3.57 3.55 3.39 3.37 3.36
Ours
FM w/ Diffusion 3.23 3.13 3.10 3.64 3.58 3.56 3.37 3.34 3.33
FM w/ OT 3.11 3.01 2.99 3.62 3.56 3.53 3.35 3.33 3.31
Table 4: Negative log-likelihood (in bits per dimension) on the test set with different values of KK using uniform dequantization.
Complete original A6.F10 from the exact-version PDF, including every panel, label, and original color.
Figure 10: Function evaluations for sampling during training, for models trained on CIFAR-10 using dopri5 solver with tolerance 1​e−51e^{-5}.
Complete original A6.F11 from the exact-version PDF, including every panel, label, and original color.
Figure 11: Non-curated unconditional ImageNet-32 generated images of a CNF trained with FM-OT.
Complete original A6.F12 from the exact-version PDF, including every panel, label, and original color.
Figure 12: Non-curated unconditional ImageNet-64 generated images of a CNF trained with FM-OT.
Complete original A6.F13 from the exact-version PDF, including every panel, label, and original color.
Figure 13: Non-curated unconditional ImageNet-128 generated images of a CNF trained with FM-OT.
Complete original A6.F14 from the exact-version PDF, including every panel, label, and original color.
Figure 14: Conditional generation 64×\times64→\rightarrow256×\times256. Flow Matching OT upsampled images from validation set.
Complete original A6.F15 from the exact-version PDF, including every panel, label, and original color.
Figure 15: Conditional generation 64×\times64→\rightarrow256×\times256. Flow Matching OT upsampled images from validation set.
Complete original A6.F17 from the exact-version PDF, including every panel, label, and original color.
Figure 16:Generated samples from the same initial noise, but with varying number of function evaluations (NFE). Flow matching with OT path trained on ImageNet-128.
Complete original A6.F18 from the exact-version PDF, including every panel, label, and original color.
Figure 17:Generated samples from the same initial noise, but with varying number of function evaluations (NFE). Flow matching with OT path trained on ImageNet 256×\times256.
Concept demonstration

Open image ↗