Literature
Display
← All papers

History-Guided Video Diffusion

Kiwhan Song · Boyuan Chen · Max Simchowitz · Yilun Du · Russ Tedrake · Vincent Sitzmann

· ICML 2025 ↗

Computer vision17 min text · CondensedOriginal paper ↗
Contents

Overview

The opening example shows frames from an 862-frame navigation video. A single test-set image initializes generation; the full showcase also uses user-directed camera poses, coarse rollout, interpolation, and stabilization, as detailed in the supplement.

Refer to caption
Figure 1: Diffusion Forcing Transformer with history guidance enables stable rollout of extremely long videos. We visualize 21 frames from an 862-frame long navigation video generated by our DFoT model from a single image in a test set video that the model has never seen before. Best viewed as videos on our project website.

At a glance

The Diffusion Forcing Transformer (DFoT) trains a video denoiser with an independently sampled noise level for each frame. Clean frames act as history, fully noised frames hide history, and partially noised frames provide intermediate conditioning. This supports arbitrary subsets of observed frames without a separate fixed-size history input.

History Guidance combines scores from complete, shortened, or partially masked histories. It improves quality and consistency in the tested video models, and can trade those gains against motion and diversity. Temporal composition also helps on unusual camera histories, Minecraft rollouts, and a fruit-swapping robot task. The long navigation showcases combine user-directed camera poses, coarse rollout, interpolation, and stabilization.

1 Introduction

Classifier-free guidance usually strengthens a text or image condition by comparing conditional and unconditional scores. This work treats video history itself as a controllable condition: which frames are visible, how many are retained, and how strongly each is corrupted.

Building on Diffusion Forcing, DFoT extends independent frame noise to non-causal Transformers. Vanilla guidance uses one history; temporal guidance combines several windows; fractional guidance uses partially noised history. Their composition provides a flexible sampling interface, while experiments test the benefits and the limits of these choices.

2 Preliminaries and Related Work

A diffusion model learns to reverse a noising process, often through a noise- or velocity-prediction parameterization. Conditional sampling uses the resulting score; classifier-free guidance extrapolates from an unconditional score toward a conditional one. Larger guidance weights can improve condition adherence while reducing diversity.

Video models often condition on a fixed first image or small frame set. DFoT instead places history and generated frames in one sequence, extending the per-frame noising idea of Diffusion Forcing to state-free, non-causal architectures such as DiT and U-ViT. Composition of several conditional scores motivates guidance beyond a single CFG difference.

Refer to caption
Figure 2: Comparison of the conventional conditional video diffusion models and Diffusion Forcing Transformer. At training time, conventional (a) approaches treat history as part of the conditioning input, first encoded by an separate encoder and then injected to the DiT via Adaptive Layer Norm and scaling. The Diffusion Forcing Transformer (b) instead does not distinguish between history and generation target frames. It trains a DiT to denoise all frames of a sequence, where frames have independently varying noise levels.
Equation 1
∇log⁡pk​(𝐱k)+ω⁡[∇log⁡pk​(𝐱k|𝐜)−∇log⁡pk​(𝐱k)],{\nabla\log\,}p_{k}({\mathbf{x}}^{k})+\omega\big[{\nabla\log\,}p_{k}({\mathbf{x}}^{k}|{\mathbf{c}})-{\nabla\log\,}p_{k}({\mathbf{x}}^{k})\big],\vskip-4.0pt (1)

3 Challenges when Guiding with History

Let H index observed history and G index frames to generate. Both are subsets of the same video, so generated frames can later become history. Supporting arbitrary H requires conditional and unconditional estimates for variable subsets, rather than a separate fixed-size conditioning vector.

A direct alternative is to randomly drop whole history frames during training. The binary-dropout ablation supports flexible conditioning but performs worse in the matched experiments. The proposed explanation is reduced use of tokens for the prediction loss; this is an empirical hypothesis, not a proof that binary dropout must always underperform.

Refer to caption
Figure 3: Sampling with DFoT and History Guidance. A DFoT can be used to estimate scores conditioned on differently masked histories using noise as masking. This includes clean (full history), fully masked (unconditional), subset masked (shorter history), or partially masked (low-frequency history). These scores can be composed when sampling to obtain a family of History Guidance methods.
Equation 2
∇log⁡pk​(𝐱𝒢k)+ω⁡[∇log⁡pk​(𝐱𝒢k|𝐱ℋ)−∇log⁡pk​(𝐱𝒢k)].{\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}})+\omega\big[{\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}}|{\mathbf{x}_{\mathcal{H}}})-{\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}})\big]. (2)

4 The Diffusion Forcing Transformer

DFoT interprets noising as masking. During training, each frame receives an independent noise level and contributes to the denoising objective. During sampling, observed frames are clean, generated frames follow the sampler’s noise schedule, and excluded history is replaced by noise. This makes interpolation, extrapolation, and mixed observed subsets available to the same trained model.

The architecture must accept frame-specific noise embeddings; compatible full-sequence Transformers can be fine-tuned rather than retrained from scratch. The appendix motivates the objective with path-wise variational bounds, but explicitly leaves cross-path consistency unresolved. Its printed ELBO formulas also contain sign and indexing problems, discussed below; the empirical results do not depend on accepting those formulas as a complete proof.

Equations 3–4
kt={0if ​t∈ℋkif ​t∈𝒢.k_{t}=\begin{cases}0&\text{if }t\in{\mathcal{H}}\\ k&\text{if }t\in{\mathcal{G}}.\\ \end{cases}\vskip-5.0pt (3)
𝔼k𝒯,𝐱𝒯,ϵ𝒯[‖ϵ𝒯−ϵθ​(𝐱𝒯k𝒯,k𝒯)‖2],\mathop{{\mathbb{E}}}_{\begin{subarray}{c}k_{{\mathcal{T}}},{\mathbf{x}}_{\mathcal{T}},\bm{\epsilon}_{\mathcal{T}}\end{subarray}}\Big[\|\bm{\epsilon}_{\mathcal{T}}-\bm{\epsilon}_{\theta}({\mathbf{x}}_{\mathcal{T}}^{k_{{\mathcal{T}}}},k_{{\mathcal{T}}})\|^{2}\Big],\vskip-5.0pt (4)
Theorem 4.1 (Informal).

The DFoT training objective (Equation 4) optimizes a reweighting of an Evidence Lower Bound (ELBO) on the expected log-likelihoods.

Table 1: Comparison with generic diffusion models on Kinetics-600. “✗”, “▲”, and “✔” indicate whether a model can condition on a “single predefined,” “arbitrary under approximation,” or “arbitrary” history. DFoT, both trained from scratch and fine-tuned, outperforms all generic diffusion baselines under the same architecture and is on par with industry models trained with more compute resources (see Section C.4).
Flexible? Method FVD ↓\downarrow
Industry size and compute ✗ MAGVIT-v2 (Yu et al., 2023b) 4.3±\scriptstyle\pm0.1
W.A.L.T (Gupta et al., 2024) 3.3±\scriptstyle\pm0.1
Rolling Diffusion (Ruhe et al., 2024) 5.2
▲ Video Diffusion (Ho et al., 2022b) 16.2±\scriptstyle\pm0.3
✔ MAGVIT (Yu et al., 2023a) 9.9±\scriptstyle\pm0.3
Same Architecture ✗ SD 4.8±\scriptstyle\pm0.0
▲ FS 95.5±\scriptstyle\pm0.4
✔ BD 6.4±\scriptstyle\pm0.1
DFoT (scratch) 4.3±\scriptstyle\pm0.1
DFoT (fine-tuned from FS) 4.7±\scriptstyle\pm0.0

5 History Guidance

Vanilla History Guidance (HG-v) applies CFG using a chosen clean history. The unconditional branch replaces that history with full noise. Temporal guidance (HG-t) combines scores from different history windows, aiming to retain useful long-range information while reducing sensitivity to unusual long contexts.

Fractional guidance (HG-f) adds a score difference based on partially noised history. This weakens constraints from less reliable signal components and reduces the tendency of strong vanilla guidance to copy a recent frame. Gaussian noise does not intrinsically remove only high frequencies: the low-pass interpretation depends on the signal’s spectral power and is justified only in a simplified model.

The general formula combines histories with different lengths and noise levels. It is a flexible sampling heuristic, not a guarantee that arbitrary weighted scores equal the true full-history conditional distribution. The retained equations distinguish ordinary conditioning, CFG, temporal composition, and fractional guidance.

The fractional-guidance coefficient is written differently in Equation 6 and the sampler’s example: one uses ω, the other ω−1 for the partially masked score difference. Both source expressions are retained; the appendix example is the one that reduces to ordinary conditioning at weight one.

Equation 5
∇log⁡pk​(𝐱𝒢k)+∑iωi​[∇log⁡pk​(𝐱𝒢k|𝐱ℋikℋi)−∇log⁡pk​(𝐱𝒢k)],\scalebox{0.85}{${\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}})+\sum_{i}\omega_{i}\big[{\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}}|{\mathbf{x}}_{{\mathcal{H}}_{i}}^{k_{{\mathcal{H}}_{i}}})-{\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}})\big]$},\vskip-5.0pt (5)
Refer to caption
Figure 4: Qualitative comparison on Kinetics-600. DFoT (both scratch and fine-tuned) generates higher-quality samples consistent with the history than baselines. FS omitted for poor quality. We show 6 of 16 frames; see Figure 17 for more comparisons.
Original vector plot, retained from the source.
Original vector plot, retained from the source.
Original vector plot, retained from the source.
Original vector plot, retained from the source.
Original vector plot, retained from the source.
Figure 5: Various metrics as a function of guidance scale ω\bm{\omega} for vanilla and fractional history guidance on Kinetics-600, comparing against ω=1\omega=1 (∙\bullet, w/o HG), SD, and ground truth (GT). FS is omitted for poor performance (FVD = 1040). Vanilla history guidance trades off dynamics ⋅\cdot diversity for quality ⋅\cdot consistency. Fractional history guidance better balances these trade-offs, achieving the best FVD.
Refer to caption
(a) Vanilla history guidance significantly improves frame quality and consistency with an increasing guidance scale. We sample with varying guidance scales ω=1\omega=1 (top, without history guidance), 1.51.5 (middle), and 33 (bottom).
Refer to caption
(b) Fractional history guidance resolves the issue of static videos, improving dynamics by guiding with lower frequencies. We sample with varying frequency scales, with kℋ=0k_{\mathcal{H}}=0 (top, vanilla guidance leading to static videos), 0.30.3 (middle), and 0.60.6 (bottom).
Figure 6: Qualitative results for vanilla ⋅\cdot fractional history guidance on Kinetics-600. Best viewed zoomed in. Red box = history frames.
Original vector plot, retained from the source.
Refer to caption
Figure 7: Robust performance of temporal history guidance given OOD history unseen in the training data. Left: Baselines sharply lose performance transitioning from in-distribution, slightly OOD, to OOD tasks, while DFoT with HG-t shows minimal drop. Right: Baselines produce blurry, inconsistent frames with artifacts on slightly OOD history and unrecognizable frames on OOD history, whereas DFoT with HG-t generates high-quality, accurate samples. Each frame shown is one of four generated; see Figure 13 for full results.
Equation 6
∇log⁡pk​(𝐱𝒢k|𝐱ℋ)+ω⁡[∇log⁡pk​(𝐱𝒢k|𝐱ℋkℋ)−∇log⁡pk​(𝐱𝒢k)],\scalebox{0.85}{${\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}}|{\mathbf{x}_{\mathcal{H}}})+\omega\big[{\nabla\log\,}p_{k}\big({\mathbf{x}_{\mathcal{G}}^{k}}|{\mathbf{x}}_{{\mathcal{H}}}^{k_{{\mathcal{H}}}}\big)-{\nabla\log\,}p_{k}({\mathbf{x}_{\mathcal{G}}^{k}})\big]$},\vskip-2.0pt (6)

6 Experiments

Experiments first compare unguided base models, then test guidance, unusual histories, long contexts, robotic control, and extended navigation videos. Separate models are trained for each dataset.

6.1 Experimental Setup

Kinetics-600 supplies the standard 128×128 prediction benchmark. RealEstate10K uses 256×256 images and camera poses; Minecraft uses action-conditioned navigation; Fruit Swapping tests memory and reactivity in a physical robot.

Matched baselines are standard fixed-history diffusion (SD), binary-dropout diffusion (BD), and unconditional full-sequence diffusion with reconstruction guidance (FS). FVD measures video-feature distribution discrepancy, VBench separates quality/consistency/dynamics, and LPIPS compares generated frames with ground truth on constrained tasks. These metrics answer different questions and are not interchangeable.

6.2 Evaluating the Diffusion Forcing Transformer

On five-frame-to-eleven-frame Kinetics prediction, unguided DFoT reaches FVD 4.3±0.1, compared with SD 4.8±0.0, BD 6.4±0.1, and FS 95.5±0.4 under the matched pipeline. Fine-tuning FS for 80,000 additional steps reaches 4.7±0.0. These extra steps are 12.5% of the 640,000-step scratch run, excluding the pretrained model’s cost.

Published literature models use different resources and architectures: W.A.L.T. scores 3.3±0.1 and MAGVIT-v2 4.3±0.1. Thus DFoT leads the matched generic baselines, ties the reported MAGVIT-v2 mean, and trails W.A.L.T. Flexible interpolation and mixed-history examples provide qualitative evidence for the shared conditioning interface.

6.3 Improving Video Generation via History Guidance

The rollout experiment generates 59 frames after five observed frames and evaluates 1,024 total 64-frame videos. Moderate vanilla guidance improves FVD from 208.0 to 181.6 at weight 1.5, while fractional guidance reaches 170.4. SD scores 247.5 and FS 1040 in this longer setting; these values should not be compared directly with the short-video benchmark.

Increasing vanilla guidance improves frame quality and consistency but eventually makes videos static and worsens FVD. Fractional masking restores some dynamics while keeping much of the quality gain. The selected masking level and guidance scale are experimental choices, not universal optima.

6.4 New Abilities via Temporal History Guidance

On RealEstate10K, histories spanning unusually large camera rotations cause large errors for the baselines. Composing overlapping shorter histories, and in this experiment shorter generation subsets, improves interpolation LPIPS across the sparsely represented rotation ranges.

For action-conditioned Minecraft, equal-weight short- and long-context scores reduce FVD from 97.625 to 79.19 over 125 generated frames. In Fruit Swapping, composed histories achieve 83% success under the main human-perturbation protocol, with 100 randomized setups. Stronger perturbations reduce the reported success to 28%. These experiments support complementary memory and reactive behavior on the tested tasks, rather than proving that other models cannot solve them.

6.5 Ultra Long Video Generation

The 862-frame showcase starts from one image, but also uses an interactive user to choose camera motions after each coarse rollout. Six intermediate frames are then generated between successive coarse frames, giving a sevenfold interpolation interval. History guidance and a small noise-level stabilization help maintain consistency.

The long sample demonstrates extended guided synthesis, not 862 independently rolled-out frames from an image alone or autonomous navigation in a verified 3D world. Additional samples and a separate comparison without interpolation appear in the supplement.

7 Conclusion

DFoT makes the amount and form of historical conditioning available as sampling-time controls. The tested guidance mixtures improve selected quality, motion, and long-context outcomes, while fine-tuning can add this flexibility to a compatible pretrained model. Remaining limitations include guidance tradeoffs, finite context, additional score evaluations, qualitative long-rollout evidence, and the lack of a general guarantee for score composition.

Acknowledgements

The authors acknowledge support from the National Science Foundation, Singapore DSTA, IARPA through DOI/IBC, Amazon Science Hub, and the MIT–Google Program for Computing Innovation.

Impact Statement

Longer and higher-quality generated videos may support robotics and other applications, while also increasing the potential for harmful or misleading synthetic content. The authors call for responsible use; the paper does not experimentally evaluate misuse safeguards.

Appendix A Proofs, Explanations, and Extensions

The theory supplement develops a path-wise variational motivation, a Gaussian explanation of noise-conditioned attenuation, and an estimator-combination interpretation. It also supplies the complete sampler and implementation extensions. Formal statements and displayed derivations are retained verbatim below.

A.1 Derivation of an ELBO

A path specifies how each frame’s noise level increases from clean data to full noise. If the forward process and learned reverse process factor along that path, Jensen’s inequality yields a path-wise likelihood bound. Averaging over suitable monotone paths motivates a weighted denoising objective. No temporal causality is required for the index set.

The source notes that learned factorizations can be inconsistent across different paths and leaves enforcing consistency open. Its printed theorem writes a positive KL term with reversed arguments; the standard ELBO expansion instead subtracts KL(q‖p). The x-prediction corollary also prints a positive squared-error term in a likelihood lower bound. These formulas are preserved as published, so the summary treats them as an intended variational argument rather than a verified bound in the printed form.

Definition A.1 (Path).

We define a path ρ\rho as a sequence (𝐤j)0≤j≤N(\mathbf{k}^{j})_{0\leq j\leq N} that begins at zero noise 𝐤0=(0,0,…,0)\mathbf{k}^{0}=(0,0,\dots,0), and terminates at full noise 𝐤N=(K,K,…,K)\mathbf{k}^{N}=(K,K,\dots,K).

Definition A.2 (Factoring Property).

We say that a model p𝜽p_{\bm{\theta}} and forward process qq factor along a path ρ\rho if for any path, ρ=(𝐤1,…,𝐤N)\rho=(\mathbf{k}^{1},\dots,\mathbf{k}^{N}) be a path, q(𝐱𝐤1:N∣𝐱𝐤0)q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}}) factors as q(𝐱𝐤1:N∣𝐱𝐤0)=∏j=1nq(𝐱𝐤j∣𝐱𝐤j−1)q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})=\prod_{j=1}^{n}q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j-1}}), and p𝜽p_{\bm{\theta}} factors as p𝜽(𝐱𝐤0:N)=∏j=1Np𝜽(𝐱𝐤j−1∣𝐱𝐤j)p𝜽(𝐱𝐤N)p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0:N}})=\prod_{j=1}^{N}p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j-1}}\mid{\mathbf{x}}^{\mathbf{k}^{j}})p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{N}}), with p𝜽​(𝐱𝐤N)p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{N}}) not depending on θ\theta.

Theorem A.3.

Suppose that (p𝛉,q)(p_{\bm{\theta}},q) factor along a path ρ=(𝐤1,…,𝐤N)\rho=(\mathbf{k}^{1},\dots,\mathbf{k}^{N}). Then, for some constant CC not depending on θ\theta, we have

Equation 7
lnp(𝐱𝐤0)≥C+𝔼𝐱𝐤1:N∼q(𝐱𝐤1:N∣𝐱𝐤0)[lnp𝜽(𝐱𝐤0∣𝐱𝐤1)+∑j=1N−1D𝕂​𝕃(p𝜽(𝐱𝐤j∣𝐱𝐤j+1)∥q(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0))].\displaystyle\ln p({\mathbf{x}}^{\mathbf{k}^{0}})\geq C+\mathbb{E}_{{\mathbf{x}}^{\mathbf{k}^{1:N}}\sim q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}_{0}})}\left[\ln{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}+\sum_{j=1}^{N-1}\mathrm{D}_{\mathbb{KL}}({p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}\parallel{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})})\right]. (7)

Consequently, if 𝔼𝐤1:N∼𝒟p\mathbb{E}_{\mathbf{k}^{1:N}\sim{\mathcal{D}}_{p}} denotes an expectation over paths ρ=(𝐤1,…,𝐤K)\rho=(\mathbf{k}^{1},\dots,\mathbf{k}^{K}) along which (p𝛉,q)(p_{\bm{\theta}},q) factor, then

Equation
lnp(𝐱𝐤0)≥C+𝔼𝐤1:N∼𝒟p𝔼𝐱𝐤1:N∼q(𝐱𝐤1:N∣𝐱𝐤0)[lnp𝜽(𝐱𝐤0∣𝐱𝐤1)+∑j=1N−1D𝕂​𝕃(p𝜽(𝐱𝐤j∣𝐱𝐤j+1)∥q(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0))].\displaystyle\ln p({\mathbf{x}}^{\mathbf{k}^{0}})\geq C+\mathbb{E}_{\mathbf{k}^{1:N}\sim{\mathcal{D}}_{p}}\mathbb{E}_{{\mathbf{x}}^{\mathbf{k}^{1:N}}\sim q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}_{0}})}\left[\ln{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}+\sum_{j=1}^{N-1}\mathrm{D}_{\mathbb{KL}}({p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}\parallel{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})})\right].
Corollary A.4.

Consider only paths ρ\rho for which 𝐤j≥𝐤j−1\mathbf{k}^{j}\geq\mathbf{k}^{j-1} entrywise, and for any tt and jj for which ktj>ktj−1k^{j}_{t}>k^{j-1}_{t}, ktj=ktj−1+1k^{j}_{t}=k^{j-1}_{t}+1 increments by one.

Equation 8
q(𝐱𝐤j+1∣𝐱t𝐤j)=∏t:ktj<ktj+1𝒩(𝐱tktj;1−βktj𝐱ktj−1,βktj𝐈),\displaystyle q({\mathbf{x}}^{\mathbf{k}^{j+1}}\mid{\mathbf{x}}_{t}^{\mathbf{k}^{j}})=\prod_{t:k^{j}_{t}<k^{j+1}_{t}}\mathcal{N}({\mathbf{x}}_{t}^{k^{j}_{t}};\sqrt{1-\beta_{k^{j}_{t}}}{\mathbf{x}}^{k^{j-1}_{t}},\beta_{k^{j}_{t}}\mathbf{I}), (8)

and define αk=(1−βk)\alpha_{k}=(1-\beta_{k}), α¯k=∏j=1kαj\bar{\alpha}_{k}=\prod_{j=1}^{k}\alpha_{j}. Suppose that we parameterize p𝛉​(𝐱𝐤j,𝐱𝐤j+1,𝐤j)=𝒩⁡(μ𝛉​(𝐱𝐤j,𝐱𝐤j+1,𝐤j),σj2)p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j})=\mathcal{N}(\mu_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j}),\sigma_{j}^{2}), where further,

Equation
μ𝜽​(𝐱𝐤j,𝐱𝐤j+1,𝐤j)=(1−α¯j−1)​αj1−α¯j​𝐱𝐤j+(1−αj)​α¯j−11−α¯j​𝐱^𝜽​(𝐱𝐤j,𝐱𝐤j+1,𝐤j),σj2:=(1−αj)​(1−α¯j−1)1−α¯j.\displaystyle\mu_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j})=\frac{(1-\bar{\alpha}_{j-1})\sqrt{{\alpha}_{j}}}{1-\bar{\alpha}_{j}}{\mathbf{x}}^{\mathbf{k}_{j}}+\frac{(1-{\alpha}_{j})\sqrt{\bar{\alpha}_{j-1}}}{1-\bar{\alpha}_{j}}\hat{{\mathbf{x}}}_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j}),\quad\sigma_{j}^{2}:=\frac{(1-\alpha_{j})(1-\sqrt{\bar{\alpha}_{j-1}})}{1-\bar{\alpha}_{j}}.

Further, let 𝐱^𝛉0​(𝐱tkj,𝐱𝐤j+1,𝐤j)=𝐱^𝛉0​(𝐱𝐤j,𝐱𝐤j+1,𝐤j)t\hat{{\mathbf{x}}}^{0}_{\bm{\theta}}({\mathbf{x}}_{t}^{k^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j})=\hat{{\mathbf{x}}}^{0}_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j})_{t} denote the tt-block component of 𝐱^𝛉0​(𝐱𝐤j,𝐱𝐤j+1,𝐤j)\hat{{\mathbf{x}}}^{0}_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j}), and suppose that if ktj=ktj+1k_{t}^{j}=k_{t}^{j+1}, then 𝐱^𝛉0​(𝐱tkj,𝐱𝐤j+1,𝐤j)=𝐱𝐤j+1\hat{{\mathbf{x}}}^{0}_{\bm{\theta}}({\mathbf{x}}_{t}^{k^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j})={\mathbf{x}}^{\mathbf{k}^{j+1}} (i.e., if no denoising occurs, we do not re-predict the denoising). Then, for some distribution 𝒟ρ{\mathcal{D}}_{\rho} over paths ρ\rho along which (p𝛉,q)(p_{\bm{\theta}},q) satisfy the requisite factoring property, and for some constant CC independent of p𝛉p_{\bm{\theta}},

Equation
lnp𝜽((𝐱𝐤0)]\displaystyle\ln p_{\bm{\theta}}(({\mathbf{x}}^{\mathbf{k}^{0}})] ≥C+𝔼ρ=𝐤0:N∼𝒟ρ𝔼p,𝐳1:T[∑j=1N∑t∈𝒯:ktj<ktj+1cktj∥𝐱^𝜽0(𝐱tkj;𝐱𝐤j+1,𝐤j)−𝐱tkt0∥2],\displaystyle\geq C+\mathbb{E}_{\rho=\mathbf{k}^{0:N}\sim{\mathcal{D}}_{\rho}}\mathbb{E}_{p,\mathbf{z}_{1:T}}\left[\sum_{j=1}^{N}\sum_{t\in{\mathcal{T}}:k_{t}^{j}<k_{t}^{j+1}}c_{k_{t}^{j}}\|\hat{{\mathbf{x}}}^{0}_{\bm{\theta}}({\mathbf{x}}_{t}^{k^{j}};{\mathbf{x}}^{\mathbf{k}^{j+1}},\mathbf{k}^{j})-{\mathbf{x}}_{t}^{k^{0}_{t}}\|^{2}\right],

where above, we define ci=(1−αj)2​α¯i−12​σ2​(1−α¯i)2c_{i}=\frac{(1-\alpha_{j})^{2}\bar{\alpha}_{i-1}}{2\sigma^{2}(1-\bar{\alpha}_{i})^{2}}.

Remark A.5 (Factoring).

Observe that forward process in Equation 8 naturally factorizes across all the paths ρ\rho considered in Corollary A.4. While p𝜽p_{\bm{\theta}} (by definition) factors across any single path ρ\rho, these factorizations may be inconsistent across paths. Enforcing some explicit consistency remains open for future work.

Equations Factoring, Definition A.2–Canceling
ln⁡p⁡(𝐱𝐤0)\displaystyle\ln p({\mathbf{x}}^{\mathbf{k}^{0}}) =ln∫𝐱𝐤1:Np(𝐱𝐤0:N)d𝐱𝐤1:N\displaystyle=\ln\int_{{\mathbf{x}}^{\mathbf{k}^{1:N}}}p({\mathbf{x}}^{\mathbf{k}^{0:N}})\mathrm{d}{\mathbf{x}}^{\mathbf{k}^{1:N}}
=ln𝔼𝐱𝐤1:N∼q(𝐱𝐤1:N∣𝐱𝐤0)p(𝐱𝐤0:N)q(𝐱𝐤1:N∣𝐱𝐤0)\displaystyle=\ln\mathbb{E}_{{\mathbf{x}}^{\mathbf{k}^{1:N}}\sim q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}\frac{p({\mathbf{x}}^{\mathbf{k}^{0:N}})}{q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}
≥𝔼𝐱𝐤1:N∼q(𝐱𝐤1:N∣𝐱𝐤0)lnp(𝐱𝐤0:N)q(𝐱𝐤1:N∣𝐱𝐤0).\displaystyle\geq\mathbb{E}_{{\mathbf{x}}^{\mathbf{k}^{1:N}}\sim q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}\ln\frac{p({\mathbf{x}}^{\mathbf{k}^{0:N}})}{q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}.
lnp𝜽(𝐱𝐤0:N)q(𝐱𝐤1:N∣𝐱𝐤0)\displaystyle\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0:N}})}{q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}
=ln⁡p𝜽​(𝐱𝐤N)+ln⁡p𝜽​(𝐱𝐤0∣𝐱𝐤1)q⁡(𝐱𝐤1∣𝐱𝐤0)+∑j=1N−1ln⁡p𝜽​(𝐱𝐤j∣𝐱𝐤j+1)q⁡(𝐱𝐤j+1∣𝐱𝐤j,𝐱𝐤0)\displaystyle=\ln p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{N}})+\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}{q({\mathbf{x}}^{\mathbf{k}^{1}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}+\sum_{j=1}^{N-1}\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}{q({\mathbf{x}}^{\mathbf{k}^{j+1}}\mid{\mathbf{x}}^{\mathbf{k}^{j}},{\mathbf{x}}^{\mathbf{k}^{0}})} (Factoring, Definition A.2)
=ln⁡p𝜽​(𝐱𝐤N)+ln⁡p𝜽​(𝐱𝐤0∣𝐱𝐤1)q⁡(𝐱𝐤1∣𝐱𝐤0)+∑j=1N−1ln⁡p𝜽​(𝐱𝐤j∣𝐱𝐤j+1)q⁡(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0)+ln⁡q⁡(𝐱𝐤j∣𝐱𝐤0)q⁡(𝐱𝐤j+1∣𝐱𝐤0)\displaystyle=\ln p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{N}})+\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}{q({\mathbf{x}}^{\mathbf{k}^{1}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}+\sum_{j=1}^{N-1}\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})}+\ln\frac{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}{q({\mathbf{x}}^{\mathbf{k}^{j+1}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})} (Bayes’ Rule on qq)
=ln⁡p𝜽​(𝐱𝐤N)+ln⁡p𝜽​(𝐱𝐤0∣𝐱𝐤1)q⁡(𝐱𝐤1∣𝐱𝐤0)+ln⁡q⁡(𝐱𝐤1∣𝐱𝐤0)q⁡(𝐱𝐤N∣𝐱𝐤0)+∑j=1N−1ln⁡p𝜽​(𝐱𝐤j∣𝐱𝐤j+1)q⁡(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0)\displaystyle=\ln p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{N}})+\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}{q({\mathbf{x}}^{\mathbf{k}^{1}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}+\ln\frac{q({\mathbf{x}}^{\mathbf{k}^{1}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}{q({\mathbf{x}}^{\mathbf{k}^{N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}+\sum_{j=1}^{N-1}\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})} (Telescoping)
=ln⁡p𝜽​(𝐱𝐤N)q⁡(𝐱𝐤N∣𝐱𝐤0)+ln⁡p𝜽​(𝐱𝐤0∣𝐱𝐤1)+∑j=1N−1ln⁡p𝜽​(𝐱𝐤j∣𝐱𝐤j+1)q⁡(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0).\displaystyle=\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{N}})}{q({\mathbf{x}}^{\mathbf{k}^{N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}+\ln p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})+\sum_{j=1}^{N-1}\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})}. (Canceling)
ln⁡p⁡(𝐱𝐤0)\displaystyle\ln p({\mathbf{x}}^{\mathbf{k}^{0}}) ≥C+𝔼𝐱𝐤1:N∼q(𝐱𝐤1:N∣𝐱𝐤0)[lnp𝜽(𝐱𝐤0∣𝐱𝐤1)+∑j=1N−1lnp𝜽​(𝐱𝐤j∣𝐱𝐤j+1)q⁡(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0)]\displaystyle\geq C+\mathbb{E}_{{\mathbf{x}}^{\mathbf{k}^{1:N}}\sim q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}^{0}})}\left[\ln{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}+\sum_{j=1}^{N-1}\ln\frac{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})}\right]
=C+𝔼𝐱𝐤1:N∼q(𝐱𝐤1:N∣𝐱𝐤0)[lnp𝜽(𝐱𝐤0∣𝐱𝐤1)+∑j=1N−1D𝕂​𝕃(p𝜽(𝐱𝐤j∣𝐱𝐤j+1)∥q(𝐱𝐤j∣𝐱𝐤j+1,𝐱𝐤0))].\displaystyle=C+\mathbb{E}_{{\mathbf{x}}^{\mathbf{k}^{1:N}}\sim q({\mathbf{x}}^{\mathbf{k}^{1:N}}\mid{\mathbf{x}}^{\mathbf{k}_{0}})}\left[\ln{p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{0}}\mid{\mathbf{x}}^{\mathbf{k}^{1}})}+\sum_{j=1}^{N-1}\mathrm{D}_{\mathbb{KL}}({p_{\bm{\theta}}({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}})}\parallel{q({\mathbf{x}}^{\mathbf{k}^{j}}\mid{\mathbf{x}}^{\mathbf{k}^{j+1}},{\mathbf{x}}^{\mathbf{k}^{0}})})\right].

A.2 Understanding Frequency Guidance

The frequency argument assumes a linear Gaussian regression model with additive observation noise. In a Fourier basis, conditioning on a noisy observation shrinks directions with weak signal variance more strongly. When natural-signal power decays with frequency, this often suppresses high-frequency information.

This is a model-dependent attenuation argument, not a literal frequency cutoff. The printed Fourier definition, scaling identities, and covariance notation contain inconsistencies, including an invalid upper index in the cosine branch and conflicting noise-normalization factors. The original formulas remain intact; their qualitative interpretation is more reliable than using the printed expressions as implementation-ready identities.

Definition A.6.

Let ℱd:ℝd→ℝd\mathcal{F}_{d}:\mathbb{R}^{d}\to\mathbb{R}^{d} denote the (real) discrete Fourier transform, specified by

Equation 9
ℱd​(𝐱)​(k)={∑i=1d𝐱⁡[i]​sin⁡(i​k/2​π)1≤k≤d/2∑i=1d𝐱⁡[i]​cos⁡(i​k/2​π)d/2<k≤1\displaystyle\mathcal{F}_{d}({\mathbf{x}})(k)=\begin{cases}\sum_{i=1}^{d}{\mathbf{x}}[i]\sin(ik/2\pi)&1\leq k\leq d/2\\ \sum_{i=1}^{d}{\mathbf{x}}[i]\cos(ik/2\pi)&d/2<k\leq 1\\ \end{cases} (9)

We note that, by Parseval’s theorem, ℱd\mathcal{F}_{d} is an isometry:

Equation 10
1d​‖ℱd​(𝐱)‖ℓ22=‖𝐱‖ℓ22\displaystyle\frac{1}{d}\|\mathcal{F}_{d}({\mathbf{x}})\|_{\ell_{2}}^{2}=\|{\mathbf{x}}\|_{\ell_{2}}^{2} (10)

Because ℱd\mathcal{F}_{d} is a bijective linear mapping, we identify it with an invertible matrix in ℝd×d\mathbb{R}^{d\times d}.

Proposition A.7.

Let 𝐱∼𝒩⁡(0,𝚺x2){\mathbf{x}}\sim\mathcal{N}(0,\bm{\Sigma}_{x}^{2}), and 𝐲|𝐱∼𝒩⁡(𝐀𝐱,𝚺y2)\mathbf{y}\mid{\mathbf{x}}\sim\mathcal{N}(\mathbf{A}{\mathbf{x}},\bm{\Sigma}_{y}^{2}). Define 𝐱σ=𝐱+σ​𝐳{\mathbf{x}}_{\sigma}={\mathbf{x}}+\sigma\mathbf{z}, where 𝐳∼𝒩⁡(0,𝐈)\mathbf{z}\sim\mathcal{N}(0,\mathbf{I}) is independent of 𝐱,𝐲{\mathbf{x}},\mathbf{y}. Define 𝐀^:=ℱm​𝐀​ℱd−1\hat{\mathbf{A}}:=\mathcal{F}_{m}\mathbf{A}\mathcal{F}_{d}^{-1}, 𝚺^x:=ℱd​𝚺x​ℱd⊤\hat{\bm{\Sigma}}_{x}:=\mathcal{F}_{d}\bm{\Sigma}_{x}\mathcal{F}_{d}^{\top} and 𝐒^​(σ):=𝚺^x​(𝚺^x+d​σ2​𝐈)−1\hat{\mathbf{S}}(\sigma):=\hat{\bm{\Sigma}}_{x}(\hat{\bm{\Sigma}}_{x}+d\sigma^{2}\mathbf{I})^{-1}, and 𝚺^y:=ℱm​𝚺y​ℱm⊤\hat{\bm{\Sigma}}_{y}:=\mathcal{F}_{m}\bm{\Sigma}_{y}\mathcal{F}_{m}^{\top}. Then,

  • ℱd​(𝐱)∼𝒩⁡(0,𝚺^x)\mathcal{F}_{d}({\mathbf{x}})\sim\mathcal{N}(0,\hat{\bm{\Sigma}}_{x})

  • ℱm​(𝐲)|ℱd​(𝐱)∼𝒩⁡(𝐀^​ℱd​(𝐱),𝚺^y)\mathcal{F}_{m}(\mathbf{y})\mid\mathcal{F}_{d}({\mathbf{x}})\sim\mathcal{N}(\hat{\mathbf{A}}\mathcal{F}_{d}({\mathbf{x}}),\hat{\bm{\Sigma}}_{y})

  • The distribution of ℱm​(𝐲)|𝐱σ\mathcal{F}_{m}(\mathbf{y})\mid{\mathbf{x}}_{\sigma} (or ℱm​(𝐲)|ℱd​(𝐱σ)\mathcal{F}_{m}(\mathbf{y})\mid\mathcal{F}_{d}({\mathbf{x}}_{\sigma})) is

    Equation 11
    𝒩⁡(𝐀^​𝐒^​(σ)​ℱd​(𝐱σ),𝚺^y+d​σ2​𝐀^​𝐒^​(σ)​𝐀^⊤)\displaystyle\mathcal{N}(\hat{\mathbf{A}}\hat{\mathbf{S}}(\sigma)\mathcal{F}_{d}({\mathbf{x}}_{\sigma}),\hat{\bm{\Sigma}}_{y}+d\sigma^{2}\hat{\mathbf{A}}\hat{\mathbf{S}}(\sigma)\hat{\mathbf{A}}^{\top}) (11)

Interpretation in Terms for Frequency Attenuation:

With diagonal power-law spectral covariance, increasing observation noise suppresses more weak-variance directions. Gaussian noising itself has no preference for low frequency: that preference arises only when low frequencies have greater signal power. The retained Gaussian conditional lemma supplies the intended shrinkage calculation, with the source’s covariance and σ/σ² notation left unchanged.

Equation
𝐒^​(σ)i​i=11+d​σ2​iα/C∼{1i≤(Cd​σ2)1/α​ or ​σ2≤C​iα/di−αi≥(Cd​σ2)1/α​ or ​σ2≥C​iα/d\displaystyle\hat{\mathbf{S}}(\sigma)_{ii}=\frac{1}{1+d\sigma^{2}i^{\alpha}/C}\sim\begin{cases}1&i\leq(\frac{C}{d\sigma^{2}})^{1/\alpha}\text{ or }\sigma^{2}\leq Ci^{\alpha}/d\\ i^{-\alpha}&i\geq(\frac{C}{d\sigma^{2}})^{1/\alpha}\text{ or }\sigma^{2}\geq Ci^{\alpha}/d\\ \end{cases}
Lemma A.8 (Gaussian Conditional Computation).

Let 𝐱∼𝒩⁡(0,𝚺x2){\mathbf{x}}\sim\mathcal{N}(0,\bm{\Sigma}_{x}^{2}), and 𝐲|𝐱∼𝒩⁡(𝐀𝐱,𝚺y2)\mathbf{y}\mid{\mathbf{x}}\sim\mathcal{N}(\mathbf{A}{\mathbf{x}},\bm{\Sigma}_{y}^{2}). Define 𝐱σ=𝐱+σ​𝐳{\mathbf{x}}_{\sigma}={\mathbf{x}}+\sigma\mathbf{z}, where 𝐳∼𝒩⁡(0,𝐈)\mathbf{z}\sim\mathcal{N}(0,\mathbf{I}) is independent of 𝐱,𝐲{\mathbf{x}},\mathbf{y}. Set 𝐒⁡(σ):=𝚺x​(𝚺x+σ2​𝐈)−1\mathbf{S}(\sigma):=\bm{\Sigma}_{x}(\bm{\Sigma}_{x}+\sigma^{2}\mathbf{I})^{-1}. Then, the distribution of 𝐲|𝐱σ\mathbf{y}\mid{\mathbf{x}}_{\sigma} is 𝒩⁡(𝐀𝐒⁡(σ)​𝐱σ,𝚺y+σ2​𝐀𝐒​(σ)​𝐀⊤)\mathcal{N}(\mathbf{A}\mathbf{S}(\sigma){\mathbf{x}}_{\sigma},\bm{\Sigma}_{y}+\sigma^{2}\mathbf{A}\mathbf{S}(\sigma)\mathbf{A}^{\top}).

Equations 12
𝐲|𝐱σ\displaystyle\mathbf{y}\mid{\mathbf{x}}_{\sigma} ∼𝒩⁡(𝚺12​𝚺22−1​𝐱σ,𝚺11−𝚺12​𝚺22−1​𝚺12)\displaystyle\sim\mathcal{N}\left(\bm{\Sigma}_{12}\bm{\Sigma}_{22}^{-1}{\mathbf{x}}_{\sigma},\bm{\Sigma}_{11}-\bm{\Sigma}_{12}\bm{\Sigma}_{22}^{-1}\bm{\Sigma}_{12}\right)
=𝒩⁡(𝐀​𝚺x​(𝚺x+σ2​𝐈)−1​𝐱σ,𝚺y+𝐀​𝚺x​𝐀⊤−𝐀​𝚺x​(𝚺x+σ2​𝐈)−1​𝚺x​𝐀⊤).\displaystyle=\mathcal{N}\left(\mathbf{A}\bm{\Sigma}_{x}(\bm{\Sigma}_{x}+\sigma^{2}\mathbf{I})^{-1}{\mathbf{x}}_{\sigma},\bm{\Sigma}_{y}+\mathbf{A}\bm{\Sigma}_{x}\mathbf{A}^{\top}-\mathbf{A}\bm{\Sigma}_{x}(\bm{\Sigma}_{x}+\sigma^{2}\mathbf{I})^{-1}\bm{\Sigma}_{x}\mathbf{A}^{\top}\right).
𝐲|𝐱σ∼𝒩⁡(𝐀𝐒⁡(σ)​𝐱σ,𝚺y+σ2​𝐀𝐒​(σ)​𝐀⊤),\displaystyle\mathbf{y}\mid{\mathbf{x}}_{\sigma}\sim\mathcal{N}(\mathbf{A}\mathbf{S}(\sigma){\mathbf{x}}_{\sigma},\bm{\Sigma}_{y}+\sigma^{2}\mathbf{A}\mathbf{S}(\sigma)\mathbf{A}^{\top}), (12)

A.3 A Maximum Likelihood Interpretation for Score Addition.

The score-addition discussion assumes that several estimators have jointly zero-mean Gaussian errors around the same target score. Under a known nonsingular covariance, generalized least squares gives a covariance-weighted combination. Independent isotropic errors reduce this to inverse-variance averaging.

These are strong assumptions that are not established for scores conditioned on different video histories. The appendix presents the argument as intuition. Its general MLE formula omits the final design matrix in the matrix to be inverted, making the printed expression dimensionally problematic; the original expression is retained rather than silently repaired.

Assumption A.9 (Gaussian Errors).

We assume that, conditioned on 𝐱{\mathbf{x}}, the errors ϵ→:=(s^1​(𝐱)−s⋆​(𝐱),s^2​(𝐱)−s⋆​(𝐱),…,s^n​(𝐱)−s⋆​(𝐱))\vec{\bm{\epsilon}}:=(\hat{s}_{1}({\mathbf{x}})-s^{\star}({\mathbf{x}}),\hat{s}_{2}({\mathbf{x}})-s^{\star}({\mathbf{x}}),\dots,\hat{s}_{n}({\mathbf{x}})-s^{\star}({\mathbf{x}})) form a Gaussian vector with mean zero and covariance 𝚺⁡(𝐱)∈ℝd​n×d​n\bm{\Sigma}({\mathbf{x}})\in\mathbb{R}^{dn\times dn}.

Equations 13–15
𝕀⊤=[𝐈d×d⊤,𝐈d×d⊤​…​𝐈d×d⊤]⊤.\displaystyle\mathbb{I}^{\top}=[\mathbf{I}_{d\times d}^{\top},\mathbf{I}^{\top}_{d\times d}\dots\mathbf{I}^{\top}_{d\times d}]^{\top}. (13)
𝐬^1:n(𝐱)=(s^1(𝐱),s^2(𝐱),…,sn(𝐱))∣𝐱∼𝒩(𝕀s⋆(𝐱),𝚺(𝐱)).\displaystyle\hat{\mathbf{s}}_{1:n}({\mathbf{x}})=(\hat{s}_{1}({\mathbf{x}}),\hat{s}_{2}({\mathbf{x}}),\dots,s_{n}({\mathbf{x}}))\mid{\mathbf{x}}\sim\mathcal{N}(\mathbb{I}s^{\star}({\mathbf{x}}),\bm{\Sigma}({\mathbf{x}})). (14)
s^mle​(𝐱)\displaystyle\hat{s}^{\textsc{mle}}({\mathbf{x}}) =argmaxs⁡(𝐱)p(𝒔^1:n(𝐱);s(𝐱))\displaystyle=\mathrm{argmax}_{s({\mathbf{x}})}p(\hat{\bm{s}}_{1:n}({\mathbf{x}});s({\mathbf{x}}))
=arg⁡maxs⁡(𝐱)​1(2​π)k​|𝚺|​exp⁡(−12​ϵ→⊤​𝚺​(𝐱)−1​ϵ→​(𝐱))\displaystyle=\arg\max_{s({\mathbf{x}})}\ \frac{1}{\sqrt{(2\pi)^{k}|\bm{\Sigma}|}}\exp\left(-\frac{1}{2}\vec{\bm{\epsilon}}^{\top}\bm{\Sigma}({\mathbf{x}})^{-1}\vec{\bm{\epsilon}}({\mathbf{x}})\right) (ϵ→=𝐬^1:n−𝕀s(𝐱)\vec{\bm{\epsilon}}=\hat{\mathbf{s}}_{1:n}-\mathbb{I}s({\mathbf{x}}))
=mins⁡(𝐱)⁡12​ϵ→⊤​𝚺​(𝐱)−1​ϵ→​(𝐱)\displaystyle=\min_{s({\mathbf{x}})}\frac{1}{2}\vec{\bm{\epsilon}}^{\top}\bm{\Sigma}({\mathbf{x}})^{-1}\vec{\bm{\epsilon}}({\mathbf{x}}) (ϵ→=𝐬^1:n−𝕀s⋆(𝐱)\vec{\bm{\epsilon}}=\hat{\mathbf{s}}_{1:n}-\mathbb{I}s^{\star}({\mathbf{x}}))
=arg⁡mins⁡(𝐱)​(𝐬^N​(𝐱)−𝕀​s⋆​(𝐱))⊤​𝚺​(𝐱)−1​(𝐬^N​(𝐱)−𝕀​s⋆​(𝐱)).\displaystyle=\arg\min_{s({\mathbf{x}})}(\hat{\mathbf{s}}_{N}({\mathbf{x}})-\mathbb{I}s^{\star}({\mathbf{x}}))^{\top}\bm{\Sigma}({\mathbf{x}})^{-1}(\hat{\mathbf{s}}_{N}({\mathbf{x}})-\mathbb{I}s^{\star}({\mathbf{x}})).
s^mle(𝐱)=(𝕀⊤𝚺(𝐱)−𝟏𝕀)−1(𝕀⊤Σ(𝐱)−1)𝐬^1:n(𝐱).\displaystyle\hat{s}^{\textsc{mle}}({\mathbf{x}})=\bm{\left}(\mathbb{I}^{\top}\Sigma({\mathbf{x}})^{-1}\mathbb{I}\right)^{-1}\left(\mathbb{I}^{\top}\Sigma({\mathbf{x}})^{-1}\right)\hat{\mathbf{s}}_{1:n}({\mathbf{x}}). (15)

Case 1: d = 1 ( 𝐱 is scalar) scores are independent.

For scalar, independent score errors with positive variances, the maximum-likelihood estimate is a convex combination with weights proportional to inverse variance. This special case explains why combining complementary estimators can help, without identifying optimal weights for history guidance.

Equation 16
s^mle​(𝐱)=∑i=1nαi(∑jαj​(𝐱))​s^i​(𝐱)\displaystyle\hat{s}^{\textsc{mle}}({\mathbf{x}})=\sum_{i=1}^{n}\frac{\alpha_{i}}{(\sum_{j}\alpha_{j}({\mathbf{x}}))}\hat{{s}}_{i}({\mathbf{x}}) (16)

Case 2: general d ( 𝐱 is scalar) scores are independent, and the errors s ^ i − s ⋆ have scaled identity covariance.

Independent vector errors with scaled-identity covariance yield the same inverse-variance weighting. A toy example then assigns disjoint error regions to individual estimators: averaging reduces squared error from 1/n to 1/n². The printed probability equality in that example is inconsistent, but the intended complementary-error illustration is straightforward.

Real guidance may use correlated estimators, negative unconditional coefficients, or weights that do not sum to one. The Gaussian averaging argument therefore does not prove that every guidance mixture is optimal or samples the exact desired conditional distribution.

Equations 17–20
s^mle​(𝐱)=∑i=1nαi​(𝐱)(∑jαj​(𝐱))​s^i​(𝐱),\displaystyle\hat{s}^{\textsc{mle}}({\mathbf{x}})=\sum_{i=1}^{n}\frac{\alpha_{i}({\mathbf{x}})}{(\sum_{j}\alpha_{j}({\mathbf{x}}))}\hat{{s}}_{i}({\mathbf{x}}), (17)
Pr[𝐱∈𝒳i]=1n,s^i(𝐱)={1𝐱∈𝒳i0otherwise\displaystyle\Pr[\mathbf{x}\in\mathcal{X}_{i}]=\frac{1}{n},\quad\hat{s}_{i}({\mathbf{x}})=\begin{cases}1&\mathbf{x}\in\mathcal{X}_{i}\\ 0&\text{otherwise}\end{cases} (18)
𝔼[(s^i)2]=ℙ[𝐱∈𝒳i]=1n.\displaystyle\mathbb{E}[(\hat{s}_{i})^{2}]=\mathbb{P}[\mathbf{x}\in\mathcal{X}_{i}]=\frac{1}{n}. (19)
𝔼[(1n∑i=1ns^i)2]=ℙ[𝐱∈𝒳i]=1n2.\displaystyle\mathbb{E}\left[\left(\frac{1}{n}\sum_{i=1}^{n}\hat{s}_{i}\right)^{2}\right]=\mathbb{P}[\mathbf{x}\in\mathcal{X}_{i}]=\frac{1}{n^{2}}. (20)

A.4 Sampling with DFoT and History Guidance

Algorithm 1 preserves the full sampling procedure. At each denoising step, it computes an unconditional score and the requested history-conditioned scores, combines their differences, and advances a chosen score-based sampler. A partially masked history must actually be diffused to the supplied noise level; changing only the noise label would create a mismatched input.

Ordinary conditioning uses one clean history with weight one. Vanilla guidance increases that weight. Temporal guidance uses several clean subsequences. Fractional guidance combines the clean-history branch with a partially noised branch. The same interface supports extrapolation and interpolation by changing the observed/generated index sets.

Algorithm 1 Flexible Sampling with DFoT and (optionally) History Guidance
 Task: specified by indices ℋ{\mathcal{H}}, 𝒢=𝒯∖ℋ{\mathcal{G}}={\mathcal{T}}\setminus{\mathcal{H}}, and history frames 𝐱ℋ{\mathbf{x}_{\mathcal{H}}}.
 Input: diffusion process defined by αk,σk\alpha_{k},\sigma_{k}, diffusion sampler 𝒮\mathcal{S} with sampling steps NN,DFoT model 𝐬𝜽​(⋅,⋅){\mathbf{s}}_{\bm{\theta}}(\cdot,\cdot), and History Guidance scheme specified by {(ℋi,kℋi,ωi)}i=1I\{({\mathcal{H}}_{i},k_{{\mathcal{H}}_{i}},\omega_{i})\}_{i=1}^{I}.
 Sample 𝐱𝒢∼𝒩⁡(0,I){\mathbf{x}}_{\mathcal{G}}~\sim\mathcal{N}(0,I), then 𝐱𝒯←𝐱ℋ⊕𝐱𝒢{\mathbf{x}}_{{\mathcal{T}}}\leftarrow{\mathbf{x}}_{{\mathcal{H}}}\oplus{\mathbf{x}}_{{\mathcal{G}}} ⊳\triangleright Sample random noise for generation frames
 for n=N,N−1,…,1n=N,N-1,\ldots,1 do
   k𝒯←(kt)t=1Tk_{{\mathcal{T}}}\leftarrow(k_{t})_{t=1}^{T} where {kt=nNif ​t∈𝒢kt=1if ​t∈ℋ\begin{cases}k_{t}=\frac{n}{N}&\text{if }t\in{\mathcal{G}}\\ k_{t}=1&\text{if }t\in{\mathcal{H}}\end{cases}
   𝐱^𝒯←𝐱𝒯\hat{{\mathbf{x}}}_{{\mathcal{T}}}\leftarrow{\mathbf{x}}_{{\mathcal{T}}}, then replace 𝐱^ℋ←ϵ\hat{{\mathbf{x}}}_{{\mathcal{H}}}\leftarrow\bm{\epsilon} where ϵ∼𝒩⁡(0,I)\bm{\epsilon}\sim\mathcal{N}(0,I) ⊳\triangleright Fully mask history
   𝐬^∅←𝐬𝜽​(𝐱^𝒯,k𝒯)\hat{{\mathbf{s}}}^{\varnothing}\leftarrow{\mathbf{s}}_{\bm{\theta}}(\hat{{\mathbf{x}}}_{{\mathcal{T}}},k_{{\mathcal{T}}}) ⊳\triangleright Estimate unconditional score
   for i=1,…,Ii=1,\ldots,I do
    k𝒯←(kt)t=1Tk_{{\mathcal{T}}}\leftarrow(k_{t})_{t=1}^{T} where {kt=nNif ​t∈𝒢kt=kℋiif ​t∈ℋikt=1if ​t∈ℋ∖ℋi\begin{cases}k_{t}=\frac{n}{N}&\text{if }t\in{\mathcal{G}}\\ k_{t}=k_{{\mathcal{H}}_{i}}&\text{if }t\in{\mathcal{H}}_{i}\\ k_{t}=1&\text{if }t\in{\mathcal{H}}\setminus{\mathcal{H}}_{i}\end{cases}
    𝐱^𝒯←𝐱𝒯\hat{{\mathbf{x}}}_{{\mathcal{T}}}\leftarrow{\mathbf{x}}_{{\mathcal{T}}}, then replace {𝐱^ℋi←αkℋi​𝐱^ℋi+σkℋi​ϵ​ where ​ϵ∼𝒩⁡(0,I)𝐱^ℋ∖ℋi←ϵ​ where ​ϵ∼𝒩⁡(0,I)\begin{cases}\hat{{\mathbf{x}}}_{{\mathcal{H}}_{i}}\leftarrow\alpha_{k_{{\mathcal{H}}_{i}}}\hat{{\mathbf{x}}}_{{\mathcal{H}}_{i}}+\sigma_{k_{{\mathcal{H}}_{i}}}\bm{\epsilon}\text{ where }\bm{\epsilon}\sim\mathcal{N}(0,I)\\ \hat{{\mathbf{x}}}_{{\mathcal{H}}\setminus{\mathcal{H}}_{i}}\leftarrow\bm{\epsilon}\text{ where }\bm{\epsilon}\sim\mathcal{N}(0,I)\end{cases} ⊳\triangleright Mask history based on ℋi{\mathcal{H}}_{i} and kℋik_{{\mathcal{H}}_{i}}
    𝐬^i←𝐬𝜽​(𝐱^𝒯,k𝒯)\hat{{\mathbf{s}}}^{i}\leftarrow{\mathbf{s}}_{\bm{\theta}}(\hat{{\mathbf{x}}}_{{\mathcal{T}}},k_{{\mathcal{T}}}) ⊳\triangleright Estimate ii-th conditional score
   end for
   𝐬^←𝐬^∅+∑i=1Iωi⋅(𝐬^i−𝐬^∅)\hat{{\mathbf{s}}}\leftarrow\hat{{\mathbf{s}}}^{\varnothing}+\sum_{i=1}^{I}\omega_{i}\cdot(\hat{{\mathbf{s}}}^{i}-\hat{{\mathbf{s}}}^{\varnothing}) ⊳\triangleright Compose scores
   𝐱𝒢←𝒮⁡(𝐱𝒢,𝐬^𝒢,nN,n−1N){\mathbf{x}}_{{\mathcal{G}}}\leftarrow\mathcal{S}({\mathbf{x}}_{{\mathcal{G}}},\hat{{\mathbf{s}}}_{{\mathcal{G}}};\frac{n}{N},\frac{n-1}{N}) ⊳\triangleright Denoise k=nN→n−1Nk=\frac{n}{N}\rightarrow\frac{n-1}{N}
 end for
 Output: 𝐱𝒢{\mathbf{x}}_{\mathcal{G}}

A.5 Simplifying Training Objective

Independent noise works well for short latent sequences, but the authors observed degraded sampling with fifty latent frames. They propose reducing the task mixture by independently masking only the history positions that need flexibility, or sampling a history length before applying noise. The Minecraft setup uses a simplified objective; the broader alternatives are not systematically evaluated.

A.6 Causal Variant

A non-causal DFoT can generate causally by replacing unavailable future frames with independent noise, preventing conditioning on their actual contents. Partially masking known future frames instead permits controlled future information. This does not automatically provide the speed benefits of a causal architecture with key/value caching; those require additional design choices.

A.7 Incorporating Other Conditioning

History conditioning can coexist with text, action, or camera-pose conditioning. An existing external conditioning pathway need not be removed: the key modification is support for per-frame noise levels and the associated training objective. Large text-conditioned foundation-model adaptation is proposed rather than demonstrated here.

A.8 Extended Temporal History Guidance

Extended temporal guidance decomposes both the history and the generated frames into overlapping subsets. Scores are composed on shorter contexts and averaged frame by frame where generated subsets overlap. This is used in the unusual-camera-history experiment to reduce the distribution shift of the full input sequence.

Equation 21

⨁j=1J∑i=1I∇log⁡pk​(𝐱𝒢jk|𝐱ℋi)\bigoplus_{j=1}^{J}\sum_{i=1}^{I}{\nabla\log\,}p_{k}({\mathbf{x}}_{{\mathcal{G}}_{j}}^{k}|{\mathbf{x}}_{{\mathcal{H}}_{i}})

(21)

Supplementary Visuals

The retained supplementary grids show four long navigation rollouts, a rollout comparison, unusual-history interpolation, flexible observed subsets, guidance effects, Minecraft memory, and additional Kinetics samples. The HTML splits multipage figures and consequently uses different later figure labels from the PDF.

Refer to caption
(a) Long navigation video generated by DFoT with HG. # frames = 862.
Refer to caption
(a) Long navigation video generated by DFoT with HG. # frames = 917.
Refer to caption
(a) Long navigation video generated by DFoT with HG. # frames = 442.
Refer to caption
(b) Long navigation video generated by DFoT with HG. # frames = 442.
Figure 10: Long navigation videos generated by DFoT with HG-v and HG-f, from a single history frame on RealEstate10K. We subsample with a stride of 8 frames for visualization. The videos exhibit consistent transitions navigating while through diverse indoor and outdoor scenes, maintaining high stability over hundreds of frames. This is enabled by the improved sample quality and consistency from HG, along with DFoT’s flexibility that allows both interpolation and extrapolation.
Refer to caption
Figure 11: Qualitative comparison of DFoT with HG vs. SD on long video generation. Given a single history frame, we task both models to generate videos of moving straight ahead and visualize them with a stride of 8 frames. While SD quickly diverges after t≈30t\approx 30 frames, DFoT with HG maintains high stability until t=72t=72 and can roll out further.
Refer to caption
(a) Given slightly OOD history with rotation angles in [120∘,130∘][120^{\circ},130^{\circ}], baselines and DFoT with HG-v generate inconsistent frames with artifacts. In contrast, DFoT with HG-t generates consistent videos that highly resemble the ground truth. This is the region where HG-t starts showing its generalization gap with other methods.
Refer to caption
(a) Given OOD history, all baselines completely fail yet DFoT with HG-t still manages to generate high-quality, accurate videos.
Figure 13: Qualitative results of testing robustness to out-of-distribution history on RealEstate10K. We provide wide-angle, 4-frame history and task the models to generate the next 4 frames that interpolate between the history frames. As the angle increases, the history becomes more out-of-distribution, and thus we split the results into slightly OOD and OOD depending on the angle range.
Refer to caption
Figure 14: An illustration of the empirical flexibility of DFoT, showing ten samples from RealEstate10K, where a single DFoT model infills the missing frames given different history. DFoT successfully generates consistent samples across ten diverse tasks, each varying in the history length from 1 to 6 frames and at different timestamps.
Refer to caption
Figure 15: Improved video generation quality with vanilla history guidance on RealEstate10K, for both extrapolation and interpolation tasks. HG-v, with an increasing guidance scale, enhances fidelity and consistency while effectively removing artifacts. Videos are sampled conditioned on two history frames, with varying guidance scales ω=1\omega=1 (top, without HG-v ), 22 (middle), and 33 (bottom). Zoom into the boxed regions to see notable differences.
Refer to caption
Figure 16: Visualization of long context generation on Minecraft. We visualize the generation up to the maximum length of the training set. Given 2525 initial frames (red), DFoT with temporal history guidance (upper) can roll out stably without blowing up even without CFG. In contrast, one can clearly see that without temporal history guidance (lower), conditional generation easily becomes blurry in later frames. This is likely because the shorter-context model is less likely to fall out of distribution, using its generation power to compensate for the unconfident, blurry prediction from the longer-context model.
Refer to caption
Figure 17: Additional qualitative comparison on Kinetics-600. We uniformly subsample 6 frames {0,3,6,9,12,15}\{0,3,6,9,12,15\} from 16-frame videos, conditioned on 5-frame histories. Both DFoT variants, scratch and fine-tuned, consistently align with the history, generating high-quality samples that closely resemble the ground truth. In contrast, the baselines, typically ordered as SD >> BD >> FS, struggle to maintain consistency and often exhibit artifacts.

Appendix B Extended Related Work

The extended comparison distinguishes fixed-frame guidance from the arbitrary subsets and partial noise levels supported by DFoT.

B.1 History-conditioned Guidance

Earlier image-to-video and multiview systems use first-frame embeddings, channel concatenation, or fixed sets of masked conditioning frames. Some already process observed and generated images as one sequence, so that architectural idea alone is not unique to DFoT. The contribution combines flexible history with independent-noise training and richer score composition; the matched binary-dropout ablation tests one relevant alternative.

Appendix C Experimental Details

The protocol supplement specifies distinct datasets, model backbones, training schedules, conditioning schemes, and evaluation sample counts. These details matter when interpreting comparisons across experiments.

C.1 Datasets

Kinetics-600 contains diverse action clips at 128×128. RealEstate10K supplies 256×256 walkthrough images and camera poses. Minecraft provides 200,000 action-labeled navigation videos of 300 frames; its original 128×128 images are upsampled to 256×256, which does not add captured detail.

Fruit Swapping contains 300 complete swap demonstrations without disturbances and 300 short regrasp demonstrations without moving fruit between slots. Average lengths are about 540 and fifty frames. A 15 cm slot allows within-slot perturbations, so the evaluation requires combining memory of the initial arrangement with local recovery behavior not jointly demonstrated in a single training trajectory.

C.2 Implementation Details

Kinetics uses a chunk-wise VideoVAE and DiT; Minecraft uses an image VAE and DiT; RealEstate10K uses pixel-space U-ViT. Attention spans spatial and temporal tokens, with 3D rotary position embeddings and frame-specific adaptive normalization. Camera rays and actions supply additional conditioning. The robot model instead uses the attention U-Net listed in Table 2.

Models use cosine-based schedules, velocity prediction except x₀ prediction for the robot, dataset-specific loss reweighting, and AdamW training. Sampling uses fifty deterministic DDIM steps; multiple guidance branches are batched together, adding computation despite parallel evaluation. Robot tokens bundle fifteen actions with the next image.

The stated video training budget is about five days on twelve H100 GPUs per model; robot training takes four hours on four RTX4090s. Both implementation tables are retained completely. The Kinetics batch size is 192 in Table 2 but 196 in the resource discussion. Robot frame-skip wording also differs between the per-image description and the fifteen-action bundle shown in the table.

Table 2: Implementation details for DFoT and baseline models.
Kinetics-600 RealEstate10K Minecraft Imitation Learning
VAEs
Input {1,4}×128×128\{1,4\}\times 128\times 128 - 1×1\times 256 ×256\times 256 -
   Compression (ft,fsf_{t},f_{s}) {1, 4}, 8 1, 8
   Latent channels 16 4
Training steps 600k 50k
   Optimizer Adam Adam
   Batch size 64 96
   Learning rate 1e-4 4e-4
   EMA 0.999 ✗
VDMs
Input 17×128×12817\times 128\times 128 8×8\times 256 ×\times 256 50×256×25650\times 256\times 256 21×32×3221\times 32\times 32
   Latent 5×16×165\times 16\times 16 ✗ 50×32×3250\times 32\times 32 ✗
   Frame skip 1 10 →\rightarrow Max 2 15
Backbone DiT U-ViT DiT Attention UNet
   Patch size 1 2 2 1
   Layer types Transformer [ResNet×2,Transformer×2]\left[\text{ResNet}\times 2,\text{Transformer}\times 2\right] Transformer Attention, Conv
   Layers 28 [3,3,6,20]\left[3,3,6,20\right] 12 8
   Hidden size 1152 [128,256,576,1152]\left[128,256,576,1152\right] 768 128
   Heads 16 9 12 4
Training steps 640k 500k 200k 100k
   Warmup steps 10k 10k 10k 10k
   Optimizer AdamW AdamW AdamW AdamW
   Batch size 192 96 96 64
   Learning rate 2e-4 5e-5 1e-4 5e-4
   Weight decay 0 1e-2 1e-3 1e-3
   EMA 0.9999 0.9999 0.9999 ✗
Diffusion type Discrete Continuous Discrete Discrete
   Noise schedule Cosine Shifted Cosine Shifted Cosine Cosine
   Noise schedule shift ✗ 0.125 0.125 ✗
   Parameterization 𝐯{\mathbf{v}} 𝐯{\mathbf{v}} 𝐯{\mathbf{v}} 𝐱0{\mathbf{x}}_{0}
   Sampler DDIM DDIM DDIM DDIM
   Sampling steps 50 50 50 50

C.3 Evaluation Metrics.

FVD compares generated and real video distributions in pretrained I3D feature space; it includes history frames here. VBench quality averages aesthetic and imaging scores, while consistency averages subject and background scores; dynamic degree measures motion. These scores evaluate generated videos without paired ground-truth comparison.

LPIPS is computed only on generated frames against their targets in highly constrained tasks. Low FVD or LPIPS does not establish physical correctness, and more motion is not universally better merely because dynamic degree increases.

C.4 Details on Video Generation Benchmark ( Section 6.2 )

The short-video benchmark predicts eleven frames from five observed frames. The latent model actually produces twelve and discards the last to match the standard sixteen-frame evaluation. FVD uses 50,000 generated videos and three random seeds.

The literature comparison mixes architectures and training regimes. The resource discussion compares parameter counts or FLOPs and batch sizes, not a fully matched total training-compute measurement. It says MAGVIT variants use comparable resources, while some other baselines use more; this is narrower than the introduction’s broad order-of-magnitude compute claim.

C.5 Details on History Guidance Experiment ( Section 6.3 )

For 64-frame Kinetics rollout, the model repeatedly conditions on two latent tokens and generates three more. Evaluation uses 1,024 videos. Guidance weights range from 1.0 to 4.0 in increments of 0.5; fractional guidance fixes the history noise level at 0.8. These results characterize the tested tradeoff curve.

C.6 Details on OOD History Experiment ( Section 6.4 , Task 1 )

RealEstate10K scenes are binned by camera-rotation range in ten-degree intervals. Up to 32 test scenes are sampled per bin, fewer in sparse bins. Histories contain four spaced frames and the model interpolates four target frames. Rotations above roughly 100° are uncommon in training, especially above 140°.

The temporal method combines two overlapping three-frame histories with weight two each and also overlaps three-frame generation subsets. The vanilla comparison uses the full history with weight two. Thus the result includes the extended generation-subset method, not solely a change in history length.

C.7 Details on Long Context Generation ( Section 6.4 , Task 2 ).

Minecraft training uses fifty subsampled frames with up to 25 history frames and an original-video stride of two. Evaluation starts with 25 context frames, then makes five 25-frame continuations, totaling 125 generated frames. Scores from 25-frame and four-frame histories receive equal weight, without subtracting an unconditional branch.

C.8 Details on Robot Imitation Learning ( Section 6.4 , Task 3 ).

The robot baselines diffuse actions and observations with current, short, or full histories. The protocol calls the short baseline two frames, while the results describe four. A full-history model can overfit to fruits being placed at slot centers; a short-history model can react locally but lose the intended final arrangement.

The reported composed policy uses one-frame, four-frame, and full-history scores with weights 0.45, 0.45, and 0.2. These sum to 1.1, so they are not a normalized convex average. The unconditional score is omitted. The observed failure modes motivate composition, but do not prove that a full-history policy is theoretically incapable of learning reactive behavior.

C.9 Details on Ultra Long Video Generation ( Section 6.5 ).

Long navigation first rolls out coarse frames: one observed image produces seven frames, then each iteration uses the last four to generate four more. A user supplies direction and distance after each iteration; these inputs become camera poses. Six frames are then interpolated between each adjacent coarse pair, using interpolated translations and SLERP rotations.

Rollout normally uses fractional guidance with weight four and history noise 0.4, switching to vanilla guidance for large direction or distance changes. Interpolation uses vanilla weight 1.5. Previously generated frames are marked slightly noisy at level 0.02 as an additional stabilization heuristic. The showcase’s length therefore reflects this entire pipeline.

Appendix D Additional Experimental Results

Additional experiments examine fine-tuning, binary dropout with guidance, long-context metrics, robot disturbances, and qualitative rollout behavior.

D.1 Additional Results on Fine-tuning to DFoT

Fine-tuning a pretrained FS model for 80,000 steps gives lower training loss than the 640,000-step scratch model, but slightly worse FVD: 4.7 versus 4.3. At 40,000 steps the fine-tuned model scores 5.2. The claimed speedup concerns the additional fine-tuning stage and selected loss comparisons, excluding pretraining.

The authors attribute the lag in sample quality to slow exponential-moving-average adaptation. At 80,000 steps, removing EMA worsens FVD to 7.3. Predicting that additional training would surpass the scratch model is a hypothesis, not a reported result.

Original vector plot, retained from the source.
(a) A comprehensive view of the training loss curves. DFoT (fine-tuned) achieves a low training loss early in the iterations and converges significantly faster than DFoT (scratch).
Original vector plot, retained from the source.
(b) A zoomed-in view of the training loss curves. Only after 80k iterations, DFoT (fine-tuned) displays a lower training loss than DFoT (scratch) trained for 640k iterations.
Figure 18: Training loss curves for DFoT, trained from scratch and fine-tuned from the pre-trained FS model, on Kinetics-600.

D.2 Ablation Study on Binary-Dropout Diffusion with Vanilla History Guidance

Binary dropout can use vanilla history guidance and reaches FVD 196.0 at its best tested weight, versus DFoT’s 181.6 with vanilla and 170.4 with fractional guidance. DFoT is not better at every tested weight: the text identifies an exception at 2.5. Fractional guidance additionally requires training on intermediate history-noise levels.

Original vector plot, retained from the source.
(a) FVD as a function of guidance scale 𝝎\bm{\omega} for DFoT and BD using HG. Both with HG-v, DFoT yields better FVD-ω\omega curves than BD and thus achieves a lower best FVD score. Applying HG-f, which is specific to DFoT, enlarges the performance gap.

Refer to caption

(b) Qualitative comparison of DFoT and BD using HG-v with optimal guidance scales ω=1.5\omega=1.5. While DFoT generates consistent, high-quality samples, BD struggles to remain consistent with the history frames and produces artifacts. Red box = history frames.
Figure 19: History Guidance works better with DFoT than with Binary-Dropout Diffusion (BD).

D.3 Detailed Results on Long Context Generation ( Section 6.4 , Task 2 )

Minecraft FVD uses 1,024 samples across all 125 generated frames. Temporal composition improves the reported metric from 97.625 to 79.19, and an example shows better retained detail. Action conditioning constrains variation, but this remains a dataset-specific result rather than a general guarantee of long-term consistency.

D.4 Detailed Results on Long-horizon yet Reactive Imitation Learning ( Section 6.4 , Task 3 )

The main robot experiment randomizes the setup 100 times and reports 83% success for composed guidance. Short-context policies fail to maintain the overall plan; the full-history policy often grasps an old slot-center location after a fruit is moved. Occasional full-history recoveries are described, so the source does not justify assigning it an unreported zero success rate.

A harder intervention knocks fruit from the closing gripper; composed guidance succeeds in 28% of those reported trials, with the separate trial count unspecified. The results are consistent with combining memory and local recovery, but do not establish the internal mechanism of subtrajectory stitching.

Refer to caption
Figure 20: Visualization of the fruit-swapping task through a DFoT generated video. Two fruits are randomly put within two random slots. The robot is tasked with swapping its slots using the third slot and moving one fruit at a time. This task requires long-horizon memory because it needs to remember the initial location of the fruit for the task completion, but also react to different fruit locations within each slot, which is combinatorically impossible form the dataset.

D.5 Additional Qualitative Results

Additional grids preserve examples of base-model comparisons, arbitrary observed subsets, increasing guidance, unusual histories, and Minecraft generation. They illustrate specific successful and failed cases rather than proving uniform performance over every supported conditioning pattern.

D.6 Detailed results on Ultra Long Video Generation ( Section 6.5 ).

A separate straight-line navigation comparison omits interpolation for both models. SD visibly degrades around frame thirty in the displayed cases, while DFoT with guidance remains coherent beyond frame 72. The longer showcase samples contain 862, 917, and two 442-frame videos after the interactive rollout/interpolation pipeline.

These are qualitative examples, without an aggregate long-rollout failure rate or controlled human-navigation study. Their length exceeds the training clips, but this does not demonstrate persistent globally consistent scene geometry.

Concept demonstration

Open image ↗