Literature
Display
← All papers

Photorealistic Video Generation with Diffusion Models

Agrim Gupta · Lijun Yu · Kihyuk Sohn · Xiuye Gu · Meera Hahn · Li Fei-Fei · Irfan Essa · Lu Jiang · José Lezama

· ECCV 2024 proceedings (2025 print edition) ↗

Computer vision7 min text · CondensedOriginal paper ↗
Contents

Overview

The complete paper and implementation appendix are condensed below. All six numbered figures, including every sample grid, eight tables with all six ablation panels, and displayed equations are preserved. The original teaser is reunited with its caption, and omitted ablation-panel descriptions are restored from the PDF.

Three five-frame text-to-video sample sequences: surfing cat, walking corgi, and dancing raccoon.
Figure 1: W.A.L.T samples for text-to-video generation. Our approach can generate high-resolution, temporally consistent photorealistic videos from text prompts. The samples shown are 512×896512\times 896 resolution over 3.63.6 seconds duration at 88 frames per second.
T2V (base)
Input 5×16×285\times 16\times 28
Spatial window 1×16×281\times 16\times 28
Spatiotermporal window 5×8×145\times 8\times 14
Training steps 250000
Batch size 512
lr schedule Constant
Optimizer Adafactor
lr 0.00005
Table 6: Hyperparameters for aspect-ratio finetuning.

At a glance

W.A.L.T uses a causal image/video autoencoder and a transformer with spatial and spatiotemporal attention windows for latent diffusion. Shared visual latents permit joint training on images and videos while temporal compression reduces sequence length. Academic benchmarks test image generation, action-conditioned video generation, and frame prediction; a much larger three-stage system generates text-conditioned 512×896 videos at eight frames per second. Classifier-free guidance is omitted in the main controlled benchmarks but used for the text-to-video results.

1 Introduction

Full attention becomes expensive as video sequences grow. Per-frame image latents allow reuse of image data but retain a long temporal sequence; conventional video compression can complicate image/video sharing. W.A.L.T addresses both with an independently encoded first frame and alternating attention patterns. This paper’s contributions are the unified continuous latent representation, an efficient transformer denoiser, controlled benchmark comparisons, and a large text-to-video cascade. Its state-of-the-art claims refer to the systems evaluated in this 2023 source.

Refer to caption
Figure 2: W.A.L.T. We encode images and videos into a shared latent space. The transformer backbone processes these latents with blocks having two layers of window-restricted attention: spatial layers capture spatial relations in both images and video, while spatiotemporal layers model temporal dynamics in videos and passthrough images via identity attention mask. Text conditioning is done via spatial cross-attention.

2 Related Work

Video Diffusion Models.

Video diffusion can operate in pixels or compressed latents. Inflating pretrained image networks with temporal layers is convenient but often retains per-frame compression. W.A.L.T instead jointly learns an image-compatible spatiotemporal representation and uses it for both data types.

Transformers for Generative Modeling.

Transformers have been used for adversarial, autoregressive, parallel-token, and diffusion generation. This work uses diffusion denoising over continuous video latents, avoiding token-by-token generation within a clip. Longer videos still require repeated clip generation.

3 Background

Training corrupts real samples with Gaussian noise and learns the reverse process. All experiments predict the velocity-style target combining clean data and noise. The autoencoder compresses space and time, and its decoder reconstructs pixels. Although motivated by vector-quantized autoencoders, this model uses continuous latents without a codebook. Adversarial and perceptual reconstruction losses improve decoded quality.

Equations 1–2
𝒙𝒕=γ⁡(t)​𝒙𝟎+1−γ⁡(t)​ϵ,\boldsymbol{x_{t}}=\sqrt{\gamma(t)}\ \boldsymbol{x_{0}}+\sqrt{1-\gamma(t)}\ \boldsymbol{\epsilon}, (1)
𝔼x∼pdata,t∼𝒰⁡(0,1),ϵ∼𝒩⁡(𝟎,𝑰)​[‖𝒚−fθ​(𝒙𝒕,𝒄,t)‖2],\mathbb{E}_{x\sim p_{\text{data}},t\sim\mathcal{U}(0,1),\boldsymbol{\epsilon}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})}\left[\left\|\boldsymbol{y}-f_{\theta}(\boldsymbol{x_{t}};\boldsymbol{c},t)\right\|^{2}\right], (2)

4 W.A.L.T

4.1 Learning Visual Tokens

A causal three-dimensional convolutional encoder processes each frame using only its present and past context. The first frame can therefore be encoded independently, so an image is simply a one-frame input. The remaining frames are temporally compressed. The MAGVIT-v2-style encoder/decoder is adapted to quantization-free latents; training batches can contain video clips and stacks of independent images.

4.2 Learning to Generate Images and Videos

Each latent frame is patchified and given learned spatial and temporal positions, with additional relative positions. Spatial attention covers tokens within one latent frame. Spatiotemporal attention spans time within a smaller spatial window; independent-image inputs use an identity mask in this layer. Alternation models spatial detail and video dynamics at lower memory cost than global space-time attention. Joint video encoding reduces a source of per-frame flicker, but does not prove that every generated sequence is artifact-free.

4.3 Conditional Generation

Text conditions use cross-attention; joint image/video training applies it in spatial layers. Adaptive layer normalization modulates features with condition and diffusion-time embeddings. AdaLN-LoRA shares most conditioning parameters and adds a low-rank per-layer correction: in the cited one-billion-parameter transformer example, rank two reduces these extra parameters from 475 million to 12 million. Self-conditioning sometimes predicts a clean latent estimate and feeds a stopped-gradient copy into a second denoising pass, trading additional training computation for quality.

Equation 3
A1\displaystyle\small A^{1} =MLP​(𝒄+𝒕),\displaystyle=\small\texttt{MLP}(\boldsymbol{c}+\boldsymbol{t}), Ai\displaystyle\small A^{i} =A1+Wbi​Wai​(𝒄+𝒕)​∀i≠1,\displaystyle=\small A^{1}+W_{b}^{i}W_{a}^{i}(\boldsymbol{c}+\boldsymbol{t})\hskip 9.24994pt\forall i\neq 1, (3)
Method K600 FVD↓\downarrow UCF FVD↓\downarrow params. steps
TrIVD-GAN-FP [51] 25.7±0.7 – – 1
Video Diffusion [36] 16.2±0.3 – 1.1B 256
RIN [41] 10.8 – 411M 1000
TATS [23] – 332±18 321M 1024
Phenaki [74] 36.4±0.2 – 227M 48
MAGVIT [81] 9.9±0.3 76±2 306M 12
MAGVITv2 [82] 4.3±0.1 58±2 307M 24
W.A.L.T-L Ours 3.3±0.0 46±2 313M 50
W.A.L.T-XL Ours – 36±2 460M 50
Table 1: Video generation evaluation on frame prediction on Kinetics-600 and class-conditional generation on UCF-101.

4.4 Autoregressive Generation

Frame-prediction training conditions on one latent frame for image animation or two for video continuation, with a mask identifying supplied context. At inference, repeated conditional generation extends the clip using classifier-free guidance. This supports longer sequences but does not establish indefinite temporal consistency.

4.5 Video Super Resolution

A base model produces low-resolution video and two super-resolution models each double spatial resolution. The refiners receive upscaled latents and noise augmentation, reducing their mismatch between clean training inputs and imperfect generated inputs. The base model is trained square, then fine-tuned for a 128×224 landscape frame through positional-embedding interpolation; later stages reach 512×896.

Method Cost (Iter×\timesBS) FID↓\downarrow IS↑\uparrow params. steps
BigGAN-deep [6] - 6.95 171.4 160M 1
LDM-4 [61] 178k×\times1200 10.56 103.5 400M 250
DiT-XL/2 [53] 7000k×\times256 9.62 121.5 675M 250
ADM [16] - 7.49 127.5 608M 2000
MDT [22] 6500k×\times256 6.23 143.0 676M 250
MaskDiT [87] 1200k×\times1024 5.69 178.0 736M 40
RIN [41] 600k×\times1024 3.42 182.0 410M 1000
simple diffusion [38] 500K×\times2048 2.77 211.8 2B 512
VDM++ [45] - 2.40 225.3 2B 512
W.A.L.T-L Ours 437k×\times1024 2.56 215.1 460M 50
Table 2: Class-conditional image generation on ImageNet 256×\times256. We adopt the evaluation protocol and implementation of ADM [16] and report results without classifier free guidance.

5 Experiments

The evaluation separates controlled academic datasets from the larger, heterogeneous text-to-video training setting.

5.1 Visual Generation

For class-conditioned UCF-101, W.A.L.T-XL reports FVD 36±2 with 460 million parameters and 50 DDIM steps, versus MAGVIT-v2’s 58±2. For Kinetics-600 prediction from five observed frames, W.A.L.T-L reports FVD 3.3 versus 4.3 for MAGVIT-v2. ImageNet evaluation generates 50,000 images at 256×256: W.A.L.T reports FID 2.56 and Inception Score 215.1 without guidance. VDM++ has a lower FID of 2.40, with a larger reported parameter count. These distributional metrics compare benchmark samples; they do not directly measure physical correctness or complete text alignment.

patch size pp FVD↓\downarrow IS↑\uparrow
1 60.7 87.2
2 134.4 82.2
4 461.8 63.9
(a) Patch size. Lower patch size is significantly better.
st window FVD↓\downarrow IS↑\uparrow sps
5×4×45\times 4\times 4 56.9 87.3 2.24
5×8×85\times 8\times 8 59.6 87.4 2.00
5×16×165\times 16\times 16 55.3 87.4 1.75
full self attn. 59.9 87.8 1.20
(b) Spatiotemporal window size. Full self-attention is not essential for good performance. sps is steps per sec.
pscp_{\text{sc}} FVD↓\downarrow IS↑\uparrow
0.0 109.9 82.6
0.3 76.0 86.5
0.6 60.0 86.8
0.9 61.4 87.1
(c) Self-conditioning. Higher p_sc is better.
rr FVD↓\downarrow IS↑\uparrow params
2 60.7 87.2 313 M
4 56.6 87.3 314 M
16 55.5 88.0 316 M
64 54.4 87.9 324 M
256 52.5 88.5 357 M
(d) AdaLN-LoRA. Bigger r is better.
FVD↓\downarrow IS↑\uparrow
w/o qk norm [14] 59.0 86.8
w/o latent norm 67.9 87.1
w/o zero snr [48] 91.0 84.2
full method 60.7 87.2
(e) Other improvements. See text for details.
cc rFVD↓\downarrow FVD↓\downarrow IS↑\uparrow
4 37.7 86.4 84.9
8 17.1 75.4 86.3
16 8.2 67.0 86.0
32 3.5 83.4 82.9
(f) Latent dimension c. Higher c is better.
Table 3: Ablation experiments on UCF-101 [70]. We compare FVD and inception scores to ablate important design decisions with the default setting: L model, 1×16×161\times 16\times 16 spatial window, 5×8×85\times 8\times 8 saptiotemporal (st) window, psc=0.9p_{\text{sc}}=0.9, c=8c=8 and r=2r=2.

5.2 Ablation Studies

UCF-101 ablations use 35,000 training steps. Smaller patches substantially improve FVD. Local temporal windows provide comparable quality with up to roughly twice the reported steps per second of full attention. Self-conditioning improves over none, although the displayed 0.6 rate slightly beats 0.9 on FVD. Larger AdaLN-LoRA ranks improve quality at increased parameter cost. A parameter-matched comparison favors different configurations on different measures: the larger shared-conditioning backbone has lower training loss but slightly worse FVD. Zero terminal signal-to-noise ratio prevents residual clean-signal leakage during the highest-noise training step and improves generation. Increasing latent channels improves reconstruction, but generation is nonmonotonic: the shown channel-16 model has better FVD than channel eight, while channel 32 worsens it. The authors retain eight as a robust default across tasks. Thus the original “higher is better” panel captions should be read alongside the actual metric columns.

Model AdaLN FVD↓\downarrow IS↑\uparrow params. final loss
L separate 34.6 90.2 458M 0.274
XL LoRA-2 36.7 89.4 460M 0.268
Table 4: Parameter matched comparison between AdaLN-LoRA and per layer adaln layers. See text for details.
Refer to caption
Figure 3: Qualitative evaluation. Example videos generated by W.A.L.T from natural language prompts at 512×896512\times 896 resolution over 3.63.6 seconds duration at 88 frames per second. The W.A.L.T model is able to generate temporally consistent photorealistic videos that align with the textual prompt.
Refer to caption
Figure 4: Examples of consistent 3D camera motion (5.1 secs). Prompts: camera turns around a {blue jay, bunny}, studio lighting, 360∘360^{\circ} rotation. Best viewed in video format.

5.3 Text-to-video

Text-to-video training uses approximately 970 million image-text pairs and 89 million video-text pairs from public internet and internal sources. The base model has three billion parameters, followed by approximately 1.3 billion- and 419 million-parameter refiners. All text-to-video results use classifier-free guidance. These data and model scales differ substantially from the controlled benchmark experiments.

5.3.1 Quantitative Evaluation

Under the zero-shot UCF-101 text protocol, adding images to a 419-million-parameter model improves FVD from 598.8 to 344.5; scaling the jointly trained base to three billion parameters yields 258.1 and Inception Score 35.1. The listed PYoCo comparison has FVD 355.2 but a higher Inception Score, 47.8. Training data and text encoders vary across systems, so this is a system-level comparison rather than a controlled architecture comparison. W.A.L.T uses T5-XL; the source suggests stronger text conditioning as one possible explanation for the score difference.

Method IS (↑\uparrow) FVD (↓\downarrow)
CogVideo (Chinese) [37] 23.6 751.3
CogVideo (English) [37] 25.3 701.6
MagicVideo [88] - 699.0
Make-A-Video [66] 33.0 367.2
Video LDM [4] 33.5 550.6
PYoCo [24] 47.8 355.2
W.A.L.T (Ours) 419M (video only) 26.8 598.8
W.A.L.T (Ours) 419M (video + image) 31.7 344.5
W.A.L.T (Ours) 3B (video + image) 35.1 258.1
Table 5: UCF-101 text-to-video generation. Joint training on image and video datasets in conjunction with scaling the model parameters is essential for high quality video generation.

5.3.2 Qualitative Results

The retained examples illustrate text-conditioned clips, image animation, and camera movement around an object. Figure 4 shows a 5.1-second continuation example. Full temporal behavior is better assessed in the accompanying videos than from selected still frames.

6 Conclusion

A shared causal visual representation and windowed transformer make joint image/video latent diffusion practical and competitive in the reported tests. Remaining practical limits include the large data and model requirements, cascaded inference, incomplete standardization of text-to-video evaluation, and lack of direct guarantees for physical realism or long-horizon consistency.

Acknowledgements

The authors acknowledge the listed colleagues for feedback and discussions.

Appendix A Implementation Details

The appendix supplies training and inference settings for the autoencoders, denoisers, and refiners. Final UCF-101 results use 60,000 steps; ablations use 35,000. The usual compression factors are eight spatially and four temporally, turning a 17-frame clip into five latent frames. Aspect-ratio fine-tuning changes spatial windows and interpolates absolute and relative positions. To continue a generated 17-frame clip, the last five decoded frames are encoded into two context latents for the next prediction. UCF class prompts are adapted from prior evaluation wording. The main text says patch size two for the final refiner, while Table 8 lists four; both source descriptions remain available rather than being silently reconciled.

Appendix B Additional Results

B.1 Image Generation

The guided ImageNet comparison uses a power-cosine guidance schedule because ordinary guidance showed little benefit. W.A.L.T reports FID 2.40 and Inception Score 290.5: the latter leads the displayed comparisons, while several alternatives have lower FID. Both rows of generated samples are retained.

Method Cost (Iter×\timesBS) FID↓\downarrow IS↑\uparrow Params. Steps
LDM-4 [61] 178k×\times1200 3.60 247.7 400M 250
DiT-XL/2 [53] 7000k×\times256 2.27 278.2 675M 250
ADM [16] - 3.94 215.8 608M 2000
MDT [22] 6500k×\times256 1.79 283.0 676M 250
MaskDiT [87] 1200k×\times1024 2.28 276.6 736M 40
simple diffusion [38] 500K×\times2048 2.44 256.3 2B 512
VDM++ [45] - 2.12 267.7 2B 512
W.A.L.T-L Ours 437k×\times1024 2.40 290.5 460M 50
Table 7: Class-conditional image generation on ImageNet 256×\times256. We adopt the evaluation protocol and implementation of ADM [16] and report results with classifier free guidance.
ImageNet UCF-101 K600 T2V (base) T2V (2×\times) T2V (2×\times2×\times)
First Stage
Input 1×256×2561\times 256\times 256 17×128×12817\times 128\times 128 17×128×12817\times 128\times 128 17×128×12817\times 128\times 128 17×256×44817\times 256\times 448 17×512×89617\times 512\times 896
fs,ftf_{s},f_{t} 8, - 8, 4 8, 4 8, 4 8, 4 8, 4
Channels 128 128 128 128 128 128
Channel multiplier 1,1,2,4 1, 2, 2, 4 1, 2, 2, 4 1, 2, 2, 4 1, 2, 2, 4 1, 2, 2, 4
Training duration 270 epochs 2000 epochs 270000 steps 1000000 steps 1000000 steps 1000000 steps
Batch size 256 256 256 256 256 256
lr schedule Cosine Cosine Cosine Cosine Cosine Cosine
Optimizer Adam Adam Adam Adam Adam Adam
Second Stage
Input 1×32×321\times 32\times 32 5×16×165\times 16\times 16 5×16×165\times 16\times 16 5×16×165\times 16\times 16 5×32×565\times 32\times 56 5×64×1125\times 64\times 112
Layers 24 28 24 52 40 24
Hidden size 1024 1152 1024 9216 1408 1024
Heads 16 16 16 16 16 16
Training duration 350 epochs 60000 steps 360 epochs 550000 steps 675000 steps 275000 steps
Batch size 1024 256 512 512 512 512
lr schedule Cosine Cosine Cosine Cosine Cosine Cosine
Optimizer AdamW AdamW AdamW Adafactor Adafactor Adafactor
lr 0.0005 0.0005 0.0005 0.0002 0.0005 0.0005
EMA ✓\checkmark ✓\checkmark ✓\checkmark ✗ ✗ ✗
Patch size 1 1 1 1 2 4
AdaLN-LoRA ✗ 2 2 2 2 2
Diffusion
Diffusion Steps 1000 1000 1000 1000 1000 1000
Noise schedule Linear Linear Linear Linear Linear Linear
β0\beta_{0} 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001
β1000\beta_{1000} 0.02 0.02 0.02 0.02 0.02 0.02
Sampler DDIM DDIM DDIM DDIM DDIM DDIM
Sampling steps 50 50 50 50 50 50
Guidance ✗ ✗ ✗ ✓\checkmark ✓\checkmark ✓\checkmark
Table 8: Training and evaluation hyperparameters.

B.2 Video Generation

All three Kinetics prediction panels preserve the ground-truth and generated sequences, with unobserved ground-truth frames shaded. These are qualitative examples rather than additional aggregate metrics.

Refer to caption
Refer to caption
Figure 5: ImageNet class-conditional generation samples.
Refer to caption
Refer to caption
Refer to caption
Figure 6: Frame prediction samples on Kinetics-600. Top: ground-truth, where unobserved frames are shaded. Bottom: generation.

B.3 Image-to-Video

Conditioning on a single latent image lets the model animate an image-generator output. The appendix points to project videos for this qualitative setting rather than reporting a separate image-to-video benchmark.

Concept demonstration

Open image ↗