Overview
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/26/26f988f445433750087a2288d9edc48d89dc9b0b7d5599376b27b3eb6fee8bcc.webp)
At a glance
WorldCrafter is a camera-controlled video generator that recalls previously seen scene content through learned, view-dependent memory. A multi-view encoder compresses selected historical frames, and a camera-conditioned readout supplies a fixed number of tokens to the generator. On an author-curated benchmark of 725 videos, the base model improves revisit consistency, camera tracking, and aggregate visual quality over eight baselines. Its distilled variant generates 16 frames per second on four GPUs, with a camera-control and visual-quality tradeoff.
1 Introduction
A recent-frame window cannot reliably remember a scene after the camera leaves it. Full-history attention is expensive; retrieving a few similar views can miss complementary evidence; explicit depth-based reconstruction can introduce geometric errors or constrain moving subjects. WorldCrafter instead adapts a representation pretrained for novel-view synthesis, preserving both appearance and spatial relationships. [63], [79], [89].
The stored archive grows as video chunks are generated, but each memory query processes a bounded selection of nine historical latent frames. Recent context maintains local motion, while memory supports revisits. This is bounded computation over a growing archive, rather than an unlimited fixed-size persistent world state.
2 Related Work
2.1 Interactive Video World Models
Streaming generation and camera conditioning make video models interactive, but neither alone provides reliable long-term recall. Prior work varies diffusion noise across time, distills autoregressive generation, or encodes discrete actions, camera parameters, or rendered geometry. WorldCrafter combines these capabilities with a separate learned historical representation. [10], [20], [28], [39], [57], [62].
2.2 Memory Mechanisms in Video World Models
Context memories reuse frames or attention features. Spatial memories reconstruct or warp observations to the requested view. Implicit memories compress history into learned features. WorldCrafter uses LagerNVS, whose novel-view reconstruction training captures appearance as well as geometry, then jointly adapts the encoder, readout, and video generator. Its memory is queried before denoising rather than retrieved inside each denoising layer. [38], [53], [63], [68], [74], [91].
3 Method
3.1 Preliminary
A VAE maps video into compact spatiotemporal latents. A DiT learns the velocity field that transforms noise into video through conditional flow matching. Autoregressive generation produces one chunk at a time, conditions on a short clean history and text, then appends the result to the archive. The original latent-flow equations are retained below. [35], [44], [52], [64].
Equations 1–3
| (1) |
| (2) |
| (3) |
3.2 Model Architecture
Each denoising step attends jointly to memory tokens, recent clean video, and the noisy current chunk. A separate camera-attention branch encodes relative camera geometry through PRoPE and UCPE, applied only to the current noisy tokens. The first chunk has no historical memory; later chunks use memory derived from generated observations. [39], [97].
Equation 4
| (4) |
3.3 Memory Encoder
The memory encoder starts from LagerNVS weights, replaces the image frontend with latent patch embeddings, and incorporates camera poses relative to the latest history frame. It aggregates appearance and geometry without constructing explicit depth maps or a 3D scene. [63].
To bound encoder cost, retrieval always retains the latest frame and greedily selects eight other frames that jointly cover the upcoming camera trajectory. The readout queries the encoded history using target camera poses and produces as many tokens as four uncompressed latent frames. Pose-guided queries outperform a pose-free readout with the same token budget. Original qualitative comparisons and result tables follow.
Equations 5–7
| (5) |
| (6) |
| (7) |

| Method | MEt3R | LPIPS | PSNR | SSIM |
|---|---|---|---|---|
| DreamX-World [16] | 0.548 | 0.627 | 12.898 | 0.243 |
| Alaya-EVOKE [88] | 0.414 | 0.565 | 12.332 | 0.290 |
| HY-WorldPlay [62] | 0.394 | 0.515 | 12.983 | 0.252 |
| Lyra 2.0 [60] | 0.334 | 0.487 | 14.050 | 0.390 |
| Echo-WM [100] | 0.449 | 0.582 | 12.592 | 0.239 |
| LingBot-World 2 [18] | 0.492 | 0.633 | 10.449 | 0.219 |
| Matrix-Game 3.5 [55] | 0.405 | 0.549 | 12.976 | 0.224 |
| SANA-WM [104] | 0.397 | 0.553 | 13.142 | 0.246 |
| WorldCrafter | 0.166 | 0.255 | 18.016 | 0.517 |
| WorldCrafter-fast | 0.129 | 0.186 | 20.868 | 0.616 |
| Method | RotErr | TransErr | CamMC |
|---|---|---|---|
| DreamX-World [16] | 54.116 | 2.759 | 3.138 |
| Alaya-EVOKE [88] | 26.042 | 2.042 | 2.199 |
| HY-WorldPlay [62] | 34.051 | 2.146 | 2.359 |
| Lyra 2.0 [60] | 16.145 | 1.538 | 1.624 |
| Echo-WM [100] | 21.455 | 2.072 | 2.189 |
| LingBot-World 2 [18] | 30.615 | 2.004 | 2.192 |
| Matrix-Game 3.5 [55] | 19.881 | 1.920 | 2.028 |
| SANA-WM [104] | 23.531 | 1.740 | 1.887 |
| WorldCrafter | 13.536 | 1.475 | 1.546 |
| WorldCrafter-fast | 18.251 | 1.638 | 1.737 |

| Method | SC | BC | TF | MS | AQ | IQ | DD | OC | Overall |
|---|---|---|---|---|---|---|---|---|---|
| DreamX-World [16] | 75.994 | 87.601 | 95.327 | 98.100 | 57.593 | 60.412 | 99.806 | 24.583 | 77.652 |
| Alaya-EVOKE [88] | 80.522 | 88.994 | 95.872 | 98.481 | 61.579 | 71.249 | 99.029 | 25.015 | 81.406 |
| HY-WorldPlay [62] | 75.407 | 86.629 | 95.553 | 97.626 | 53.454 | 61.700 | 97.087 | 21.182 | 76.539 |
| Lyra 2.0 [60] | 77.277 | 87.935 | 95.167 | 98.652 | 57.832 | 65.149 | 99.806 | 21.484 | 78.943 |
| Echo-WM [100] | 81.671 | 90.206 | 95.771 | 96.751 | 61.818 | 66.564 | 98.835 | 26.082 | 80.211 |
| LingBot-World 2 [18] | 75.584 | 87.725 | 94.206 | 96.657 | 60.815 | 68.776 | 100.000 | 25.605 | 78.174 |
| Matrix-Game 3.5 [55] | 70.370 | 83.626 | 94.395 | 98.087 | 58.615 | 69.374 | 100.000 | 18.813 | 76.967 |
| SANA-WM [104] | 79.413 | 89.625 | 95.768 | 98.608 | 63.467 | 66.080 | 98.835 | 25.234 | 80.841 |
| WorldCrafter | 82.695 | 90.589 | 96.217 | 98.787 | 61.432 | 69.058 | 96.893 | 26.745 | 81.910 |
| WorldCrafter-fast | 81.365 | 89.559 | 96.444 | 98.736 | 58.650 | 63.982 | 96.505 | 24.306 | 80.285 |
| Memory evaluation | Camera evaluation | ||||||
|---|---|---|---|---|---|---|---|
| Method | MEt3R | LPIPS | PSNR | SSIM | RotErr | TransErr | CamMC |
| Context memory | 0.382 | 0.497 | 13.907 | 0.315 | 26.522 | 2.083 | 2.150 |
| Frozen memory encoder | 0.227 | 0.305 | 16.873 | 0.472 | 15.428 | 1.793 | 1.886 |
| Pose-free memory readout | 0.251 | 0.333 | 16.486 | 0.467 | 18.307 | 1.701 | 1.828 |
| Similarity-based history retrieval | 0.213 | 0.296 | 17.125 | 0.485 | 14.657 | 1.579 | 1.664 |
| WorldCrafter | 0.166 | 0.255 | 18.016 | 0.517 | 13.536 | 1.475 | 1.546 |
3.4 Base Model Training
Training combines Open-Sora-Plan videos, DL3DV scenes, and synthetic MIND videos; Depth Anything 3 supplies camera estimates and Qwen2.5-VL supplies captions. A filtered subset emphasizes cameras following moving subjects. [7], [41], [42], [43], [83].
Four stages adapt the Helios-base context window, train camera attention with the backbone frozen, warm up the latent memory encoder, and jointly optimize the full system. The final stage uses 8,000 iterations on real scenes followed by 1,000 with synthetic motion data. The encoder sees nine latent frames and the generator receives a four-frame-equivalent memory budget; reported training stages use 16–32 GPUs. [96].
3.5 Distillation for Real-time Interaction
Distribution matching distillation reduces sampling to two denoising steps at each of three resolutions. A high-noise model trained with synthetic motion data handles the early steps; a low-noise model trained without synthetic data performs the final step to reduce texture smearing. WorldCrafter-fast reaches 16 frames per second on a machine with four GPUs; this is not a single-device latency result. [85], [86], [96].
4 Experiments
4.1 Experimental Setup
The benchmark contains 145 initial images: 83 dynamic scenes and 62 static scenes. Five camera trajectories per image yield 725 videos per method, spanning 528–1,648 frames and including closed-loop revisits. All models receive the same initial images, descriptions, and target trajectories; outputs are resized to 640 × 384 for evaluation. Baselines include both full-step and distilled configurations, so results are not an equal-compute comparison. [16], [18], [55], [60], [62], [88], [100], [104].
4.2 Memory Evaluation
Revisit metrics compare a later return with the first generated observation at the same location. WorldCrafter reduces LPIPS from Lyra 2.0’s 0.487 to 0.255, a 47.6% relative reduction, while PSNR increases from 14.050 to 18.016 dB. The distilled model scores still better on all four revisit metrics. These measures assess self-consistency of generated views, rather than agreement with an independently observed physical scene. [3], [60], [70], [99].
4.3 Camera Control Evaluation
Camera trajectories are estimated from every fourth generated frame using VGGT-Ω and aligned to the targets with a similarity transform. The base model has the lowest rotation, translation, and pose-matrix errors; the distilled model ranks third. Because trajectories are estimated and aligned, the scores reflect that evaluation pipeline rather than direct measurements of a known 3D world. [66], [69].
4.4 Visual Quality Evaluation
On VBench, WorldCrafter obtains the highest aggregate visual-quality score, 81.910, among the tested methods. WorldCrafter-fast scores 80.285 overall and has the best temporal-flickering metric. The pair leads five of eight component measures, but neither leads every aspect of appearance or dynamics. [29].
4.5 Ablation Study
Matched ablations support learned memory over four retrieved context frames, joint encoder–generator training over a frozen encoder, target-pose readout over pose-free readout, and complementary-view selection over independent similarity ranking. Revisit error grows more slowly with longer revisit intervals than for context memory. The full model achieves LPIPS 0.255 versus 0.497 with context memory.
The reported memory-processing comparison is 0.062 seconds per chunk for encoding and readout versus 1.346 seconds for depth estimation, alignment, and warping: a 21.7-fold component speedup. This excludes shared video decoding and denoising, so it is not an end-to-end generation speedup.
5 Discussion
The learned memory improves recall within the tested trajectories, but particularly complex or extended camera paths can still break consistency. Re-encoding selected history for every chunk adds latency; incremental memory updates remain future work. The study evaluates camera-directed scene generation, not general physical interaction or unrestricted world simulation. Project and demonstrations.