Literature
Display
← All papers

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

Wangbo Yu · Kunhao Liu · Wenbo Hu · Shenghai Yuan · Chaoran Feng · Haiyang Zhou · Yukun Huang · Yiran Wang · Wang Zhao · Yingmin Luo · Ying Shan

· arXiv (preprint) ↗

World models5 min text · CondensedOriginal paper ↗
Contents

Overview

[Uncaptioned image]
Figure 1: WorldCrafter enables consistent, camera-controlled exploration. It preserves scene appearance and structure across revisits in static (a) and dynamic (b) scenes, as illustrated by reconstructed point clouds. Subjects remain consistent after leaving and re-entering view (c). Users can also generate and explore scenes from text descriptions alone (d).

At a glance

WorldCrafter is a camera-controlled video generator that recalls previously seen scene content through learned, view-dependent memory. A multi-view encoder compresses selected historical frames, and a camera-conditioned readout supplies a fixed number of tokens to the generator. On an author-curated benchmark of 725 videos, the base model improves revisit consistency, camera tracking, and aggregate visual quality over eight baselines. Its distilled variant generates 16 frames per second on four GPUs, with a camera-control and visual-quality tradeoff.

1 Introduction

A recent-frame window cannot reliably remember a scene after the camera leaves it. Full-history attention is expensive; retrieving a few similar views can miss complementary evidence; explicit depth-based reconstruction can introduce geometric errors or constrain moving subjects. WorldCrafter instead adapts a representation pretrained for novel-view synthesis, preserving both appearance and spatial relationships. [63], [79], [89].

The stored archive grows as video chunks are generated, but each memory query processes a bounded selection of nine historical latent frames. Recent context maintains local motion, while memory supports revisits. This is bounded computation over a growing archive, rather than an unlimited fixed-size persistent world state.

Refer to caption
Figure 2: Overview of the WorldCrafter pipeline. The video DiT generates the first chunk conditioned on camera poses, before any history is available. At subsequent rollout steps, max-coverage history retrieval selects latent frames based on the current camera poses, and the memory encoder maps them into a compact 3D-aware representation. A pose-conditioned memory readout module extracts fixed-size memory tokens that, together with recent history, guide generation of the current chunk. Generated frames are added to the history for subsequent rollouts.

2 Related Work

2.1 Interactive Video World Models

Streaming generation and camera conditioning make video models interactive, but neither alone provides reliable long-term recall. Prior work varies diffusion noise across time, distills autoregressive generation, or encodes discrete actions, camera parameters, or rendered geometry. WorldCrafter combines these capabilities with a separate learned historical representation. [10], [20], [28], [39], [57], [62].

2.2 Memory Mechanisms in Video World Models

Context memories reuse frames or attention features. Spatial memories reconstruct or warp observations to the requested view. Implicit memories compress history into learned features. WorldCrafter uses LagerNVS, whose novel-view reconstruction training captures appearance as well as geometry, then jointly adapts the encoder, readout, and video generator. Its memory is queried before denoising rather than retrieved inside each denoising layer. [38], [53], [63], [68], [74], [91].

3 Method

3.1 Preliminary

A VAE maps video into compact spatiotemporal latents. A DiT learns the velocity field that transforms noise into video through conditional flow matching. Autoregressive generation produces one chunk at a time, conditions on a short clean history and text, then appends the result to the archive. The original latent-flow equations are retained below. [35], [44], [52], [64].

Equations 1–3
𝐳t=(1−t)​𝐳+t​ϵ,𝐯t∗=ϵ−𝐳.\mathbf{z}_{t}=(1-t)\mathbf{z}+t\boldsymbol{\epsilon},\qquad\mathbf{v}^{*}_{t}=\boldsymbol{\epsilon}-\mathbf{z}. (1)
ℒFM=𝔼𝐳,t,ϵ​[‖𝐯θ​(𝐳t,t∣𝐲)−𝐯t∗‖22].\mathcal{L}_{\mathrm{FM}}=\mathbb{E}_{\mathbf{z},t,\boldsymbol{\epsilon}}\left[\left\|\mathbf{v}_{\theta}(\mathbf{z}_{t},t\mid\mathbf{y})-\mathbf{v}^{*}_{t}\right\|_{2}^{2}\right]. (2)
d​𝐳td​t=𝐯θ(𝐳t,t∣𝐳r,𝐲).\frac{\mathrm{d}\mathbf{z}_{t}}{\mathrm{d}t}=\mathbf{v}_{\theta}\!\left(\mathbf{z}_{t},t\mid\mathbf{z}^{\mathrm{r}},\mathbf{y}\right). (3)

3.2 Model Architecture

Each denoising step attends jointly to memory tokens, recent clean video, and the noisy current chunk. A separate camera-attention branch encodes relative camera geometry through PRoPE and UCPE, applied only to the current noisy tokens. The first chunk has no historical memory; later chunks use memory derived from generated observations. [39], [97].

Equation 4
d​𝐳td​t=𝐯θ(𝐳t,t∣𝐌,𝐳r,𝐂,𝐲).\frac{\mathrm{d}\mathbf{z}_{t}}{\mathrm{d}t}=\mathbf{v}_{\theta}\!\left(\mathbf{z}_{t},t\mid\mathbf{M},\mathbf{z}^{\mathrm{r}},\mathbf{C},\mathbf{y}\right). (4)

3.3 Memory Encoder

The memory encoder starts from LagerNVS weights, replaces the image frontend with latent patch embeddings, and incorporates camera poses relative to the latest history frame. It aggregates appearance and geometry without constructing explicit depth maps or a 3D scene. [63].

To bound encoder cost, retrieval always retains the latest frame and greedily selects eight other frames that jointly cover the upcoming camera trajectory. The readout queries the encoded history using target camera poses and produces as many tokens as four uncompressed latent frames. Pose-guided queries outperform a pose-free readout with the same token budget. Original qualitative comparisons and result tables follow.

Equations 5–7
𝐑=Φ⁡(𝐳h,𝐂h)∈ℝ|𝐳h|​L×d,\mathbf{R}=\Phi(\mathbf{z}^{\mathrm{h}},\mathbf{C}^{\mathrm{h}})\in\mathbb{R}^{|\mathbf{z}^{\mathrm{h}}|L\times d}, (5)
𝐌=Readout⁡(Φ⁡(𝐳s,𝐂s)).\mathbf{M}=\operatorname{Readout}\!\left(\Phi(\mathbf{z}^{\mathrm{s}},\mathbf{C}^{\mathrm{s}})\right). (6)
𝐌=Readout⁡(Φ⁡(𝐳s,𝐂s),𝐂q).\mathbf{M}=\operatorname{Readout}\!\left(\Phi(\mathbf{z}^{\mathrm{s}},\mathbf{C}^{\mathrm{s}}),\mathbf{C}^{\mathrm{q}}\right). (7)
Refer to caption
Figure 3: Qualitative long-horizon revisit comparison in a static scene. Rows show the first observation, an intermediate view during exploration, the matched revisit, and the point cloud reconstruction of the generated video, respectively.
Table 1: Long-horizon revisit consistency. Results are averaged over 725 generated videos using matched first-visit and revisit frames. Progressively lighter shades of blue mark the best, second-best, and third-best results, respectively.
Method MEt3R↓\downarrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow
DreamX-World [16] 0.548 0.627 12.898 0.243
Alaya-EVOKE [88] 0.414 0.565 12.332 0.290
HY-WorldPlay [62] 0.394 0.515 12.983 0.252
Lyra 2.0 [60] 0.334 0.487 14.050 0.390
Echo-WM [100] 0.449 0.582 12.592 0.239
LingBot-World 2 [18] 0.492 0.633 10.449 0.219
Matrix-Game 3.5 [55] 0.405 0.549 12.976 0.224
SANA-WM [104] 0.397 0.553 13.142 0.246
WorldCrafter 0.166 0.255 18.016 0.517
WorldCrafter-fast 0.129 0.186 20.868 0.616
Table 2: Camera-control accuracy. Results are averaged over 725 generated videos using Sim(3)-aligned estimated and target trajectories. Progressively lighter shades of blue mark the best, second-best, and third-best results, respectively.
Method RotErr↓\downarrow TransErr↓\downarrow CamMC↓\downarrow
DreamX-World [16] 54.116 2.759 3.138
Alaya-EVOKE [88] 26.042 2.042 2.199
HY-WorldPlay [62] 34.051 2.146 2.359
Lyra 2.0 [60] 16.145 1.538 1.624
Echo-WM [100] 21.455 2.072 2.189
LingBot-World 2 [18] 30.615 2.004 2.192
Matrix-Game 3.5 [55] 19.881 1.920 2.028
SANA-WM [104] 23.531 1.740 1.887
WorldCrafter 13.536 1.475 1.546
WorldCrafter-fast 18.251 1.638 1.737
Refer to caption
Figure 4: Qualitative long-horizon revisit comparison in a dynamic scene. Rows show the first observation, an intermediate view during exploration, the matched revisit, and the point cloud reconstruction of the generated video, respectively.
Table 3: Visual quality on VBench. Scores are computed in custom-input mode over 725 generated videos. SC, BC, TF, MS, AQ, IQ, DD, and OC denote subject consistency, background consistency, temporal flickering, motion smoothness, aesthetic quality, imaging quality, dynamic degree, and overall consistency, respectively. The Overall column reports the aggregate VBench score. Progressively lighter shades of blue mark the best, second-best, and third-best results, respectively.
Method SC BC TF MS AQ IQ DD OC Overall
DreamX-World [16] 75.994 87.601 95.327 98.100 57.593 60.412 99.806 24.583 77.652
Alaya-EVOKE [88] 80.522 88.994 95.872 98.481 61.579 71.249 99.029 25.015 81.406
HY-WorldPlay [62] 75.407 86.629 95.553 97.626 53.454 61.700 97.087 21.182 76.539
Lyra 2.0 [60] 77.277 87.935 95.167 98.652 57.832 65.149 99.806 21.484 78.943
Echo-WM [100] 81.671 90.206 95.771 96.751 61.818 66.564 98.835 26.082 80.211
LingBot-World 2 [18] 75.584 87.725 94.206 96.657 60.815 68.776 100.000 25.605 78.174
Matrix-Game 3.5 [55] 70.370 83.626 94.395 98.087 58.615 69.374 100.000 18.813 76.967
SANA-WM [104] 79.413 89.625 95.768 98.608 63.467 66.080 98.835 25.234 80.841
WorldCrafter 82.695 90.589 96.217 98.787 61.432 69.058 96.893 26.745 81.910
WorldCrafter-fast 81.365 89.559 96.444 98.736 58.650 63.982 96.505 24.306 80.285
Table 4: Ablation of the memory design. Each variant changes one design choice while keeping the training and inference settings and the DiT memory-token budget fixed.
Memory evaluation Camera evaluation
Method MEt3R↓\downarrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow RotErr↓\downarrow TransErr↓\downarrow CamMC↓\downarrow
Context memory 0.382 0.497 13.907 0.315 26.522 2.083 2.150
Frozen memory encoder 0.227 0.305 16.873 0.472 15.428 1.793 1.886
Pose-free memory readout 0.251 0.333 16.486 0.467 18.307 1.701 1.828
Similarity-based history retrieval 0.213 0.296 17.125 0.485 14.657 1.579 1.664
WorldCrafter 0.166 0.255 18.016 0.517 13.536 1.475 1.546
Original figure
Figure 5: Ablations of implicit 3D-aware memory, joint optimization, and memory-processing efficiency. (a) Long-horizon revisit consistency of context memory and WorldCrafter’s implicit 3D-aware memory, measured by LPIPS across revisit intervals. (b) Validation LPIPS for joint optimization and frozen memory-encoder training at matched iterations. (c) Per-chunk memory-processing latency of depth-based spatial memory and WorldCrafter.

3.4 Base Model Training

Training combines Open-Sora-Plan videos, DL3DV scenes, and synthetic MIND videos; Depth Anything 3 supplies camera estimates and Qwen2.5-VL supplies captions. A filtered subset emphasizes cameras following moving subjects. [7], [41], [42], [43], [83].

Four stages adapt the Helios-base context window, train camera attention with the backbone frozen, warm up the latent memory encoder, and jointly optimize the full system. The final stage uses 8,000 iterations on real scenes followed by 1,000 with synthetic motion data. The encoder sees nine latent frames and the generator receives a four-frame-equivalent memory budget; reported training stages use 16–32 GPUs. [96].

3.5 Distillation for Real-time Interaction

Distribution matching distillation reduces sampling to two denoising steps at each of three resolutions. A high-noise model trained with synthetic motion data handles the early steps; a low-noise model trained without synthetic data performs the final step to reduce texture smearing. WorldCrafter-fast reaches 16 frames per second on a machine with four GPUs; this is not a single-device latency result. [85], [86], [96].

4 Experiments

4.1 Experimental Setup

The benchmark contains 145 initial images: 83 dynamic scenes and 62 static scenes. Five camera trajectories per image yield 725 videos per method, spanning 528–1,648 frames and including closed-loop revisits. All models receive the same initial images, descriptions, and target trajectories; outputs are resized to 640 × 384 for evaluation. Baselines include both full-step and distilled configurations, so results are not an equal-compute comparison. [16], [18], [55], [60], [62], [88], [100], [104].

4.2 Memory Evaluation

Revisit metrics compare a later return with the first generated observation at the same location. WorldCrafter reduces LPIPS from Lyra 2.0’s 0.487 to 0.255, a 47.6% relative reduction, while PSNR increases from 14.050 to 18.016 dB. The distilled model scores still better on all four revisit metrics. These measures assess self-consistency of generated views, rather than agreement with an independently observed physical scene. [3], [60], [70], [99].

4.3 Camera Control Evaluation

Camera trajectories are estimated from every fourth generated frame using VGGT-Ω and aligned to the targets with a similarity transform. The base model has the lowest rotation, translation, and pose-matrix errors; the distilled model ranks third. Because trajectories are estimated and aligned, the scores reflect that evaluation pipeline rather than direct measurements of a known 3D world. [66], [69].

4.4 Visual Quality Evaluation

On VBench, WorldCrafter obtains the highest aggregate visual-quality score, 81.910, among the tested methods. WorldCrafter-fast scores 80.285 overall and has the best temporal-flickering metric. The pair leads five of eight component measures, but neither leads every aspect of appearance or dynamics. [29].

4.5 Ablation Study

Matched ablations support learned memory over four retrieved context frames, joint encoder–generator training over a frozen encoder, target-pose readout over pose-free readout, and complementary-view selection over independent similarity ranking. Revisit error grows more slowly with longer revisit intervals than for context memory. The full model achieves LPIPS 0.255 versus 0.497 with context memory.

The reported memory-processing comparison is 0.062 seconds per chunk for encoding and readout versus 1.346 seconds for depth estimation, alignment, and warping: a 21.7-fold component speedup. This excludes shared video decoding and denoising, so it is not an end-to-end generation speedup.

5 Discussion

The learned memory improves recall within the tested trajectories, but particularly complex or extended camera paths can still break consistency. Re-encoding selected history for every chunk adds latency; incremental memory updates remain future work. The study evaluates camera-directed scene generation, not general physical interaction or unrestricted world simulation. Project and demonstrations.

Concept demonstration

Open image ↗