Literature
Display
← All papers

GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

Jiahao Lu · Minghao Yin · Wenbo Hu · Hengyu Liu · Wang Zhao · Sai-Kit Yeung · Ying Shan · Yuan Liu

· arXiv (preprint) ↗

World models10 min text · CondensedOriginal paper ↗
Contents

Overview

GAE teaser showing generated RGB sequences, corresponding depth maps, and point clouds for indoor and outdoor scenes.
Figure 1: Geometry-native world generation. Given a reference image, a camera trajectory, and optional text, GAE jointly generates RGB views and 3D geometry from a shared, compact latent space. Each example pairs generated RGB and depth sequences with the resulting 3D point cloud.

At a glance

GAE compresses features from a frozen geometry model into one latent state that decodes to both RGB images and geometry. Under a shared generation protocol, its 64-channel version reduces FVD by 12.7% on RealEstate10K and 23.1% on DL3DV against the best competing latent. Independently recovered camera trajectories also improve. The controlled tests use nine-view clips; longer and text-conditioned examples come from a separately trained model. Project · Code.

1 Introduction

A video can look plausible while its scene geometry changes between views. GAE makes geometry part of the generated representation: it compresses all four DA3 feature levels, reconstructs them for the original frozen geometry head, and adds an RGB decoder. Semantic and spatial supervision organize the compact state for flow matching. This studies the choice of latent representation, with generator architecture and training held largely fixed. [18], [41], [12], [33].

2 Related Work

Generative representation spaces.

Pixel VAEs compress appearance; RAEs reuse pretrained semantic features. Reconstruction quality alone does not establish how easily a latent can be generated. GAE adds native geometry decoding while learning a compact representation. [29], [57], [33], [58].

Geometry-aware visual generation.

Earlier methods introduce geometry through explicit scenes, camera controls, joint output modalities, or additional training objectives. GAE instead generates a single state derived from a geometry foundation model. [25], [13], [7], [10], [1].

Geometry foundation models as generative states.

GLD generates a cascade of geometry features; latent Riemannian flow matching evolves four feature levels jointly. GAE learns a compact Euclidean bottleneck and reconstructs the hierarchy afterward, so one standard flow model can generate it. [12], [47].

3 Method

Training has two stages: learn a codec and RGB decoder around a frozen geometry backbone, then freeze the codec and train a conditional flow over its standardized posterior means. At sampling time, only requested output latents evolve; reference images, cameras, and text remain fixed conditions.

Two-stage GAE pipeline: compress frozen geometry features, then generate compact latents with conditional flow matching.
Figure 2: Overview of Geometry-Native Autoencoder (GAE). Stage 1 (codec training) compresses frozen multi-level geometry foundation features into a compact, diffusable geometry latent; the feature decoder rebuilds the full hierarchy for frozen geometry readout, while a separate learned head decodes RGB. Stage 2 (flow training) performs conditional flow matching only over generated-view latents, while clean reference latents, camera rays, and text act as controls.
Equations 1–2
𝐈1:V→ℰ,𝒫𝐗1:V→Encϕ𝐳1:V→Decϕ𝐗^1:V⏟Stage 1: train the codec 𝒜ϕ,𝐳¯(1)1:V→Flowψ(⋅∣𝒞)𝐳¯^1:V→Std−1𝐳^1:V⏟Stage 2: train the flow model 𝐟ψ,\underbrace{\mathbf{I}_{1:V}\xrightarrow{\;\mathcal{E},\,\mathcal{P}\;}\mathbf{X}_{1:V}\xrightarrow{\;\mathrm{Enc}_{\phi}\;}\mathbf{z}_{1:V}\xrightarrow{\;\mathrm{Dec}_{\phi}\;}\hat{\mathbf{X}}_{1:V}}_{\text{Stage 1: train the codec }\mathcal{A}_{\phi}},\qquad\underbrace{\bar{\mathbf{z}}^{(1)}_{1:V}\xrightarrow{\;\mathrm{Flow}_{\psi}(\cdot\mid\mathcal{C})\;}\hat{\bar{\mathbf{z}}}_{1:V}\xrightarrow{\;\mathrm{Std}^{-1}\;}\hat{\mathbf{z}}_{1:V}}_{\text{Stage 2: train the flow model }\mathbf{f}_{\psi}}, (1)
𝐈^1:V=𝒟θrgb(𝐳^1:V),𝐆^1:V=ℋDPT(Decϕ(𝐳^1:V)).\hat{\mathbf{I}}_{1:V}=\mathcal{D}_{\theta}^{\mathrm{rgb}}(\hat{\mathbf{z}}_{1:V}),\qquad\hat{\mathbf{G}}_{1:V}=\mathcal{H}_{\mathrm{DPT}}\!\left(\mathrm{Dec}_{\phi}(\hat{\mathbf{z}}_{1:V})\right). (2)

3.1 Geometry-native codec

Fusing all four DA3 levels retains complementary geometry cues while avoiding direct generation of the full high-dimensional hierarchy.

From a feature hierarchy to one state.

Normalize each feature level with fixed training-set channel statistics, reshape tokens to spatial grids, and concatenate channels. Shallow features retain detail and correspondence; deeper features supply complementary inputs expected by the geometry decoder.

Equation 3
𝐗v=𝒫(𝐅v0:3):=∥ℓ=03reshape(𝐟v,ℓ−𝐦ℓ𝐬ℓ+ϵ)∈ℝCx×hp×wp,Cx=∑ℓ=03Cℓ,\mathbf{X}_{v}=\mathcal{P}(\mathbf{F}_{v}^{0:3}):=\big\|_{\ell=0}^{3}\mathrm{reshape}\!\left(\frac{\mathbf{f}_{v,\ell}-\mathbf{m}_{\ell}}{\mathbf{s}_{\ell}+\epsilon}\right)\in\mathbb{R}^{C_{x}\times h_{p}\times w_{p}},\qquad C_{x}=\sum_{\ell=0}^{3}C_{\ell}, (3)

Compact geometry codec.

A convolutional codec with spatial attention preserves the patch grid and compresses channels to 64 or 128, versus 3,072 for one raw DA3 level. Codec training samples a lightly KL-regularized posterior; generation uses its deterministic mean.

Equations 4–6
(𝝁v,log⁡𝝈v2)\displaystyle(\bm{\mu}_{v},\log\bm{\sigma}_{v}^{2}) =Encϕ​(𝐗v),\displaystyle=\mathrm{Enc}_{\phi}(\mathbf{X}_{v}), (4)
𝐳v\displaystyle\mathbf{z}_{v} =𝝁v+ϵv⊙exp⁡(12​log⁡𝝈v2),ϵv∼𝒩⁡(𝟎,𝐈),\displaystyle=\bm{\mu}_{v}+\bm{\epsilon}_{v}\odot\exp\!\left(\tfrac{1}{2}\log\bm{\sigma}_{v}^{2}\right),\qquad\bm{\epsilon}_{v}\sim\mathcal{N}(\mathbf{0},\mathbf{I}), (5)
𝐅^v0:3\displaystyle\hat{\mathbf{F}}_{v}^{0:3} =Decϕ​(𝐳v).\displaystyle=\mathrm{Dec}_{\phi}(\mathbf{z}_{v}). (6)

Joint RGB and geometry decoding.

The learned RGB head reads the bottleneck directly. A second decoder reconstructs all four feature levels for the frozen DPT geometry head, producing depth, rays, and point maps. Freezing that head requires retained information to remain readable through DA3’s original interface.

Equation 7
𝐈^v=𝒟θrgb(𝐳v),𝐆^v=ℋDPT(𝐅^v0:3)=ℋDPT(Decϕ(𝐳v)),\hat{\mathbf{I}}_{v}=\mathcal{D}^{\mathrm{rgb}}_{\theta}(\mathbf{z}_{v}),\qquad\hat{\mathbf{G}}_{v}=\mathcal{H}_{\mathrm{DPT}}(\hat{\mathbf{F}}_{v}^{0:3})=\mathcal{H}_{\mathrm{DPT}}\!\left(\mathrm{Dec}_{\phi}(\mathbf{z}_{v})\right), (7)

Codec objective.

Training combines feature reconstruction, weak KL regularization, RGB pixel/perceptual losses, geometry losses, and representation shaping. Geometry supervision consists of DA3 pseudo-targets from original features, rather than measured depth ground truth.

Equation 8
ℒcodec=ℒfeat+λkl​ℒkl+ℒrgb+ℒgeo+ℒrepr.\mathcal{L}_{\mathrm{codec}}=\mathcal{L}_{\mathrm{feat}}+\lambda_{\mathrm{kl}}\mathcal{L}_{\mathrm{kl}}+\mathcal{L}_{\mathrm{rgb}}+\mathcal{L}_{\mathrm{geo}}+\mathcal{L}_{\mathrm{repr}}. (8)

3.2 Organizing the latent for generation

Token-wise alignment with C-RADIO improves semantic neighborhoods and transport diagnostics, but damages pairwise spatial structure. Matching posterior-token similarities to DINOv2 restores that structure. The representation weight is 0.25 and the relational multiplier is 8. After training, discard the teachers and projector, freeze the codec, and standardize latent channels. [8], [26], [54].

Table 1: Three-step latent shaping on DA3-GIANT. Fusion and reconstruction repair the conditioning of the raw hierarchy; token alignment improves transport and semantic organization but collapses relational structure; ℒstruct\mathcal{L}_{\mathrm{struct}} restores spatial and cross-view structure while retaining gains.
Diffusability (single-view) Diffusability (multi-view) Structure (single-view) Structure (cross-view)
Representation objective ρ↓\rho\downarrow κ↓\kappa\downarrow effrank↑\uparrow ρ↓\rho\downarrow κ↓\kappa\downarrow effrank↑\uparrow LNC@5↑\uparrow LDS↑\uparrow SRSS↑\uparrow xLNC∗↑{}^{*}\uparrow
Raw geometry (DA3-GIANT L0) 0.876 2.8×1082.8{\times}10^{8} 11.3 0.793 3.9×1083.9{\times}10^{8} 9.3 0.353 0.102 0.123 0.642
Raw geometry (DA3-GIANT L3) 0.769 6.6×10166.6{\times}10^{16} 11.5 0.539 2.0×10112.0{\times}10^{11} 9.0 0.054 0.054 0.070 0.380
None 0.820 130 58.0 0.630 580 20.0 0.310 0.240 0.300 0.590
ℒtok\mathcal{L}_{\mathrm{tok}} 0.700 170 53.0 0.570 450 26.0 0.490 0.020 0.030 0.580
ℒtok+ℒstruct\mathcal{L}_{\mathrm{tok}}+\mathcal{L}_{\mathrm{struct}} 0.674 227 51.3 0.564 356 45.2 0.498 0.444 0.553 0.591
Equations 9–12
ℒtok\displaystyle\mathcal{L}_{\mathrm{tok}} =1V​N​∑v,i[1−⟨gη​(𝝁v,i)^,𝐜^v,i⟩],\displaystyle=\frac{1}{VN}\sum_{v,i}\left[1-\left\langle\widehat{g_{\eta}(\bm{\mu}_{v,i})},\hat{\mathbf{c}}_{v,i}\right\rangle\right], (9)
ℒstruct\displaystyle\mathcal{L}_{\mathrm{struct}} =1V​N​(N−1)​∑v∑i≠j(⟨𝝁^v,i,𝝁^v,j⟩−⟨𝐝^v,i,𝐝^v,j⟩)2,\displaystyle=\frac{1}{VN(N-1)}\sum_{v}\sum_{i\neq j}\left(\left\langle\hat{\bm{\mu}}_{v,i},\hat{\bm{\mu}}_{v,j}\right\rangle-\left\langle\hat{\mathbf{d}}_{v,i},\hat{\mathbf{d}}_{v,j}\right\rangle\right)^{2}, (10)
ℒrepr\displaystyle\mathcal{L}_{\mathrm{repr}} =λrepa​(ℒtok+λμ​ℒstruct),λrepa=0.25,λμ=8.0.\displaystyle=\lambda_{\mathrm{repa}}\left(\mathcal{L}_{\mathrm{tok}}+\lambda_{\mu}\mathcal{L}_{\mathrm{struct}}\right),\qquad\lambda_{\mathrm{repa}}=0.25,\quad\lambda_{\mu}=8.0. (11)
𝐳¯v=𝐳v−𝐦z𝐬z+ϵ.\bar{\mathbf{z}}_{v}=\frac{\mathbf{z}_{v}-\mathbf{m}_{z}}{\mathbf{s}_{z}+\epsilon}. (12)

3.3 Conditional flow matching and generation

Flow matching.

A transformer jointly predicts clean target-view latents along a linear data-to-noise path. Convert these predictions to velocity and minimize conditional flow-matching error; all views share a timestep. [33], [20].

Equation 13
ℒflow=𝔼⁡[ρ⁡(t)​‖𝐮^ψ​(𝐳¯t,t,𝒞)−(ϵ−𝐳¯gt)‖22].\mathcal{L}_{\mathrm{flow}}=\mathbb{E}\!\left[\rho(t)\left\|\hat{\mathbf{u}}_{\psi}(\bar{\mathbf{z}}_{t},t,\mathcal{C})-(\bm{\epsilon}-\bar{\mathbf{z}}^{\mathrm{gt}})\right\|_{2}^{2}\right]. (13)

Reference, camera, and text conditioning.

Reference features must be encoded using observed references alone, because joint encoding with unavailable target views would leak context. Prepend clean reference tokens at timestep zero, exclude them from the prediction head, and keep every output slot noisy. Metric Plücker rays modulate self-attention; text enters cross-attention. Condition dropout enables several generation modes. [52].

Training and inference.

Train the codec on single- and multi-view data, then train the frozen-codec flow with both text-to-image and view-conditioned batches. Sampling integrates Gaussian noise from time one to zero and decodes the resulting state into appearance and geometry.

4 Experiments

Experiments separate intrinsic latent properties, real-image reconstruction, and generation. Reconstruction fidelity alone is insufficient evidence for generation quality.

Experimental setup.

Seven controlled variants use pixel, semantic, raw geometry, or GAE latents with the same flow family, camera controls, training budget, and sampling. Evaluation uses held-out pools of 64 scenes from each dataset, nine 252 × 252 views, one reference, 50 Euler steps, and CFG scale 2. The higher-resolution final model is evaluated qualitatively. [61], [19].

Ranking convention.

Blue cells rank only controlled variants, with displayed ties resolved using unrounded values. Gray GLD and Gen3R rows are complete external pipelines and are excluded from that ranking.

4.1 Latent-space study

Raw DA3 levels have 3,072 channels but effective ranks near ten and severe covariance anisotropy. GAE improves conditioning and compresses to 64 or 128 channels while retaining useful geometry. It does not win every diagnostic: semantic RAEv2 has stronger semantic neighborhoods, and raw L0 has stronger cross-view retrieval. GAE-64 favors compactness and spatial structure; GAE-128 retains more cross-view information.

Radar chart comparing latent representation diagnostics, RGB generation quality, and 3D consistency across eight representations.
Figure 3: Latent representation quality and matched DiT generation. Normalized overview of representation (green), RGB generation (blue), and 3D consistency (orange).
Table 2: Latent-space diagnostics. GAE combines smooth transport and good conditioning with strong spatial and cross-view structure while retaining useful semantic organization.
Diffusability (single-view) Diffusability (multi-view) Structure (single-view) Structure (cross-view)
Latent C×H×WC{\times}H{\times}W ρ↓\rho\downarrow κ↓\kappa\downarrow effrank↑\uparrow ρ↓\rho\downarrow κ↓\kappa\downarrow effrank↑\uparrow LNC@5↑\uparrow LDS↑\uparrow SRSS↑\uparrow xLNC∗↑{}^{*}\uparrow
Pixel VAE, single-image (SD-VAE) 4×32×324{\times}32{\times}32 0.888 3.7 3.0 0.765 4.1 3.1 0.051 0.262 0.324 0.124
Pixel VAE, video (WAN2.1 VAE) 16×32×3216{\times}32{\times}32 0.847 127 5.3 0.609 160 4.3 0.084 0.234 0.313 0.402
Semantic RAE (RAEV2) 1024×16×161024{\times}16{\times}16 0.711 1.8×1041.8{\times}10^{4} 289.3 0.640 6.0×1046.0{\times}10^{4} 93.3 0.932 0.171 0.220 0.587
Raw geometry (DA3-GIANT L0) 3072×18×183072{\times}18{\times}18 0.876 2.8×1082.8{\times}10^{8} 11.3 0.793 3.9×1083.9{\times}10^{8} 9.3 0.353 0.102 0.123 0.642
Raw geometry (DA3-GIANT L3) 3072×18×183072{\times}18{\times}18 0.769 6.6×10166.6{\times}10^{16} 11.5 0.539 2.0×10112.0{\times}10^{11} 9.0 0.054 0.054 0.070 0.380
GLD (L0/L1 stage avg.) – 0.854 9.5×1079.5{\times}10^{7} 34.8 0.742 1.7×1081.7{\times}10^{8} 22.7 0.274 0.103 0.131 0.594
Ours (GAE-128) 128×18×18128{\times}18{\times}18 0.674 227 51.3 0.564 356 45.2 0.498 0.444 0.553 0.591
Ours (GAE-64) 64×18×1864{\times}18{\times}18 0.634 53.1 37.0 0.519 117 33.2 0.488 0.631 0.767 0.569

4.2 Codec reconstruction quality

4.2.1 RGB reconstruction

GAE-128 matches or slightly exceeds raw L0 RGB reconstruction with 24 times fewer channels: multi-view PSNR is 34.78 versus 34.70, and rFVD is 4.3 versus 4.4. GAE-64 loses some reconstruction fidelity. Single-view evaluation uses ImageNet; multi-view results average five datasets.

Table 3: RGB reconstruction. Single-view uses ImageNet; multi-view averages five video datasets.
Single-view Multi-view
Latent PSNR↑\uparrow LPIPS↓\downarrow rFID↓\downarrow PSNR↑\uparrow LPIPS↓\downarrow rFVD↓\downarrow
Pixel VAE, single-image (SD-VAE) 18.49 0.140 10.1 19.97 0.121 37.7
Pixel VAE, video (WAN2.1 VAE) 18.34 0.125 8.9 19.83 0.163 46.7
Semantic RAE (RAEV2) 22.08 0.134 8.8 24.73 0.094 32.1
Raw geometry (DA3-GIANT L0) 28.64 0.038 5.5 34.70 0.014 4.4
Raw geometry (DA3-GIANT L3) 19.90 0.219 25.3 23.98 0.122 76.2
Ours (GAE-128) 28.76 0.036 5.4 34.78 0.014 4.3
Ours (GAE-64) 27.30 0.061 7.1 32.37 0.040 9.1

4.2.2 Geometry reconstruction

Both GAE widths improve depth, point-map, and pose reconstruction over the raw-L0 baseline. Depth and point references come from Pi3 estimates on real frames; poses use dataset cameras. External Gen3R is slightly better on some depth measures. Pixel and semantic codecs lack the required native geometry decoder. [45].

Table 4: Geometry reconstruction from encoded latents. Depth and point maps are compared with Pi3 (Wang et al., 2026c) on the real target frames; camera poses are compared with the input real trajectory. Displayed ties between GAE-64 and GAE-128 are ordered using unrounded values.
RealEstate10K (Pi3 ref) DL3DV (Pi3 ref)
Method AbsRel↓\downarrow δ1↑\delta_{1}\uparrow Cham.↓\downarrow PMap↓\downarrow ATE↓\downarrow RPEr∘↓{}^{\circ}\downarrow AbsRel↓\downarrow δ1↑\delta_{1}\uparrow Cham.↓\downarrow PMap↓\downarrow ATE↓\downarrow RPEr∘↓{}^{\circ}\downarrow
Pi3 on real (ref) — — — — 0.009 0.17 — — — — 0.011 0.26
Gen3R 0.085 0.906 0.430 0.829 0.038 0.63 0.114 0.862 0.237 0.542 0.022 0.72
GLD 0.271 0.694 0.857 1.580 0.039 1.23 0.303 0.608 0.462 1.083 0.026 1.14
Raw geometry (DA3-GIANT L0) 0.178 0.759 0.635 1.160 0.068 1.33 0.234 0.639 0.575 1.130 0.057 1.09
Ours (GAE-128) 0.090 0.905 0.400 0.700 0.006 0.19 0.125 0.862 0.145 0.433 0.008 0.25
Ours (GAE-64) 0.090 0.905 0.386 0.691 0.007 0.20 0.125 0.861 0.145 0.433 0.008 0.27

4.3 Flow-matching generation quality

The same controlled flow recipe tests how latent choice affects generated RGB and geometry.

4.3.1 RGB generation

GAE-64 reduces FVD from the strongest non-GAE controlled baseline’s 258.6 to 225.7 on RealEstate10K and from 373.2 to 287.0 on DL3DV. It leads the paired image metrics; GAE-128 has the best RealEstate10K FID. Scene examples illustrate preserved appearance as the camera moves.

Table 5: Camera-conditioned appearance generation. All controlled latent variants use the matched protocol described above.
RealEstate10K DL3DV
Latent FVD↓\downarrow FID↓\downarrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow FVD↓\downarrow FID↓\downarrow LPIPS↓\downarrow PSNR↑\uparrow SSIM↑\uparrow
Pixel VAE, single-image (SD-VAE) 258.6 36.4 0.171 19.32 0.666 373.2 57.1 0.232 17.05 0.495
Pixel VAE, video (WAN2.1 VAE) 362.9 41.0 0.219 16.52 0.575 596.7 77.3 0.313 14.31 0.407
Semantic RAE (RAEV2) 379.4 31.4 0.191 17.60 0.585 453.4 49.2 0.231 16.01 0.434
Raw geometry (DA3-GIANT L0) 298.6 35.5 0.177 18.31 0.643 376.5 60.2 0.235 16.68 0.481
Raw geometry (DA3-GIANT L3) 488.9 50.0 0.273 16.82 0.579 584.8 80.6 0.314 16.22 0.447
Ours (GAE-128) 233.4 25.6 0.146 19.54 0.701 345.2 44.2 0.200 17.39 0.541
Ours (GAE-64) 225.7 27.1 0.143 20.02 0.711 287.0 41.3 0.194 18.00 0.553
GLD 445.1 44.0 0.337 14.91 0.518 587.4 66.4 0.356 14.51 0.406
Gen3R 269.7 25.7 0.196 17.79 0.621 580.5 48.3 0.270 15.62 0.440
Two RealEstate10K scenes comparing reference and target RGB views across nine methods.
Figure 4: Qualitative RGB comparison on RealEstate10K. Columns are methods and rows show one reference and two target views for each scene under identical camera conditioning. Insets report per-scene PSNR over the two displayed target views; Ours denotes the C=64C{=}64 model.

4.3.2 3D consistency

Independent VGGT reconstruction gives GAE-64 ATE of 0.0034 versus the best competing controlled latent’s 0.0072 on RealEstate10K, and 0.0056 versus 0.0073 on DL3DV. These correspond to 52.8% and 23.3% reductions after Sim(3) alignment. The two GAE widths also lead controlled MEt3R scores. DA3 results are a backbone-overlap cross-check; external Gen3R has a better RealEstate10K MEt3R score. [41], [2].

Table 6: Generated-view 3D consistency. VGGT is independent of all latent backbones; DA3-GIANT is reported as a backbone-overlap cross-check.
VGGT (independent) DA3-GIANT (cross-check)
Latent ATE↓\downarrow RPEt↓\downarrow RPEr∘↓{}^{\circ}\downarrow Reproj.↓\downarrow ATE↓\downarrow RPEt↓\downarrow RPEr∘↓{}^{\circ}\downarrow Reproj.↓\downarrow MEt3R↓\downarrow
RealEstate10K
Pixel VAE, single-image (SD-VAE) 0.0072 0.0182 0.648 0.0093 0.0046 0.0191 0.338 0.0018 0.1385
Pixel VAE, video (WAN2.1 VAE) 0.0225 0.0252 0.746 0.0105 0.0135 0.0176 0.751 0.0019 0.1441
Semantic RAE (RAEV2) 0.0085 0.0233 0.420 0.0104 0.0066 0.0219 0.507 0.0019 0.1263
Raw geometry (DA3-GIANT L0) 0.0085 0.0202 0.435 0.0096 0.0047 0.0167 0.358 0.0020 0.1420
Raw geometry (DA3-GIANT L3) 0.0112 0.0211 0.516 0.0124 0.0036 0.0148 0.355 0.0019 0.1575
Ours (GAE-128) 0.0041 0.0173 0.332 0.0075 0.0026 0.0143 0.304 0.0017 0.1213
Ours (GAE-64) 0.0034 0.0060 0.380 0.0068 0.0026 0.0141 0.225 0.0013 0.1208
GLD 0.0124 0.0258 0.886 0.0133 0.0088 0.0221 0.950 0.0038 0.1660
Gen3R 0.0090 0.0213 0.423 0.0072 0.0063 0.0211 0.387 0.0014 0.1157
DL3DV
Pixel VAE, single-image (SD-VAE) 0.0073 0.0098 0.432 0.0108 0.0055 0.0076 0.358 0.0024 0.1542
Pixel VAE, video (WAN2.1 VAE) 0.0243 0.0288 1.419 0.0258 0.0206 0.0251 1.016 0.0019 0.1777
Semantic RAE (RAEV2) 0.0104 0.0143 0.638 0.0130 0.0088 0.0113 0.585 0.0031 0.1549
Raw geometry (DA3-GIANT L0) 0.0110 0.0136 0.634 0.0129 0.0079 0.0102 0.482 0.0042 0.1721
Raw geometry (DA3-GIANT L3) 0.0097 0.0120 0.603 0.0151 0.0052 0.0073 0.380 0.0039 0.1754
Ours (GAE-128) 0.0056 0.0072 0.294 0.0080 0.0043 0.0053 0.260 0.0029 0.1347
Ours (GAE-64) 0.0056 0.0066 0.293 0.0077 0.0038 0.0053 0.254 0.0028 0.1370
GLD 0.0136 0.0164 1.242 0.0241 0.0104 0.0136 1.048 0.0087 0.2183
Gen3R 0.0096 0.0133 0.678 0.0143 0.0096 0.0131 0.608 0.0065 0.1662
Generated views, independent VGGT point clouds, and recovered camera paths for appearance, semantic, and geometry latents.
Figure 5: Independent 3D check across latent families. The appearance-native Wan2.1 VAE, semantic-native RAEv2, and geometry-native GAE are conditioned on the same reference image and prescribed camera trajectory. Generated views, detail crops, VGGT-reconstructed point clouds, and recovered camera trajectories highlight the impact of the latent representation on cross-view visual consistency and camera control.

4.3.3 Geometry generation

Decoded depth and point maps are compared with Pi3 on real target frames, while decoded poses are compared with dataset cameras. GAE-64 leads controlled geometry-native variants on all six DL3DV metrics; GAE-128 has better RealEstate10K point-cloud and point-map errors. These reference estimates are independent of DA3, but are still model-derived geometry.

Table 7: Geometry generation from sampled latents. Depth and point maps are compared with Pi3 on the real target frames; camera poses are compared with the real trajectory.
RealEstate10K (Pi3 ref) DL3DV (Pi3 ref)
Method AbsRel↓\downarrow δ1↑\delta_{1}\uparrow Cham.↓\downarrow PMap↓\downarrow ATE↓\downarrow RPEr∘↓{}^{\circ}\downarrow AbsRel↓\downarrow δ1↑\delta_{1}\uparrow Cham.↓\downarrow PMap↓\downarrow ATE↓\downarrow RPEr∘↓{}^{\circ}\downarrow
Pi3 on real (ref) — — — — 0.009 0.17 — — — — 0.011 0.26
Gen3R 0.164 0.807 0.727 1.428 0.046 0.69 0.232 0.688 0.587 1.180 0.031 0.91
GLD 0.271 0.694 0.858 1.580 0.039 1.23 0.323 0.594 0.462 1.097 0.028 1.19
Raw geometry (DA3-GIANT L0) 0.157 0.822 0.540 1.058 0.019 0.35 0.245 0.715 0.330 0.815 0.019 0.48
Ours (GAE-128) 0.139 0.849 0.432 0.813 0.013 0.37 0.225 0.750 0.325 0.832 0.016 0.40
Ours (GAE-64) 0.134 0.857 0.501 0.924 0.010 0.33 0.204 0.765 0.262 0.698 0.014 0.36
Gen3R, GLD, and GAE comparisons of generated RGB views, point clouds, and camera trajectories across indoor scenes.
Figure 6: Multi-frame RGB, geometry, and camera-pose comparison across RealEstate10K scenes. Gen3R, GLD, and Ours receive the same reference image and prescribed camera trajectory. For each scene, early, middle, and final generated views are paired with the generated point cloud and recovered pose. The multi-scene comparison exposes cross-frame structure, depth-layer distortions, and trajectory drift that are less apparent in a single final view.

4.3.4 Additional qualitative capabilities

Long-rollout showcase.

A separate model jointly samples 81 views at 672 × 378 from one reference. Its point cloud concatenates decoded, unprojected depth maps using cameras recovered from decoded rays, without cross-view fusion or test-time optimization. These selected examples demonstrate longer generation, not a controlled long-horizon benchmark.

Five selected views and directly decoded point clouds from four 81-view GAE rollouts.
Figure 7: Long-rollout showcase. Five views from each 81-view rollout and the point cloud decoded from the same latents.

Text-conditioned generation.

The same final checkpoint generates RGB and depth from text. Selected prompts demonstrate capability; no standalone text-to-image benchmark is reported.

Nine text prompts paired with generated RGB images and depth maps.
Figure 8: Text-conditioned RGB and geometry generation. Each pair shows an RGB sample and depth decoded from the same latent. These selected examples demonstrate capability rather than benchmark text-to-image quality.

4.4 Ablation studies

Representation shaping improves GIANT-128 generation, while the larger backbone improves appearance and pose but slightly worsens MEt3R against LARGE-128. Clean reference tokens lower the reference–target point-cloud gap from 0.1016 to 0.0318. Metric camera scale lowers scale-sensitive ATE from 0.0346 to 0.0093. Text-to-image co-training lowers FVD from 472.9 to 225.7.

Table 8: Codec and conditioning ablations on RealEstate10K. Diag. denotes ATE, point-cloud gap, scale-sensitive SE(3) ATE, or LPIPS by block. Except PSNR, lower is better.
Setting FVD ↓\downarrow PSNR ↑\uparrow MEt3R ↓\downarrow Diag. ↓\downarrow
Codec design (Diag. == VGGT ATE)
LARGE-128, with ℒrepr\mathcal{L}_{\mathrm{repr}} 270.5 19.12 0.1192 0.0048
GIANT-128, no ℒrepr\mathcal{L}_{\mathrm{repr}} 266.4 19.19 0.1226 0.0044
GIANT-128, with ℒrepr\mathcal{L}_{\mathrm{repr}} 233.4 19.54 0.1213 0.0041
Reference conditioning (Diag. == Ref–tgt PC gap)
Reference latent in ODE state 257.1 18.91 0.1217 0.1016
Clean conditioning token 225.7 20.02 0.1208 0.0318
Pose scale (Diag. == SE(3) ATE, no Sim(3))
Per-scene normalized pose 350.7 16.28 0.1290 0.0346
Metric pose 225.7 20.02 0.1208 0.0093
Text-to-image co-training (Diag. == LPIPS)
Without T2I co-training 472.9 16.77 0.1262 0.199
With T2I co-training 225.7 20.02 0.1208 0.143

5 Conclusion

A compact state that preserves geometry can improve controlled view generation. The evidence is limited to the reported datasets, estimator-based geometry checks, and training recipes; the paper provides no repeated-run uncertainty or quantitative long-horizon evaluation. The larger model also uses internally curated training data, so its selected examples do not isolate the latent’s contribution.

Appendix A Implementation and evaluation details

A.1 Geometry-native codec details

The frozen DA3-GIANT encoder and geometry head surround the learned codec. Training reconstructs its four feature levels, RGB, and geometry pseudo-targets. C-RADIO, DINOv2, and their projector are needed only while learning the codec.

A.2 Flow-matching parameterization

Predict a clean latent, then divide the noisy-to-clean difference by max(t, 0.05) to obtain velocity. Controlled models use a DDT-style encoder–decoder transformer with 28 width-768 encoder blocks and six width-2048 decoder blocks. Cross-view attention processes 1 × 1 latent tokens; temporal models add frame-axis rotary embeddings.

Equations 14–16
𝐳¯t=(1−t)​𝐳¯gt+t​𝐳¯(1),t∈[0,1],\bar{\mathbf{z}}_{t}=(1-t)\bar{\mathbf{z}}^{\mathrm{gt}}+t\bar{\mathbf{z}}^{(1)},\qquad t\in[0,1], (14)
𝐳¯^gt=𝐟ψ​(𝐳¯t,t,𝒞),\hat{\bar{\mathbf{z}}}^{\mathrm{gt}}=\mathbf{f}_{\psi}(\bar{\mathbf{z}}_{t},t,\mathcal{C}), (15)
𝐮^t=𝐳¯t−𝐳¯^gtmax⁡(t,tϵ),tϵ=0.05.\hat{\mathbf{u}}_{t}=\frac{\bar{\mathbf{z}}_{t}-\hat{\bar{\mathbf{z}}}^{\mathrm{gt}}}{\max(t,t_{\epsilon})},\qquad t_{\epsilon}=0.05. (16)

A.3 Inference-aligned multimodal conditioning

Only requested outputs are numerical flow states. References provide clean evidence, cameras provide spatial control, and text provides semantic context.

Reference tokens.

Encode the observed reference set once, prepend its standardized latents at timestep zero, and keep these tokens outside the output head and solver. Outputs at reference-camera poses are still generated normally; reference tokens are never repeatedly clamped into the noisy state.

Equation 17
𝐳¯ref=Std[Encϕμ[𝒫(ℰ(𝐈1:Kref))]].\bar{\mathbf{z}}^{\mathrm{ref}}=\operatorname{Std}\!\left[\mathrm{Enc}_{\phi}^{\mu}\!\left[\mathcal{P}\!\left(\mathcal{E}(\mathbf{I}^{\mathrm{ref}}_{1:K})\right)\right]\right]. (17)

Camera rays.

Represent each camera ray by its direction, normalized moment, and logarithmic moment magnitude. The last component preserves metric translation scale when metric camera data are available. Evaluate rays on the transformer token grid and use them to modulate attention queries and keys. [52].

Equation 18
𝐝,𝐦^=𝐦‖𝐦‖+ϵ,s=log⁡(‖𝐦‖+ϵ).\mathbf{d},\qquad\hat{\mathbf{m}}=\frac{\mathbf{m}}{\|\mathbf{m}\|+\epsilon},\qquad s=\log(\|\mathbf{m}\|+\epsilon). (18)

Text.

Frozen language-model features condition cross-attention. Independent text, camera, and reference dropout supports different condition combinations and supplies branches for classifier-free guidance. [9].

A.4 Two-stage training and sampling

Freeze all codec modules before training the flow. Every sampling run starts with Gaussian output latents, uses fixed clean reference conditions, and finishes by denormalizing and decoding.

Controlled latent comparison.

Controlled models mix RealEstate10K and DL3DV equally, with nine-view training clips and one to four prefix references. The shared flow has approximately 0.93 billion parameters, excluding frozen modules, and trains on eight GPUs with BF16, AdamW, and EMA. Learning rate warms up for 2,000 steps to 10⁻⁴ and decays toward 10⁻⁶ over 100,000 steps. Qwen3-0.6B supplies text features; one text-to-image batch follows every three multi-view updates. Pixel and semantic outputs are resized from 256 to 252 pixels for evaluation. [50].

Final model.

The separate final model trains on 81-view clips at 672 × 378 using 80 GPUs and a 60,000-step schedule. Data include six named public collections plus internally curated sets. The approximately 0.93-billion-parameter count covers only the trainable flow. One text-to-image batch follows every four multi-view updates; text, camera, and reference dropout rates are 0.10, 0.20, and 0.20. Showcases use 50 Euler steps and CFG 2, retaining camera control in the unconditional branch. [51], [11], [40], [62].

A.5 Controlled evaluation protocol

Distributional metrics use frames and clips; paired metrics compare requested views with real targets. Camera metrics use Sim(3)-aligned recovered trajectories. Reprojection retains the highest-confidence half of points; MEt3R averages visibility-masked feature disagreement across view pairs. Native depth is median-scale aligned per view, while point clouds use trajectory alignment. Consequently, the standard geometry results do not establish absolute metric-scale accuracy; the scale ablation addresses that separately.

Appendix B Latent-space diagnostic definitions

The diagnostics describe transport ambiguity, covariance conditioning, semantic organization, spatial structure, and cross-view correspondence. They are complementary proxies, not interchangeable measures of generation quality.

Evaluation data and preprocessing.

Single-view tests use ImageNet-S classes and foreground masks; multi-view tests group frames by scene. Spatial thresholds use shared image coordinates, and cosine comparisons normalize tokens. Reported latents receive no diagnostic-only PCA removal or centering. [5].

Within-group velocity fraction ρ (lower is better).

The variance ratio divides within-class or within-scene covariance trace by total covariance trace. Lower values indicate more compact groups relative to overall variation, interpreted here as less ambiguous transport.

Equation 19
ρ=tr⁡(𝚺within)tr⁡(𝚺total).\rho=\frac{\operatorname{tr}(\bm{\Sigma}_{\mathrm{within}})}{\operatorname{tr}(\bm{\Sigma}_{\mathrm{total}})}. (19)

Condition number κ and effective rank (lower/higher is better).

Condition number measures the largest-to-smallest covariance eigenvalue ratio. Effective rank is the squared eigenvalue sum divided by the sum of squared eigenvalues. Together they distinguish nominal channel count from the number and balance of active directions.

Equation 20
κ=λ1λC,effrank=(∑iλi)2∑iλi2.\kappa=\frac{\lambda_{1}}{\lambda_{C}},\qquad\operatorname{effrank}=\frac{\left(\sum_{i}\lambda_{i}\right)^{2}}{\sum_{i}\lambda_{i}^{2}}. (20)

Latent neighbor consistency LNC@5 (higher is better).

LNC@5 foreground-pools each image latent and measures how often its five nearest descriptors share its class. It evaluates local semantic organization, not patch-level geometry.

Latent distance structure LDS (higher is better).

LDS subtracts average similarity of distant token pairs from that of nearby pairs, using nearby (approximately 24 pixels) and distant (approximately 96 pixels) neighborhoods. A higher score means similarity better reflects spatial proximity.

Equation 21
LDS=𝔼(i,j)∈𝒩​[⟨𝐳^i,𝐳^j⟩]−𝔼(i,j)∈ℱ​[⟨𝐳^i,𝐳^j⟩].\mathrm{LDS}=\mathbb{E}_{(i,j)\in\mathcal{N}}[\langle\hat{\mathbf{z}}_{i},\hat{\mathbf{z}}_{j}\rangle]-\mathbb{E}_{(i,j)\in\mathcal{F}}[\langle\hat{\mathbf{z}}_{i},\hat{\mathbf{z}}_{j}\rangle]. (21)

Spatial relative similarity score SRSS (higher is better).

SRSS compares a foreground token’s similarity to nearby foreground positives against distant background negatives. A higher score indicates spatially coherent foreground regions that remain distinct from background.

Equation 22
SRSS=𝔼i∈FG​[𝔼j∈𝒫⁡(i)​⟨𝐳^i,𝐳^j⟩−𝔼k∈ℬ⁡(i)​⟨𝐳^i,𝐳^k⟩],\mathrm{SRSS}=\mathbb{E}_{i\in\mathrm{FG}}\left[\mathbb{E}_{j\in\mathcal{P}(i)}\langle\hat{\mathbf{z}}_{i},\hat{\mathbf{z}}_{j}\rangle-\mathbb{E}_{k\in\mathcal{B}(i)}\langle\hat{\mathbf{z}}_{i},\hat{\mathbf{z}}_{k}\rangle\right], (22)

Cross-view latent neighbor consistency xLNC ∗ (higher is better).

UFM-Refine provides cross-view correspondences. Retrieve each matched token’s nearest latent neighbor in the other view, count neighborhood matches, and normalize accuracy against chance. xLNC* equals zero at chance and one for perfect retrieval; evaluation requires sufficient view overlap. [56].

Equation 23
xLNC∗=a−achance1−achance,\mathrm{xLNC}^{*}=\frac{a-a_{\mathrm{chance}}}{1-a_{\mathrm{chance}}}, (23)

Appendix C Additional qualitative comparisons

C.1 Independent 3D check

Additional scenes compare generated RGB, point clouds, and camera paths under identical controls. VGGT reconstructs all outputs independently of the generating latent backbone.

Independent VGGT reconstruction and camera trajectories for two additional scenes across three latent families.
Figure 9: Independent 3D check of generated views. The appearance-native WAN2.1 VAE, semantic-native RAEV2, and our geometry-native latent receive the same reference and camera controls. The right-hand panels show point clouds and camera trajectories reconstructed from the generated RGB clips by an independent geometry model VGGT.

C.2 RGB comparisons

The appendix displays all nine views for one RealEstate10K scene and one DL3DV scene. Each shown PSNR covers that scene’s eight generated target views, not the complete dataset.

Nine-view RealEstate10K scene comparing ground truth and eight generation methods, with scene-specific PSNR.
Figure 10: RealEstate10K comparison. All methods receive the same reference image and target camera trajectory, and the eight remaining views are generated jointly.
Nine-view DL3DV scene comparing ground truth and eight generation methods, with scene-specific PSNR.
Figure 11: DL3DV comparison. All methods receive the same reference image and target camera trajectory, and the eight remaining views are generated jointly.
Concept demonstration

Open image ↗