Literature
Display
← All papers

BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving

Zeming Chen · Hang Zhao

Preprint posted · arXiv (preprint) ↗

Computer Vision10 min text · CondensedOriginal paper ↗
Contents

Overview

The supplement expands the generative-model background, spatial-consistency metric, camera-rotation and object-editing galleries, and limitations. All 16 numbered figures, five numbered tables with seven data panels, eight appendix equations and the main-text mathematical definitions are retained.

At a glance

BEV-VAE compresses synchronized surround-camera images into one spatially organized bird’s-eye-view latent representation. A camera-conditioned decoder reconstructs views, while a separately trained diffusion transformer generates new latent scenes from noise and optional 3D box occupancy. The experiments show useful reconstruction, modest camera-rotation synthesis and object-layout edits on nuScenes and Argoverse 2. Shared coordinates provide a geometric inductive bias; they do not guarantee physically correct images or consistent object identity.

1 Introduction

Generating each camera image independently makes overlapping views difficult to align, while projecting 3D boxes into 2D can discard depth relationships. This work instead assigns the scene a common BEV latent grid and conditions generation on 3D occupancy. Camera extrinsics choose which view the decoder produces, without retraining for each demonstrated rotation.

The evidence concerns static multi-view image generation at 256×256 resolution. Claims about synthetic data helping autonomous driving motivate the work; no downstream driving, planning or safety improvement is measured. “Arbitrary view” is broader than the demonstrated rotations, which range up to ±15° around the original cameras.

Refer to caption
Figure 1: Comparison of two paradigms for multi-view image generation. (a) Image latent space generation relies on 2D projections of 3D objects to guide image synthesis and cross-view attention to enforce spatial consistency. (b) BEV latent space generation is conditioned on 3D occupancy to produce a unified representation, from which all views are decoded, naturally preserving spatial consistency and enabling novel view synthesis by adjusting camera poses.

2 Related Work

The design combines BEV perception, a variational image bottleneck and latent diffusion. Its distinction is where generation occurs and how geometric conditioning enters, rather than inventing these components.

2.1 Bird’s-Eye-View Perception

Bottom-up BEV methods lift image features using estimated depth. Top-down approaches query spatial reference points and gather projected image features with deformable attention [8][9]. Each attention head learns sampling offsets and weights, with bilinear interpolation of sampled features. The original attention formula and weight normalization are retained below; q is a query, p a reference location, v the value features, M the heads and K the sampled keys.

EquationsDA⁡(q,p,v)=∑m=1M𝒲m​∑k=1K𝒜m​k⋅𝒱m​k,\operatorname{DA}(q,p,v)=\sum_{m=1}^{M}\mathcal{W}_{m}\sum_{k=1}^{K}\mathcal{A}_{mk}\cdot\mathcal{V}_{mk},∑k=1K𝒜m​k=1\sum_{k=1}^{K}\mathcal{A}_{mk}\!=\!1

2.2 Variational Autoencoder

The VAE learns a Gaussian posterior and reconstruction decoder. Prior vector-quantized and adversarial variants improve perceptual quality, while latent diffusion moves iterative denoising into a compressed representation [10][11][18]. Here the latent representation covers a multi-camera scene, and a DiT transformer models its distribution [19]. The method uses pretrained perceptual features even though its generative backbone is trained without a Stable Diffusion prior.

2.3 Autonomous Driving Multi-view Generation

BEVGen uses autoregressive generation from BEV layouts [20]. MagicDrive, DrivingDiffusion and Panacea use diffusion with different camera/layout encodings and cross-view interactions [1][2][4], while DriveWM models multi-view relations in a world-model setting [3]. Comparisons therefore differ in pretraining, conditions and architecture; they do not isolate the effect of BEV representation alone.

3 Method

Training is split into a multi-view autoencoder stage and a latent generative stage. The architectural figure preserves both paths and their conditioning.

Refer to caption
Figure 2: Overall architecture of BEV-VAE with DiT for multi-view image generation. In Stage 1, BEV-VAE learns to encode multi-view images into a spatially compact latent space in BEV and reconstruct them, ensuring spatial consistency. In Stage 2, DiT is trained with Classifier-Free Guidance (CFG) in this latent space to generate BEV representations from random noise, which are then decoded into multi-view images.

3.1 Overall Architecture

The encoder processes images, assembles multi-height scene features and compresses them into a sampled state latent. The decoder reverses this process through state, scene and image modules. A StyleGAN discriminator supplies an adversarial training signal alongside reconstruction and KL terms. The second-stage DiT generates in the learned scene space; it is not a temporally predictive driving policy.

3.2 Encoder

A ViT uses 8×8 patches on each 256×256 image, producing a 32×32 token grid. An upsampling-only three-level FPN supplies image features with 96 channels [23]. The scene encoder places eight height samples at each position of a 128×128 BEV grid, projects them with known camera calibration and gathers features only from views containing the projected points.

Queries share parameters along a pillar, with height positional encodings distinguishing samples. The state encoder concatenates eight heights into 768 channels, reduces the horizontal grid to 32×32 and applies self-attention. The resulting latent has d channels, evaluated at 4, 8, 16 and 32. Eight here counts height samples; the paper does not specify that the vertical extent is eight meters. The original feature aggregation and sequence-length definitions follow, with V denoting camera views and L sequence lengths.

EquationsFi​m​g=FPN⁡(𝐄img​(x))=Concat⁡(Fi​m​g0,Fi​m​g1,Fi​m​g2),F_{img}=\operatorname{FPN}(\mathbf{E}_{\mathrm{img}}(x))=\operatorname{Concat}(F_{img}^{0},F_{img}^{1},F_{img}^{2}),Fs​c​n=1|𝒱hit|​∑v∈𝒱hit DA⁡(QBEV,PBEV,Fi​m​g(v)),F_{scn}=\frac{1}{|\mathcal{V}_{\text{hit}}|}\sum_{v\in\mathcal{V}_{\text{hit }}}\operatorname{DA}(Q_{\text{BEV}},P_{\text{BEV}},F_{img}^{(v)}),LQ=8×128×128L_{Q}=8\times 128\times 128LV=∑i=02(32×32×22​i)L_{V}=\sum_{i=0}^{2}(32\times 32\times 2^{2i})

3.3 Decoder

The state decoder expands 32×32 features to a 128×128 grid, splits channels into eight height levels, and builds a downsampled scene pyramid. For each camera, the scene decoder samples 60 depths on each of 32×32 rays and aggregates scene features with learned depth weights. Samples outside the represented scene receive zero weight. Camera parameters determine the mapping between frustum and BEV coordinates.

The image decoder maps the 96-channel features to width 768, adds interactions between rays with self-attention, and upsamples to 256×256 RGB. This is learned feature aggregation, not an explicit guarantee of correct visibility, surface geometry or radiance. The source equations retain their original notation, including the argument written as x̂ in the state-decoder formula.

EquationsF^s​c​n=FPN⁡(𝐆stt​(x^))=Concat⁡(F^s​c​n0,F^s​c​n1,F^s​c​n2),\hat{F}_{scn}=\operatorname{FPN}(\mathbf{G}_{\mathrm{stt}}(\hat{x}))=\operatorname{Concat}(\hat{F}_{scn}^{0},\hat{F}_{scn}^{1},\hat{F}_{scn}^{2}),F^i​m​g(v)=∑d∈𝒟hitWd⊙DA⁡(QCFV,PCFV,F^s​c​n),\hat{F}^{(v)}_{img}=\sum_{d\in\mathcal{D}_{\text{hit}}}W_{d}\odot\operatorname{DA}(Q_{\text{CFV}},P_{\text{CFV}},\hat{F}_{scn}),LQ=60×32×32L_{Q}=60\times 32\times 32LV=∑i=02(8×128×128×2−3​i)L_{V}=\sum_{i=0}^{2}(8\times 128\times 128\times 2^{-3i})

3.4 Loss

The posterior is regularized toward a standard Gaussian, and reparameterization permits gradient training. Reconstruction combines squared pixel error with squared VGG-16 feature differences. A hinge discriminator loss separates real and reconstructed images; the generator receives the negative discriminator score. The total encoder/decoder objective uses β=10⁻⁶ and adversarial coefficient 0.1λ, with δ=10⁻⁶ in the adaptive gradient ratio.

All seven original definitions follow. The printed λ expression divides gradients and does not state a norm or scalar reduction convention; practical reproduction should check the implementation instead of assuming a missing operation. The small KL coefficient is a chosen regularization weight, not evidence that the learned posterior exactly matches the sampling prior.

EquationsℒKL=DKL(qϕ(z∣x)∥p(z))=12∑i=1d(σi2+μi2−1−logσi2),\mathcal{L}_{\mathrm{KL}}=D_{\mathrm{KL}}(q_{\phi}(z\mid x)\|p(z))=\frac{1}{2}\sum_{i=1}^{d}(\sigma_{i}^{2}+\mu_{i}^{2}-1-\log\sigma_{i}^{2}),z=μ+σ⊙ϵ,(μ,σ)=E⁡(x),ϵ∼𝒩⁡(0,I).z=\mu+\sigma\odot\epsilon,\quad(\mu,\sigma)=E(x),\quad\epsilon\sim\mathcal{N}(0,I).ℒR=ℒ2+ℒperceptual=‖x−x^‖2+∑l‖ψl​(x)−ψl​(x^)‖2.\mathcal{L}_{\mathrm{R}}=\mathcal{L}_{2}+\mathcal{L}_{\mathrm{perceptual}}=\|x-\hat{x}\|^{2}+\sum_{l}\|\psi_{l}(x)-\psi_{l}(\hat{x})\|^{2}.ℒD=max⁡(0,1−D⁡(x))+max⁡(0,1+D⁡(x^)),\mathcal{L}_{\mathrm{D}}=\max(0,1-D(x))+\max(0,1+D(\hat{x})),ℒA=−D⁡(x^)\mathcal{L}_{\mathrm{A}}=-D(\hat{x})ℒG=β⋅ℒKL+ℒR+0.1⋅λ⋅ℒA\mathcal{L}_{\mathrm{G}}=\beta\cdot\mathcal{L}_{\mathrm{KL}}+\mathcal{L}_{\mathrm{R}}+0.1\cdot\lambda\cdot\mathcal{L}_{\mathrm{A}}λ=∇GL[ℒR]∇GL​[ℒA]+δ\lambda=\frac{\nabla_{G_{L}}[\mathcal{L}_{\mathrm{R}}]}{\nabla{G_{L}}[\mathcal{L}_{\mathrm{A}}]+\delta}

3.5 Generation

Each box stores quaternion orientation, center, size and semantic class. Boxes become a class-specific binary occupancy tensor over eight heights and a 128×128 horizontal grid. Maximum occupancy merges overlapping boxes of the same class, so the condition does not retain separate instance identities. Patch processing produces a 768×32×32 conditional feature added to a corresponding state feature.

The main text writes a scale s on feature addition, while Appendix A also uses s in conventional classifier-free noise-prediction guidance. Both source formulas are retained; their shared symbol alone does not fully specify the implementation of guidance. The paper does not detail every projection between the d-channel VAE latent and the 768-channel conditioning representation.

Equations𝐛=(qw,qx,qy,qz,xc,yc,zc,l,w,h,c),\mathbf{b}=(q_{w},q_{x},q_{y},q_{z},x_{c},y_{c},z_{c},l,w,h,c),𝐂box∈{0,1}C×8×128×128\mathbf{C}_{\text{box}}\in\{0,1\}^{C\times 8\times 128\times 128}𝐂box(c,z,y,x)=maxi:ci=c𝟏[(z,y,x)∈Ω(𝐛i)]\mathbf{C}_{\mathrm{box}}(c,z,y,x)=\max_{i:c_{i}=c}\mathbf{1}[(z,y,x)\in\Omega(\mathbf{b}_{i})]Fb​o​x∈ℝ768×32×32F_{{box}}\in\mathbb{R}^{768\times 32\times 32}Fs​t​t′=Fs​t​t+s⋅Fb​o​x,F_{stt}^{\prime}=F_{stt}+s\cdot F_{box},

4 Experiments

Experiments evaluate reconstruction and generation separately. The galleries preserve all cameras, editing examples and original captions, including examples where generated vehicle appearance changes.

Refer to caption
Figure 3: Multi-view image reconstruction on AV2. Row 1 shows validation images, and Rows 2-5 display reconstructed images with latent dimensions of 4, 8, 16, and 32. With higher dimensions, the reconstruction more accurately preserves fine details, such as the manhole covers in the white box.
Refer to caption
Figure 4: Novel view synthesis via camera pose modifications. Row 1 presents validation images, and Row 2 shows reconstructions. Rows 3 and 4 display reconstructed images with all cameras rotated 15° left and 15° right, respectively. Note: Latent dimension is set to 32.
Refer to caption
Figure 5: Multi-view image generation on nuScenes. Row 1 shows validation images, and Row 2 presents images generated from the corresponding 3D bounding boxes.
Refer to caption
Figure 6: Multi-view image generation on AV2. Row 1 presents real images from the validation set. Row 2 shows images generated from the corresponding 3D bounding boxes. Rows 3-5 present generated images after removing a specific vehicle, with the removed vehicles indicated by numerical labels. Note: The same 3D bounding box may produce different objects across generated images.

4.1 Datasets

nuScenes supplies six cameras, 700 training and 150 validation scenes: 155,245 training samples and 33,142 validation samples, with 28,126 and 6,019 annotated samples respectively, across ten categories. AV2 supplies seven cameras and the same scene-split counts: 224,175 training and 47,946 validation samples, with 109,907 and 23,521 annotated samples, across 30 categories. The front AV2 camera uses a rotated orientation. Sample counts refer to synchronized multi-camera sets, not single images; labeled subsets are smaller than the complete image collections.

4.2 Metrics

PSNR and SSIM measure reconstruction against the input. FID measures feature-distribution differences for either reconstruction or generation; those two uses should not be compared as the same task. MVSC divides average LoFTR adjacent-view matching confidence by that of real images [24]. It is a relative feature-matching statistic, not a direct geometric error, a percentage of physically correct scenes or a guarantee of accurate layout adherence.

4.3 Settings

Training uses eight A800 GPUs and AdamW. The VAE stage runs 100,000 iterations at one sample per GPU, learning rate 4×10⁻⁵ on nuScenes or 8×10⁻⁵ on AV2, 5,000 warm-up steps, betas (0.9,0.99), weight decay 10⁻⁴ and EMA 0.9999.

The diffusion stage uses eight samples per GPU, learning rate 10⁻⁴, 400,000 iterations for nuScenes or 200,000 for AV2, 5,000 warm-up steps, betas (0.9,0.95), weight decay 0.1, zero bias decay and EMA 0.999. Wall-clock cost, sampling latency, repeated-seed uncertainty and several spatial-range/architecture details are not specified in the paper.

4.4 Reconstruction

One 32×32×d latent represents the entire camera set, whereas the SD-VAE reference compresses each individual image into 32×32×4. Table 1 treats SD-VAE as a pretrained image-reconstruction reference, with substantially different training resources, rather than a controlled equal-budget baseline.

Larger d generally improves PSNR, SSIM and MVSC. On nuScenes, reconstruction FID improves to 11.42 at d=16 but worsens to 13.72 at d=32; suggested overfitting is not independently established. At d=32, AV2 reaches PSNR 26.68, SSIM 0.8004, MVSC 0.9505 and FID 3.02, compared with SD-VAE’s 27.81, 0.8229, 0.8962 and 1.87. Thus it improves this spatial-consistency proxy while remaining behind in per-image fidelity. On nuScenes its best listed MVSC 0.9291 is still just below SD-VAE’s 0.9292.

The camera-rotation galleries use d=32 and illustrate novel views around original poses. They do not include quantitative ground truth for newly synthesized viewpoints. Larger reconstruction capacity does not imply a better generative latent, as the next experiment shows.

Model Latent Shape Training Data PSNR↑\uparrow SSIM↑\uparrow MVSC↑\uparrow FID↓\downarrow
SD-VAE 32×32×432\times 32\times 4 5.85B images 29.63 0.8283 0.9292 2.18
BEV-VAE 32×32×432\times 32\times 4 155K × 6 views 23.48 0.6039 0.8994 17.83
BEV-VAE 32×32×832\times 32\times 8 155K × 6 views 24.53 0.6569 0.9107 13.08
BEV-VAE 32×32×1632\times 32\times 16 155K × 6 views 25.73 0.7124 0.9222 11.42
BEV-VAE 32×32×3232\times 32\times 32 155K × 6 views 26.32 0.7455 0.9291 13.72
(a) Reconstruction metrics on nuScenes across different dimensions, with SD-VAE as reference.
Model Latent Shape Training Data PSNR↑\uparrow SSIM↑\uparrow MVSC↑\uparrow FID↓\downarrow
SD-VAE 32×32×432\times 32\times 4 5.85B images 27.81 0.8229 0.8962 1.87
BEV-VAE 32×32×432\times 32\times 4 224K × 7 views 22.99 0.6318 0.8270 7.47
BEV-VAE 32×32×832\times 32\times 8 224K × 7 views 24.02 0.6870 0.8827 5.10
BEV-VAE 32×32×1632\times 32\times 16 224K × 7 views 25.49 0.7529 0.9226 3.62
BEV-VAE 32×32×3232\times 32\times 32 224K × 7 views 26.68 0.8004 0.9505 3.02
(b) Reconstruction metrics on AV2 across different dimensions, with SD-VAE as reference.
Table 1: Comparison of BEV-VAE with varying latent dimensions and SD-VAE for multi-view reconstruction. BEV-VAE performs spatial modeling by encoding multi-view images into a unified BEV representation and decoding them back into images, while SD-VAE, trained on 5.85 billion images, serves only as a reference for image reconstruction rather than a direct baseline.

4.5 Generation

The best generative latent in the reported sweep is d=8. nuScenes reaches FID 21.14 at guidance s=5; AV2 reaches 10.68 at s=3. Higher d makes reconstruction easier but makes modeling the latent distribution harder under the reported training setup.

On nuScenes, 21.14 improves on from-scratch BEVGen’s 25.54 but trails the listed Stable Diffusion-based results: Panacea 16.96, MagicDrive 16.20, DrivingDiffusion 15.83 and DriveWM 12.99. Similar absence of pretrained generative priors does not by itself equalize all resources or conditions. AV2 has no direct seven-view baseline here, and its lower FID than nuScenes cannot establish a scaling law because dataset content, cameras, labels and training iterations also differ.

Object-removal examples show layout control qualitatively, with no numerical removal, box-adherence or identity-retention metric. The same box can produce different appearances across generated samples or views. The source cross-references around Tables 2–4 are imperfect; the original numbered table captions and panels are retained.

Scale d=4d=4 d=8d=8 d=16d=16 d=32d=32
s=0s=0 30.04 32.01 40.31 47.10
s=1s=1 27.19 24.17 30.03 38.58
s=2s=2 24.14 22.43 24.63 32.78
s=3s=3 23.41 22.22 23.45 31.03
s=4s=4 22.94 22.11 23.42 30.31
s=5s=5 23.06 21.14 23.74 30.52
(a) FID on nuScenes across scales and dimensions.
Scale d=4d=4 d=8d=8 d=16d=16 d=32d=32
s=0s=0 23.63 30.05 33.12 32.66
s=1s=1 16.87 15.28 23.28 21.82
s=2s=2 13.48 11.15 16.65 16.14
s=3s=3 12.72 10.68 14.96 15.31
s=4s=4 13.03 11.06 14.73 15.97
s=5s=5 13.34 11.92 15.15 17.01
(b) FID on AV2 across scales and dimensions.
Table 2: Impact of guidance scale across latent dimensions for multi-view image generation.
Method Paradigm Extra Prior FID↓\downarrow
BEVGen Autoregression None 25.54
Panacea Diffusion Stable Diffusion 16.96
MagicDrive Diffusion Stable Diffusion 16.20
DrivingDiffusion Diffusion Stable Diffusion 15.83
DriveWM Diffusion Stable Diffusion 12.99
BEV-VAE w/ DiT Diffusion None 21.14
Table 3: Benchmark results on nuScenes. BEV-VAE w/ DiT significantly bridges the gap from-scratch and SD-finetuned methods.
Setting PSNR↑\uparrow SSIM↑\uparrow MVSC↑\uparrow FID↓\downarrow
wo/ ℒperceptual\mathcal{L}_{\mathrm{perceptual}} 25.19 0.7203 0.8698 68.99
wo/ ℒA\mathcal{L}_{\mathrm{A}} 24.87 0.7135 0.9053 13.62
w/ ℒ1\mathcal{L}_{1} instead of ℒ2\mathcal{L}_{2} 23.41 0.6780 0.8632 8.09
ℒG\mathcal{L}_{\mathrm{G}} 24.02 0.6870 0.8827 5.10
Table 4: Ablation on loss function for reconstruction performance on AV2. The latent dimension is fixed at 8.

4.6 Analysis of Loss Function

The AV2 d=8 loss ablation gives reconstruction FID 5.10 with the full objective, 68.99 without perceptual loss, 13.62 without adversarial loss, and 8.09 when replacing squared error with L1. The full objective’s PSNR/SSIM/MVSC are 24.02/0.6870/0.8827. Removing the adversarial term raises MVSC to 0.9053 and PSNR to 24.87, illustrating a fidelity/consistency trade-off rather than uniform improvement across metrics. These are single reported configurations without confidence intervals.

5 Conclusion

A structured scene latent enables shared-coordinate image decoding and useful qualitative control. The evidence supports a representation and generative-model design, while leaving high-resolution fidelity, temporal coherence, physical correctness, cross-dataset transfer and downstream driving benefit unresolved. Temporal models and physical priors are future directions rather than tested capabilities.

Appendix A Preliminary for Generative Models

The eight original equations cover the VAE evidence lower bound, Gaussian forward diffusion, reparameterization, the reverse process, noise-prediction loss and classifier-free guidance. The VAE maximizes its ELBO, equivalently minimizing the negative bound; the source’s later phrase about minimizing the ELBO should be interpreted with that sign distinction. Conditioning dropout trains both conditional and unconditional predictions. The final guidance relation is schematic, so its proportionality should not be read as a fully specified equality without the appropriate score/noise scaling.

Equations 1–8
logpθ(x)≥𝔼qϕ​(z|x)[logpθ(x|z)]−DKL(qϕ(z|x)∥pθ(z)),\log p_{\theta}(x)\geq\mathbb{E}_{q_{\phi}(z|x)}\left[\log p_{\theta}(x|z)\right]-D_{\mathrm{KL}}\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z)\right), (1)
q⁡(xt∣x0)=𝒩⁡(xt,α¯t​x0,(1−α¯t)​𝐈),q(x_{t}\mid x_{0})=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathbf{I}), (2)
xt=α¯t​x0+1−α¯t​ϵt,ϵt∼𝒩⁡(0,𝐈).x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon_{t},\quad\epsilon_{t}\sim\mathcal{N}(0,\mathbf{I}). (3)
pθ​(xt−1∣xt)=𝒩⁡(xt−1,μθ​(xt),σt2​𝐈),p_{\theta}(x_{t-1}\mid x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t}),\sigma_{t}^{2}\mathbf{I}), (4)
ℒsimple​(θ)=𝔼⁡[‖ϵθ​(xt)−ϵt‖22].\mathcal{L}_{\text{simple}}(\theta)=\mathbb{E}[\|\epsilon_{\theta}(x_{t})-\epsilon_{t}\|_{2}^{2}]. (5)
∇x​log​p​(c∣x)=∇x​log​p​(x∣c)−∇x​log​p​(x),\nabla_{x}\log p(c\mid x)=\nabla_{x}\log p(x\mid c)-\nabla_{x}\log p(x), (6)
pθ​(xt−1∣xt,c)=𝒩⁡(xt−1∣μθ​(xt,c),σt2​𝐈).p_{\theta}(x_{t-1}\mid x_{t},c)=\mathcal{N}(x_{t-1}\mid\mu_{\theta}(x_{t},c),\sigma_{t}^{2}\mathbf{I}). (7)
ϵ^θ​(xt,c)=ϵθ​(xt,∅)+s⋅(ϵθ​(xt,c)−ϵθ​(xt,∅))∝ϵθ​(xt,∅)+s⋅∇x​log​p​(c∣xt).\hat{\epsilon}_{\theta}(x_{t},c)=\epsilon_{\theta}(x_{t},\emptyset)+s\cdot(\epsilon_{\theta}(x_{t},c)-\epsilon_{\theta}(x_{t},\emptyset))\propto\epsilon_{\theta}(x_{t},\emptyset)+s\cdot\nabla_{x}\log p(c\mid x_{t}). (8)

Appendix B Evaluation with Multi-View Spatial Consistency

LoFTR matches the bordering half-images of each adjacent camera pair; MVSC is the mean confidence relative to real-image matches. The source calls it both Multi-View Spatial Consistency and Multi-View Spatial Confidence. Neither name changes the measured quantity. The appendix comparison reports MagicDrive 0.8310, Panacea 0.9189 and BEV-VAE 0.8902; BEV-VAE is between the two and has worse FID than both.

Panacea also uses text, map and depth conditions, and MagicDrive uses text and maps; BEV-VAE uses 3D occupancy plus extrinsics. The highlighted misaligned vehicle in Figure 7 illustrates that matching confidence need not imply agreement with the intended boxes. The paper’s explanation that extra conditioning partly causes Panacea’s higher score is not isolated by an ablation.

Method FID↓\downarrow MVSC↑\uparrow Object Layouts Camera Poses Other Conditions
MagicDrive 16.20 0.8310 Fourier embedding(1D) Fourier embedding Text, map.
Panacea 16.96 0.9189 Perspective projection (2D) Pseudo-color image Text, map, depth.
Ours 21.14 0.8902 Binary occupancy (3D) Extrinsic matrix None
Table 5: Comparison on nuScenes: image quality, spatial consistency, and conditions

Appendix C Reconstruction with Camera Pose Control

Figures 8–13 preserve three scenes from each dataset. Each has the original camera images plus seven reconstructed rows at +15°, +10°, +5°, 0°, −5°, −10° and −15°, using d=32. They demonstrate modest rotation control from an encoded scene, not unrestricted camera translation or guaranteed reconstruction of previously unseen surfaces.

Appendix D Generation with Precise 3D Object Control

Figures 14–15 retain the real images, reconstruction, generation from all boxes and five separate vehicle-removal settings. Figure 16 rotates one vehicle’s box ±15° on the ego horizontal plane. These are selected visual demonstrations; the paper reports no aggregate edit-success rate. Appearance may change even when a box is unchanged, so occupancy control does not imply persistent vehicle identity.

Appendix E Limitations in Resolution and Data Scale

The authors acknowledge blur, 256×256 resolution and limited training data, especially on nuScenes. They attribute limitations to data scale and lack of pretrained image priors, but do not separately rule out architecture or optimization constraints. AV2 is about 1.44 times as large by the exact unannotated training-sample counts, described approximately as 1.5×. Better AV2 results are suggestive, not a controlled scaling experiment. All supplementary gallery panels remain below, including full labels supplied by the native images where the PDF page margin clips them.

Refer to caption
Figure 7: Multi-View Spatial Consistency (MVSC) on nuScenes. The comparison is based on images generated by different methods. Row 1 shows the projections of 3D object layouts onto the image plane. Row 2 presents the corresponding validation images. Rows 3–5 display the results generated by MagicDrive, Panacea, and our method, respectively. To better visualize spatial consistency across adjacent views, each row of images is shifted to the right by half an image width. Vertical black lines mark the centerlines of each camera view. Red boxes indicate regions where the generated vehicles are significantly misaligned with the ground-truth layouts.
Refer to caption
Figure 8: Example 1 on nuScenes: Novel view synthesis via camera pose modifications.
Refer to caption
Figure 9: Example 1 on AV2: Novel view synthesis via camera pose modifications.
Refer to caption
Figure 10: Example 2 on nuScenes: Novel view synthesis via camera pose modifications.
Refer to caption
Figure 11: Example 2 on AV2: Novel view synthesis via camera pose modifications.
Refer to caption
Figure 12: Example 3 on nuScenes: Novel view synthesis via camera pose modifications.
Refer to caption
Figure 13: Example 3 on AV2: Novel view synthesis via camera pose modifications.
Refer to caption
Figure 14: Multi-view image generation on nuScenes with 3D object layout editing.
Refer to caption
Figure 15: Multi-view image generation on AV2 with 3D object layout editing.
[Uncaptioned image]
Figure 16: Rotating the orientation of a specific vehicle on AV2. Row 1 presents validation images and Row 2 shows generated images. Rows 3 and 4 depict the same vehicle rotated 15° clockwise and counterclockwise on the ego vehicle’s horizontal plane.
Concept demonstration

Open image ↗