Literature
Display
← All papers

Gen-NCAP: A Generative Simulator for Corner Case Benchmarking in End-to-End Autonomous Driving

Gen Li · Nan Wang · YunLong Li · Ziyang Yan · Yinghao Shuai · Saining Zhang · Shu Han · Shaocong Xu · Baijun Ye · Lu Zhang · Luan Zhang · Paul Barsch · Zebang Shen · Chaojian Li · Hang Zhao · Ya-Qin Zhang · Hao Zhao

· IASEAI 2026 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

Gen-NCAP combines text-driven object and trajectory generation with a reconstructed driving scene and video refinement. The resulting closed-loop simulator evaluates how an autonomous-driving model reacts to rare obstacles and traffic conflicts, rather than replaying a fixed recorded trajectory.

The paper reports 300 scenarios derived from 47 nuScenes clips, covering eight corner-case categories. It compares four driving models, introduces collision-severity and responsiveness scores, and tests synthetic-data fine-tuning. Results support a useful simulation stress test and improved rendered imagery; they do not establish real-road safety or complete physical realism.

1. Introduction

On-road comparisons are expensive and difficult to reproduce, while log replay cannot show how a model’s decisions alter future observations. Closed-loop simulation offers controlled feedback, but manually creating rare scenarios is costly and rendering errors can distort the evaluated model’s behavior. Figure 1 illustrates the proposed combination of scene editing and refinement.

figure 1
Figure 1: We propose Gen-NCAP, a generative closed-loop simulator demonstrating strong extensibility across diverse corner cases and reduced sim-to-real gap through high-fidelity, temporally consistent video refinement module.

Scenario coverage and flexibility

The first challenge is scenario variety: generating many cases without specifying every object and trajectory manually. The second is asset flexibility: introducing new objects beyond those already reconstructed in a scene. Gen-NCAP addresses both with language-conditioned generation, although the paper does not quantify how often generated assets or trajectories require correction.

Rendering fidelity and temporal stability

Inserted objects can look inconsistent with the background, and novel camera viewpoints can reveal reconstruction artifacts. Independent frame enhancement can also flicker or alter dynamic objects. The proposed refinement model uses a video history to improve appearance across time. Visual consistency is one component of the simulation gap; it does not by itself validate dynamics or collision physics.

2.1. End-to-end autonomous driving

End-to-end driving models predict trajectories or controls from sensor inputs. The evaluated systems are UniAD, VAD, SparseDrive, and DiffusionDrive [18] [19] [16] [17], which differ in perception, scene representation, and planning design.

2.2. Evaluation benchmarks

CARLA offers editable synthetic environments; other benchmarks reuse recorded scenes or render reconstructed sensor views. NeuroNCAP [9] supplies an especially relevant comparison for photorealistic closed-loop safety tests. Gen-NCAP adds generative scene edits and video refinement to broaden the available stress scenarios.

2.3. Generative models for simulation

Prior generative renderers repair novel views or refine reconstructions. The paper focuses on jointly correcting background artifacts and inserted moving foreground objects while maintaining temporal consistency. Difix3D+ [31] is the direct refinement baseline.

3.1. Framework overview

Figure 2 shows two connected modules. Corner-case generation creates a new object, estimates its dimensions, and proposes its motion in a reconstructed scene. Video refinement then improves rendered observations before they reach the driving model. The closed loop updates observations according to that model’s decisions.

figure 2
Figure 2: Overview of Gen-NCAP. Our Gen-NCAP framework consists of two key modules: (a) a Corner Case Generation Module, which generates a diverse range of safety-critical driving scenarios, and (b) a Video Refinement Module, which reduces rendering artifacts using targeted video diffusion models, minimizing the sim-to-real gap.

3.2. Object generation

A text prompt first produces an image through FLUX.1-dev [32]; TRELLIS [33] converts the image into a three-dimensional Gaussian asset. A language model estimates object dimensions from its category and scene context. Figure 3 retains both asset and trajectory examples.

The text names GPT-4o and then GPT-4 for the sizing step without resolving that naming difference. Model-generated dimensions are estimates rather than measured geometry.

figure 3
Figure 3: Visualizations of generated objects and trajectories.

Trajectory generation

A vision-language model (VLM) receives scene imagery, a bird’s-eye view (BEV) with existing object positions and dimensions, the new object’s attributes, and the requested corner case. It proposes planar positions in ego coordinates, which are transformed into world coordinates using the ego-to-world pose.

The method therefore relies on supplied spatial scene information as well as language. The main PDF does not specify a trajectory-feasibility validator or quantify violations of motion constraints.

Driving-scene reconstruction

Three-dimensional Gaussian splatting reconstructs the driving environment. A scene graph [36] separates sky, background, rigid vehicles, and non-rigid actors into editable nodes. This allows object insertion and movement while retaining the rest of the reconstructed scene.

4. Video refinement

The refinement module learns to map degraded rendered video toward higher-quality imagery. Its training examples target two specific artifact families, then a common video model handles both.

Training-data curation

For foreground artifacts, vehicles segmented from nuScenes are reconstructed with TRELLIS and substituted into the scene, producing deliberately imperfect foreground/background alignment. For novel-view artifacts, under-trained scene reconstructions produce blurry or distorted renderings. These degraded frames are paired with clean targets; Figure 4 retains examples of both constructions.

figure 4
Figure 4: Visualization of the paired dataset. (a) simulates artifacts in novel-view rendering, (b) simulates foreground–background misalignment.

Training the video refinement model

A video diffusion transformer, based on the cited LTX-Video model [37], is conditioned on degraded observations and trained against clean-video latent representations. Equation 1 preserves the original squared-error objective and notation. The main paper describes the data construction and architecture at a high level but omits training-set size, detailed schedules, and runtime measurements.

Equation
equation 1

Applying refinement in the closed loop

At time t, the refiner receives past observations and the current rendered frame. Its refined current frame becomes the driving model’s input, so the stated procedure does not require future frames. Using video history aims to reduce temporal inconsistency.

This causal input structure does not establish real-time throughput. It also leaves open whether generative refinement always preserves the precise obstacle geometry and identity needed for safety evaluation.

5. Gen-NCAP benchmark

The benchmark contains 300 interactive scenarios built from 47 nuScenes clips [38], with eight categories inspired by Euro NCAP collision-avoidance scenarios. Figures 5–6 preserve every category diagram and the dynamic examples. Categories include vehicle conflicts, pedestrians, road obstacles, and animals.

The published PDF refers readers to an appendix for category definitions and simulator details, but that appendix is absent from the available thirteen-page publication. The scenario abbreviations in the figures remain as printed.

figure 5
Figure 5: Illustration of the eight distinct categories of safety-critical scenarios in Gen-NCAP.
figure 6
Figure 6: Visualization of dynamic corner cases at different timesteps generated by Gen-NCAP.

5.1. Evaluation metrics

The evaluation supplements collision occurrence with a severity-inspired score and a response-magnitude score. These measure different properties, so a model need not rank best on all of them.

Gen-NCAP Score

The Gen-NCAP Score (GNS) is 100 without a collision and 100/(1 + ksv) otherwise. Here k is an object-category weight, s is the collided object’s volume, and v is ego speed at collision. Higher is better.

This is a chosen severity proxy inspired by momentum, rather than a validated injury model. The main PDF does not provide the category weights needed to reproduce the score independently.

Equation
equation 2

Gen-NCAP Responsiveness Score

The Gen-NCAP Responsiveness Score (GNRS) is log(1 + a²) × (1 + L), using ego acceleration a and deviation L from the recorded reference trajectory. Higher values indicate larger responses.

Because acceleration is squared and deviation increases the score, a large value does not distinguish beneficial avoidance from excessive or unsafe motion. Responsiveness must be interpreted alongside collision outcomes.

Equation
equation 3

6. Experimental setup

The four driving systems use author-provided pretrained weights. Closed-loop evaluation reports NeuroNCAP Score, collision rate, acceleration, GNS, and GNRS. Refinement uses Fréchet Inception Distance (FID), vehicle-crop FID (FID-A), and Fréchet Video Distance (FVD). An additional set of rendered novel views is mentioned, but its detailed construction is deferred to the unavailable appendix.

6.1. Comparing driving models

Table 1 retains every model, scenario category, metric, and printed average. DiffusionDrive has the lowest displayed average collision rate (0.850), while VAD has the best GNS (39.39) and NeuroNCAP Score (1.610). The displayed GNRS average is highest for DiffusionDrive (5.65), contradicting the prose statement that UniAD is best on that metric.

The table labels collision rate with a percent sign but displays fraction-like values, and labels acceleration with a percent sign without explaining its units. The edition preserves the original numbers and labels rather than silently rescaling them. Cross-model associations between acceleration and collisions do not establish a causal comfort–safety tradeoff. Figure 7 retains the failure examples for all four systems.

figure 7
Figure 7: Qualitative comparisons of the failure cases of four E2E-AD models in handling corner cases within closed-loop simulation.
table 1
Table 1: Quantitative results of four E2E-AD models on the Gen-NCAP benchmark. Horizontal bars provide an intuitive comparison of performance across different metrics.

6.2. Fine-tuning with generated data

Fine-tuning VAD with Gen-NCAP data improves the reported NeuroNCAP score from 0.81 to 1.24 and reduces collision rate from 0.90 to 0.85. Table 2 preserves the comparison. Interpreting these rates as fractions gives a five-percentage-point reduction, matching the paper’s stated improvement; this is a simulated evaluation, not an observed road-crash reduction.

table 2
Table 2: Quantitative comparison between the original VAD and the VAD fine-tuned(VAD*) using Gen-NCAP.

6.3. Real-to-sim evaluation in open loop

Table 3 compares open-loop predictions from original images, NeuroNCAP renderings, and Gen-NCAP renderings on aligned scenarios. UniAD and VAD show broadly similar displacement errors across image sources, with several lower reported collision values for Gen-NCAP.

This tests how rendering changes planning outputs relative to a recorded reference. Similar open-loop errors do not establish closed-loop transfer, faithful rare-event dynamics, or safety under unrestricted real-world distribution shift.

table 3
Table 3: Evaluation of real-to-sim performance on NeuronCap scenarios using typical planning metrics in an open-loop setting.ADE stands for Average Displacement Error.

6.4. Video refinement results

Figures 8–9 retain every original crop and comparison, including difficult moving vehicles. They show restoration of foreground alignment and blurry novel views relative to raw renderings and Difix. The images support visual improvement on the shown examples; no independent geometric-preservation test is reported.

figure 8
Figure 8: Vehicle motion blur restoration with zoom-in results (below each example).
figure 9
Figure 9: Visualization of the Video Refinement Module’s performance in correcting the two types of artifacts. Compared to other methods, our method significantly improves image quality.

Quantitative refinement results

Table 4 reports FID 48.48 versus 77.97 without refinement and 52.63 for Difix. Vehicle-crop FID is 21.86 versus 34.46 and 21.97; FVD is 816.8 versus 952.4 and 1113. Lower is better for all three.

The full-image and video scores improve, while the crop-FID difference from Difix is only 0.11. These distributional quality measures do not certify physical correctness, and the paper supplies no uncertainty estimates for these comparisons.

table 4
Table 4: Quantitative results of video refinement module.

7. Conclusion

Gen-NCAP provides a concrete way to generate and render diverse simulated corner cases, compare driving models under feedback, and augment training data. Its evidence supports a controlled simulation benchmark with improved visual quality. Reproducibility of several settings, fidelity of generated geometry and dynamics, and transfer to real-road safety remain unresolved by the available paper.

Source availability note

This edition retains all nine figures, four tables, three numbered equations, and 41 references in the published thirteen-page PDF. That PDF repeatedly cites an appendix that it does not contain; no appendix link was found in the author’s public research listing. The absent implementation details and any unprovided supplemental media could not be reviewed or restored.

The author list follows the publication metadata and author’s research listing, which include Baijun Ye; the downloaded PDF’s title-page list omits that name.

Concept demonstration

Open image ↗