Overview
The visual abstract links four types of difficult driving scenes with the reported benchmark gains. All scientific figures, original annotation examples, the classification prompt, five data/result/configuration tables and the complete reference list are retained.
At a glance
Impromptu VLA curates approximately 80,000 difficult driving clips from eight public datasets, using a vision-language model to identify scenarios and generate task annotations followed by human verification. Pretraining Qwen2.5-VL on these examples before nuScenes fine-tuning improves simulated closed-loop driving and open-loop trajectory error. The resource targets unstructured roads and unusual situations; the results do not establish safe real-world autonomous driving.
1 Introduction
Standard driving datasets emphasize common structured roads. The authors instead target ambiguous road boundaries, temporary traffic rules, unusual dynamic obstacles and challenging road conditions. Their contribution combines a taxonomy, curated clips, question-answer annotations and action trajectories, with model weights and data intended for open release. The study tests both downstream driving benchmarks and whether its own QA tasks diagnose perception, prediction and planning.
The strongest controlled comparison is between a Qwen model fine-tuned directly on nuScenes and one additionally trained on Impromptu VLA first. That comparison establishes the usefulness of the added training recipe; it does not separate dataset selection from additional training compute or demonstrate superiority to every driving specialist.
2 Impromptu VLA Dataset: Learning to Drive on Unstructured Roads
The dataset combines existing real-world driving imagery rather than generating synthetic corner cases. Its automated selection and annotation pipeline is followed by human checks, with task availability varying by the source annotations.
| Attribute | Mapillary | ONCE | NAVSIM | nuScenes | Waymo | Argoverse-V2 | KITTI | IDD | Sum. |
| Camera Views | 1 | 7 | 8 | 6 | 5 | 7 | 1 | 1 | - |
| Resolutions | Variable | 1920 x 1020 | 1920 x 1080 | 1600 x 900 | 640 x 360 | 1550 x 2048 | 1242 x 375 | 1920 x 1080 | - |
| FPS | 2Hz* | 2Hz | 2Hz | 2Hz | 10Hz | 20Hz | 10Hz | 15Hz | |
| Raw Clip Count | 1000k | 800k | 90k | 40k | 40k | 320k | 20k | 7k | 2000k |
| Labeled Clip Count | 22062 | 18093 | 18600 | 11370 | 7530 | 2490 | 930 | 930 | 80k |
| Raw Data Size | 60GB | 2TB | 450GB | 1.5TB | 5TB | 1TB | 180GB | 18GB | 10TB |
| Labeled Data Size | 1GB | 5GB | 12GB | 10GB | 7GB | 7GB | 1GB | 0.5GB | 43.5GB |
2.1 Overview
The authors start with a claimed pool of over two million clips and more than 10 TB of data, then select roughly 80,000 examples. Table 1 names Mapillary, ONCE, NAVSIM, nuScenes, Waymo, Argoverse-V2, KITTI and IDD as the eight sources. References to nuPlan in the broader background should not be read as an additional ninth row of the constructed dataset.
2.2 Defining a Taxonomy for Unstructured Driving Scenarios
Approximately 10% of the aggregated data is sampled systematically for open-ended descriptions from Qwen2.5-VL-72B [3]. A second VLM classification step filters conventional scenes. Its prompt is refined against around 1,000 descriptions labeled independently by two humans and reconciled to consensus. The paper claims high agreement but gives no numerical agreement rate or held-out prompt-selection test.
Semantic grouping of the unusual descriptions, followed by manual consolidation, yields four categories: unclear road boundaries; temporary traffic-rule changes; unconventional dynamic obstacles; and difficult road surfaces or weather. These categories mix infrastructure, actor behavior and visibility. They are a practical curation taxonomy, not a demonstrated exhaustive or mutually exclusive ontology of driving risk.
2.3 Data Processing and Annotation
Sequences are aligned to 2 Hz. Following NAVSIM [14], a keyclip contains 1.5 seconds of past context and five seconds of future trajectory. Temporary packs of up to 15 seconds check that a scene characteristic persists across more than a single keyclip. The paper does not give a precise universal persistence threshold; the released unit remains an individually annotated keyclip.
Hierarchical prompting describes the scene, static road features and moving actors before assigning a category. The seven annotation tasks cover scene description, traffic lights, vulnerable road users, other actors’ motion intentions, ego meta-actions, planning explanations and future ego trajectory. Road-user annotations rely on existing ground truth, trajectories use recorded vehicle motion, and several semantic tasks use the VLM. These label sources have different reliability and should not be treated as uniformly measured ground truth.
Annotators reportedly review every keyclip and label, accepting, rejecting or making small corrections. A separate 200-image nuScenes check reports classification F1 of 0.90 for temporary rules, 0.81 for unusual dynamic obstacles and 0.91 for difficult road conditions. Unclear boundaries are too rare in that sample for a meaningful F1. This small classification check does not quantify every QA task’s accuracy across all eight sources.
2.4 Dataset Statistics
The dataset is split 80:20 by clips, stratified by scenario category. Episode-, drive- and source-disjoint splitting is not specified, so independence of nearby clips and separation from downstream nuScenes evaluation cannot be confirmed from the paper alone. That is a reporting gap, not evidence that leakage occurred.
Table 1 reports 22,062 Mapillary, 18,093 ONCE, 18,600 NAVSIM, 11,370 nuScenes, 7,530 Waymo, 2,490 Argoverse-V2, and 930 each KITTI and IDD clips. These entries sum to 82,005, consistent with an approximate 80K description but not an exact total. The raw-count rows sum to 2.317 million rather than the rounded two million, while the labeled-size entries sum to 43.5 GB. Mapillary’s 2 Hz rate is explicitly assumed because FPS is unavailable; it should not be interpreted as a verified native capture rate. Figure 2 retains source/category imbalance and the absence of IDD examples in the challenging-road category.
3 Experiments
Evaluation asks whether the curated data improves existing driving benchmarks and whether its QA validation set reveals capability changes. All comparisons and borrowed baseline values remain in their original tables.
| Source | Method | NeuroNCAP Score | Collision rate (%) | ||||||
| Avg. | Stat. | Frontal | Side | Avg. | Stat. | Frontal | Side | ||
| CVPR 2023 | UniAD2 | 0.73 | 0.84 | 0.10 | 1.26 | 88.6 | 87.8 | 98.4 | 79.6 |
| ICCV 2023 | VAD2 | 0.66 | 0.47 | 0.04 | 1.45 | 92.5 | 96.2 | 99.6 | 81.6 |
| ICRA 2025 | SparseDrive1 | 0.92 | - | - | - | 93.9 | - | - | - |
| CVPR 2025 | BridgeAD-S1 | 1.52 | - | - | - | 76.2 | - | - | - |
| CVPR 2025 | BridgeAD-B1 | 1.60 | - | - | - | 72.6 | - | - | - |
| - | Base+nuScenes | 1.77 | 1.80 | 1.67 | 1.75 | 72.5 | 68.0 | 73.0 | 71.5 |
| - | Base+Impromptu+nuScenes | 2.15 | 1.77 | 2.31 | 2.10 | 65.5 | 70.0 | 59.0 | 65.0 |
3.1 Pushing Forward Boundaries of Existing End-to-end Autonomous Driving Benchmarks
NeuroNCAP [41] uses simulated driving scenes based on nuScenes. Its score is 5 when no collision occurs; otherwise it is 4·max(0, 1 − vᵢ/vᵣ), where vᵢ is actual relative impact speed and vᵣ the reference impact speed without avoidance. Higher is better; collision rate independently records how often a collision occurs. Neither metric is a real-road safety certification.
For the 3B model, Impromptu pretraining followed by nuScenes fine-tuning raises average NeuroNCAP score from 1.77 to 2.15 and lowers collision rate from 72.5% to 65.5%, a seven-percentage-point reduction. Improvement is uneven: static collisions rise from 68.0% to 70.0%, while frontal collisions fall from 73.0% to 59.0% and side collisions from 71.5% to 65.0%. Static score also slips from 1.80 to 1.77. The substantial remaining collision rate and these regressions qualify the overall improvement.
Open-loop nuScenes [7] measures distance from a recorded future trajectory, not successful closed-loop interaction. The 3B average L2 error falls from 0.34 m to 0.30 m and the 7B model from 0.32 m to 0.30 m over one-, two- and three-second horizons. EMMA+ [25] reports 0.29 m with a different, much larger proprietary training setup. Thus “near state of the art” is a narrow numerical comparison in this table, not a compute- or data-matched ranking. The HTML labels this open-loop table as Figure 4; it is Table 3 in the PDF, shifting subsequent figure/table numbering.
| Method | L2 Error (m) | |||
| 1s | 2s | 3s | Avg. | |
| Closed-source API-only Models | ||||
| GPT-4o1 [24] | 0.28 | 0.93 | 2.02 | 1.07 |
| Claude-3.5-Sonnet1 | 0.29 | 0.98 | 2.12 | 1.13 |
| Claude-3.7-Sonnet1 | 0.28 | 0.94 | 2.04 | 1.09 |
| Gemini-2.0-Flash1 | 0.31 | 1.08 | 2.36 | 1.25 |
| Gemini-2.5-Pro1 | 0.37 | 1.35 | 2.96 | 1.56 |
| Open-source Generalist VLMs | ||||
| LLaVA-1.6-Mistral-7B2 | 1.49 | 3.38 | 4.09 | 2.98 |
| Llama-3.2-11B-Vision-Instruct2 | 1.54 | 3.31 | 3.91 | 2.92 |
| Qwen2-VL-7B-Instruct2 [80] | 1.45 | 3.21 | 3.76 | 2.81 |
| DeepSeek-VL2-16B1 [71] | 0.66 | 1.68 | 2.92 | 1.75 |
| DeepSeek-VL2-28B1 [71] | 0.37 | 1.35 | 2.96 | 1.56 |
| LLaMA-3.2-11B-Vision-Instruct1 | 0.52 | 1.42 | 2.68 | 1.54 |
| LLaMA-3.2-90B-Vision-Instruct1 | 0.66 | 1.71 | 3.01 | 1.79 |
| Qwen-2.5-VL-7B-Instruct1 [4] | 0.46 | 1.33 | 2.55 | 1.45 |
| Training-based Driving Specialists (Existing Methods) | ||||
| UniAD3 [23] | 0.42 | 0.64 | 0.91 | 0.66 |
| VAD3 [29] | 0.17 | 0.34 | 0.60 | 0.37 |
| BEV-Planner3 [37] | 0.16 | 0.32 | 0.57 | 0.35 |
| Ego-MLP3* [37] | 0.15 | 0.32 | 0.59 | 0.35 |
| Ours and Key Competitors (Specialized Driving Models) | ||||
| DriveVLM3 [58] | 0.18 | 0.34 | 0.68 | 0.40 |
| OmniDrive3 [61] | 0.14 | 0.29 | 0.55 | 0.33 |
| DriveVLM-Dual3 [58] | 0.15 | 0.29 | 0.48 | 0.31 |
| EMMA (random init) [25]3 | 0.15 | 0.33 | 0.63 | 0.37 |
| EMMA [25]3 | 0.14 | 0.29 | 0.54 | 0.32 |
| EMMA+3 [25] | 0.13 | 0.27 | 0.48 | 0.29 |
| 3B Base+nuScenes | 0.14 | 0.30 | 0.58 | 0.34 |
| 3B Base+Impromptu+nuScenes | 0.13 | 0.27 | 0.52 | 0.30 |
| 7B Base+nuScenes | 0.13 | 0.28 | 0.55 | 0.32 |
| 7B Base+Impromptu+nuScenes | 0.13 | 0.27 | 0.53 | 0.30 |
3.2 Diagnostic Evaluation of VLM Capabilities on Impromptu VLA
On Impromptu’s validation split, 3B road-user and traffic-light accuracies rise from 0.87/0.95 to 0.91/0.96, dynamic-object prediction from 0.20 to 0.92, and meta-planning from 0.56 to 0.84. Mean one-to-four-second trajectory error falls from 6.62 m to 0.69 m. For 7B, the corresponding accuracies become 0.91/0.97/0.92/0.83 and trajectory error falls from 5.74 m to 0.54 m.
These large gains compare the base model with a model trained for the dataset’s task formats. They demonstrate adaptation but do not isolate semantic understanding from format learning, category selection or label-model alignment. The caption incorrectly groups planning L2 with upward accuracy metrics; the actual trajectory column is an error and lower is better. No uncertainty intervals, repeated-run variation or detailed per-category diagnostics accompany these aggregate results.
| Method | Q&A Accuracy | Traj. Pred. L2 Error (m) | |||||||
| V.R.U. | T. Light | Dyn. Obj. | M.P. | 1s | 2s | 3s | 4s | Avg. | |
| 3B Base | 0.87 | 0.95 | 0.20 | 0.56 | 3.39 | 5.31 | 7.70 | 10.08 | 6.62 |
| 3B Base+Impromptu | 0.91 | 0.96 | 0.92 | 0.84 | 0.16 | 0.43 | 0.82 | 1.34 | 0.69 |
| 7B Base | 0.86 | 0.92 | 0.22 | 0.55 | 2.99 | 4.80 | 6.64 | 8.52 | 5.74 |
| 7B Base+Impromptu | 0.91 | 0.97 | 0.92 | 0.83 | 0.10 | 0.31 | 0.65 | 1.11 | 0.54 |
4 Related Works
The work sits between vision-language driving models such as DriveVLM [58] and Senna [28], QA driving benchmarks, and large imitation-learning datasets. It also relates to simulation, adversarial scene generation, mapping and trajectory forecasting. Its distinguishing choice is to curate naturally occurring difficult scenes and attach multiple QA/planning tasks, rather than introduce a new simulator or specialized motion-planning architecture. Background examples establish context; they are not all experimental baselines.
5 Conclusion
The evidence supports curated difficult-scene data as useful additional training for the tested Qwen-based driving pipeline. The authors explicitly acknowledge bias from using Qwen2.5-VL to generate labels. Human verification reduces but does not quantify all such bias, especially when training/evaluation use the same model family. Unreported split isolation, annotation agreement, learning budgets and uncertainty, source imbalance, possible annotation inconsistencies, and high remaining collision rates limit stronger conclusions.
Supplementary Material
The supplement specifies training formats and a small hyperparameter table, gives the scene-classification prompt, and preserves concrete QA records from each source dataset. These records show both the intended task structure and its limitations.
Appendix A Implementation Details
The implementation uses full fine-tuning of Qwen2.5-VL variants. The stated configuration applies across the pipeline; exact dataset-stage training budgets are not supplied.
A.1 Data format for nuScenes Fine-tuning and Evaluation
nuScenes training/evaluation uses the front camera and three seconds each of past and future trajectory. This differs from Impromptu’s 1.5-second past/five-second future convention. Past ego trajectory points are training inputs; evaluation also incorporates velocity, acceleration and steering where available. The example distinguishes the train and test prompt formats, but the paper does not fully analyze the effect of the changed state inputs.
A.2 Hyperparameters
The shared recipe gives a 4,096-token cutoff, full fine-tuning, image-resolution setting 262,144, learning rate 5×10⁻⁶, cosine schedule and 0.03 warmup ratio. These are empirical settings, not an exhaustive search. Batch size, epoch/update counts, optimizer details and per-stage compute are absent, limiting exact reproduction and compute-matched interpretation of the added pretraining benefit.
| Hyperparameter | Value |
| Cutoff len | 4096 |
| Finetuning type | full |
| Image resolution | 262144 |
| Learning rate | 5.0e-06 |
| Scheduler type | cosine |
| Warmup ratio | 0.03 |
Appendix B VLM-based Unstructured Scene Identification
The original classification prompt asks for exactly one word, “Conventional” or “Unconventional.” It checks unexpected events, severe weather, degraded or ambiguous roads, atypical road-user behavior and complex situations requiring special intervention. This broad gate precedes category refinement. The approximately 1,000-description human comparison guides iterative prompt changes; no separate agreement result is reported. The complete prompt is restored from the PDF because its HTML object was not represented by the reader’s normal image inventory.
Appendix C Q&A Examples
The appendix gives two nuScenes examples and paired examples for NAVSIM, Argoverse-V2, ONCE, KITTI, Mapillary, Waymo and IDD. Task tags such as <PLANNING> and <DYNAMIC_OBJECTS> separate similar-looking trajectory and motion-intention outputs. The authors say disambiguation prevents task confusion but provide no quantitative token-ablation result.
Not every source supplies every task: road-user and actor labels depend on the source ground truth. Trajectory prompts specify past states and future coordinates, while descriptive and planning labels are language-generated. Some examples visibly warrant caution—for instance, the first NAVSIM planning explanation says a red traffic light is ahead while its traffic-light answer is “None.” The Mapillary records also show numeric trajectories despite the assumed frame rate and incomplete provenance explanation. These exact examples are retained as source evidence; their presence does not establish that every annotation was independently correct.








