Literature
Display
← All papers

Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

Haohan Chi · Huan-ang Gao · Ziming Liu · Jianing Liu · Chenyu Liu · Jinwei Li · Kaisen Yang · Yangcheng Yu · Zeda Wang · Wenyi Li · Leichen Wang · Xingtao Hu · Hao Sun · Hang Zhao · Hao Zhao

· NeurIPS 2025 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

The visual abstract links four types of difficult driving scenes with the reported benchmark gains. All scientific figures, original annotation examples, the classification prompt, five data/result/configuration tables and the complete reference list are retained.

Refer to caption
Figure 1: Visual Abstract of Impromptu VLA. We construct Impromptu VLA Dataset, which contains over 80K clips curated from 8 open-sourced datasets, focusing on four critical types of unstructured "corner case" scenarios that challenge current autonomous driving vehicles. It supports interconnected VLA tasks including scene understanding, prediction, meta planning and trajectory planning. Key experimental results demonstrates that VLA models trained with Impromptu VLA Dataset achieve significant performance improvements in both closed-loop and open-loop metrics.

At a glance

Impromptu VLA curates approximately 80,000 difficult driving clips from eight public datasets, using a vision-language model to identify scenarios and generate task annotations followed by human verification. Pretraining Qwen2.5-VL on these examples before nuScenes fine-tuning improves simulated closed-loop driving and open-loop trajectory error. The resource targets unstructured roads and unusual situations; the results do not establish safe real-world autonomous driving.

1 Introduction

Standard driving datasets emphasize common structured roads. The authors instead target ambiguous road boundaries, temporary traffic rules, unusual dynamic obstacles and challenging road conditions. Their contribution combines a taxonomy, curated clips, question-answer annotations and action trajectories, with model weights and data intended for open release. The study tests both downstream driving benchmarks and whether its own QA tasks diagnose perception, prediction and planning.

The strongest controlled comparison is between a Qwen model fine-tuned directly on nuScenes and one additionally trained on Impromptu VLA first. That comparison establishes the usefulness of the added training recipe; it does not separate dataset selection from additional training compute or demonstrate superiority to every driving specialist.

2 Impromptu VLA Dataset: Learning to Drive on Unstructured Roads

The dataset combines existing real-world driving imagery rather than generating synthetic corner cases. Its automated selection and annotation pipeline is followed by human checks, with task availability varying by the source annotations.

Refer to caption
Figure 2: Characteristics comparison of different driving scene datasets. Figure (a) illustrates the distribution of scene categories across various datasets and the number of video clips contained in each dataset, providing a direct view of the emphasis on different scene types and the data scale of each dataset. Figure (b) compares the trajectory distribution in the original with the trajectory distribution in our constructed dataset, explaining the trajectory diversity of our dataset. Figure (c) shows examples of different scene categories from 8 source datasets. Notably, the IDD dataset lacks data for the "Challenging Road Conditions" category.
Attribute Mapillary ONCE NAVSIM nuScenes Waymo Argoverse-V2 KITTI IDD Sum.
Camera Views 1 7 8 6 5 7 1 1 -
Resolutions Variable 1920 x 1020 1920 x 1080 1600 x 900 640 x 360 1550 x 2048 1242 x 375 1920 x 1080 -
FPS 2Hz* 2Hz 2Hz 2Hz 10Hz 20Hz 10Hz 15Hz
Raw Clip Count 1000k 800k 90k 40k 40k 320k 20k 7k 2000k
Labeled Clip Count 22062 18093 18600 11370 7530 2490 930 930 80k
Raw Data Size 60GB 2TB 450GB 1.5TB 5TB 1TB 180GB 18GB 10TB
Labeled Data Size 1GB 5GB 12GB 10GB 7GB 7GB 1GB 0.5GB 43.5GB
Table 1: Dataset Information. Summary of key attributes for the datasets used in this study. Note that the Mapillary dataset exhibits variable resolutions. For the Mapillary dataset, the frequency is assumed to be 2Hz as specific FPS information is not explicitly provided.

2.1 Overview

The authors start with a claimed pool of over two million clips and more than 10 TB of data, then select roughly 80,000 examples. Table 1 names Mapillary, ONCE, NAVSIM, nuScenes, Waymo, Argoverse-V2, KITTI and IDD as the eight sources. References to nuPlan in the broader background should not be read as an additional ninth row of the constructed dataset.

Refer to caption
Figure 3: Data Processing and Annotation Pipeline for the Impromptu VLA Dataset. The diagram outlines the sequential process for creating our dataset, starting from raw data collection and scenario taxonomy definition (Sec. 2.2, through frequency alignment and keyclip selection, to multi-task annotation generation via Qwen2.5-VL (including scene description, object/feature analysis, and labeling), and concluding with rigorous human verification (Sec. 2.3).

2.2 Defining a Taxonomy for Unstructured Driving Scenarios

Approximately 10% of the aggregated data is sampled systematically for open-ended descriptions from Qwen2.5-VL-72B [3]. A second VLM classification step filters conventional scenes. Its prompt is refined against around 1,000 descriptions labeled independently by two humans and reconciled to consensus. The paper claims high agreement but gives no numerical agreement rate or held-out prompt-selection test.

Semantic grouping of the unusual descriptions, followed by manual consolidation, yields four categories: unclear road boundaries; temporary traffic-rule changes; unconventional dynamic obstacles; and difficult road surfaces or weather. These categories mix infrastructure, actor behavior and visibility. They are a practical curation taxonomy, not a demonstrated exhaustive or mutually exclusive ontology of driving risk.

2.3 Data Processing and Annotation

Sequences are aligned to 2 Hz. Following NAVSIM [14], a keyclip contains 1.5 seconds of past context and five seconds of future trajectory. Temporary packs of up to 15 seconds check that a scene characteristic persists across more than a single keyclip. The paper does not give a precise universal persistence threshold; the released unit remains an individually annotated keyclip.

Hierarchical prompting describes the scene, static road features and moving actors before assigning a category. The seven annotation tasks cover scene description, traffic lights, vulnerable road users, other actors’ motion intentions, ego meta-actions, planning explanations and future ego trajectory. Road-user annotations rely on existing ground truth, trajectories use recorded vehicle motion, and several semantic tasks use the VLM. These label sources have different reliability and should not be treated as uniformly measured ground truth.

Annotators reportedly review every keyclip and label, accepting, rejecting or making small corrections. A separate 200-image nuScenes check reports classification F1 of 0.90 for temporary rules, 0.81 for unusual dynamic obstacles and 0.91 for difficult road conditions. Unclear boundaries are too rare in that sample for a meaningful F1. This small classification check does not quantify every QA task’s accuracy across all eight sources.

2.4 Dataset Statistics

The dataset is split 80:20 by clips, stratified by scenario category. Episode-, drive- and source-disjoint splitting is not specified, so independence of nearby clips and separation from downstream nuScenes evaluation cannot be confirmed from the paper alone. That is a reporting gap, not evidence that leakage occurred.

Table 1 reports 22,062 Mapillary, 18,093 ONCE, 18,600 NAVSIM, 11,370 nuScenes, 7,530 Waymo, 2,490 Argoverse-V2, and 930 each KITTI and IDD clips. These entries sum to 82,005, consistent with an approximate 80K description but not an exact total. The raw-count rows sum to 2.317 million rather than the rounded two million, while the labeled-size entries sum to 43.5 GB. Mapillary’s 2 Hz rate is explicitly assumed because FPS is unavailable; it should not be interpreted as a verified native capture rate. Figure 2 retains source/category imbalance and the absence of IDD examples in the challenging-road category.

3 Experiments

Evaluation asks whether the curated data improves existing driving benchmarks and whether its QA validation set reveals capability changes. All comparisons and borrowed baseline values remain in their original tables.

Source Method NeuroNCAP Score ↑\uparrow Collision rate (%) ↓\downarrow
Avg. Stat. Frontal Side Avg. Stat. Frontal Side
CVPR 2023 UniAD2 0.73 0.84 0.10 1.26 88.6 87.8 98.4 79.6
ICCV 2023 VAD2 0.66 0.47 0.04 1.45 92.5 96.2 99.6 81.6
ICRA 2025 SparseDrive1 0.92 - - - 93.9 - - -
CVPR 2025 BridgeAD-S1 1.52 - - - 76.2 - - -
CVPR 2025 BridgeAD-B1 1.60 - - - 72.6 - - -
- Base+nuScenes 1.77 1.80 1.67 1.75 72.5 68.0 73.0 71.5
- Base+Impromptu+nuScenes 2.15 1.77 2.31 2.10 65.5 70.0 59.0 65.0
Table 2: Results on NeuroNCAP. (where 1 indicates sourced from [81] and 2 indicates sourced from [40]) Best scores in each category (without/with post-processing) are in bold, second best are underlined. The improvements in both the overall NeuroNCAP score and, crucially, the reduction in collision rates suggest that our dataset helps the model develop a more nuanced understanding of complex road interactions, leading to more robust and safer driving policies.

3.1 Pushing Forward Boundaries of Existing End-to-end Autonomous Driving Benchmarks

NeuroNCAP [41] uses simulated driving scenes based on nuScenes. Its score is 5 when no collision occurs; otherwise it is 4·max(0, 1 − vᵢ/vᵣ), where vᵢ is actual relative impact speed and vᵣ the reference impact speed without avoidance. Higher is better; collision rate independently records how often a collision occurs. Neither metric is a real-road safety certification.

For the 3B model, Impromptu pretraining followed by nuScenes fine-tuning raises average NeuroNCAP score from 1.77 to 2.15 and lowers collision rate from 72.5% to 65.5%, a seven-percentage-point reduction. Improvement is uneven: static collisions rise from 68.0% to 70.0%, while frontal collisions fall from 73.0% to 59.0% and side collisions from 71.5% to 65.0%. Static score also slips from 1.80 to 1.77. The substantial remaining collision rate and these regressions qualify the overall improvement.

Open-loop nuScenes [7] measures distance from a recorded future trajectory, not successful closed-loop interaction. The 3B average L2 error falls from 0.34 m to 0.30 m and the 7B model from 0.32 m to 0.30 m over one-, two- and three-second horizons. EMMA+ [25] reports 0.29 m with a different, much larger proprietary training setup. Thus “near state of the art” is a narrow numerical comparison in this table, not a compute- or data-matched ranking. The HTML labels this open-loop table as Figure 4; it is Table 3 in the PDF, shifting subsequent figure/table numbering.

Method L2 Error (m) ↓\downarrow
1s 2s 3s Avg.
Closed-source API-only Models
GPT-4o1 [24] 0.28 0.93 2.02 1.07
Claude-3.5-Sonnet1 0.29 0.98 2.12 1.13
Claude-3.7-Sonnet1 0.28 0.94 2.04 1.09
Gemini-2.0-Flash1 0.31 1.08 2.36 1.25
Gemini-2.5-Pro1 0.37 1.35 2.96 1.56
Open-source Generalist VLMs
LLaVA-1.6-Mistral-7B2 1.49 3.38 4.09 2.98
Llama-3.2-11B-Vision-Instruct2 1.54 3.31 3.91 2.92
Qwen2-VL-7B-Instruct2 [80] 1.45 3.21 3.76 2.81
DeepSeek-VL2-16B1 [71] 0.66 1.68 2.92 1.75
DeepSeek-VL2-28B1 [71] 0.37 1.35 2.96 1.56
LLaMA-3.2-11B-Vision-Instruct1 0.52 1.42 2.68 1.54
LLaMA-3.2-90B-Vision-Instruct1 0.66 1.71 3.01 1.79
Qwen-2.5-VL-7B-Instruct1 [4] 0.46 1.33 2.55 1.45
Training-based Driving Specialists (Existing Methods)
UniAD3 [23] 0.42 0.64 0.91 0.66
VAD3 [29] 0.17 0.34 0.60 0.37
BEV-Planner3 [37] 0.16 0.32 0.57 0.35
Ego-MLP3* [37] 0.15 0.32 0.59 0.35
Ours and Key Competitors (Specialized Driving Models)
DriveVLM3 [58] 0.18 0.34 0.68 0.40
OmniDrive3 [61] 0.14 0.29 0.55 0.33
DriveVLM-Dual3 [58] 0.15 0.29 0.48 0.31
EMMA (random init) [25]3 0.15 0.33 0.63 0.37
EMMA [25]3 0.14 0.29 0.54 0.32
EMMA+3 [25] 0.13 0.27 0.48 0.29
3B Base+nuScenes 0.14 0.30 0.58 0.34
3B Base+Impromptu+nuScenes 0.13 0.27 0.52 0.30
7B Base+nuScenes 0.13 0.28 0.55 0.32
7B Base+Impromptu+nuScenes 0.13 0.27 0.53 0.30
Figure 4: Open-loop trajectory prediction L2 errors (m) on the nuScenes dataset. (where 1 indicates sourced from [48], 2 indicates sourced from [75] and 3 indicates sourced from [25]). Best results within each category are in bold, second best are underlined.

3.2 Diagnostic Evaluation of VLM Capabilities on Impromptu VLA

On Impromptu’s validation split, 3B road-user and traffic-light accuracies rise from 0.87/0.95 to 0.91/0.96, dynamic-object prediction from 0.20 to 0.92, and meta-planning from 0.56 to 0.84. Mean one-to-four-second trajectory error falls from 6.62 m to 0.69 m. For 7B, the corresponding accuracies become 0.91/0.97/0.92/0.83 and trajectory error falls from 5.74 m to 0.54 m.

These large gains compare the base model with a model trained for the dataset’s task formats. They demonstrate adaptation but do not isolate semantic understanding from format learning, category selection or label-model alignment. The caption incorrectly groups planning L2 with upward accuracy metrics; the actual trajectory column is an error and lower is better. No uncertainty intervals, repeated-run variation or detailed per-category diagnostics accompany these aggregate results.

Method Q&A Accuracy ↑\uparrow Traj. Pred. L2 Error (m) ↓\downarrow
V.R.U. T. Light Dyn. Obj. M.P. 1s 2s 3s 4s Avg.
3B Base 0.87 0.95 0.20 0.56 3.39 5.31 7.70 10.08 6.62
3B Base+Impromptu 0.91 0.96 0.92 0.84 0.16 0.43 0.82 1.34 0.69
7B Base 0.86 0.92 0.22 0.55 2.99 4.80 6.64 8.52 5.74
7B Base+Impromptu 0.91 0.97 0.92 0.83 0.10 0.31 0.65 1.11 0.54
Table 3: Quantitative Evaluation on the Impromptu VLA validation set. Performance comparison on various Q&A tasks within our validation set. The table shows metrics for 3B and 7B Qwen2.5-VL models, comparing a Base version against one fine-tuned on Impromptu VLA (’Ours’). Accuracy ↑\uparrow is reported for perception (V.R.U., T. Light), prediction (Dyn. Obj.), meta-planning (M.P.) and Planning (L2). Best results are in bold.
Refer to caption
Figure 5: NeuroNCAP performance in challenging scenarios. This figure compares the driving behavior of the two models in three representative challenging scenarios: static, frontal, and side. For each scenario, the left column shows the behavior of the base model, which is fine-tuned on nuScenes. The right column shows the performance of the model trained on a subset of our proposed dataset and then fine-tuned on nuScenes. Compared to the base model, the model using our data can better avoid vehicles by turning, slowing down, etc.

4 Related Works

The work sits between vision-language driving models such as DriveVLM [58] and Senna [28], QA driving benchmarks, and large imitation-learning datasets. It also relates to simulation, adversarial scene generation, mapping and trajectory forecasting. Its distinguishing choice is to curate naturally occurring difficult scenes and attach multiple QA/planning tasks, rather than introduce a new simulator or specialized motion-planning architecture. Background examples establish context; they are not all experimental baselines.

5 Conclusion

The evidence supports curated difficult-scene data as useful additional training for the tested Qwen-based driving pipeline. The authors explicitly acknowledge bias from using Qwen2.5-VL to generate labels. Human verification reduces but does not quantify all such bias, especially when training/evaluation use the same model family. Unreported split isolation, annotation agreement, learning budgets and uncertainty, source imbalance, possible annotation inconsistencies, and high remaining collision rates limit stronger conclusions.

Supplementary Material

The supplement specifies training formats and a small hyperparameter table, gives the scene-classification prompt, and preserves concrete QA records from each source dataset. These records show both the intended task structure and its limitations.

Appendix A Implementation Details

The implementation uses full fine-tuning of Qwen2.5-VL variants. The stated configuration applies across the pipeline; exact dataset-stage training budgets are not supplied.

A.1 Data format for nuScenes Fine-tuning and Evaluation

nuScenes training/evaluation uses the front camera and three seconds each of past and future trajectory. This differs from Impromptu’s 1.5-second past/five-second future convention. Past ego trajectory points are training inputs; evaluation also incorporates velocity, acceleration and steering where available. The example distinguishes the train and test prompt formats, but the paper does not fully analyze the effect of the changed state inputs.

A.2 Hyperparameters

The shared recipe gives a 4,096-token cutoff, full fine-tuning, image-resolution setting 262,144, learning rate 5×10⁻⁶, cosine schedule and 0.03 warmup ratio. These are empirical settings, not an exhaustive search. Batch size, epoch/update counts, optimizer details and per-stage compute are absent, limiting exact reproduction and compute-matched interpretation of the added pretraining benefit.

Hyperparameter Value
Cutoff len 4096
Finetuning type full
Image resolution 262144
Learning rate 5.0e-06
Scheduler type cosine
Warmup ratio 0.03
Table 4: Hyperparameters used in training. This table details the empirical values for crucial hyperparameters employed across our entire pipeline. These settings were consistently applied in all experiments and were derived from general observations rather than specific task-based tuning.
Refer to caption
Figure 6: Example of data used when training and testing on nuScenes

Appendix B VLM-based Unstructured Scene Identification

The original classification prompt asks for exactly one word, “Conventional” or “Unconventional.” It checks unexpected events, severe weather, degraded or ambiguous roads, atypical road-user behavior and complex situations requiring special intervention. This broad gate precedes category refinement. The approximately 1,000-description human comparison guides iterative prompt changes; no separate agreement result is reported. The complete prompt is restored from the PDF because its HTML object was not represented by the reader’s normal image inventory.

Original PDF figure A2.F7
Figure 7: Prompt used during Unconventional Driving Scenario Identification

Appendix C Q&A Examples

The appendix gives two nuScenes examples and paired examples for NAVSIM, Argoverse-V2, ONCE, KITTI, Mapillary, Waymo and IDD. Task tags such as <PLANNING> and <DYNAMIC_OBJECTS> separate similar-looking trajectory and motion-intention outputs. The authors say disambiguation prevents task confusion but provide no quantitative token-ablation result.

Not every source supplies every task: road-user and actor labels depend on the source ground truth. Trajectory prompts specify past states and future coordinates, while descriptive and planning labels are language-generated. Some examples visibly warrant caution—for instance, the first NAVSIM planning explanation says a red traffic light is ahead while its traffic-light answer is “None.” The Mapillary records also show numeric trajectories despite the assumed frame rate and incomplete provenance explanation. These exact examples are retained as source evidence; their presence does not establish that every annotation was independently correct.

Refer to caption

Figure 8: Q&A example from the nuScenes dataset

Refer to caption

Figure 9: Q&A example from the nuScenes dataset

Refer to caption

Figure 10: Q&A examples from the NAVSIM dataset

Refer to caption

Figure 11: Q&A examples from the Argoverse-V2 dataset

Refer to caption

Figure 12: Q&A examples from the ONCE dataset

Refer to caption

Figure 13: Q&A examples from the KITTI dataset

Refer to caption

Figure 14: Q&A examples from the Mapillary dataset

Refer to caption

Figure 15: Q&A examples from the Waymo dataset

Refer to caption

Figure 16: Q&A examples from the IDD dataset
Concept demonstration

Open image ↗