At a glance
This workshop position paper maps opportunities and difficulties in bringing multimodal language models, predictive world models, preference alignment, and state-space architectures into autonomous driving. It argues for critical integration of these tools, including their possible negative effects, rather than treating language models as a sufficient driving solution.
The contribution is a research agenda and taxonomy, with two descriptive figures and two method-comparison tables. It does not introduce a driving system, controlled benchmark comparison, or new deployment experiment. This edition covers the fixed WACV 2025 proceedings version and its separate supplementary preliminaries; the authors describe the broader project as a living document.
1. Introduction
Earlier surveys catalogued models and datasets; this position paper emphasizes integration choices, limitations, and open questions [33] [185]. Figure 1 illustrates increasing use of selected research keywords, while Figure 2 organizes the agenda. Keyword growth is evidence of attention to a topic, not evidence of safer driving.
The discussion grew out of the LLVM-AD workshop series at WACV 2024, ITSC 2024, and WACV 2025. Detailed implementations are outside its stated scope.
2. Multimodal LLMs in autonomous driving
Two roles are distinguished: interpreting and reasoning about multimodal driving information, and participating in an end-to-end sensor-to-action pipeline. These roles impose different supervision, latency, and validation demands.
2.1. Multimodal understanding and reasoning
Inputs may include images, navigation instructions, vehicle state, and other sensor representations; outputs may be explanations, instructions, actions, or control values. Monolithic models combine perception and language in one network. Compositional systems instead decompose tasks and retrieve external knowledge or prior experiences.
DiLu, Language Agent, and RAG-Driver illustrate the latter direction [166] [116] [190]. Decomposition is presented as a route toward interpretability and reduced data or compute requirements, without a new controlled comparison establishing those benefits here.
2.2. End-to-end autonomous driving
Modular systems separate perception, prediction, and planning. End-to-end systems jointly learn plans or actions from sensor inputs, but imitation of recorded trajectories may not capture the reasoning required by unusual situations. Multimodal foundation models are proposed as a source of semantic knowledge and reasoning capabilities.
Table 1 retains all seventeen systems, their adaptation methods, inputs, outputs, and evaluation environments. Discrete actions, numerical control signals, natural-language outputs, and trajectories are distinct output types; rows are not directly comparable performance results. The text highlights EMMA and other recent systems while acknowledging substantial driving-specific finetuning, compute cost, and latency.
3. World models in autonomous driving
The paper uses “world model” broadly: an internal representation useful for decisions, or an explicit predictor of future observations or states. Explicit prediction supports scenario generation and closed-loop simulation. Table 2 classifies sixteen systems by output representation: video, 3D geometry or occupancy, and multiple modalities.
This is a taxonomy of methods with different inputs and supervision, not an evaluation establishing a best representation. Visual realism alone does not establish reliable physical prediction or safe control.
3.1. Video-based world models
GAIA-1 combines autoregressive token prediction with video diffusion. ADriver-I couples image-text processing to future frames; DriveDreamer uses structured conditioning, and DriveDreamer-2 translates language into trajectories and map information. Drive-WM generates multiple views and supports planning, while Vista emphasizes generalization and controllability [66] [51]. These are summaries of cited work rather than reproduced experiments.
3.2. 3D world models
Point-cloud predictors include Copilot4D, which uses discrete diffusion, and ViDAR, which predicts geometry from historical multi-view images. Occupancy models represent free or occupied space, sometimes with semantics: UniWorld predicts binary occupancy; OccWorld predicts semantic occupancy and ego motion; DFIT-OccWorld and DOME explore alternative temporal decoders.
DriveWorld and Drive-OccWorld use past camera views, while RenderWorld uses rendered 3D Gaussian representations and 2D supervision. These distinctions matter: access to historical occupancy, raw point clouds, and camera images implies different sensing and annotation requirements.
3.3. Multimodal world models
MUVO learns from cameras and LiDAR and predicts images, point clouds, and occupancy conditioned on actions. BEVWorld tokenizes camera and LiDAR information into a shared bird’s-eye-view latent space and predicts future sensor observations with diffusion. Predicting several modalities is proposed to strengthen the encoder’s environmental representation.
4. Trustworthiness in autonomous driving
The agenda includes alignment with human intentions, robustness under changing conditions, security and privacy, and fairness. The paper surveys possible techniques and threats; it does not certify a deployed vehicle or demonstrate that any one technique resolves them.
4.1. AI alignment
Supervised finetuning teaches desired responses. RLHF adds preference modeling and policy optimization; direct preference optimization and ranking objectives provide alternative training procedures. Human, AI-generated, or combined feedback can supply preferences [133].
Alignment must also withstand spurious correlations, unfamiliar situations, and distribution shifts caused by an agent’s own actions. Invariance, adversarial augmentation, and risk extrapolation are discussed as candidate approaches. Language-model judges are proposed for evaluating rare scenarios and rule adherence, but no driving-specific judge calibration or safety guarantee is established.
4.2. Security and privacy
The threat inventory covers adversarial sensor perturbations, poisoned training data and backdoors, indirect prompt injection, and jailbreaks. Multimodal interactions can create attack paths across text and vision. Privacy concerns include information inferred about training membership or recovered from models.
The proposed responses include more robust training, anomaly detection, cross-modal defenses, differential privacy, federated learning, and explicit security benchmarks. These remain research directions; defenses effective against one threat or modality need not generalize to another.
4.3. Fairness
Personalized driver profiles can reproduce biases, while biased perception may misclassify or inadequately prioritize particular road users. The authors call for interpretable decisions and broader risk evaluation. The paper does not supply subgroup measurements or establish the fairness of the surveyed systems.
5. Emerging foundation models
Transformers capture global context, but their inference cost motivates alternative sequence architectures. State-space models and Mamba are considered for language, vision, point clouds, and multi-sensor processing.
State-space models and Mamba
Mamba introduces input-dependent selection and efficient scanning to model long sequences with linear scaling in sequence length [53]. This architectural property is distinct from measured end-to-end vehicle latency, which also depends on sensor processing, model size, and hardware support.
Mamba for language
MambaByte operates on byte sequences, while Jamba and BlackMamba combine state-space processing with mixture-of-experts designs. The paper presents them as alternatives for efficient language backbones, not autonomous-driving evaluations.
Mamba for vision
Image patches require spatial structure that a single directional scan does not automatically preserve. Vision Mamba uses bidirectional processing and position embeddings; VMamba adds two-dimensional selective scanning. VideoMamba and Vivim extend scanning across space and time to capture video dependencies.
Mamba for autonomous driving
PointMamba and Point Cloud Mamba tokenize irregular, sparse point clouds before sequence modeling. CoMamba applies state-space processing to cooperative 3D perception; hierarchical Mamba adaptation is cited for multimodal driving-video understanding. These task-specific results do not amount to a unified driving-system comparison in this paper.
6. Deployment and difficulty
Deployment is treated as a system problem spanning compute, timing, verification, memory, and lifecycle cost. References to commercial and research vehicles are contextual examples, not new deployment evidence collected by the authors.
Computational constraints
Automotive processors have less capacity than development hardware and may favor different operations. Efficient selective scans on a development GPU need not map efficiently to an automotive neural processing unit optimized for convolutions. Distillation and training-time transfer into smaller driving networks are suggested ways to use foundation-model knowledge within device limits.
Real-time performance
The authors give illustrative requirements below 100 ms for scene understanding and below 50 ms for critical decisions, contrasting these with language-model reasoning that may take seconds. These are stated deployment targets, not measured performance achieved by a proposed system. Consistency under changing computational load matters as well as average latency.
Validation and safety assurance
Validation should include rare events, degraded sensing, fallback behavior, and traceable decisions. Alignment methods and natural-language explanations do not substitute for testing actual vehicle behavior across operating conditions.
Memory and bandwidth
Limited onboard memory and bandwidth constrain both model storage and movement of multimodal data between processors. Compute optimizations must therefore be assessed alongside data-transfer costs.
Deployment costs
Hardware investment, predeployment validation, maintenance, calibration, upgrades, and continuing software updates all contribute to cost. Updated models, security patches, and added features require continued checks of system behavior.
7. Discussion
The paper identifies three broader directions: personalized interaction, embodied reasoning grounded in environments, and cooperation among vehicles. All introduce additional deployment and evaluation questions.
7.1. Human-centric autonomous driving
Language interaction may express passenger preferences and explain actions. Retrieval, feedback learning, edge-cloud cooperation, and federated learning are proposed for personalization. Preference adaptation must be considered together with the profiling bias and privacy concerns discussed earlier.
7.2. Embodied intelligence
DriVLMe adds multi-turn dialogue to driving assistance; LeapAD combines complementary decision processes with an experience memory. Richer simulated urban environments are proposed for testing adaptive agents. These examples motivate grounded interaction but do not establish reliable reasoning in unrestricted real traffic.
7.3. Cooperative driving
Connected vehicles can exchange observations, plans, and actions through V2V or V2X communication. Language-based multi-agent frameworks may improve interpretability and coordination, but scalability, communication security, and real-world implementation remain open.
8. Third LLVM-AD workshop
The third LLVM-AD workshop accepted ten papers spanning reasoning, novel-object detection, 3D representations, safety-event analysis, and multimodal driving. The closing outlook is optimistic about foundation models and embodied AI, while the paper’s own contribution remains an agenda rather than an experimentally validated integration recipe.
Appendix A. Preliminaries
The official one-page supplement supplies background for the main taxonomy. Its three subsections are included here; it contains no additional figures, tables, or displayed equations.
A.1. Multimodal LLMs
Multimodal language models connect pretrained visual and language representations through alignment, instruction tuning, or small trainable adapters. Examples distinguish fully tuned systems from frozen backbones joined by a projection layer or instruction-aware visual queries. The supplement lists both closed and open model families as context for the driving discussion.
A.2. AI alignment
Forward alignment shapes model behavior during development through preferences, feedback, and robustness to distribution shifts. Backward alignment evaluates and governs behavior through safety assessments, interpretability, human-value verification, and continuing oversight after deployment. These are complementary lifecycle activities.
A.3. State-space models and Mamba
The state-space background maps a scalar input sequence x(t) to y(t) through an N-dimensional hidden state h(t). Structured state-space models discretize continuous dynamics and structure the transition matrix; Mamba adds selective updates and a parallel scanning implementation.
Bidirectional and two-dimensional scans adapt the idea to vision. Multimodal fusion models, including Cobra and VL-Mamba, extend it further. Their linear sequence scaling motivates research into faster driving perception, with hardware and full-system timing still requiring separate validation.