Literature
Display
← All papers

Position: Prospective of Autonomous Driving - Multimodal LLMs, World Models, Embodied Intelligence, AI Alignment, and Mamba

Yunsheng Ma · Wenqian Ye · Can Cui · Haiming Zhang · Shuo Xing · Fucai Ke · Jinhong Wang · Chenglin Miao · Jintai Chen · Hamid Rezatofighi · Zhen Li · Guangtao Zheng · Chao Zheng · Tianjiao He · Manmohan Chandraker · Burhaneddin Yaman · Xin Ye · Hang Zhao · Xu Cao

· WACV (Workshops) 2025 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

This workshop position paper maps opportunities and difficulties in bringing multimodal language models, predictive world models, preference alignment, and state-space architectures into autonomous driving. It argues for critical integration of these tools, including their possible negative effects, rather than treating language models as a sufficient driving solution.

The contribution is a research agenda and taxonomy, with two descriptive figures and two method-comparison tables. It does not introduce a driving system, controlled benchmark comparison, or new deployment experiment. This edition covers the fixed WACV 2025 proceedings version and its separate supplementary preliminaries; the authors describe the broader project as a living document.

1. Introduction

Earlier surveys catalogued models and datasets; this position paper emphasizes integration choices, limitations, and open questions [33] [185]. Figure 1 illustrates increasing use of selected research keywords, while Figure 2 organizes the agenda. Keyword growth is evidence of attention to a topic, not evidence of safer driving.

The discussion grew out of the LLVM-AD workshop series at WACV 2024, ITSC 2024, and WACV 2025. Detailed implementations are outside its stated scope.

Complete original figure 1
Figure 1. Keywords of these new topics are increasing in the autonomous driving domain.
Complete original figure 2
Figure 2. Different aspects for prospective of autonomous driving.

2. Multimodal LLMs in autonomous driving

Two roles are distinguished: interpreting and reasoning about multimodal driving information, and participating in an end-to-end sensor-to-action pipeline. These roles impose different supervision, latency, and validation demands.

2.1. Multimodal understanding and reasoning

Inputs may include images, navigation instructions, vehicle state, and other sensor representations; outputs may be explanations, instructions, actions, or control values. Monolithic models combine perception and language in one network. Compositional systems instead decompose tasks and retrieve external knowledge or prior experiences.

DiLu, Language Agent, and RAG-Driver illustrate the latter direction [166] [116] [190]. Decomposition is presented as a route toward interpretability and reduced data or compute requirements, without a new controlled comparison establishing those benefits here.

2.2. End-to-end autonomous driving

Modular systems separate perception, prediction, and planning. End-to-end systems jointly learn plans or actions from sensor inputs, but imitation of recorded trajectories may not capture the reasoning required by unusual situations. Multimodal foundation models are proposed as a source of semantic knowledge and reasoning capabilities.

Table 1 retains all seventeen systems, their adaptation methods, inputs, outputs, and evaluation environments. Discrete actions, numerical control signals, natural-language outputs, and trajectories are distinct output types; rows are not directly comparable performance results. The text highlights EMMA and other recent systems while acknowledging substantial driving-specific finetuning, compute cost, and latency.

Complete original table 1
Table 1. Overview of state-of-the-art approaches on the applications of large language models for autonomous driving. RAG: retrievalaugmented generation. IL: imitation learning. RL: reinforcement learning. NL: natural language. Signal: control signals represented as numerical values. Action: a set of predefined discrete actions that correspond to short-term decisions. Human-Like: leverage the reasoning and generalization capabilities of LLMs to enhance the decision-making processes in autonomous vehicles, achieving human-like driving.

3. World models in autonomous driving

The paper uses “world model” broadly: an internal representation useful for decisions, or an explicit predictor of future observations or states. Explicit prediction supports scenario generation and closed-loop simulation. Table 2 classifies sixteen systems by output representation: video, 3D geometry or occupancy, and multiple modalities.

This is a taxonomy of methods with different inputs and supervision, not an evaluation establishing a best representation. Visual realism alone does not establish reliable physical prediction or safe control.

Complete original table 2
Table 2. Summary of works on world models for autonomous driving. This table outlines the distinct characteristics of each method, including their input, output, and key decoder technique. I, Act, PC, Traj, and Occ denote Images, Action, Point Cloud, Trajectory, and Occupancy respectively. MSSM and GRU represent the Memory State Space Model and Gated Recurrent Unit respectively.

3.1. Video-based world models

GAIA-1 combines autoregressive token prediction with video diffusion. ADriver-I couples image-text processing to future frames; DriveDreamer uses structured conditioning, and DriveDreamer-2 translates language into trajectories and map information. Drive-WM generates multiple views and supports planning, while Vista emphasizes generalization and controllability [66] [51]. These are summaries of cited work rather than reproduced experiments.

3.2. 3D world models

Point-cloud predictors include Copilot4D, which uses discrete diffusion, and ViDAR, which predicts geometry from historical multi-view images. Occupancy models represent free or occupied space, sometimes with semantics: UniWorld predicts binary occupancy; OccWorld predicts semantic occupancy and ego motion; DFIT-OccWorld and DOME explore alternative temporal decoders.

DriveWorld and Drive-OccWorld use past camera views, while RenderWorld uses rendered 3D Gaussian representations and 2D supervision. These distinctions matter: access to historical occupancy, raw point clouds, and camera images implies different sensing and annotation requirements.

3.3. Multimodal world models

MUVO learns from cameras and LiDAR and predicts images, point clouds, and occupancy conditioned on actions. BEVWorld tokenizes camera and LiDAR information into a shared bird’s-eye-view latent space and predicts future sensor observations with diffusion. Predicting several modalities is proposed to strengthen the encoder’s environmental representation.

4. Trustworthiness in autonomous driving

The agenda includes alignment with human intentions, robustness under changing conditions, security and privacy, and fairness. The paper surveys possible techniques and threats; it does not certify a deployed vehicle or demonstrate that any one technique resolves them.

4.1. AI alignment

Supervised finetuning teaches desired responses. RLHF adds preference modeling and policy optimization; direct preference optimization and ranking objectives provide alternative training procedures. Human, AI-generated, or combined feedback can supply preferences [133].

Alignment must also withstand spurious correlations, unfamiliar situations, and distribution shifts caused by an agent’s own actions. Invariance, adversarial augmentation, and risk extrapolation are discussed as candidate approaches. Language-model judges are proposed for evaluating rare scenarios and rule adherence, but no driving-specific judge calibration or safety guarantee is established.

4.2. Security and privacy

The threat inventory covers adversarial sensor perturbations, poisoned training data and backdoors, indirect prompt injection, and jailbreaks. Multimodal interactions can create attack paths across text and vision. Privacy concerns include information inferred about training membership or recovered from models.

The proposed responses include more robust training, anomaly detection, cross-modal defenses, differential privacy, federated learning, and explicit security benchmarks. These remain research directions; defenses effective against one threat or modality need not generalize to another.

4.3. Fairness

Personalized driver profiles can reproduce biases, while biased perception may misclassify or inadequately prioritize particular road users. The authors call for interpretable decisions and broader risk evaluation. The paper does not supply subgroup measurements or establish the fairness of the surveyed systems.

5. Emerging foundation models

Transformers capture global context, but their inference cost motivates alternative sequence architectures. State-space models and Mamba are considered for language, vision, point clouds, and multi-sensor processing.

State-space models and Mamba

Mamba introduces input-dependent selection and efficient scanning to model long sequences with linear scaling in sequence length [53]. This architectural property is distinct from measured end-to-end vehicle latency, which also depends on sensor processing, model size, and hardware support.

Mamba for language

MambaByte operates on byte sequences, while Jamba and BlackMamba combine state-space processing with mixture-of-experts designs. The paper presents them as alternatives for efficient language backbones, not autonomous-driving evaluations.

Mamba for vision

Image patches require spatial structure that a single directional scan does not automatically preserve. Vision Mamba uses bidirectional processing and position embeddings; VMamba adds two-dimensional selective scanning. VideoMamba and Vivim extend scanning across space and time to capture video dependencies.

Mamba for autonomous driving

PointMamba and Point Cloud Mamba tokenize irregular, sparse point clouds before sequence modeling. CoMamba applies state-space processing to cooperative 3D perception; hierarchical Mamba adaptation is cited for multimodal driving-video understanding. These task-specific results do not amount to a unified driving-system comparison in this paper.

6. Deployment and difficulty

Deployment is treated as a system problem spanning compute, timing, verification, memory, and lifecycle cost. References to commercial and research vehicles are contextual examples, not new deployment evidence collected by the authors.

Computational constraints

Automotive processors have less capacity than development hardware and may favor different operations. Efficient selective scans on a development GPU need not map efficiently to an automotive neural processing unit optimized for convolutions. Distillation and training-time transfer into smaller driving networks are suggested ways to use foundation-model knowledge within device limits.

Real-time performance

The authors give illustrative requirements below 100 ms for scene understanding and below 50 ms for critical decisions, contrasting these with language-model reasoning that may take seconds. These are stated deployment targets, not measured performance achieved by a proposed system. Consistency under changing computational load matters as well as average latency.

Validation and safety assurance

Validation should include rare events, degraded sensing, fallback behavior, and traceable decisions. Alignment methods and natural-language explanations do not substitute for testing actual vehicle behavior across operating conditions.

Memory and bandwidth

Limited onboard memory and bandwidth constrain both model storage and movement of multimodal data between processors. Compute optimizations must therefore be assessed alongside data-transfer costs.

Deployment costs

Hardware investment, predeployment validation, maintenance, calibration, upgrades, and continuing software updates all contribute to cost. Updated models, security patches, and added features require continued checks of system behavior.

7. Discussion

The paper identifies three broader directions: personalized interaction, embodied reasoning grounded in environments, and cooperation among vehicles. All introduce additional deployment and evaluation questions.

7.1. Human-centric autonomous driving

Language interaction may express passenger preferences and explain actions. Retrieval, feedback learning, edge-cloud cooperation, and federated learning are proposed for personalization. Preference adaptation must be considered together with the profiling bias and privacy concerns discussed earlier.

7.2. Embodied intelligence

DriVLMe adds multi-turn dialogue to driving assistance; LeapAD combines complementary decision processes with an experience memory. Richer simulated urban environments are proposed for testing adaptive agents. These examples motivate grounded interaction but do not establish reliable reasoning in unrestricted real traffic.

7.3. Cooperative driving

Connected vehicles can exchange observations, plans, and actions through V2V or V2X communication. Language-based multi-agent frameworks may improve interpretability and coordination, but scalability, communication security, and real-world implementation remain open.

8. Third LLVM-AD workshop

The third LLVM-AD workshop accepted ten papers spanning reasoning, novel-object detection, 3D representations, safety-event analysis, and multimodal driving. The closing outlook is optimistic about foundation models and embodied AI, while the paper’s own contribution remains an agenda rather than an experimentally validated integration recipe.

Appendix A. Preliminaries

The official one-page supplement supplies background for the main taxonomy. Its three subsections are included here; it contains no additional figures, tables, or displayed equations.

A.1. Multimodal LLMs

Multimodal language models connect pretrained visual and language representations through alignment, instruction tuning, or small trainable adapters. Examples distinguish fully tuned systems from frozen backbones joined by a projection layer or instruction-aware visual queries. The supplement lists both closed and open model families as context for the driving discussion.

A.2. AI alignment

Forward alignment shapes model behavior during development through preferences, feedback, and robustness to distribution shifts. Backward alignment evaluates and governs behavior through safety assessments, interpretability, human-value verification, and continuing oversight after deployment. These are complementary lifecycle activities.

A.3. State-space models and Mamba

The state-space background maps a scalar input sequence x(t) to y(t) through an N-dimensional hidden state h(t). Structured state-space models discretize continuous dynamics and structure the transition matrix; Mamba adds selective updates and a parallel scanning implementation.

Bidirectional and two-dimensional scans adapt the idea to vision. Multimodal fusion models, including Cobra and VL-Mamba, extend it further. Their linear sequence scaling motivates research into faster driving perception, with hardware and full-system timing still requiring separate validation.

Concept demonstration

Open image ↗