Literature
Display
← All papers

Evolution path of multimodal collaborative optimization and training techniques for end-to-end autonomous driving systems

ShengBo LI · HongYang LI · Hang Zhao · YingFeng CAI · Bing ZHU · Ji TAO · JianQiu WANG · Bin SHUAI · Chen CHEN · JiaXin GAO · Fei ZHAO · Jie LI · QingChao LIU · PeiXing ZHANG · KeQiang LI

Published online · Scientia Sinica Technologica 55(10), 1638–1658 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

This Chinese-language review surveys end-to-end autonomous driving across model architecture, multimodal data, training, compute infrastructure, simulation, and evaluation. Its organizing idea is a vehicle–cloud development cycle: collect useful driving data, train and test models, then deploy a smaller model to the vehicle.

This edition condenses the review into English while retaining all fourteen original figures and their bilingual captions. The paper synthesizes existing systems and proposes directions; it does not introduce a new controlled benchmark or establish a general safety guarantee.

1. Introduction

Traditional perception, prediction, planning, and control modules are easy to separate and debug, but intermediate representations can lose information and independently optimized objectives can conflict. End-to-end learning aims to coordinate these components through a shared training objective [1] [2].

Figure 1 depicts the proposed vehicle–cloud cycle: fleet collection, filtering, simulated scene generation, mixed real/synthetic training, model compression, and over-the-air deployment. The review treats this as a development architecture, rather than evidence that online deployment can proceed without independent validation.

Complete original Figure 1
图1 车云协同的自动驾驶大模型开发方案 Figure 1 Framework of cloud-vehicle collaborative large model for autonomous driving.

2. Model architectures

The survey groups the literature into perception foundation models, decision/control models, segmented end-to-end systems, unified networks, and multimodal vision–language–action systems. These categories describe architectural trends and can overlap.

2.1 Perception models

Perception models preprocess sensor streams, extract features, fuse modalities, and produce objects, tracks, depth, or semantic maps. Modular specialists permit replacement and task-specific debugging; joint models can share representations and reduce explicit handoffs. Figure 2 retains the complete perception workflow [3].

Examples include SAM and SEEM for segmentation, Depth Anything for monocular depth, E2E-MFD for multimodal fusion/detection, and PnPNet for joint perception and prediction [5] [6] [7] [8] [9]. Their inclusion is illustrative; the review does not re-evaluate them under one common protocol.

Complete original Figure 2
图2 自动驾驶感知大模型运行流程[3] Figure 2 Operational workflow of autonomous driving perception foundation model [3].

2.2 Decision and control models

The paper contrasts expert rules, imitation from human driving, and reinforcement learning from interaction. Rules expose decisions clearly but become difficult to maintain across many situations. Imitation uses accessible demonstrations, while reinforcement learning optimizes an explicit reward and must manage exploration and transfer.

Integrated decision and control combines path evaluation with policy outputs and vehicle-model constraints [14]. TransFuser and VAD illustrate learned sensor fusion and compact scene representations [18] [19]. These are cited examples, not new experiments in this review.

2.3 Segmented end-to-end architectures

Segmented end-to-end systems retain distinct perception, prediction, and decision/control networks while coordinating their learning. The review describes iDrive as a three-part realization, supported by reinforcement-learning algorithms, motion prediction, control tools, and optimization methods [20] [21] [22] [23]. Figure 3 preserves its original system diagram.

The review argues that imitation-only development is constrained by rare-event data and distribution shift, and proposes simulation-based reinforcement learning as a complement. Its strong claims about exceeding human driving should be read as motivation, not a proven consequence of changing the training method.

Complete original Figure 3
图3 国内首个端到端系统iDrive的原理图 Figure 3 China’s first end-to-end system iDrive.

2.4 Unified end-to-end architectures

Unified models jointly optimize perception, prediction, and planning. UniAD organizes multiple subtasks around planning, and Figure 4 contrasts independent modules, a shared backbone with task heads, and a planning-oriented end-to-end architecture [25].

The paper also discusses commercial systems as they were described in 2025. Their cited parameter, data, and compute scales are contextual claims rather than measurements produced by this review. Unified training does not by itself resolve unusual interactions, sensor failures, or interpretation of decisions.

Complete original Figure 4
图4 UniAD端到端整体架构[25] Figure 4 Overall architecture of UniAD end-to-end system [25].

2.5 Multimodal driving models

Vision-language models add semantic interpretation and instruction understanding. DriveVLM uses a slower semantic reasoning system alongside a faster driving system; Figure 5 preserves the asynchronous interaction structure [26]. The goal is to benefit from language reasoning without requiring its full computation at every control step.

The review identifies inference latency, weak three-dimensional grounding, hallucinated objects, and incorrect direction judgments as limitations. Extending the model to generate actions is a research direction, not proof that language explanations reliably justify those actions.

Complete original Figure 5
图5 DriveVLM快慢双系统整体框架图[26] Figure 5 DriveVLM dual fast-slow system framework [26].

3. Data support

The data system covers collection, storage, retrieval, annotation, and feedback into training. Scale alone is insufficient: useful coverage, synchronization, quality control, and lawful reuse determine whether the data improves behavior.

3.1 Data collection

Dedicated collection vehicles offer controlled routes and sensor quality but limited scale. Production fleets broaden coverage and can capture rare events, while making collection less controllable. Simulation deliberately generates difficult conditions with automatic labels, at the cost of a reality gap.

The review describes world models such as DriveDreamer as another source of training data [27]. Generated and real samples must be balanced, and downstream benefits depend on the task and generation quality. Figures 6–7 retain the collection architecture and comparison of collection processes.

Complete original Figure 6
图6 实车数据采集 Figure 6 Real-world vehicle data acquisition.
Complete original Figure 7
图7 常见数据采集方式流程 Figure 7 Common data collection process approaches to data collection.

3.2 Storage and management

Large, geographically distributed fleets motivate distributed transfer/storage and searchable scene metadata. Compound queries may combine road structure, traffic signals, congestion, maneuvers, and surface conditions. Figure 8 shows the storage and management cycle.

The authors discuss access controls, encryption, audits, and privacy alongside convenient retrieval. Their exabyte-scale examples illustrate a possible infrastructure burden; they are not minimum requirements established for every end-to-end system.

Complete original Figure 8
图8 数据存储与管理流程图 Figure 8 Flowchart of data storage and management.

3.3 Data annotation

Automatic labeling combines model predictions, confidence estimates, temporal/structural consistency, and iterative correction. Shared encoders and task-specific decoders can jointly label objects, road elements, trajectories, and behavior. Dynamic task weighting aims to reduce interference.

Domain adaptation, continual learning, and active selection of uncertain samples are proposed to improve coverage while reserving human effort for difficult examples. The survey also advocates shared standards and datasets. It provides no experiment showing that these mechanisms eliminate annotation error or distribution shift.

4. Training paradigms and tools

The training pipeline must connect learning objectives with data loading, optimization, validation, and deployment. The review treats algorithm choice and the supporting toolchain as coupled engineering decisions.

4.1 Training algorithms

Imitation fits human state–action pairs; reinforcement learning gathers interaction experience and optimizes a reward [13]. The two approaches differ in available supervision, exploration costs, and how performance objectives are expressed.

Imitation learning

Imitation can predict low-level controls or future trajectories. Direct controls are easy to log but can be sensitive to vehicle dynamics and temporal inconsistency. Trajectory outputs require a compatible tracking controller. Figure 9 preserves the supervised-learning loop.

Advantages include straightforward optimization and no manually designed reward. Limitations include rare demonstrations, imbalanced data, and behavior outside the training distribution [31] [32]. The review’s very large mileage claim is not treated here as a verified universal sample-complexity requirement.

Complete original Figure 9
图9 模仿学习训练示意图 [13] Figure 9 Illustration of imitation learning training [13].

Reinforcement learning

Reinforcement learning alternates environment interaction, value estimation, and policy improvement. Parallel simulation supplies experience while avoiding physical exploration during training; Figure 10 preserves the learning loop. Reward design and transfer from simulation remain major constraints.

The review describes combining a supervised perception representation with a shallower reinforcement-learning policy as one route to manage optimization difficulty [38]. Its statement about the absence of successful unified reinforcement-learning systems reflects the paper’s 2025 assessment, not a current comprehensive survey.

Complete original Figure 10
图10 强化学习训练示意图[13] Figure 10 Reinforcement learning training framework [13].

4.2 Training tools

The toolchain spans stream/batch preprocessing, model training, hyperparameter search, compression, and inference. Examples include Kafka/Spark for data processing, PyTorch/TensorFlow for learning, optimization tools, and deployment runtimes. These examples are descriptive rather than a tested reference stack.

Pruning, quantization, and distillation trade model capacity and numerical precision for smaller memory and latency demands. Accuracy and timing must be checked together; the review does not establish that compression preserves behavior in every scenario.

5. Compute and software systems

Compute design separates high-volume cloud development from time-sensitive vehicle execution. Hardware, software, communication, storage, and scheduling all affect end-to-end throughput and latency.

5.1 Compute architecture

The proposed hierarchy assigns large-scale training, map/data processing, and simulation to the cloud, while the vehicle handles immediate perception and control. Heterogeneous processors divide general orchestration, parallel tensor computation, and specialized low-power work.

Distributed training uses data/model/pipeline parallelism, fast interconnects, and shared storage. Vehicle systems integrate processors, accelerators, and sensor interfaces under power and timing constraints. The review’s named chip and cluster specifications are historical examples; it gives no consistent measured comparison across platforms.

5.2 Software architecture

Cloud services manage data, training, simulation, deployment preparation, and resource allocation. Vehicle software integrates operating-system scheduling, inference runtimes, middleware, and communication. Figure 11 preserves the complete cloud/vehicle architecture.

The survey cites frameworks, real-time operating systems, robotics middleware, automotive interfaces, and vehicle/cloud protocols. Their presence in an architecture does not itself establish a certified safety level or a guaranteed latency bound.

Complete original Figure 11
图11 自动驾驶车云端算力系统架构 Figure 11 Autonomous driving cloud-edge computing architecture.

6. Simulation and testing

Testing combines recorded real inputs with generated interactive environments. The review emphasizes coverage of rare conditions and evaluation of joint perception/control behavior, while recognizing the need for meaningful metrics.

6.1 Simulation test methods

The distinction is whether the evaluated policy’s actions affect subsequent observations and traffic behavior. Recorded replay is useful for repeatable input comparisons; interactive simulation can expose action-dependent failures.

Recorded-data open-loop tests

Open-loop replay feeds synchronized recorded sensor streams into the system and compares perception or predictions under realistic observations. It is efficient and repeatable, but the recorded future does not respond to the evaluated policy’s alternative actions. Good replay scores therefore do not by themselves demonstrate closed-loop driving performance.

Generated-data closed-loop tests

Generated environments update observations in response to the ego vehicle and other agents. Diffusion, neural radiance fields, and Gaussian representations support scene reconstruction and sensor generation; Figures 12–13 preserve the sensor-level and trajectory-generation approaches [45] [46] [47].

Adversarial agents, optimization, and importance sampling can concentrate tests on risky interactions. Their value depends on realism and search coverage. A generated image’s visual quality alone does not establish correct vehicle dynamics or realistic traffic responses.

Complete original Figure 12
图12 典型的端到端感知系统闭环测试方法 [45] Figure 12 Closed-loop evaluation method of typical end-to-end perception systems [45].
Complete original Figure 13
图13 面向端到端决策的轨迹数据生成方法 Figure 13 Method of trajectory data generation for end-to-end decision-making.

6.2 Evaluation metrics

Task-oriented evaluations measure completion, time, speed, interventions, and failures. Behavioral evaluations instead measure properties such as safety, comfort, efficiency, energy use, and rule compliance. Weighting or aggregating these measures introduces assumptions and may be scenario-dependent.

Figure 14 preserves STCER-H: five instantaneous dimensions—safety, efficiency, comfort, energy, and regulation—plus an aggregate measure of humanlike behavior [57]. The survey identifies risk measurement as especially difficult and favors data-driven risk proxies. Such proxies are not direct proof of safety.

Complete original Figure 14
图14 驾驶行为能力STCER-H指标体系[57] Figure 14 STCER-H framework for driving behavior capability [57].

7. Directions and challenges

The research agenda concerns multimodal consistency, data coverage, efficient training, interpretable decisions, domain transfer, real-time execution, and governance. The article frames these as interdependent issues.

7.1 Technical directions

Proposed directions include attention-based sensor fusion and shared spatial representations; realistic, targeted synthetic data; and stronger links between learned decisions, physical constraints, and debugging tools. Language can add semantic context, while generation can broaden scenario coverage.

The review discusses adversarial simulation, causal structure, uncertainty, and redundant responses as complementary mechanisms [61]. It supplies an agenda rather than a validated recipe that jointly solves these challenges.

7.2 Long-tail and domain transfer challenges

Rare conditions and combinations of events make exhaustive data collection infeasible. Pretraining, few-shot adaptation, structured priors, and adversarial simulation are proposed to improve coverage. Residual differences between synthetic and real sensor/interaction distributions remain unresolved.

Several percentages in this discussion lack clearly matched experimental support in the review. The notes retain the qualitative direction without converting those numbers into verified generalization claims.

Compute efficiency and real-time response

The cloud can support large models, but deployed systems must meet vehicle power, memory, communication, and response-time constraints. Compression and heterogeneous scheduling are proposed remedies. Remote semantic reasoning is useful only within a design that accounts for communication delay and availability; the survey provides no formal timing analysis.

Causal confusion and reliability

Correlations in driving logs can produce shortcuts rather than dependable responses to causes. The paper proposes causal structure, attention to decision-changing moments, and uncertainty estimates for fallback decisions, citing world-model and shortcut-learning research [64] [65]. It does not establish that a language explanation or confidence value identifies the true cause of a decision.

Governance and public acceptance

The authors raise privacy, data sharing, cross-border governance, auditability, and public understanding as deployment concerns. Federated training, privacy techniques, and clearer explanations are discussed as possible tools; this is a research overview rather than a jurisdiction-specific compliance guide.

There are material citation mismatches: the Tesla-compute paragraph cites a real-time Linux reference, and the blockchain/cross-border-data claim cites a driving imitation-learning paper [39] [66]. The bibliography also mismatches some named planning/attention methods. All source entries and captions are retained; these attributions and associated numerical claims are not independently validated by this edition.

8. Conclusion

The survey’s central message is that unified driving models require coordinated progress in data, learning, compute, and evaluation. Long-tail transfer, timing, causal interpretation, and governance remain barriers. Future work should pair model improvements with explicit validation of the complete deployed system.

Concept demonstration

Open image ↗