At a glance
TRIAD Planner trains a driving policy with three complementary signals: constrained attention over ego-state channels, offline vision-language guidance about relevant scene entities, and mode-selection targets shaped by tail excess in predicted clearance risk. Language-model queries and risk-target construction occur during training; deployment uses the learned feed-forward planner.
On nuPlan, TRIAD leads the learning-based methods included in the main comparison. On test14-hard, its reactive closed-loop score is 67.12 versus CAR Planner’s previously reported 64.90. The evidence is limited to this benchmark and does not establish superiority over the stronger rule-based or hybrid closed-loop results, or guarantee safe trajectories.
1. Introduction
Imitation learning trains on expert states but deploys on states induced by its own actions. Early errors can therefore move a planner outside its training distribution. The paper addresses three possible failure points: overreliance on a few ego-state channels, insufficient attention to relevant agents and map elements, and selection of undesirable trajectory modes.
The proposal combines regularization and teacher supervision within one architecture. It seeks robustness through better learned representations and mode probabilities rather than through an online language-model controller.
3.1. Planner overview
Figure 1 retains the full architecture. Inputs are the current ego state, surrounding-agent histories, and map polygons with attributes. The model predicts several ego trajectories and their logits, plus auxiliary surrounding-agent predictions. The three added components act on ego-state aggregation, scene-token relevance, and training targets for trajectory modes.
3.2. Base architecture
The six ego-state channels are x-position, y-position, heading, velocity, acceleration, and steering angle. Each is embedded separately before learned-query aggregation. Agent histories and map elements become tokens, a shared Transformer mixes the tokens, and the final ego token produces K candidate trajectories and mode scores. This is planning from structured state and map inputs, not direct camera-only control.
3.3. Constrained ego-state attention
Head-averaged attention over the six ego channels is compared with a uniform distribution using mean absolute deviation. The constraint limits the minibatch-average deviation to a threshold, allowing individual channels to receive unequal weights. It constrains an average training statistic, not every sample or attention head independently.
Equation 1 gives the projected augmented Lagrangian method (ALM) penalty. The nonnegative multiplier starts at zero and is updated after each optimizer step using the same forward-pass constraint residual. The constraint acts before ego information is fused with external scene tokens.
Equations
3.4. Scene-level token relevance
The scene component trains ego-query attention to retain relevant external agents and route-related map evidence. It uses a cached teacher distribution on guided training samples, while unguided samples retain the other training objectives.
3.4.1. Offline multimodal guidance
The paper queries GPT-5.4-Mini offline with an ego-centered bird’s-eye view (BEV) and structured scene text. Inputs include agent identifiers, motion cues, traffic signals, route context, and the target maneuver derived from the logged expert trajectory. The teacher outputs agent-priority weights and short reasons; Figure 2 preserves an example.
The expert maneuver is privileged training information. It is used to construct supervision, not supplied as a future trajectory at deployment. The complete prompt and examples are retained in Appendix A.
3.4.2. Prior construction and attention supervision
Invalid agent identifiers are discarded and remaining priorities are normalized. Route-aware geometric scores supply map priorities. Agent and map scores are then normalized jointly over valid external tokens.
Equation 2 uses cross-entropy from this scene prior to the ego-query attention distribution, averaging only over samples with valid guidance. Thus the VLM supplies agent priorities while the map component remains a geometric prior.
Equation
3.5. Tail-excess supervision
Each trajectory mode receives a time series of clearance deficits relative to predicted surrounding agents, accounting for an effective safety radius and margin. Equation 3 averages the positive excess above that mode’s own empirical value-at-risk (VaR) quantile, with scaling by the upper-tail fraction.
Reader qualification: this tail-excess statistic measures variation above a mode-specific threshold. It omits the quantile term present in a full conditional value-at-risk expression; adding the same constant to all risks of a mode leaves its tail excess unchanged. A consistently high-risk sequence can therefore have low tail excess. The statistic is not an absolute collision-risk guarantee.
Equation
Hard and soft mode targets
For hard supervision, Equation 4 chooses the mode minimizing negative log probability plus weighted tail excess. That selected trajectory is regressed toward the expert trajectory.
Equation 5 also constructs a soft target by multiplying stopped-gradient mode probabilities by an exponential penalty on normalized tail excess. A Kullback–Leibler (KL) loss trains the probability head toward this target. Risk sensitivity used in target construction and the weight applied to the risk loss are separate parameters.
Equations
3.6. Training objective and inference
The base objective combines ego-trajectory regression, selected-mode classification, and auxiliary agent prediction. Equation 7 adds the channel penalty, scene-guidance loss, and risk-target loss.
At inference, the planner selects a trajectory from its learned mode distribution in a normal forward pass. The offline teacher and risk-target computation are disabled. Their influence persists through the trained parameters.
Equations
4.1. Experimental setup
Training uses nuPlan v1.1 with the training_scenarios_1M filter and no extra city, map, or scenario filtering. Guidance is generated for about 20% of cached training examples, then guided examples are sampled with probability 0.5. Ego-state perturbation is applied with probability 0.5 and coordinates are recentered in the perturbed frame.
Models train for 25 epochs with batch size 64, AdamW, learning rate 0.001, weight decay 0.0001, and a cosine schedule with three warm-up epochs. Hardware includes a 32-core processor, 251 GiB memory, and four RTX PRO 5000 GPUs.
Both test14 splits cover 14 scenario types. Open-loop score (OLS) measures agreement with logged expert behavior; non-reactive closed-loop score (NR-CLS) uses replayed agents; reactive closed-loop score (R-CLS) uses simulated responding agents.
4.2. Benchmark results
Table 1 retains every expert, rule-based, hybrid, and learning-based result. TRIAD scores 89.72/87.65/82.55 on the random split and 87.13/73.39/67.12 on the hard split for OLS/NR-CLS/R-CLS. It leads the learning-based models listed in the table. Pluto is evaluated without post-processing.
Against CAR Planner’s reported results, the hard-split improvements are 0.45 in NR-CLS and 2.22 in R-CLS. CAR’s scores are taken from the earlier report, rather than established here as a repeated identical-run comparison. The rule-based PDM-Closed and hybrid PDM-Hybrid retain higher reactive closed-loop scores than TRIAD on both splits.
Inference efficiency
Table 2 reports wall time per evaluation scenario across 261 scenarios: TRIAD averages 15.18 seconds, compared with 15.00 for CAR and 15.01 for PlanTF. Total time is 66.0 minutes for TRIAD and 65.2 for CAR.
These are scenario-level measurements, not latency per planner step. They support comparable evaluation cost for these models on the reported setup; they do not imply exactly zero runtime difference or establish a real-time control deadline.
Ablation of the three levels
Figure 3 retains all channel (C), scene (S), and decision (D) combinations. The full combination gives the strongest scores in that comparison, but gains are not additive: Base+S+D has lower reactive scores than either Base+S or Base+D on both splits. Better expert imitation also does not always produce better reactive closed-loop behavior.
Channel-level analysis
In the illustrated right-turn rollout, PlanTF concentrates attention on acceleration as it drifts and collides, whereas TRIAD distributes attention across more dynamic channels and stays closer to the route centerline. Figure 4 keeps the entire time sequence and attention plots. This example shows an association between attention and behavior; it does not isolate attention concentration as the sole cause of collision.
Scene-prior comparison
Two substitute teacher priors rank agents by current distance to ego or distance to the expert’s future path. The latter also uses privileged information only during offline supervision. Table 3 and Figure 5 preserve the quantitative and visual comparisons.
TRIAD’s VLM-guided prior performs best on all three hard-split metrics and is best or tied-best on val14. The future-path heuristic achieves higher NR-CLS on test14-random (88.38 versus 87.65), while TRIAD has higher OLS and R-CLS. This supports benefits beyond simple proximity cues without establishing dominance on every metric.
Scene attention analysis
Figure 6 aggregates attention by agent, ego, and map token groups. Ego-to-agent mass rises from 0.168 for CAR to 0.198 for TRIAD; ego-to-map mass remains dominant, changing from 0.819 to 0.802. Ego-to-ego mass falls from 0.013 to near zero after ego-state encoding.
Increases in agent attention are larger in several interaction-dependent scenarios, including pedestrian waiting. These are descriptive attention patterns consistent with the training objective, not direct measurements of causal reasoning or calibrated uncertainty.
Decision-level analysis
Figure 7 plots selected-mode tail excess against mean minimum clearance margin. TRIAD has a lower fitted tail-excess trend than CAR over much of the observed range on both splits. This is consistent with learning the targeted statistic at similar proximity levels. Lower excess relative to each mode’s own quantile should not be confused with lower absolute risk in every scenario.
5. Conclusion
Jointly training ego-state attention, scene relevance, and mode supervision improves the evaluated imitation planner on nuPlan. The main contribution is a training recipe that retains standard feed-forward deployment, with the strongest evidence coming from benchmark scores and the full-component ablation.
6. Limitations
Only about one-fifth of the cached corpus receives teacher guidance. Around 3% of teacher outputs have weights summing above one and are filtered or renormalized. Missing agents, incomplete scene descriptions, and incorrect priorities can still produce noisy targets.
Guidance lacks timestep-level temporal supervision and may miss interactions that emerge later. Evaluation is confined to nuPlan; cross-dataset and real-vehicle transfer remain untested. The paper does not report uncertainty across repeated training runs, and the appendix does not supply a complete set of loss weights and risk hyperparameters.
Acknowledgments
The work acknowledges support from the National Research Foundation of Korea and the Institute of Information & Communications Technology Planning & Evaluation, funded by the Korean Ministry of Science and ICT.
Appendix A. Guidance generation
Appendix A documents the offline teacher inputs, complete prompt, example output, and validation pipeline. All three original text listings are retained.
A.1. Multimodal input
The BEV spans 100 × 100 meters at 1024 × 1024 pixels, with ego centered and heading upward. Structured text includes the nine nearest agents, ego dynamics, map context, and a maneuver inferred from the logged target trajectory’s accumulated heading change. The original stationary-scene example follows.
{
"scenario": {"scenario_type": "stationary"},
"ego": "The ego vehicle’s speed is 0.0015 m/s with acceleration of
0.0099 m/s^2. The steering angle is 0.0 degrees and the yaw
rate is 0.0 degrees/s.",
"agents": "There are 9 agents near the ego vehicle:
1: VEHICLE, distance 3.81 m, speed 4.36 m/s;
2: VEHICLE, distance 10.24 m, speed 0.0 m/s;
3: VEHICLE, distance 8.23 m, speed 0.0 m/s;
4: VEHICLE, distance 19.16 m, speed 2.11 m/s;
5: VEHICLE, distance 20.21 m, speed 0.0 m/s;
6: VEHICLE, distance 22.92 m, speed 0.0 m/s;
7: BICYCLE, distance 23.21 m, speed 0.09 m/s;
8: PEDESTRIAN, distance 23.27 m, speed 0.01 m/s;
9: VEHICLE, distance 29.52 m, speed 6.3 m/s.",
"map": "The typical speed limit is approximately 11.93 m/s.",
"target_trajectory": "Stop."
}A.2. Prompt design
The reported teacher is gpt-5.4-mini, queried at temperature zero without an explicit maximum completion-token limit. The prompt specifies the coordinate frame, color legend, input fields, and priority schema. Its original wording is retained below. The float.2f notation describes numeric weights rounded to two decimal places, not literal text to return.
[System]
You are an expert autonomous driving system analyst.
[User]
You are given a bird’s-eye-view (BEV) map image.
The map is ego-centric. The ego vehicle is always at the center of the image.
The map is heading-up: the ego vehicle’s forward driving direction always
points toward the top of the image.
Therefore, left/right in the image correspond directly to left/right of the
ego vehicle, and the top/bottom of the image correspond to ahead/behind the
ego vehicle.
The color legend for the BEV map is as follows:
- Ego vehicle: cyan (#00ffff) with the label of 0
- Vehicles: yellow-green (#b0bb4a)
- Cyclists: magenta (#ff00ff)
- Pedestrians: yellow squares (#ffff33)
- Regular road lanes: gray (#7a7a7a)
- Intersection area: dark gray (#3a3a3a)
- Lane connectors / turning paths: blue (#115d91)
- Crosswalks: beige (#c8b89a) with white dashed hatching
- Sidewalks / walkways: brown (#a0522d)
- Parking areas: dark navy (#021e30)
- Traffic lights: colored circles (green (#00ff00), yellow (#ffff00),
red (#ff0000), or gray (#aaaaaa)) at the entry of lane connectors.
Each agent box has an ID label and a velocity arrow showing its direction
and speed. Analyze the current driving scene based on the BEV map and the
state information.
State information:
- Scenario: {scenario}
- Ego: {ego}
- Agents: {agents}
- Map: {map}
- Target trajectory: {target_trajectory}
I need to use a guidance factor to represent the weighting characteristics of
agents and scene context in autonomous driving for avoiding collision and
off-road trajectory.
agents_priority: For each identified agent in the scene excluding the ego
vehicle, report its priority weight based on its type, position, navigation
relevance, and distance to the ego vehicle. Higher weights represent higher
priority. Normalize the weights so that their sum is 1.
Output only a valid Python dictionary. Use exactly these keys and value
formats:
{
"agents_priority": {
"AGENT_ID": {"weight": float.2f, "reason": "string"}
}
}Teacher output example
The example assigns weights and reasons to nine agents, prioritizing the nearest moving vehicle. This is an illustrative teacher response rather than independently verified ground truth about the scene.
{
"agents_priority": {
"1": {"weight": 0.28,
"reason": "Closest moving vehicle ahead-right in the ego lane
corridor, posing the highest immediate collision risk."},
"3": {"weight": 0.16,
"reason": "Stationary vehicle directly ahead in the lane."},
"2": {"weight": 0.14,
"reason": "Stationary vehicle ahead in the same approach path."},
"4": {"weight": 0.11,
"reason": "Vehicle farther ahead near the intersection connector."},
"5": {"weight": 0.10,
"reason": "Stationary vehicle near the intersection area."},
"6": {"weight": 0.09,
"reason": "Stationary vehicle near the upper-left approach."},
"7": {"weight": 0.06,
"reason": "Slow bicycle near the upper-right connector."},
"8": {"weight": 0.04,
"reason": "Pedestrian near the upper-right area."},
"9": {"weight": 0.02,
"reason": "Far vehicle moving away from the ego path."}
}
}A.3. Output validation and prior construction
Returned dictionaries are structurally checked and serialized as JSON. Failed API calls, parsing errors, and formatting errors are logged for offline retry. Invalid identifiers and padded agents are ignored; numeric weights are clipped to [0, 1], combined with route-aware map scores, and normalized.
If no usable teacher priority remains, inverse-distance scores provide a fallback. Guidance focuses on interaction and maneuver scenarios, and weighted sampling with replacement controls its proportion during training.