Overview
SemanticPlan tests planning around unusual, instruction-conditioned road participants. This edition retains the complete figures, result tables, metric display, scenario inventory and both agent-output schemas from arXiv v1. The two evaluation tracks use different interaction protocols.
At a glance
Language-guided agents augment real nuPlan scenes with unusual intentions and reactions. A constrained action interface translates their decisions into simulator motion and state changes. SemanticPlan contains more than 50 scenario types and 230 scenarios. Tested planners struggle, but the collision-prone track replays generated trajectories; only the semantic track keeps the controlled agents reacting online during evaluation.
1 Introduction
Log replay cannot react to the tested vehicle, while simple traffic rules cover limited intentions. The framework adds instruction-following participants to nuPlan scenes and grounds their commands in simulator constraints. Its contribution is controllable interaction-focused stress testing rather than a representative estimate of real-world accident risk. The dataset is manually designed and filtered for scenarios the simulator can express.
2 Related Work
Related work spans simulation environments, generative traffic models, language-guided agents and ego planners. The proposed combination emphasizes interpretable participant intentions and simulator-executed actions within existing driving scenes.
Driving simulators and benchmarks.
Driving benchmarks trade off controlled synthetic environments, logged real scenes, interactive traffic and cheaper non-reactive evaluation. SemanticPlan augments nuPlan’s map and recorded scene context with selected long-tail participants; the semantic track additionally requires scenario-specific interpretation.
Traffic simulation.
Token prediction and diffusion models improve traffic diversity and realism, while adversarial, reinforcement-learning and language-conditioned approaches improve control over generated behavior. Here, language controls participant intent through a restricted action vocabulary, with motion generated or executed separately.
Simulation using LLMs.
Earlier language-model agents simulate vehicle interactions and broader urban activity. This work concentrates on ego-planning evaluation in nuPlan, with explicit observation, feedback and action interfaces for selected surrounding participants.
Planning approaches.
Baselines include rule-based, learned and hybrid planners; semantic baselines add language-model decisions to a conventional longitudinal controller. Their results measure the specified simulator tasks and interfaces, not an unrestricted comparison of general driving intelligence.
3 Method
The simulator separates high-level intent from physically executed motion. Agent inputs include role, local observations, feedback and history; outputs are validated before changing trajectories or interaction state.
3.1 Overview
Only a selected subset of non-ego participants becomes instruction-following agents. Humans can move, handle objects or enter/exit vehicles; vehicles can park, merge or change lanes. Deterministic executors impose preconditions and physical state transitions, allowing semantic intentions to interact with the simulated world.
3.2 Agent Interface: Inputs
Each query combines persistent intent with current local context and the outcome of previous commands. The visual and textual observation channels supply complementary scene information.
Inputs.
An agent conditions its next structured action on a role instruction, simulator feedback, an agent-centric text/image observation and conversation history. Human agents use a vision-language model. The explicit feedback loop allows later actions to respond to failed interactions and changing surroundings.
Simulator feedback.
Executors return structured success/failure events. Agents can retry a rejected interaction, yield when blocked or choose a different maneuver; these responses are generated from feedback rather than guaranteed by a fixed behavior script.
Agent local observation.
Text summarizes the agent state, nearby participants, map context and recent events. The rendered view adds semantic colors, identifiers, object geometry and recent trajectories. Both are local to the controlled participant.
3.3 Agent Interface: Outputs
Human outputs specify movement plus optional physical or social actions. Vehicle outputs specify maneuvers whose feasibility and execution are handled by the simulator.
Human agent outputs.
Humans request one or two movement axes, their ratio, speed and optionally a target. Flow matching converts this intent into a trajectory. Accepted object/vehicle actions change attachment relationships, such as carrying an object or riding in a vehicle. Gestures and utterances become stimuli for nearby agents without directly modifying physical state.
Vehicle agent outputs.
Vehicles issue parameterized maneuvers instead of steering/throttle commands or arbitrary waypoints. The native route planner supplies map-compliant execution and rejects infeasible requests. Consequently, reported improvements include the benefit of this hand-engineered execution layer.
3.4 Flow-Matching Trajectory Generation
A conditional flow-matching model takes recent motion, requested speed and the continuous direction resolved from the axis command. It samples a short local trajectory, converts it into world coordinates and executes it over the control horizon. The interface constrains intent while the learned motion model supplies smoother trajectories.
3.5 Simulator Internals
Local observations summarize nearby entities and semantic events. Ego planning retains nuPlan map/route/object inputs, with semantic-track action descriptions added for controlled agents. Vehicle replanning and honking use explicit simulator mechanisms rather than unconstrained language-model control.
4 Experiments
Experiments compare planning scores, simulation behavior quality, motion distributions, vehicle interfaces and honking strategies. The appendices specify the substantial runtime cost, dataset construction, scoring rules and complete output schemas.
4.1 SemanticPlan Dataset
More than 50 designed scenario types are placed into over 230 nuPlan validation scenes. Main text reports 1–8 agents and 3–5 base scenes per type; Appendix B instead gives 1–5 agents and 3–8 scenes. These inconsistent ranges are retained as a source caveat rather than reconciled. Scenarios are grouped into collision-prone and semantic tracks, with some types shared.
4.1.1 Collision-prone Track
The collision-prone track uses ordinary nuPlan planner observations and scores route progress multiplied by safety/drivable-area compliance. Ego-at-fault collisions receive zero credit. Controlled human trajectories are generated beforehand and replayed for each planner, while vehicles use the ordinary traffic model; the ego remains closed-loop but those replayed humans do not react to its particular rollout.
4.1.2 Semantic Track
The semantic track combines task-specific spatial/instruction constraints with honk-sensitive interactions. Controlled participants remain online so the tested ego’s behavior can alter their responses.
General semantic scenarios.
General semantic cases penalize footprint overlap with specified regions or failure to obey a traffic officer’s stop/lane-change command. Progress is truncated at the first safety violation and discounted by the semantic penalty. Goals can be adjusted when safe waiting is the intended task.
Honk-sensitive scenarios.
Some scenarios require honking to clear a path; others penalize it according to the recipient’s role. Progress and inappropriate-honk penalties jointly score behavior. These role weights encode benchmark design choices rather than empirically validated universal social rules.
4.2 Experimental Setup
Planners receive no fine-tuning on the augmented scenarios. Collision-prone tests reuse three generated overlays per scene to make repeated baseline evaluation cheaper. Semantic tests run agent inference online. The zero-shot claim concerns ego-planner adaptation; the pedestrian trajectory generator is separately trained on nuPlan validation logs.
Experimental details.
Simulation advances at 10 Hz. Agents are queried every two seconds and execute 20-waypoint, two-second plans between queries. The default model is Qwen3.6-27B in eight-bit floating point, with sampling temperature 0.7 and three stochastic rollouts per instance. Some honk-required scenarios run longer. On eight RTX 3090 GPUs and 20 CPU cores, the reported IDM evaluation takes 1.5 hours for replay-based collision tests versus 15.1 hours for online semantic tests.
Baselines.
Collision baselines span IDM, PDM Closed, UrbanDriver, GC-PGP, PlanTF, Diffusion Planner, PDM Hybrid and PLUTO. Semantic baselines keep IDM longitudinal control and optionally add Qwen3.6-27B decisions every two seconds for lane changes, honking, constraints and officer instructions. A further variant permits explicit conservative stops.
4.3 Baseline Results
PDM Hybrid leads collision-prone overall score at 0.651, narrowly above PDM Closed’s 0.644; all safety scores remain below one. On semantic tasks, adding language decisions raises overall score from 0.353 to 0.389. Allowing conservative stops lowers region penalty from 0.561 to 0.146 but reduces progress and overall score to 0.309. These are means over three simulations, without reported confidence intervals.
| Method | |||
|---|---|---|---|
| Rule-based | |||
| IDM | 0.818 | 0.507 | 0.409 |
| PDM Closed | 0.855 | 0.767 | 0.644 |
| Learning-based & Hybrid | |||
| UrbanDriver | 0.890 | 0.301 | 0.242 |
| GC-PGP | 0.604 | 0.432 | 0.258 |
| PlanTF | 0.861 | 0.459 | 0.380 |
| Diffusion Planner | 0.935 | 0.425 | 0.373 |
| PDM Hybrid | 0.856 | 0.774 | 0.651 |
| PLUTO | 0.924 | 0.664 | 0.616 |
4.4 Simulation Quality
Ten scenario types have deterministic behavior checks over actions, feedback, state transitions and proximity. Flow matching raises Qwen3.6-27B behavior success from 0.792 to 0.889; smaller models with it score 0.653 and 0.708. Acceleration and jerk move closer to nuPlan statistics, but mean speed remains 2.04 versus 0.97 m/s in logs. Thus improved selected statistics and predicate success do not establish full human-behavior realism.
| Flow Mat. | Model | Success Rate |
|---|---|---|
| ✓ | Qwen3.5-4B | 0.653 |
| ✓ | Qwen3.5-9B | 0.708 |
| Qwen3.6-27B | 0.792 | |
| ✓ | Qwen3.6-27B | 0.889 |
| Source | Speed | Accel. | Jerk | Turn | Direction |
|---|---|---|---|---|---|
| m/s | m/s2 | m/s3 | deg | Straight/L/R | |
| nuPlan logs | 0.97 | 2.96 | 50.10 | 22.9 | 0.42/0.25/0.33 |
| Ours w/o FM | 1.86 | 0.90 | 15.57 | 20.4 | 0.74/0.12/0.14 |
| Ours | 2.04 | 2.90 | 47.34 | 26.1 | 0.54/0.23/0.23 |
4.5 Vehicle Action Comparison
High-level vehicle actions achieve 0.905 no-collision and 0.956 drivable-area scores, versus 0.285/0.442 for direct trajectories and 0.507/0.697 for direct controls. The simulator’s route planner handles feasibility and kinematics, so these results support constrained execution interfaces rather than raw model control ability.
| Action interface | No Col. | Driv. |
|---|---|---|
| Low-level traj. | 0.285 | 0.442 |
| Low-level ctrl. | 0.507 | 0.697 |
| High-level actions | 0.905 | 0.956 |
| Strategy | Prog. | Pen. | |
|---|---|---|---|
| Never honk | 0.457 | 0.000 | 0.452 |
| Always honk | 0.913 | 0.201 | 0.493 |
4.6 Honking Analysis
Always honking doubles progress on honk-required scenarios from 0.457 to 0.913 but incurs a 0.201 penalty on honk-penalized scenarios; the aggregate honk score rises only from 0.452 to 0.493. The extreme policies illustrate the metric’s progress/appropriateness tradeoff, not an optimal honking policy.
4.7 Qualitative Examples
The original qualitative panels show collision mechanisms and spatial semantic constraints. Their captions are preserved even where a source caption’s bicycle description differs from the visibly depicted workers/trucks.
Collision-prone scenarios.
IDM examples show pedestrians rushing toward a bus and participants entering the ego lane. These selected visualizations explain failure modes but do not quantify their prevalence or provide an independent realism assessment.
Semantic scenarios.
Examples include a school bus stop sign, military convoy, sprinkler truck and police standoff. Desired behavior can be detouring, waiting or obeying an instruction. For intended stopping cases, the progress goal is changed so waiting safely can count as completion.
5 Conclusion
Instruction-following agents plus constrained executors enable diverse interaction-focused driving tests. Results expose failures of the tested planners, with important limits: manually curated scenarios, appearance-based cases filtered out, model/runtime dependence, a replay-based collision track and no real-road validation. The work supplies a stress-test platform rather than evidence of deployment safety.
Appendix A Simulation Details
The appendix specifies the online semantic interface, route construction, feedback events, honking and the learned pedestrian motion generator. These details determine how much behavior comes from the model versus the simulator.
Ego observation.
Ego receives standard nuPlan map, route and tracked states. Semantic evaluation adds controlled-agent identifiers and brief visible-action descriptions generated at each query, such as a person waving for a car to stop. This information channel is an explicit part of the benchmark.
Semantic-track evaluation.
Online agents use Qwen3.6-27B with vLLM, prefix caching and up to 16 concurrent agents on eight RTX 3090 GPUs. Memory is reserved for a learned ego planner where needed. A complete semantic evaluation with IDM takes 15.1 hours under the stated setup.
Route planner.
Vehicles follow lane-graph routes extended through valid traffic-light-compatible edges, preferring low-curvature continuations. Maneuvers select a target lane and join it with a short S-curve before IDM executes the route under occupancy constraints. Missing lanes or road space cause rejection with feedback. A persistent driver identity can combine driving, parking, exiting and later pedestrian actions.
Simulator feedback.
Human feedback reports accepted/rejected object and vehicle interactions, invalid targets, insufficient proximity, infeasible preconditions and contact warnings. Vehicle events report maneuver start/completion/failure, blocked routes and timeouts. These deterministic events ground the agent’s next decision.
Honking channel.
An ego honk injects a structured stimulus into nearby agents’ observations. Their role instructions determine the response. The simulation models the event and its interpretation, rather than acoustic propagation or a learned audio signal.
A.1 Flow-matching-based Trajectory Generation
The human trajectory model is trained and tested separately from ego planning. It is conditioned on short history, desired direction and speed.
Model.
The adapted flow-matching backbone integrates an ordinary differential equation from Gaussian noise to produce 20 future points over two seconds. Conditioning includes two seconds of local history and the requested direction/speed.
Training data.
About 1.4 million pedestrian trajectories are sampled from nuPlan validation logs after balancing speed/direction bins. Testing uses a tenth as many trajectories from disjoint logs. This is a held-out generator test within that source split; it should not be described as training solely on nuPlan’s official training split.
Evaluation.
With ground-truth direction/speed, the generator achieves 0.151 m average and 0.203 m final displacement error. Direction-control tests average 3.68° error and 97.0% within 15°. The prose’s “≥97%” should be read as aggregate: Table 8 reports 94.7% at 90° and 95.8% at 135°. Sharp turns are harder than continuing the recent motion.
| Metric | Mean | |||||
|---|---|---|---|---|---|---|
| Direction Error (deg) | 1.87 | 2.66 | 4.64 | 4.73 | 3.51 | 3.68 |
| hit@ (%) | 99.2 | 99.3 | 94.7 | 95.8 | 97.6 | 97.0 |
| Average Displacement Error (ADE, m) | – | – | – | – | – | |
| Final Displacement Error (FDE, m) | – | – | – | – | – |
Appendix B Dataset Details
Dataset documentation covers manual scenario selection, prompt/placement refinement, the complete type list, scoring definitions and exact agent-output formats.
B.1 Dataset Construction
Annotators propose about 80 types, discard cases poorly supported by the simulator—especially appearance-dependent snow, potholes and small debris—and retain over 50 interaction types. Prompts are refined in simple scenes, then fixed while placements are adapted in real nuPlan validation scenes. Appendix ranges are 1–5 agents and 3–8 instances per type, differing from the main text. This iterative screening favors behaviors the current simulator and models can reliably express.
B.2 Scenario Type List
The full original table retains every listed collision-prone, general-semantic, honk-required and honk-penalized type. Some appear in both tracks. Examples span crossing crowds, unusual loads, emergency/official interactions, blocked roads and socially inappropriate honking.
| Collision-prone track | |
|---|---|
| Elderly pushes wheelchair across road | Children on scooters repeatedly cross the road |
| Pedestrian uses phone and drifts | Pedestrian chases windblown hat into road |
| Child breaks free from parent and runs into road | Walked dog chases squirrel into lane |
| Children play on road | Child runs to retrieve ball |
| Drunk person staggers into road | Courier parks tricycle then crosses with package |
| Delivery rider weaves fast through traffic | Salesperson solicits cars in roadway |
| Police question then chase suspect | Wedding convoy moves slowly with followers |
| Marathon runners race along roadside at varied speed | Crowd spreads into road as event ends |
| Crowd gathers at bike share station to scan and ride off | Pedestrian carries long pole sweeping adjacent lane |
| People dry crops on half roadway | Photographer shoots on road with cables |
| Courier places parcels onto roadside delivery motorcycle | One pedestrian crosses on red and others follow |
| Pedestrian dashes across to catch bus | Rider drags dead e-scooter into lane |
| Pedestrians weave between multiple buses at stop | Workers move cones to clear driving space |
| Drivers talk in lane after minor crash | Students cross and some stop to tie shoe |
| Pedestrians push strollers and carts while crossing abreast | Pedestrians file out of construction gap |
| Overwide foam-box cargo on tricycle spills into adjacent lane | Aid station runners converge and spread into lane |
| Food carts with large canopies cluster at intersection | Workers move between two parked trucks |
| Scattered bikes force pedestrians to detour into lane | Deer dart across road erratically |
| Semantic track | |
| General semantic | |
| Traffic police direct vehicles to stop | Traffic police direct vehicles to change lanes |
| Children dart out around stopped school bus | Military convoy with armed soldiers |
| Workers move between two parked trucks | Sprinkler truck operates slowly in center lane |
| Long large truck drives ahead | Police standoff with suspect |
| Vehicle on fire and bystanders panic | Teacher leads students crossing |
| Ambulance and medics treat injured | |
| Honk-required | |
| Street performer draws crowd | Bystanders block road at crash scene |
| Spectators line up and wait | Residents do square dancing |
| Promotional event draws lingering crowd | Tourists swarm around tour bus to take photos |
| Cat sleeps on the road | |
| Honk-penalized | |
| Municipal workers repair manhole and cables | Scavenger pushes overloaded cart slowly |
| Traffic police handle accident and direct flow | Protesters march with banners |
| Religious pilgrims process and kneel | Vendor cart overturns and pedestrians help pick up goods |
| Injured rider lies by fallen e-bike as bystanders gather | Animal rights protesters block livestock truck |
B.3 Metric Details
Scores combine route progress with collision/drivable-area compliance or semantic penalties. The metrics operationalize task success through explicit designer choices and should be compared within the appropriate track.
Collision-prone score.
Collision-prone score averages progress multiplied by drivable-area compliance and a no-at-fault-collision indicator. A collision zeros the rollout; otherwise driving outside the permitted area reduces its contribution.
General semantic score.
General semantic evaluation truncates progress at the first safety violation. Equation 1 accumulates the fraction of ego footprint overlapping a penalty region and scales it by a scenario-specific tolerance before clipping to [0,1]. Larger tolerance permits more overlap. Traffic-officer cases substitute a command-compliance penalty. Final score multiplies truncated progress by one minus the penalty.
Equation 1
| (1) |
Honk-sensitive score.
Each honk recipient has a role weight, and repeated honks accumulate a saturating penalty using decay 0.7. Weighted penalties are summed and clipped to [0,1], then discount truncated progress. The complete rule makes social appropriateness an explicit benchmark parameter.
B.4 Agent Output Format
Both complete YAML schemas are retained below as exact source text with indentation, comments and original field spellings. They document the permitted interface, not verified sample outputs.
Human agent output format.
The human schema includes analysis, target/stop distance, one or two perpendicular movement axes and their ratio, speed up to 4 m/s, physical actions with object/vehicle identifiers, soft actions with recipients, pose, stationary status, visible-action description and view rotation. A put-down action requires a target; unspecified actions can be null.
# Brief analysis.
analysis: ""
# Rough movement intent for the next about 2 seconds.
move:
# If you want to move directly toward a visible target agent, write its id here; otherwise null.
target_id:
# Optional stop distance when approaching target_id; otherwise null.
stop_at_distance:
# Main movement axis. Must be exactly one of W/A/S/D.
wasd_primary_axis: "W|A|S|D"
# Optional auxiliary axis for diagonal movement. Use null for pure primary-axis movement.
# It must be perpendicular to the primary axis, e.g. primary W can use auxiliary A or D.
wasd_secondary_axis: null # null | "W" | "A" | "S" | "D"
# Ratio primary:secondary. Output your own positive numeric ratio, e.g. "1:0", "1:1", "3:1", "5:3".
wasd_axis_ratio: ""
# Rough walking speed in m/s. Use 0.0 if you decide not to move now.
# range from 0.0 to 4.0, do not exceed 4.0
speed: 0.0
# Predefined simulator action. Set to null if no predefined action is needed.
predefined_action:
type: "null|enter_vehicle|exit_vehicle|pick_up_object|put_down_object|push_object"
# For enter_vehicle or exit_vehicle:
vehicle_id: "<some_vehicle_id>"
# For pick_up_object|put_down_object|push_object:
object_id: "<some_object_id>"
# For put_down_object, put_down_target is required: use "ground" to drop on ground, or an agent id.
# If it is null/empty, put_down_object will not take effect.
put_down_target: "<some_agent_id>|ground"
# Soft action outside predefined_action that may affect other agents, such as humans or vehicles.
# action_description: describe the action using a phrase
# action_description will be delivered to the target agents.
# Set soft_action to null if there is no action that affects other agents.
soft_action:
action_description: ""
target_agent_ids: []
pose: "stand|walk|run|sprint|sit|kneel|lie|crouch"
# Whether you are stationary at the current moment (true/false).
is_stationary:
# Briefly describe your current visible action in a short phrase.
brief_visable_action: ""
# Optional camera/view control for Stage 2 and future observations.
# Use none by default.
# left_90 means rotate the view counter-clockwise by 90 degrees to look left.
# right_90 means rotate the view clockwise by 90 degrees to look right.
# back_180 means rotate the view by 180 degrees to look backward.
view_rotation: "none|left_90|right_90|back_180"
Vehicle agent output format.
The vehicle schema includes parking, lane changes, starting, lateral offsets, target-lane driving, speed changes, reversing, exiting and honking. Command-specific fields control distance, timing, direction and acceleration bounds. Reversing requires zero current speed; honking may accompany another command. The schema retains all original comments and parameter names.
# Brief analysis.
analysis: ""
# High-level vehicle command executed by the simulator. If the agent does not want to execute any command or wants to keep the current state, set `command: null`.
command:
# Allowed `type` values:
# park: Park at `forward_distance` meters ahead; optional `lateral_distance` (meters, left negative / right positive).
# lane_change: Change lane. Set `direction=left|right`, `lane_change_time` (seconds, 2~10), and `forward_distance` (meters) to indicate where the lane change should finish.
# start_driving: Start from a parked state and merge into the nearest lane; `forward_distance` indicates the insertion distance along the lane.
# lateral_offset: Slightly offset within the current lane; set `direction`, `lateral_offset_time` (seconds), and `forward_distance` (meters) to indicate where the offset should end. For an actual lane change, use `lane_change` instead.
# drive_to_lane: Drive to a specified lane (useful for turns / U-turns).
# accelerate: Set `target_velocity` (m/s) with `max_accel` (m/s^2, 0.5~3.0).
# decelerate: Set `target_velocity` (m/s) with `max_decel` (m/s^2, 0.5~3.0).
# reverse: Reverse for `reverse_distance` meters (only takes effect when current speed is 0). Optional `use_last_path=true` to reverse along the recently driven path.
# exit_vehicle: Let the human driver exit the vehicle (only when a human driver is inside).
# honk: Honk once; nearby agents will receive a honk stimulus.
type: "park|lane_change|start_driving|lateral_offset|drive_to_lane|accelerate|decelerate|reverse|exit_vehicle|honk|null"
# For park / start_driving / lane_change / lateral_offset:
forward_distance:
# For lane_change: smaller => more urgent (higher curvature), larger => more comfortable.
lane_change_time:
# For park (optional):
lateral_distance: null
# For lane_change / lateral_offset:
direction: "left|right"
# For lateral_offset:
lateral_offset_time:
offset: 0.5
# For drive_to_lane:
lane_id:
# For accelerate / decelerate:
target_velocity:
max_accel:
max_decel:
# For reverse:
reverse_distance: 3.0
use_last_path: false
# Whether honking, together with other commands (does not affect the main command).
honk: false