Overview
This edition follows the complete twenty-page, 15 June 2026 v1, retaining all fifteen figures, seven numbered tables, the score definition and the full thirty-two-category threat taxonomy. The benchmark judges additional risk relative to a recorded expert; a no-threat label is not an unconditional driving-safety certificate.
At a glance
FluidTest evaluates whether a planner’s trajectory adds a plausible hazard beyond an expert reference in the same scene. Trained human annotation is combined with a threat taxonomy, structured evidence checks and a three-agent review loop. On a deliberately filtered 151-scene WOD-E2E subset, final human-reviewed additional-threat rates are 65% for the authors’ Poutine implementation and 51% for RAP. These are benchmark judgments about proposed trajectories, not measured real-world collision probabilities.
1 Introduction
Low displacement error and high aggregate simulator or human-feedback scores can conceal semantically poor plans. FluidTest replaces an absolute scalar rating with a pairwise question about additional unnecessary maneuvers or future risk. Its taxonomy adapts the California Driver’s Handbook into thirty-two operational categories, including both physical hazards and driving-quality issues.
The study filters WOD-E2E validation scenes to remove obvious future uncertainty, trivial constant-velocity expert plans and severe camera/projection errors. The resulting val151 is a selected diagnostic subset, not a representative estimate over ordinary driving or every long-tail condition. Poutine is a self-reimplementation because official code/checkpoints were unavailable; RAP and an internal VMA planner provide comparisons. Matching aggregate Poutine metrics does not prove behavioral identity with the original model.
2 Evaluations for Long-tail Planning
The paper argues that evaluation should separate genuinely hazardous alternatives from harmless deviations and explain why a plan is problematic. The proposed expert-relative judgment makes that comparison explicit, while depending on the chosen reference, available evidence and calibrated human preferences.
Simulation benchmarks are saturated.
nuPlan, NAVSIM and pseudo-simulation evaluate predefined outcomes under particular scene and agent models. Limited event sets, nonreactive replay and simplified interaction can reward behavior that would be undesirable around real people. Figure 1 collects reported scores over release dates; score saturation is motivation, not an independent validation of FluidTest’s labels.
Human rater feedback is useful but not sufficient.
WOD-E2E’s Rater Feedback Score incorporates human preferences but assigns a scalar through its own sparse reference procedure. The study finds cases where high RFS misses additional threats. This supports auditing what a metric measures, rather than treating every human-derived score as interchangeable with the proposed conservative threat judgment.
VLMs as verifiers and simulators.
Accurate interaction simulation remains difficult, as illustrated by InterSim. Inspired by Logic-LM, PAL and LLM+P, FluidTest uses language/vision models to propose semantic interpretations and structured decision graphs to organize verification. The graph makes evidence and routing inspectable; its perceptual inputs and judgments remain fallible.
3 FluidTest
3.1 Reasoning and Metrics
For each scene, a confirmed threat set is intersected with a chosen category subset. The violation rate is the fraction of scenes with at least one such label; NATR is one minus that rate. Overall NATR uses the entire taxonomy, so multiple threats in one scene count once.
The taxonomy includes inefficiency, purposeless drift and route compliance as well as collision and right-of-way risks. NATR therefore measures absence of the benchmark’s confirmed additional threats, not merely absence of collisions. It also depends on evidence sufficiency, conservative future-risk judgments and how unresolved cases are handled.
Equation
3.2 Human Annotation Protocol and WebUI
Annotators compare the planner and expert trajectories under identical visual context, choosing Y, N or Not Sure. Y means added risk or an unnecessary maneuver; N allows geometrically different alternatives that add nothing worse. Not Sure is an abstention when future uncertainty, unexplained expert behavior or projection noise prevents a fair comparison.
The expert provides context, but is not assumed to make the scene perfect. The protocol asks whether the prediction is worse relative to that reference; it cannot determine absolute safety solely from agreement with expert behavior.
3.3 Testing Protocol and FluidTest Safety Arena (Gold)
The Prosecutor proposes threat classes, the Detector runs evidence-grounded checks, and the Judge reflects on consistency and requests revision or further visual evidence. Cropping and zooming support inspection, following the tool-and-reflection ideas of ReAct and Reflexion. Abstentions go to human reviewers.
The paper describes a shared Safety Arena server with a dynamic closed set and trained human review for controlled submissions. Local use is indicative; Table 1 is the final human-reviewed result. Structured explanations and agreement do not independently establish that every missed hazard has been found.
4 Experiment Results and Insights
The experiments examine annotator consistency, final planner scores, overlap of selected difficult cases and the relation between standard metrics and added-threat labels. These analyses concern different label stages and subsets; their denominators should not be silently interchanged.
4.1 Human Label Consistency and NATR Learning
Human consistency requires task-specific prior knowledge.
Five annotators evaluate Poutine outputs, including three project researchers. After excluding uncertain cases, the researcher group achieves reported overlap 79.31% and Fleiss’ κ=0.7145. Figure 3 shows stronger agreement within that group than across all five.
Appendix B explains that examples and task-specific calibration materially improved two researchers’ initially noisier labels. The two nonresearchers did not receive the same training because of time limits. The comparison therefore does not isolate expertise from training, and agreement among calibrated annotators is not independent ground-truth safety validation.
Learning human safety preferences requires additional alignment.
Table 2 reports 55% accuracy for untuned Qwen3.5-9B, 50% for ChatGPT-5.4-Mini xHigh and 83% for fine-tuned Qwen3.5-4B on the Poutine-Union evaluation. The latter has precision 75% and recall 100%; the separate 9B SFT/Poutine-1K row reports 79% accuracy.
These are different model/data conditions, not a controlled size-only comparison. The Union description uses a Y label from any researcher; class prevalence and exact test counts are not supplied here, so “near random” should be understood as the authors’ characterization rather than a verified prevalence-adjusted baseline. SFT uses a reported 10% test split.
4.2 Final FluidTest Benchmark Results
Final NATR is 0.35 for Poutine, 0.47 for VMA and 0.49 for RAP, while 5-second ADE is 2.44, 2.46 and 2.62 respectively. Thus the best imitation-distance result is not the best expert-relative threat result on this subset. VMA leads Quality/Priority, while RAP leads Collision/Lateral category scores.
The category rates overlap and are not additive; overall NATR counts any confirmed category. VMA is an internal stress-test model. These finite-sample results do not establish a deployment-wide safety ranking or universal superiority of one planner.
4.3 Reliable Hard-Case Metrics Should be Stable Across Planners
The authors propose shared difficult scenes across different planners as a desirable benchmark property, motivated by item response analysis and error-overlap analysis. Shared failures can reveal common scene difficulty, but genuine weaknesses can also be planner-specific.
The reported heatmaps and Table 4 use directional intersection divided by each individual failure-set size. Although the prose mentions Jaccard, these percentages are not intersection over union. They also vary with the fraction of scenes selected, so a larger raw overlap alone does not establish planner independence or safety validity.
Closed-loop PDMS is weakly universal.
For five NAVSIM planners, PDMS=0 sets show directional overlaps around 17–44%. This documents differing selected failures under those planners and that threshold. It does not by itself prove that the unshared failures are metric artifacts rather than real model-specific errors.
RFS is less universal than open-loop displacement.
Poutine and RAP’s worst 10% ADE sets share nine scenes and have union twenty-three, giving 56.25% directional overlap but Jaccard 39.13%. The corresponding RFS sets share four with union twenty-eight: 25% directional overlap and Jaccard 14.29%. These distinguish the exact statistic from the surrounding Jaccard terminology.
NATR is more reliable and informative.
The NATR(Y) row shares forty-six scenes with union ninety-seven. Its two directional rates are 73.02% and 57.50%, while Jaccard is 47.42%. The underlying sets have sixty-three and eighty scenes, far larger than the sixteen-scene worst-10% tails.
The reported 73% is therefore one direction, not a symmetric agreement statistic. Appendix D explores larger displacement/RFS thresholds, but selection prevalence remains important: even independent random sets have expected directional overlap equal to the other set’s selected fraction. These data support inspecting shared threats, not an unrestricted claim of metric universality.
4.4 Relationship with Existing Metrics
For each planner, the study bins scenes by 3-second ADE, 5-second ADE, FDE or RFS and plots the fraction with additional threats. The visual associations are descriptive on val151; bin counts and uncertainty intervals are not provided, and no calibrated universal hazard predictor is established.
Average displacement errors are limited safety indicators.
Displacement captures imitation mismatch but not its semantic direction. A small deviation toward another vehicle can be hazardous, while a larger safe alternative can differ from the expert. Figure 7 preserves the motivating examples and their magnified details.
Longer-horizon displacement is more threat-sensitive.
Longer-horizon ADE and final displacement are more associated with threats in the shown examples because committed interactions, lane choices and obstacles emerge later. This is an empirical association within the tested data, not a proof that increasing prediction horizon always improves safety assessment.
Scalar human feedback does not reliably track additional threats.
The highest RFS region still has about 40% additional threats in the plotted analysis. Appendix C shows perfect-score examples involving failure to yield, creeping into an intersection and heading toward the road edge. These demonstrate specific blind spots of the reference/scoring procedure rather than equating every high-RFS plan with unsafe behavior.
5 Conclusion
FluidTest contributes an expert-relative annotation question and an auditable semantic review process. Its evidence shows that selected strong planners can receive favorable existing metrics while exhibiting additional threats under this protocol. The human-calibrated definitions and multi-stage review are central to what the score means.
6 Limitations
The study does not yet establish scalable annotation across diverse backgrounds. Reliable projection, visual grounding and sufficient scene context remain prerequisites, and the thirty-two-category taxonomy is incomplete. Filtering difficult-to-judge scenes, using a small selected benchmark, imperfect automated perception and unresolved judgments further limit any interpretation as comprehensive safety certification.
Appendix A WebUI for Annotators and Labeling Guidance
The WebUI shows scene metadata, ego/trajectory states, three stitched front views and three rear views, with one second of past playback. Lazy loading and a magnifier support inspection. Annotators register driving experience and review instructions before choosing Y, N or Space/Unsure.
How to label potential hazards.
The guidance asks whether a prediction introduces something meaningfully worse than the expert, including a plausible future hazard or unnecessary maneuver. Existing hazards in the expert path are not automatically a Y for the planner. When the expert’s behavior cannot be explained well enough to compare fairly, the correct response is abstention.
Appendix B Human Label Cost
Labeling cost analysis.
The prose reports average labeling time 5.292 seconds; Figure 11 instead labels mean 6.9 seconds, median 4.9 seconds, 90th percentile 12.7 seconds and a 0–90% average of 5.2 seconds for VMA trajectories. The exact relationship between the prose figure and trimmed plot statistic is not explained.
Either displayed mean puts one pass over 151 scenes below twenty minutes, but this excludes the separate cost of training annotators, collecting multiple judgments, tool-based verification and final human adjudication.
Training annotators for additional-threat labels.
Only one researcher initially knew the entire labeling logic, taxonomy and classifier pipeline. Expanded examples improved agreement for the other two. The nonresearchers were not comparably retrained, so their lower consistency is a limitation of this study design, not evidence that nonresearchers cannot learn the task. Standardized calibration and consistency checks are proposed for the shared arena.
Annotator disagreement cases.
The four disagreement examples involve ambiguous one-way/road-width interpretations and missed small deviations toward cones or double-yellow markings. Magnification can recover evidence, but differing defensive-driving preferences also matter. Multiple trained annotators are needed to reduce dependence on one person’s assumptions or missed detail.
Appendix C RFS Failure Cases
The three RFS=10 examples compare cyan expert, green maximum-score RFS reference and red planner paths. Both the reference and Poutine fail to yield to a cyclist in one case; RAP moves into an intersection while the expert waits; VMA heads toward an edge despite a safer turning reference.
These are qualitative counterexamples to perfect RFS implying no added threat. The third also illustrates a claimed weakness of displacement-heavy scoring on high-curvature paths. The images retain all original trajectory overlays, zooms and scene labels.
Appendix D Worst-Case Overlap Depends on the Tail Threshold
Directional overlap rises as larger tails are selected. At 50%, ADE reaches 60.53% and RFS 69.74%, compared with their 10% values of 56.25% and 25%. NATR’s directional pair is 73.02%/57.50%, so comparisons depend on direction as well as selected fraction.
Figure 13 compares thresholded metric selections with binary human labels; its row labels use ADE above average and RFS below average, whereas the surrounding prose describes worst-50% selections, which need not be equivalent. The visual patterns show disagreement, but the caption’s general claims should be read as subset-specific. Table 4’s NATR sets and Figure 14’s pre-review Judge counts are intermediate analyses, distinct from Table 1’s final human-reviewed rates.
Appendix E Prosecutor Threat Classifier
Table 3 reports unfiltered 2B/4B accuracy of 0.7200 for both, and filtered accuracy 0.9286/1.0000 after removing human-disagreement cases. The filtered 2B precision is 1.0 and recall 0.8571; the 4B reports 1.0 on all displayed measures.
The appendix prose also refers back to the 79–83% Table 2 results, so these rows should not be silently merged. Removing ambiguous cases changes the evaluation distribution, and exact held-out sample counts are not reported. Perfect accuracy on that filtered set neither proves broad threat detection nor establishes that all original errors arose solely from label ambiguity.
Appendix F Judge Result Details
Figure 14 shows pre-adjudication Judge counts: RAP 69 threats/77 no-threats/5 abstentions, Poutine 76/69/6 and VMA 66/79/6, each totaling 151. Similar abstention counts do not prove unbiased missingness or unchanged final ranking: the content of unresolved cases also matters.
These raw Judge outputs differ from the final human-reviewed rates in Table 1. The paper routes abstentions to human reviewers; the raw blue cells should not be substituted for final NATR or treated as certified safe outcomes.
Appendix G Threat Definitions and Boundaries
Table 7 retains every definition and boundary for the thirty-two labels: signal/stop control, yielding, collision courses, following distance, lane and roadway use, special lanes, speed, weaving, navigation/temporary instructions and six driving-quality categories. Several definitions require a plausible future collision course without visible overlap inside the current five-second path.
The rules explicitly use context and lawful/necessary exceptions. They are the benchmark’s authored operational definitions, not a complete legal analysis. Comparing expert and planner swept corridors, rather than only centerlines, is especially important for collision judgments.
Original selectable table
| Criterion | Union scenes | Shared scenes | Shared / |RAP set| | Shared / |Poutine set| |
|---|---|---|---|---|
| 5sADE worst 3% | 9 | 1 | 20.00% | 20.00% |
| 5sADE worst 10% | 23 | 9 | 56.25% | 56.25% |
| 5sADE worst 30% | 65 | 27 | 58.70% | 58.70% |
| 5sADE worst 50% | 106 | 46 | 60.53% | 60.53% |
| RFS worst 3% | 9 | 1 | 20.00% | 20.00% |
| RFS worst 10% | 28 | 4 | 25.00% | 25.00% |
| RFS worst 30% | 67 | 25 | 54.35% | 54.35% |
| RFS worst 50% | 99 | 53 | 69.74% | 69.74% |
| NATR (Ours) | 97 | 46 | 73.02% | 57.50% |
Original selectable table
| State | Meaning |
|---|---|
| confirmed | Evidence sufficiently supports the threat. |
| unproven | Threat hypothesis is plausible but unsupported by sufficient evidence. |
| lawful_exception | Behavior appears risky but is justified by traffic context or legal exception. |
| routed | Evidence better matches another sibling threat. |
| not_applicable | Preconditions for this threat are absent. |
Original selectable table
| Node ID | Input | Function |
|---|---|---|
| NODE_LANE_OCCUPANCY | Trajectory + lane masks | Determines whether the predicted trajectory occupies a valid lane region and measures lane overlap duration. |
| NODE_ROAD_SIDE | Trajectory + roadway segmentation | Determines whether the trajectory travels on the wrong roadway side or exits drivable space. |
| NODE_SIGNAL_STATE | Traffic-light crop | Detects signal state and movement permissions. |
| NODE_STOP_CHECK | Ego velocity profile | Determines whether the ego reaches a full stop before entering a controlled area. |
| NODE_PEDESTRIAN_CONFLICT | Pedestrian detections + trajectory | Detects whether the trajectory conflicts with pedestrian-priority zones. |
| NODE_VEHICLE_CONFLICT | Vehicle tracks + trajectory | Determines whether another vehicle possesses right-of-way conflict priority. |
| NODE_STATIC_COLLISION | Static-object masks + trajectory | Detects overlap or collision course with barriers, cones, curbs, poles, or other fixed objects. |
| NODE_STATIONARY_VEHICLE | Parked-vehicle masks + trajectory | Detects overlap or insufficient clearance to stationary vehicles. |
| NODE_FOLLOWING_DISTANCE | Relative speed + spacing | Estimates whether following distance is unreasonable for visible conditions. |
| NODE_LANE_CHANGE | Temporal trajectory sequence | Determines whether the ego commits to a lane change maneuver. |
| NODE_NAVIGATION_MATCH | Navigation command + lane structure | Checks whether the planner prepares correctly for a required turn or route branch. |
| NODE_SPEED_REASONABLENESS | Ego speed + scene context | Determines whether the speed is unsafe for visible environmental conditions. |
| NODE_WORKZONE | Cone/barrier segmentation | Detects temporary work-zone narrowing and clearance risk. |
| NODE_TEMP_CONTROL | Flagger/officer/cone cues | Detects temporary traffic-control instructions and diversion paths. |
Original selectable table
| Group | Threat label | Definition |
|---|---|---|
| Signal/stop control | Red light turning no full stop | Vehicle makes a turn on red without first coming to a full stop where the red-turn movement is allowed only after stopping and yielding. |
| Signal/stop control | Red light violation | Vehicle enters or proceeds through a steady red signal or makes a prohibited movement on red. |
| Signal/stop control | Stop sign no full stop | Vehicle fails to come to a full stop where a stop is required. |
| Signal/stop control | Yellow phase noncompliance | Vehicle enters on a steady yellow when a normal safe stop before the controlled entry was reasonably available. |
| Right-of-way | Failure to yield pedestrian | Vehicle fails to yield to a pedestrian in a marked or unmarked crosswalk or equivalent pedestrian priority zone. |
| Right-of-way | Failure to yield vehicle | Vehicle enters, merges, or turns across another vehicle’s or bicyclist’s right-of-way unsafely. |
| Collision | Collision with static objects | Predicted ego travel remains on a path that would strike, scrape, or is clearly driving into a plausible future collision course with, a static object; judge the ego swept footprint/corridor, not only the centerline overlay, and visible overlap within the current 5s path is not required when ego is aimed toward the object and GT is not. |
| Collision | Collision with stationary vehicle | Predicted ego travel remains on a path that would strike a parked or otherwise stationary vehicle. |
| Collision | Vehicle collision course | Predicted ego travel remains on a path that would strike another vehicle unless one of the vehicles changes course, outside narrower right-of-way, tailgating, or lane-change explanations. |
| Following distance | Following too closely | Vehicle follows another vehicle more closely than is reasonable and prudent for the visible speed and traffic context. |
| Lane use | Lane straddling | Vehicle drives for a sustained period between two lanes instead of committing to one lane. |
| Lane use | Unsafe lane change | Vehicle changes lanes without reasonable safety. |
| Lane use | Unsafe passing | Vehicle passes in a prohibited or unsafe manner. |
| Lane use | Wrong side of road | Vehicle travels on the oncoming or left side of the roadway outside a clearly lawful exception. |
| Lane use | Wrong way one way road | Vehicle travels opposite the designated direction on a clearly one-way roadway. |
| Roadway position | Off road driving | Vehicle leaves the ordinary drivable lane or roadway and drives on a sidewalk, curb area, shoulder, dirt, gore, median edge, parking apron, private-property drive/frontage area, landscaped area, or another non-roadway area without a clearly lawful or necessary explanation. |
| Special-lane misuse | Driving in bike lane | Vehicle uses a bicycle lane as a normal travel lane beyond narrow lawful exceptions. |
| Special-lane misuse | Driving in bus only lane | Vehicle uses a lane reserved for public transit buses outside allowed conditions. |
| Special-lane misuse | Improper shoulder pass or bypass | Vehicle uses the shoulder, emergency lane, or other off-main-traveled roadway edge as a pass, bypass, or ordinary travel path outside clear lawful exceptions. |
| Speed | Speeding posted limit | Vehicle clearly exceeds a posted speed limit when the limit and excess are visually well supported. |
| Speed | Unsafe speed for conditions | Vehicle is traveling faster than is reasonable for the visible scene conditions. |
| Reckless pattern | Speed contest or exhibition | Vehicle behavior suggests racing, competitive acceleration, or exhibition of speed. |
| Pattern | Aggressive weaving | Vehicle repeatedly changes lanes or swerves laterally in a rapid, conflict-seeking, or intimidation-like manner that materially raises interaction risk. |
| Route compliance | Late navigation lane preparation | Vehicle stays out of the lane position needed for a near-term instructed turn, exit, or branch until the last reasonable preparation chance is closing or already missed, making future route failure or a forced risky late lane change likely. |
| Route compliance | Not following navigation instruction | Vehicle materially deviates from a clear near-horizon navigation instruction by committing to the wrong turn, branch, or lane stack without a visible reason. |
| Temporary control | Not following temporary traffic instructions | Vehicle materially disregards a clear temporary traffic-control instruction from an officer, flagger, construction guide, cone-directed diversion, or similar active traffic-direction setup. |
| Policy quality | Blocking traffic without necessity | Vehicle stops or crawls in a way that unnecessarily obstructs a lane, crosswalk, or intersection without a legitimate traffic-control reason. |
| Policy quality | Low efficiency driving | Vehicle remains static or crawls at an obviously inefficient pace without a visible traffic-control, safety, conflict, or route-based reason. |
| Policy quality | Meaningless lane change | Vehicle changes lanes without a reasonable traffic-related purpose or weaves left-right-left without improving progress. |
| Policy quality | Meaningless lateral drift | Compared with GT, the predicted vehicle adds or retains material lateral deviation without a clear purpose, while not fully becoming a lane change, lane straddling, or roadway departure. |
| Policy quality | Noncommittal merge | Vehicle begins a merge or turn commitment and then hesitates or aborts in a way that reflects noncommittal go-no-go behavior and may disrupt surrounding traffic. |
| Policy quality | Stops without escape room | Vehicle stops too close to an obstacle or blocked area and leaves insufficient space for a normal forward go-around if the obstacle remains stationary. |
Appendix H Threat Decision Graphs and Reusable Nodes
The paper includes one complete authored graph, for unsafe_lane_change, and links the external threat-graph site for the remaining full graphs. This edition preserves the entire graph and all accompanying reusable-node and terminal-state tables; the external collection is separate from the paper’s contents.
Original decision-graph labels
N1. Completed lane change visible? N2. Better explained as vehicle right-of-way taking? N3. Better explained as aggressive weaving? N4. Inadequate safety margin to adjacent traffic? N5. Forced reaction, near miss, or collision? N6. Legitimate merge, route, obstacle, or emergency reason? N7. Better explained as noncommittal merge? N8. Lane change lacks traffic or route purpose? Confirmed unsafe_lane_change Routed failure_to_yield_vehicle Routed aggressive_weaving Routed noncommittal_merge Routed meaningless_lane_change Not applicable Not applicable Unproven Unproven yes no abstain no yes abstain no yes abstain yes no abstain yes/no/abstain yes no abstain yes no/abstain yes no abstain
H.1 Graph Design Principles
Graphs require observable evidence, reusable checks, recorded routing/evidence and multiple terminal outcomes. Table 6 lists fourteen shared nodes covering lane/road side, signals/stopping, conflicts/collisions, following distance, lane changes, navigation, reasonable speed, work zones and temporary controls.
Figure 15 first requires a completed lane change, then routes right-of-way taking or aggressive weaving to narrower labels. Inadequate margin leads to confirmation through a forced-reaction question whose yes/no/abstain branches all confirm; an observed near miss or collision is therefore not required on that branch. The alternative branch checks legitimate maneuver reasons, noncommittal merge and purposeless lane changes. This explains the actual graph flow rather than relying only on its caption.
H.2 Terminal States
The five terminal states are confirmed, unproven, lawful_exception, routed and not_applicable. Routed triggers another graph. The other three nonconfirmed states are treated as no threat for the current category before Judge review, even though unproven specifically means insufficient supporting evidence.
Absence of confirmation is consequently different from proof of absence. Auditable intermediate states help diagnose that distinction; final abstentions still require the human-review procedure described earlier.