Literature
Display
← All papers

SidewalkBench: Benchmarking Visual Navigation on Urban Sidewalks

Zhizheng Liu · Honglin He · Vivek Alumootil · Akshat Pandya · Brad Squicciarini · Wayne Wu · Bolei Zhou

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

SidewalkBench provides controlled sidewalk-navigation tests spanning static geometry, specified pedestrian responses, and long routes. Its appendices document simulation interfaces, data ablations, complete benchmark results, and the small physical-transfer experiment.

Refer to caption
Figure 1: Left: Challenges of sidewalk navigation include static obstacles, dynamic pedestrians, diverse structures and layouts, and long-horizon distances. Right: SidewalkBench can simulate diverse and realistic scenarios for standardized evaluation of visual navigation on urban sidewalks.

At a glance

The benchmark combines procedural urban environments, scanned streets, and event-triggered pedestrians in Isaac Sim. Nine visual-navigation policies are evaluated on 330 basic-layout tests, 800 pedestrian-reactive tests, and 105 long routes. Sidewalk-specific training markedly improves navigation, yet pedestrian interaction and sustained operation remain difficult. Synthetic-data finetuning improves two targeted behaviors in simulation and a small physical study; this is a benchmark and initial transfer result, not a deployment-safety certification.

1 Introduction

Short, separately designed evaluations obscure how models fail on curved sidewalks, crossings, social groups, and extended travel. SidewalkBench standardizes those challenges while exposing data-domain effects and failure categories. The central observation is that sidewalk-specific data helps more than model size alone in the tested comparisons. The protocol, environment assumptions, and targeted ablations limit how broadly that ranking should be generalized.

2 Related Work

The work connects navigation foundation models, social-navigation benchmarks, and urban simulation. Its contribution is a common sidewalk-focused evaluation with both static-layout and controlled pedestrian challenges.

Visual Navigation Foundation Models

Prior policies use regression, diffusion, world models, reinforcement learning, or language-conditioned action generation. Their training domains and interfaces vary substantially. The benchmark compares these families under a common sidewalk-following setup rather than evaluating every model’s full original goal-conditioned capability.

Simulation Benchmarks for Visual Navigation

Indoor benchmarks, short reconstructed outdoor routes, and existing social-navigation simulators cover different subsets of the problem. SidewalkBench combines larger outdoor layouts, scanned appearance/geometry, and dynamic pedestrians, enabling controlled comparisons without claiming simulated humans capture every real interaction.

Urban Simulators for Embodied AI

Isaac Sim and Isaac Lab supply physics and parallel camera rendering. A separate pedestrian pipeline reduces animation overhead and allows more varied motions than fixed walking/standing assets. Rendering throughput is a component measurement, distinct from end-to-end policy latency.

3 SidewalkBench Design

The benchmark separates scene construction, pedestrian behavior/animation, and scenario definitions. Procedural scenes favor controlled variation; scans favor visual and geometric realism. Both remain simulated evaluations.

Refer to caption
Figure 2: Overview of scene types in SidewalkBench.

3.1 Scene Types

Procedural and scanned environments provide complementary tests. Agreement in rankings across them is useful evidence about those scene distributions, but is not by itself validation on physical streets.

Procedurally Generated Scenes

Seven primitive block types are joined into urban topologies. Five lateral functional zones separate roads, sidewalks, curbs/gutters, verges, and frontage; textures, sky maps, objects, and placements vary using UrbanVerse-100K assets. The resulting one hundred environments each span two by two kilometres. These scene dimensions are not the distance actually evaluated.

Real-world Scanned Scenes

An XGRIDS device with LiDAR and four cameras supplies scans. Gaussian splats render appearance while scanned meshes provide geometry and physics, followed by sidewalk/crosswalk annotation. Eleven reconstructed scenes average 150 by 150 metres. Scanning improves realism but does not remove simulator, camera, or control differences.

3.2 Pedestrian Simulation

A high-level state machine selects behavior; a separate low-level motion/rendering pipeline animates it. Controlled event tests and reciprocal long-route pedestrians serve different purposes and should not be treated as the same crowd model.

Event-based High-level Behaviors

Behavior graphs connect walking, waiting, object interaction, and social interaction. Relative robot position triggers eight scenarios: obstruction, conversation, queueing, frontal/lateral approach, overtaking, crossing, and a stopping gesture. Handwritten trajectories and speeds force relevant encounters, making them reproducible stress tests rather than samples of natural encounter frequency.

Refer to caption
Figure 3: Illustration of the event-based behaviors. The blue arrow indicates the moving direction of the pedestrian after the event is triggered and the red arrow indicates the robot’s intended reaction.

Flexible and Efficient Low-level Animation

SMPL poses drive textured meshes or Gaussian human avatars, drawing motion from generators and datasets. Nvdiffrast pedestrian rendering is composited with simulator backgrounds. The headline roughly sixty-fold improvement concerns mesh-animation/rendering versus Isaac’s native character module, not full simulation or every avatar representation.

3.3 Scenarios

The 330 static tests cover straight, curved, and crossing blocks over ten to twenty metres: one hundred procedural and ten scanned examples per type. Reaching the goal within sixty seconds succeeds; obstacle contact or leaving the sidewalk fails. Eight hundred pedestrian tests add one hundred cases per behavior, entirely in procedural scenes; crossing uses crosswalks and the other behaviors use straight sidewalks.

The 105 long routes exceed one hundred metres and total 36.5 km, with one hundred procedural and five scanned cases. Alternative routes are blocked to leave a single path. After failure, the robot resets to the closest waypoint and continues, so the metric measures interventions along reset-assisted travel rather than autonomous end-to-end completion.

Evaluation Metrics

Static tests emphasize route completion; pedestrian tests use task success. Long routes count collisions, out-of-lane events, and freezing per one hundred metres, together with average speed. Appendix metrics add success-weighted path length, pedestrian collisions, and minimum distance conditioned on successful trials. Those quantities answer different questions and are not interchangeable.

4 Experiments

Evaluation spans nine pretrained or adapted navigation models and three scenario families, followed by data ablations and a narrow synthetic-to-physical finetuning experiment. Full original tables retain five-run standard deviations where reported.

4.1 Navigation Models

General models are ViNT, NoMaD, MBRA, CityWalker, and InternVLA-N1; sidewalk models are MIMIC, S2E, FlowPilot, and OpenPilot. The last is a reimplementation retrained on sidewalk data, not the unchanged public driving checkpoint. It has eight million parameters and twenty-hertz observations; InternVLA has seven billion and five-hertz inputs. Architecture, data, resolution, and input frequency all differ, so the main cross-model ranking alone cannot isolate their causal effects.

Method Data (# Hours) Goal Encoder Decoder Resolution FPS Model Size
ViNT [50] General Image CNN Regression 85×\times64 4 24 M
NoMaD [51] General Image / None CNN Diffusion 96×\times96 4 16 M
MBRA [26] General Point CNN Regression 96×\times96 5 63 M
CityWalker [39] General Point ViT Regression 630×\times350 1 201 M
InternVLA-N1 [55] General Language VLM Diffusion 384×\times384 5 7 B
MIMIC [24] Sidewalk (50) None ViT Reg. + Cls. 256×\times256 5 79 M
S2E [25] Sidewalk (100) Point ViT Reg. + Cls. 256×\times256 5 95 M
FlowPilot [23] Sidewalk (300) Point / None ViT FM. + Cls. 512×\times288 5 230 M
OpenPilot [10] Sidewalk (1000) None ViT Regression 352×\times128 20 8 M
Table 1: Detailed comparison of visual navigation models evaluated on SidewalkBench.

4.2 Unit-test Scenarios

Sidewalk-trained MIMIC obtains straight-route completion of 77.9% in procedural and 97.1% in scanned scenes, versus ViNT’s 15.4% and 20.8%. OpenPilot performs strongly with one thousand training hours. The main prose calls 86.5% and 89.8% its averages, but Tables 6–7 identify those as curved-block results; Table 5 gives 91.1% average procedural completion. The notes use the tabulated metric labels. Scanned-scene results remain simulated, and completion is not success without a final failure.

Refer to caption
Figure 4: Evaluation results of unit-test scenarios. We show route completion rate (%) on different sidewalk structures in both procedurally generated (PG) and real-world scanned scenes.
Refer to caption
Figure 5: Qualitative results. (a)-(d): Success examples. (a) Avoiding a frontal approaching pedestrian. (b) Identifying the ramp. (c) Following the sidewalk curve. (d) Narrow passing. (e)-(h): Typical failure cases. (e) Ignoring social groups. (f) Failure to yield to a laterally crossing pedestrian. (g) Colliding with pedestrians on the crosswalk. (h) Inability to interpret pedestrian gestures.

4.3 Pedestrian-reactive Scenarios

Table 2’s across-model means fall from 38.5% without pedestrians to 32.3% with a stationary obstruction, 17.2% for conversing groups, and 12.3% for queues. Frontal/lateral approach average 17.4%/19.8%; overtaking reaches 43.2%. Crossing and gesturing remain especially weak at 3.8% and 1.7%. Several narrative averages differ slightly from the table, which is retained unchanged. OpenPilot can lead static tests yet score zero crossing success; broad lane-following competence does not imply these interaction skills.

Method Static Obstructing Conversing Queueing Frontal Lateral Overtaking Ped-Crossing Gesturing
ViNT [50] 0.6±0.90.6_{\pm 0.9} 0.2±0.40.2_{\pm 0.4} 0.0±0.00.0_{\pm 0.0} 0.2±0.40.2_{\pm 0.4} 0.6±0.90.6_{\pm 0.9} 0.6±0.50.6_{\pm 0.5} 0.0±0.00.0_{\pm 0.0} 0.6±1.30.6_{\pm 1.3} 0.0±0.00.0_{\pm 0.0}
NoMaD [51] 14.4±3.514.4_{\pm 3.5} 13.4±5.013.4_{\pm 5.0} 8.6±1.18.6_{\pm 1.1} 4.4±2.54.4_{\pm 2.5} 13.8±2.513.8_{\pm 2.5} 3.4±1.13.4_{\pm 1.1} 10.4±2.610.4_{\pm 2.6} 4.6±2.14.6_{\pm 2.1} 0.0±0.00.0_{\pm 0.0}
MBRA [26] 32.4±9.932.4_{\pm 9.9} 19.2±2.019.2_{\pm 2.0} 9.4±1.99.4_{\pm 1.9} 6.0±2.86.0_{\pm 2.8} 19.0±3.219.0_{\pm 3.2} 4.0±1.24.0_{\pm 1.2} 15.8±2.415.8_{\pm 2.4} 5.8±1.95.8_{\pm 1.9} 0.0±0.00.0_{\pm 0.0}
CityWalker [39] 2.0±1.22.0_{\pm 1.2} 2.6±1.12.6_{\pm 1.1} 1.0±0.71.0_{\pm 0.7} 1.4±0.51.4_{\pm 0.5} 0.6±0.90.6_{\pm 0.9} 0.8±0.80.8_{\pm 0.8} 8.6±3.58.6_{\pm 3.5} 7.2±2.07.2_{\pm 2.0} 11.6±3.211.6_{\pm 3.2}
InternVLA-DualVLN [55] 20.0±3.720.0_{\pm 3.7} 17.8±4.117.8_{\pm 4.1} 12.8±3.512.8_{\pm 3.5} 6.0±3.26.0_{\pm 3.2} 18.8±3.318.8_{\pm 3.3} 3.2±1.63.2_{\pm 1.6} 11.4±2.111.4_{\pm 2.1} 1.2±1.11.2_{\pm 1.1} 0.0±0.00.0_{\pm 0.0}
MIMIC [24] 60.2±6.560.2_{\pm 6.5} 31.4±2.331.4_{\pm 2.3} 5.2±2.05.2_{\pm 2.0} 2.8±1.52.8_{\pm 1.5} 2.6±1.12.6_{\pm 1.1} 65.2±6.165.2_{\pm 6.1} 93.6±2.393.6_{\pm 2.3} 0.4±0.90.4_{\pm 0.9} 0.0±0.00.0_{\pm 0.0}
S2E [25] 68.4±2.768.4_{\pm 2.7} 63.0±5.263.0_{\pm 5.2} 36.4±4.436.4_{\pm 4.4} 22.2±3.822.2_{\pm 3.8} 43.8±9.443.8_{\pm 9.4} 46.8±3.846.8_{\pm 3.8} 80.0±3.980.0_{\pm 3.9} 1.0±1.01.0_{\pm 1.0} 0.2±0.40.2_{\pm 0.4}
FlowPilot [23] 66.8±3.666.8_{\pm 3.6} 60.2±4.160.2_{\pm 4.1} 33.6±7.233.6_{\pm 7.2} 32.4±5.032.4_{\pm 5.0} 40.6±5.340.6_{\pm 5.3} 19.8±3.119.8_{\pm 3.1} 74.0±4.174.0_{\pm 4.1} 13.6±4.813.6_{\pm 4.8} 0.0±0.00.0_{\pm 0.0}
OpenPilot [10] 81.8±3.481.8_{\pm 3.4} 83.0±4.483.0_{\pm 4.4} 48.2±5.548.2_{\pm 5.5} 35.0±3.535.0_{\pm 3.5} 16.4±5.616.4_{\pm 5.6} 34.8±5.934.8_{\pm 5.9} 95.2±3.795.2_{\pm 3.7} 0.0±0.00.0_{\pm 0.0} 3.8±2.23.8_{\pm 2.2}
Avg 38.5±1.238.5_{\pm 1.2} 32.3±1.532.3_{\pm 1.5} 17.2±1.117.2_{\pm 1.1} 12.3±0.812.3_{\pm 0.8} 17.4±2.017.4_{\pm 2.0} 19.8±0.719.8_{\pm 0.7} 43.2±0.943.2_{\pm 0.9} 3.8±0.43.8_{\pm 0.4} 1.7±0.31.7_{\pm 0.3}
Table 2: Model success rates (%) of pedestrian-reactive scenarios. Static denotes a scenario without pedestrians as reference. The top three positive ranks in each column are highlighted from dark to light blue; ties share a shade. Avg is the mean across all nine models. We also report the standard deviation across 5 runs with different random seeds.
Refer to caption
Figure 6: Evaluation results of long-horizon scenarios.
Scenario Before After
Ped-Crossing (Sim) 13.6 69.0
Ped-Crossing (Real) 0.0 40.0
Gesturing (Sim) 0.0 49.0
Gesturing (Real) 0.0 50.0
Table 3: Success rates (%) of FlowPilot [23] before and after finetuning using our simulation platform for synthetic data generation.

4.4 Long-horizon Scenarios

OpenPilot has 0.68 failures per hundred metres at 1.19 m/s procedurally, and 0.74 in scanned scenes. These are reset-assisted simulated counts. Extrapolating 0.68 to a two-kilometre delivery gives 13.6 interventions, but the paper’s stated 32.6 interventions/hour is inconsistent with 0.68 and 1.19 m/s, which imply about 29.1. Neither is a measured physical takeover rate.

Scanned scenes generally increase collision rates and reduce out-of-lane events relative to procedural scenes, showing that failure composition changes with geometry and rendering. Low total counts alone should not obscure a model’s collision behavior.

Refer to caption
Figure 7: Qualitative results of the long-horizon scenarios. We visualize the failure cases of the two best performing models, FlowPilot [23] and OpenPilot [10]. (a) Failure to follow the crosswalk. (b) Collision when facing both a pedestrian and an obstacle in the front. (c) Out of the boundary while turning. We strongly encourage viewing these results as videos on the project page.

4.5 Synthetic Training Data Generation

Targeted synthetic finetuning of FlowPilot improves crossing success from 13.6% to 69% in simulation and 0% to 40% physically. Gesture stopping rises from 0% to 49% and 50%, respectively. The physical study uses ten trials per scenario across two locations; it does not establish broad urban transfer, long-route gains, or robust generalization to arbitrary pedestrian gestures.

Refer to caption
Figure 8: Qualitative results of finetuning with synthetic data. We visualize the model predictions of FlowPilot [23] before and after finetuning using our simulation platform for synthetic data generation. We strongly encourage viewing these results as videos on the project page.

5 Conclusion

The benchmark reveals a large gap between useful sidewalk-following policies and sustained, socially competent navigation. Domain-specific data and targeted synthetic demonstrations help in the tested settings. Reproducible simulation supports diagnosis and iteration while requiring additional physical and model-specific validation.

6 Limitations

Rule-based events omit subtle natural pedestrian responses, procedural rendering can introduce artifacts, and the benchmark does not explain model internals. Outcomes may depend on textures and low-level controllers. Synchronized testing ignores inference latency, limiting deployment interpretation. Physical finetuning evidence is small; these limits preclude a broad claim that high simulator scores guarantee safe street operation.

Acknowledgment

The work acknowledges Office of Naval Research and National Science Foundation grants plus NVIDIA Academic Grant Program hardware support.

Appendix A Implementation Details

Implementation details specify pedestrian rules, collision proxies, robot limits, synchronization, and model input adaptation. These choices define what the reported benchmark measures.

Pedestrian Simulation

Reactive tests handcraft pedestrian speed and direction, while long routes use ORCA reciprocal avoidance with walking speeds from 1.1 to 1.65 m/s. Motion-LCM generates common movements and Motion-X supplies uncommon gestures. Pedestrians use radius-0.3-metre cylinders for separate collision checking, so detailed body-contact physics is not modeled.

For fifty pedestrians in one environment, the mesh pipeline renders at 226.6 frames/s versus native Isaac’s 3.4. Gaussian avatars reach 10.7, compared with MetaUrban’s 7.5. These amortized measurements include motion generation, rigging, skinning, and pedestrian rendering; they do not establish the same speedup for the complete closed-loop system. Original Figures 9–10 show throughput and qualitative alternate embodiments.

Scenario Description Triggering Condition
Obstructing A stationary pedestrian stands at the center of the sidewalk looking around. Within 10m ahead of the robot
Conversing Multiple pedestrians stand facing each other at the center of the path, forming a social group. Within 10m ahead of the robot
Queueing Pedestrians form a structured line starting from one side of the sidewalk with tight spacing. Within 10m ahead of the robot
Frontal Approaching A pedestrian walks directly toward the current position of the robot from the opposite direction. Within 20m ahead of the robot
Lateral Approaching A pedestrian crosses the robot’s projected path laterally, requiring the robot to yield. Within 4m ahead of the robot
Overtaking A pedestrian approaches and passes the robot from behind with a faster speed. Within 1m behind the robot
Ped-Crossing One or more pedestrians crossing from front and behind across a designated crosswalk. Robot enters the crosswalk zone
Gesturing A pedestrian continuously faces the robot while executing a distinct stopping gesture. Within 10m ahead of the robot
Table 4: Descriptions of event-based pedestrian behaviors.
Refer to caption
Figure 9: Comparisons of the pedestrian rendering efficiency. We report the amortized rendering FPS of pedestrians including motion generation, rigging, skinning, and rendering.
Refer to caption
Figure 10: Qualitative results of other robot embodiments. We show examples of the Unitree Go2 robot dog (left) and the Unitree G1 humanoid (right) evaluated with the FlowPilot [23] model. Video results are available on the project page.

Testing Scenario Configurations

The primary four-wheel delivery robot is limited to 1.5 m/s and 0.65 rad/s. A common PD controller converts predicted waypoints into velocities. Ten parallel environments run on one L40S with 0.005-second physics steps. The simulator waits for each policy command, excluding computation latency even while matching model observation frequency. Gesture success means stopping within three metres in front of the person; conversation/queue success requires going around the entire group, not between its members.

Models

Privileged route information is removed: image-goal models run exploration modes, point-goal models receive random forward points two to ten metres away with lateral offsets within ±2 metres, and language models get “Follow the sidewalk.” This standardizes the intended sidewalk-following task but changes goal-conditioned models’ normal use. Route-consistent goals are supported separately by the code. OpenPilot is retrained on one thousand sidewalk hours and uses two recent images plus recurrent context.

Appendix B Ablation on Sidewalk Training Data

Retraining ViNT on three hundred sidewalk hours increases average procedural completion from 20.9% to 85.9% and reduces long-route failures from 25.24 to 1.24 per hundred metres. OpenPilot improves from 73.7%/7.57 with ten hours to 91.1%/0.68 with one thousand. At matched three hundred hours, ViNT obtains 85.9%/1.24 versus OpenPilot’s 84.8%/1.80. Domain and scale are influential here; the comparison does not prove architecture never matters, and individual block scores are not monotonically improved at every data scale.

Row Model Data (# Hours) Straight ↑\uparrow Curve ↑\uparrow Crosswalk ↑\uparrow Average ↑\uparrow Long-horizon ↓\downarrow
1 ViNT [50] General 15.4 16.5 30.9 20.9 25.24
2 ViNT [50] Sidewalk (300) 72.6 85.8 99.4 85.9 1.24
3 OpenPilot [10] Sidewalk (10) 68.2 53.2 99.8 73.7 7.57
4 OpenPilot [10] Sidewalk (100) 78.0 72.3 95.4 81.9 2.59
5 OpenPilot [10] Sidewalk (300) 81.1 78.0 95.2 84.8 1.80
6 OpenPilot [10] Sidewalk (1000) 87.6 86.5 99.3 91.1 0.68
Table 5: Ablation on scaling training data. We report the route completion rate (%) on three types of unit-test scenarios and their average, and the average failure/100m of long-horizon scenarios in the procedurally generated scenes.

Appendix C More Results of Unit-test Scenarios

Tables 6–7 report five-run success, success-weighted path length, and completion. ViNT’s poor general-data performance improves after the separate domain retraining, so it should not be interpreted as an immutable architectural limitation. Curves and final crosswalk ramps are difficult: S2E’s procedural crossing completion is 95.9% but success only 63.2%. Small gaps between success and path-weighted success indicate efficient successful routes, not that unsuccessful episodes are satisfactory.

Refer to caption
Figure 11: Qualitative comparisons of the unit-test scenarios. ViNT [50], NoMaD [51], and InternVLA-N1 [55] are not able to follow the straight sidewalk lane. Only MBRA [26] has some basic lane-following ability. We strongly encourage viewing these results (and other models) as videos on the project page.
Procedurally Generated Scenes
Straight Curve Crosswalk
Method SR% ↑\uparrow SPL% ↑\uparrow RC% ↑\uparrow SR% ↑\uparrow SPL% ↑\uparrow RC% ↑\uparrow SR% ↑\uparrow SPL% ↑\uparrow RC% ↑\uparrow
ViNT [50] 0.6±0.90.6_{\pm 0.9} 0.6±0.80.6_{\pm 0.8} 15.4±1.315.4_{\pm 1.3} 0.6±1.30.6_{\pm 1.3} 0.6±1.30.6_{\pm 1.3} 16.5±3.716.5_{\pm 3.7} 1.4±1.11.4_{\pm 1.1} 1.4±1.11.4_{\pm 1.1} 30.9±3.130.9_{\pm 3.1}
NoMaD [51] 14.4±3.514.4_{\pm 3.5} 13.9±3.413.9_{\pm 3.4} 46.6±2.546.6_{\pm 2.5} 11.2±4.311.2_{\pm 4.3} 11.2±4.311.2_{\pm 4.3} 44.8±3.244.8_{\pm 3.2} 26.6±7.926.6_{\pm 7.9} 25.6±7.625.6_{\pm 7.6} 94.4±0.894.4_{\pm 0.8}
MBRA [26] 32.4±9.932.4_{\pm 9.9} 31.5±9.631.5_{\pm 9.6} 64.7±4.764.7_{\pm 4.7} 0.2±0.40.2_{\pm 0.4} 0.2±0.40.2_{\pm 0.4} 51.9±0.851.9_{\pm 0.8} 35.2±3.935.2_{\pm 3.9} 34.1±3.834.1_{\pm 3.8} 95.5±1.295.5_{\pm 1.2}
CityWalker [39] 2.0±1.22.0_{\pm 1.2} 1.9±1.21.9_{\pm 1.2} 34.1±2.334.1_{\pm 2.3} 1.6±1.81.6_{\pm 1.8} 1.6±1.81.6_{\pm 1.8} 37.5±2.837.5_{\pm 2.8} 12.6±4.312.6_{\pm 4.3} 12.1±4.112.1_{\pm 4.1} 77.4±4.377.4_{\pm 4.3}
InternVLA-DualVLN [55] 20.0±3.720.0_{\pm 3.7} 19.4±3.619.4_{\pm 3.6} 47.2±3.647.2_{\pm 3.6} 3.0±1.63.0_{\pm 1.6} 3.0±1.63.0_{\pm 1.6} 47.7±1.347.7_{\pm 1.3} 3.2±1.33.2_{\pm 1.3} 3.1±1.33.1_{\pm 1.3} 72.8±1.872.8_{\pm 1.8}
MIMIC [24] 60.2±6.560.2_{\pm 6.5} 60.2±6.560.2_{\pm 6.5} 77.9±2.577.9_{\pm 2.5} 43.6±4.743.6_{\pm 4.7} 43.6±4.743.6_{\pm 4.7} 71.0±2.471.0_{\pm 2.4} 76.8±5.076.8_{\pm 5.0} 76.8±5.076.8_{\pm 5.0} 98.8±1.298.8_{\pm 1.2}
S2E [25] 68.4±2.768.4_{\pm 2.7} 67.2±2.667.2_{\pm 2.6} 82.4±1.982.4_{\pm 1.9} 33.4±5.033.4_{\pm 5.0} 33.0±5.133.0_{\pm 5.1} 65.9±2.165.9_{\pm 2.1} 63.2±6.863.2_{\pm 6.8} 61.6±6.661.6_{\pm 6.6} 95.9±0.995.9_{\pm 0.9}
FlowPilot [23] 66.8±3.666.8_{\pm 3.6} 65.7±3.665.7_{\pm 3.6} 79.0±3.079.0_{\pm 3.0} 55.4±4.355.4_{\pm 4.3} 55.3±4.355.3_{\pm 4.3} 75.9±3.575.9_{\pm 3.5} 71.8±5.971.8_{\pm 5.9} 69.7±5.769.7_{\pm 5.7} 98.3±1.098.3_{\pm 1.0}
OpenPilot [10] 81.8±3.4\mathbf{81.8}_{\pm 3.4} 81.3±3.4\mathbf{81.3}_{\pm 3.4} 87.6±3.3\mathbf{87.6}_{\pm 3.3} 71.4±6.8\mathbf{71.4}_{\pm 6.8} 71.3±6.8\mathbf{71.3}_{\pm 6.8} 86.5±4.3\mathbf{86.5}_{\pm 4.3} 95.2±1.8\mathbf{95.2}_{\pm 1.8} 94.3±1.8\mathbf{94.3}_{\pm 1.8} 99.3±1.0\mathbf{99.3}_{\pm 1.0}
Table 6: Full evaluation results of unit-test scenarios on procedurally-generated scenes. We evaluate success rate (SR%), success weighted by path length (SPL%), and route completion rate (RC%) on different sidewalk structures. We also report the standard deviation across 5 runs with different random seeds.
Real-World Scanned Scenes
Straight Curve Crosswalk
Method SR% ↑\uparrow SPL% ↑\uparrow RC% ↑\uparrow SR% ↑\uparrow SPL% ↑\uparrow RC% ↑\uparrow SR% ↑\uparrow SPL% ↑\uparrow RC% ↑\uparrow
ViNT [50] 0.0±0.00.0_{\pm 0.0} 0.0±0.00.0_{\pm 0.0} 20.8±1.920.8_{\pm 1.9} 0.0±0.00.0_{\pm 0.0} 0.0±0.00.0_{\pm 0.0} 19.1±3.119.1_{\pm 3.1} 0.0±0.00.0_{\pm 0.0} 0.0±0.00.0_{\pm 0.0} 37.4±2.137.4_{\pm 2.1}
NoMaD [51] 52.0±8.452.0_{\pm 8.4} 50.7±8.350.7_{\pm 8.3} 76.6±4.476.6_{\pm 4.4} 4.0±5.54.0_{\pm 5.5} 3.9±5.33.9_{\pm 5.3} 46.0±3.746.0_{\pm 3.7} 18.0±4.518.0_{\pm 4.5} 17.7±4.417.7_{\pm 4.4} 59.9±2.159.9_{\pm 2.1}
MBRA [26] 52.0±8.452.0_{\pm 8.4} 51.3±8.551.3_{\pm 8.5} 77.2±3.177.2_{\pm 3.1} 0.0±0.00.0_{\pm 0.0} 0.0±0.00.0_{\pm 0.0} 42.5±3.642.5_{\pm 3.6} 16.0±13.416.0_{\pm 13.4} 15.8±13.315.8_{\pm 13.3} 65.1±4.265.1_{\pm 4.2}
CityWalker [39] 14.0±5.514.0_{\pm 5.5} 13.9±5.413.9_{\pm 5.4} 56.5±5.556.5_{\pm 5.5} 0.0±0.00.0_{\pm 0.0} 0.0±0.00.0_{\pm 0.0} 43.3±5.343.3_{\pm 5.3} 0.0±0.00.0_{\pm 0.0} 0.0±0.00.0_{\pm 0.0} 53.7±9.153.7_{\pm 9.1}
InternVLA-N1 [55] 36.0±11.436.0_{\pm 11.4} 35.2±10.735.2_{\pm 10.7} 66.7±5.166.7_{\pm 5.1} 4.0±5.54.0_{\pm 5.5} 4.0±5.44.0_{\pm 5.4} 37.2±3.037.2_{\pm 3.0} 14.0±5.514.0_{\pm 5.5} 13.2±5.413.2_{\pm 5.4} 53.1±1.453.1_{\pm 1.4}
MIMIC [24] 84.0±5.584.0_{\pm 5.5} 83.3±5.583.3_{\pm 5.5} 97.1±0.4\mathbf{97.1}_{\pm 0.4} 30.0±7.130.0_{\pm 7.1} 29.4±7.029.4_{\pm 7.0} 66.0±2.766.0_{\pm 2.7} 94.0±5.5\mathbf{94.0}_{\pm 5.5} 92.8±5.2\mathbf{92.8}_{\pm 5.2} 99.2±0.7\mathbf{99.2}_{\pm 0.7}
S2E [25] 90.0±0.0\mathbf{90.0}_{\pm 0.0} 88.8±0.188.8_{\pm 0.1} 92.3±0.092.3_{\pm 0.0} 54.0±5.554.0_{\pm 5.5} 53.9±5.553.9_{\pm 5.5} 76.5±3.776.5_{\pm 3.7} 44.0±8.944.0_{\pm 8.9} 43.3±8.843.3_{\pm 8.8} 79.3±2.579.3_{\pm 2.5}
FlowPilot [23] 90.0±0.0\mathbf{90.0}_{\pm 0.0} 89.1±0.189.1_{\pm 0.1} 92.2±0.192.2_{\pm 0.1} 46.0±8.946.0_{\pm 8.9} 45.3±8.745.3_{\pm 8.7} 71.0±4.271.0_{\pm 4.2} 90.0±0.090.0_{\pm 0.0} 89.2±0.089.2_{\pm 0.0} 94.9±1.094.9_{\pm 1.0}
OpenPilot [10] 90.0±0.0\mathbf{90.0}_{\pm 0.0} 89.3±0.0\mathbf{89.3}_{\pm 0.0} 90.1±0.090.1_{\pm 0.0} 82.0±4.5\mathbf{82.0}_{\pm 4.5} 81.8±4.5\mathbf{81.8}_{\pm 4.5} 89.8±2.3\mathbf{89.8}_{\pm 2.3} 92.0±4.592.0_{\pm 4.5} 91.4±4.691.4_{\pm 4.6} 92.1±4.492.1_{\pm 4.4}
Table 7: Full evaluation results of unit-test scenarios on real-world scanned scenes. We evaluate success rate (SR%), success weighted by path length (SPL%), and route completion rate (RC%) on different sidewalk structures. We also report the standard deviation across 5 runs with different random seeds.
Refer to caption
Figure 12: Failure to identify the ramp at the end of the crosswalk.

Appendix D More Results of Pedestrian-reactive Scenarios

InternVLA’s frontal-approach success is only 18.8%, yet its pedestrian collision rate is 8.4% versus S2E’s 45.6%. Minimum distance on successful trials is 1.35 m versus 0.86 m. This supports considering social clearance separately from progress, but conditioning distance on successful trials and allowing freezing prevents interpreting low collision rates as complete social competence. The proposed combination of large-model reasoning and lightweight control remains future work rather than a tested hybrid.

Metric Method Obstructing Conversing Queueing Frontal Lateral Overtaking Ped-Crossing Gesturing
Pedestrian Collision Rate (%) ↓\downarrow ViNT [50] 0.80.8 1.01.0 2.22.2 0.20.2 1.21.2 0.00.0 3.63.6 0.60.6
NoMaD [51] 11.211.2 16.616.6 23.423.4 8.28.2 28.228.2 14.014.0 42.642.6 9.49.4
MBRA [26] 26.026.0 43.843.8 57.657.6 26.426.4 68.668.6 36.636.6 60.460.4 25.225.2
CityWalker [39] 9.89.8 34.834.8 42.842.8 25.625.6 22.222.2 0.20.2 37.637.6 12.412.4
InternVLA-DualVLN [55] 9.69.6 16.216.2 28.028.0 8.48.4 29.629.6 20.820.8 14.614.6 8.08.0
MIMIC [24] 67.267.2 84.084.0 88.288.2 96.696.6 33.033.0 0.00.0 96.496.4 69.269.2
S2E [25] 22.822.8 50.250.2 66.866.8 45.645.6 48.448.4 1.41.4 83.483.4 20.420.4
FlowPilot [23] 33.833.8 56.656.6 57.457.4 55.455.4 78.278.2 8.88.8 76.876.8 30.630.6
OpenPilot [10] 11.411.4 38.638.6 44.644.6 82.682.6 64.664.6 0.00.0 99.899.8 15.615.6
Avg 21.421.4 38.038.0 45.745.7 38.838.8 41.641.6 9.19.1 57.257.2 21.321.3
Minimum Pedestrian Distance (m) ↑\uparrow ViNT [50] 1.841.84 – 1.141.14 1.681.68 1.361.36 – 0.800.80 –
NoMaD [51] 1.481.48 1.231.23 0.910.91 1.441.44 1.071.07 0.900.90 0.680.68 –
MBRA [26] 1.041.04 0.850.85 0.760.76 1.031.03 0.770.77 0.780.78 0.630.63 –
CityWalker [39] 1.001.00 1.061.06 0.940.94 0.710.71 0.890.89 1.171.17 0.800.80 1.721.72
InternVLA-DualVLN [55] 1.171.17 1.261.26 0.860.86 1.351.35 1.881.88 0.790.79 0.700.70 –
MIMIC [24] 0.720.72 0.740.74 0.740.74 0.630.63 0.980.98 0.970.97 0.520.52 –
S2E [25] 0.890.89 0.770.77 0.730.73 0.860.86 0.910.91 0.920.92 0.540.54 0.910.91
FlowPilot [23] 0.870.87 0.780.78 0.780.78 0.760.76 0.800.80 0.870.87 0.620.62 –
OpenPilot [10] 0.710.71 0.660.66 0.620.62 0.630.63 0.940.94 0.950.95 – 1.191.19
Avg 1.081.08 0.920.92 0.830.83 1.011.01 1.071.07 0.920.92 0.660.66 1.271.27
Table 8: More evaluation results of pedestrian-reactive scenarios. We evaluate pedestrian collision rate (%) and average minimum pedestrian distance in the successful trials. ‘–’ means the model has no successful trials. Avg is the mean across all nine models.
Refer to caption
Figure 13: Qualitative comparisons of the pedestrian-reactive scenarios. MIMIC [24] and OpenPilot [10] freeze in front of a conversing pedestrian group. InternVLA-N1 [55] leaves a much larger space when passing the pedestrian group compared to FlowPilot [23], demonstrating its better social compliance capability than the sidewalk-specific models. We strongly encourage viewing these results (and other models) as videos on the project page.

Appendix E More Results of Long-horizon Scenarios

Table 9 decomposes long-route failures. OpenPilot’s procedural 0.68 comprises 0.32 collisions, 0.24 out-of-lane, and 0.12 freezing events per hundred metres; scanned scenes report 0.74 total with 0.55 collisions. MIMIC and FlowPilot are competitive in total procedural counts but differ in speed and collision frequency. Long-route aggregate scores are reported without the five-run deviations provided for the shorter tests.

Procedurally Generated Scenes Real-World Scanned Scenes
Method FC ↓\downarrow C ↓\downarrow OL ↓\downarrow F ↓\downarrow 𝑽\boldsymbol{V} ↑\uparrow FC ↓\downarrow C ↓\downarrow OL ↓\downarrow F ↓\downarrow 𝑽\boldsymbol{V} ↑\uparrow
ViNT [50] 25.24 3.48 21.71 0.05 1.03 24.11 7.36 16.38 0.37 1.19
NoMaD [51] 8.70 2.08 6.50 0.13 1.43 5.89 2.94 2.94 0.00 1.48
MBRA [26] 7.10 2.10 4.84 0.16 1.44 5.15 2.58 2.58 0.00 1.46
CityWalker [39] 11.10 1.39 7.10 2.62 0.34 10.86 4.05 1.29 5.52 0.14
InternVLA-N1 [55] 12.92 2.43 10.33 0.15 1.33 9.39 4.23 4.97 0.18 1.28
MIMIC [24] 1.41 0.85 0.48 0.08 0.61 1.66 1.47 0.00 0.18 0.34
S2E [25] 2.36 0.69 1.53 0.14 0.85 0.92 0.74 0.18 0.00 1.15
FlowPilot [23] 1.34 0.71 0.55 0.08 1.15 0.92 0.92 0.00 0.00 1.06
OpenPilot [10] 0.68 0.32 0.24 0.12 1.19 0.74 0.55 0.00 0.18 1.19
Table 9: Evaluation results of long-horizon scenarios. We report the average failure counts (FC) per 100 meters traveled, including collision (C), out-of-lane (OL), and freezing (F). We also report the average velocity (VV, m/s).

Appendix F Details of Synthetic Training Data Generation

Five hundred successful demonstration episodes are collected for each of crossing and gesturing using ORCA reference trajectories in the policy’s observation/action format. Finetuning freezes the visual backbone and updates lightweight action/adaptation modules for fifty epochs with AdamW, learning rate 5×10⁻⁶ decaying to 10⁻⁶ and gradient norm capped at one. The physical evaluation uses ten trials per scenario across two locations under the same task rules. A small targeted study demonstrates promise while leaving wider behavior and route coverage untested.

Concept demonstration

Open image ↗