Literature
Display
← All papers

Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Anthony G. Francis · Claudia Pérez-D'Arpino · Chengshu Li · Fei Xia · Alexandre Alahi · Rachid Alami · Aniket Bera · Abhijat Biswas · Joydeep Biswas · Rohan Chandra · Hao-Tien Lewis Chiang · Michael Everett · Sehoon Ha · Justin W. Hart · Jonathan P. How · Haresh Karnan · Tsang-Wei Edward Lee · Luis J. Manso · Reuth Mirsky · Sören Pirk · Phani-Teja Singamaneni · Peter Stone · Ada V. Taylor · Peter Trautman · Nathan Tsoi · Marynel Vázquez · Xuesu Xiao · Peng Xu · Naoki Yokoyama · Alexander Toshev · Roberto Martín-Martín

· ACM Transactions on Human-Robot Interaction ↗

Robotics19 min text · CondensedOriginal paper ↗
Contents

Overview

This long review organizes social navigation around explicit behavioral principles, research questions, evaluation metrics, scenarios, benchmarks, datasets, and simulators. It proposes reporting and interface guidelines rather than a new navigation policy or a universally validated social-performance score. The comparisons describe the field as reviewed in 2023.

Refer to caption
Fig. 1: We identify eight broad principles of social robot navigation - including safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness - which motivate specific guidelines for experiments, metrics, scenarios, benchmarks, datasets, and simulators. Principles and guidelines are labeled with two-letter codes, with P for principles, R for real-world issues, M for metrics, N for scenarios, B for benchmarks, D for datasets, and S for simulators.

At a glance

Social navigation evaluation must assess both navigation outcomes and how people experience a robot’s behavior. The paper proposes eight principles, a taxonomy of experimental conditions, guidelines for evaluation resources, and a common metrics-interface design. Its aim is more comparable studies while preserving task and cultural context; it does not demonstrate that a single benchmark can measure every aspect of socially appropriate behavior.

I Introduction

Discussions at the 2022 Social Navigation Symposium motivated this synthesis. Principles describe desired behavior; guidelines specify practical choices for measuring it and building reusable evaluation resources. The article connects metrics to scenarios, benchmarks, datasets, and simulators. It offers a framework for principled comparisons across varied research questions rather than one mandatory experimental protocol.

II Related Work

The review builds on prior evaluation surveys and analyses of social-navigation challenges. Its contribution is a shared definition, expanded principles, research lifecycle, resource taxonomy, scenario cards, and interface proposal. These recommendations extend existing work; the paper is not a comprehensive survey of every navigation algorithm or an empirical leaderboard of the proposed principles.

III Towards a Definition of Social Navigation

The term social navigation covers dynamic-obstacle avoidance, social convention, and intentional communication. The authors separate these meanings before defining behavior that accommodates others while pursuing the robot’s own task. This distinction determines which evidence is needed: collision avoidance alone cannot establish comfort or politeness.

III-A What is a Social Robot?

A socially navigating robot acts around humans or robots while adjusting its behavior so their experience is preserved or improved. Expressive hardware or interactive features alone do not guarantee beneficial interaction. The definition concerns accommodation of other agents’ needs, feelings, capabilities, and conventions; how those effects are measured remains an experimental question.

Original figure
Fig. 2: We define a socially navigating robot as one that interacts with humans and other robots in a way that achieves its navigation goals while enabling other agents to achieve theirs. To make this objective achievable, we propose eight principles for social robot navigation: safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness.

III-B Principles of Social Navigation

The eight principles are safety, comfort, legibility, politeness, social competency, understanding other agents, proactivity, and contextual appropriateness. Safety includes damage and induced harm beyond direct collision. Comfort concerns annoyance and stress; legibility makes goals understandable; politeness concerns consideration in motion and communication; social competency follows local conventions. Understanding others supports prediction and accommodation, while proactivity resolves emerging conflicts or deadlocks rather than simply waiting.

Context changes how these objectives matter: culture, individual abilities and histories, geometry, operational setting, task urgency, and interpersonal relationships can all alter appropriate behavior. The principles overlap and can conflict. Illustrated personal-space distances are contextual examples, not universal thresholds. The article recommends describing intended contexts precisely enough for readers to assess transfer, without claiming a complete computational definition of context.

Original figure
Fig. 3: Contextual factors of social navigation. While the first seven principles represent factors to optimize, the eighth principle, contextual appropriateness, calls out that the weighting of these factors can be affected by many features, including cultural, diversity, environmental, task and interpersonal context. Lines in the diagram are representative of common interactions but are not exclusive.

IV Research Methodologies of Social Navigation

Evaluation methods should follow the scientific question. Algorithmic comparisons may use measurable navigation outcomes; questions about human perceptions require evidence about human responses. A benchmark’s scope therefore depends on whether it studies performance, mechanisms, generalization, perceived social quality, behavior change, or failures found in deployment.

IV-A Research Questions of Social Navigation

The paper distinguishes comparisons against baselines, component ablations, and environmental generalization from human ratings, observed behavior changes, and exploratory issue discovery. Simulation can make ablations practical, but claims about comfort or politeness often need validated human studies. Live deployments are especially useful for discovering the distribution of encounter types and failures. The recommendation is to combine suitable navigation tests and human evidence, while documenting what each study can actually establish.

IV-B Types of Social Navigation Studies

The review distinguishes uncontrolled field studies, partly controlled robot deployments, laboratory experiments, repeatable social-navigation scenarios, and staged interactions. Natural settings reveal unanticipated behavior but make individual encounters hard to reproduce. Familiarity and participant instructions can change reactions in deployments and laboratories. Staged gatherings increase interaction density but remain staged. Ecological validity and construct validity require evidence, not simply choosing one of these study labels.

Original figure
Fig. 4: Lifecycle of social navigation research. Field studies, robot deployments, and staged social interactions can be used to collect data, which helps identify issues and their prevalence. Issues discovered guide laboratory experiments and the development of social navigation scenarios, which in turn can inform data collection. Issue discovery also helps guide the development of benchmarks that test these issues, along with public benchmark challenges; attempts at solutions of these challenges can also help identify issues.

IV-C Lifecycle of Social Navigation Research

The proposed lifecycle links data collection, issue discovery, laboratory experiments, scenario development, benchmarking, and public challenges. Frequent or high-risk failures found in deployments should inform reproducible scenarios, while targeted experiments should feed findings back into evaluation resources. Controlled scenarios cannot replace observing new real-world conditions. Shared datasets and clearly defined success criteria help the wider community benefit from costly deployments.

IV-D Guidelines for Real-world Studies

Three real-study guidelines are to preserve safety through monitoring or protective controls, respect participant consent and privacy, and define a clear scientific objective. Ablated policies may behave unexpectedly and may be better studied in simulation when adequate controls are absent. These are the paper’s experimental-design recommendations; institution-specific review and legal requirements depend on the study context.

V A Taxonomy of Social Navigation

The taxonomy provides common terms for comparing metrics, scenarios, benchmarks, datasets, and simulators. It supports explicit descriptions of what an evaluation resource measures and what conditions it represents, rather than treating resources with different assumptions as interchangeable.

Original figure
Fig. 5: A taxonomy of social navigation. Most social navigation instruments share common factors like overall context, physical environments, human user type, robot role and task, and so on. However, datasets, benchmarks and simulators have additional factors particular to them.

V-A A Taxonomy for Analysis

Metrics quantify behavior; scenarios specify interactions; benchmarks combine protocols and metrics; public challenges add submission and comparison mechanisms. Datasets supply observations and annotations, while simulators generate controlled interactions. The proposed shared interface targets incompatible simulator outputs. Each resource also needs factors specific to its purpose, such as sampling coverage for datasets or fidelity for simulators.

V-B Factors Common to Social Navigation

The seventeen shared factors cover context, physical environment, human-user population, human behavior, robot task and role, scenarios and their coverage, robot embodiment, sensors, actuators, communication, collected data, metrics, human and robot behavior-generation methods, and simulation versus reality. Descriptions should disclose whether observations are noisy robot-view measurements or privileged ground truth, whether humans react to the robot, and how test cases and rare events are sampled.

Robot familiarity, participant instructions, morphology, visibility, sensing delay, and noncooperative behavior can all change outcomes. A shared vocabulary does not require identical robots or one universal setting. Dataset annotations/privacy, benchmark baselines/accessibility, and simulation fidelity require additional reporting beyond these common factors.

VI Social Navigation Metrics

Social navigation involves several partly overlapping constructs, so a scalar navigation-efficiency score cannot capture its full quality. Even safety can refer to physical harm, perceived threat, or social disruption. The paper recommends connecting each metric to a defined construct and context, and treating combinations of metrics as design choices that require justification.

VI-A Taxonomy of Existing Social Navigation Metrics

Metrics are classified along three independent axes: how they are obtained, what variable they model, and their temporal scope. Describing all three makes it easier to distinguish a human perception measurement from a reproducible proxy or a time-local control signal.

Original figure
Fig. 6: The proposed taxonomy suggests classifying metrics according to three aspects: the type of variable (or variables) they model, their nature, and their temporal scope. To quickly identify metric types, we suggest using a three-letter code based on these factors. For example Success Rate (SR) is a Non-Social, Hand-Crafted, Task-Wise metric, or NHT metric; a sensor metric gauging moment-to-moment human facial reactions to robot behavior would be a Social, Sensor, Stepwise metric, or SSS metric; and a questionnaire asking about the overall quality of a robot’s navigation would be an All-Encompassing, Questionnaire, Task-Wise metric, or AQT metric.

VI-A 1 Taxonomy based on their nature

Surveyed measurements include questionnaires and sensor-based indicators of human responses. Algorithmic metrics are hand-designed or learned from human-labeled data. Hand-designed measures are objective in their computation, while their interpretation may be subjective or incomplete. Learned scores can be cheap to apply but inherit their training data and assumptions. Human studies are expensive and variable, yet remain important evidence for claims about human experience.

VI-A 2 Taxonomy based on the variable being modeled

Non-social measures assess task success, efficiency, or energy; social measures target comfort, acceptance, trust, or related constructs; all-encompassing measures attempt both. A reliable task-efficiency number still does not provide social-performance evidence. The authors regard comprehensive human-aligned scores as desirable but note that few had been proposed and none resolves all contextual issues.

VI-A 3 Taxonomy based on temporal scope

Step-wise metrics score individual instants; task-wise metrics score a whole episode. Averaging time steps can conceal brief consequential events or reversals, so episode measures may need temporal structure or event weighting. For reinforcement learning, an immediate score is useful only if its accumulated value matches the intended task outcome; delayed episode feedback can be more faithful to that objective.

VI-B The Challenges of Measuring Social Navigation

Measurement has expanded from whether navigation succeeds, to how efficiently it succeeds, to its effects on others. Four difficulties remain central: changing environments, participant adaptation, changes in deployed robot behavior, and limits of the metrics themselves. These complicate reproducibility and comparisons across studies.

VI-B 1 The Challenges of Dynamic Environments

Lighting, weather, pedestrian behavior, and robot battery state can alter measured performance during deployment. Controlling every factor may be impractical, but reports should identify relevant changes and limits of comparability rather than attributing all variation to the navigation policy.

VI-B 2 Challenges Based on How Robots Change People

People adapt to robot signals and behavior across encounters, changing fluency, conflict rates, and acceptance even if the policy stays fixed. Student or robotics-expert participant pools may be unrepresentative of the public. Studies should report prior exposure and examine novelty and long-term adaptation rather than treating all encounters as independent experiences of equally unfamiliar users.

VI-B 3 Challenges Based on How People Change Robots

Experienced users may implicitly compensate for a robot’s weaknesses, making its deployment appear safer or more acceptable than encounters with visitors. Anecdotes about collision-free development teams illustrate this confound; they are not quantified performance results. Long-term acceptance can also remain high despite faults, so task statistics and overall attitudes need separate interpretation.

VI-B 4 The Limitations of Social Metrics Themselves

Human ratings depend on culture, goals, setting, and familiarity, while realistic studies and large participant pools are expensive. Choosing a small set of observable signals can omit gaze, gestures, or facial expressions. Combining metrics requires context-sensitive weights that remain an open problem, and heterogeneous sensing or communication capabilities can make a standardized score unfair across robot embodiments.

VI-C Assessment of Existing Social Navigation Metrics

The authors favor ratings from people experiencing an interaction, while acknowledging scaling, disruption, and reproducibility limits. They find no convincing universal procedure for ranking one metric as better than another. Reports should identify both a metric’s original validation context and the context of its new use. Learned comprehensive scores and narrowly diagnostic metrics each have limited applicability.

Metric Short Description Class. Parameters Unit Range Cited
Success metrics     Success SS Binary variable describing whether the robot reaches the goal. When averaged, it is referred as Success Rate (SR). NHT - boolean {0,1}\{0,1\} [15]
Collision CC Number of collisions in the trajectory. When averaged it is referred to as Collision Rate (CR). NHT Collisions to terminate episode collision [0,∞)[0,\infty) [21]
Wall Collisions W​CWC Number of collisions against walls. NHT - collision [0,∞)[0,\infty) R@G
Agent Collisions A​CAC Number of collisions against humans or robots. NHT - collision [0,∞)[0,\infty) R@G
Human Collisions H​CHC Number of collisions against humans. Also called H-collisions [88]. NHT - collision [0,∞)[0,\infty) R@G [88]
Timeout before reaching goal T​OTO Binary variable accounting for failures caused by a timeout. NHT Time threshold timeout {0,1}\{0,1\} R@G
Failure to progress F​PFP Number of failures caused by not decreasing the distance to the goal for a given period of time. NHT Distance & time thresholds failure [0,∞)[0,\infty) R@G
Stalled time S​TST Time where the magnitude of the speed of the robot falls within a given threshold. NHT Distance & time thresholds ss [0,∞)[0,\infty) [89]
Time to reach goal TT Time between task assignment and completion. NHT - ss [0,∞)[0,\infty) [90, 21]
Path length P​LPL Length of the trajectory. NHT - mm [0,∞)[0,\infty) [90, 21]
Success weighted by path length S​P​LSPL Success weighted using normalized inverse path length, i.e., weighted using path length divided by the max of the min distance and path length [15]. NHT - success [0,1][0,1] [15]
Quality and social metrics     Velocity-based features Vm​i​nV_{min}, Va​v​gV_{avg}, Vm​a​xV_{max} Minimum, average and maximum linear velocity on a trajectory. SHT - m/sm/s (−∞,∞)(-\infty,\infty) [21]
Linear acceleration based features Am​i​nA_{min}, Aa​v​gA_{avg}, Am​a​xA_{max} Minimum, average and maximum linear acceleration on a trajectory. SHT - m/s2m/s^{2} (−∞,∞)(-\infty,\infty) [21]
Movement jerk Jm​i​nJ_{min}, Ja​v​gJ_{avg}, Jm​a​xJ_{max} Minimum, average and maximum linear jerk (i.e., the second-order derivative of the linear speed). SHT - m/s3m/s^{3} (−∞,∞)(-\infty,\infty) [21]
Clearing distance C​Dm​i​n{CD}_{min}, C​Dm​a​x{CD}_{max} Minimum and average distance to obstacles in a trajectory. SHT - mm [0,∞)[0,\infty) [21]
Space compliance S​CSC Ratio of the trajectory with the minimum distance to a human under a given threshold. If the threshold is 0.5​m0.5m, it is referred to as Personal Space Compliance (P​S​CPSC) [26]. SHT Distance threshold mm [0,1)[0,1) [71]
Minimum distance to human D​Hm​i​nDH_{min} Minimum distance to a human in a given trajectory. SHT - mm [0,∞)[0,\infty)
Minimum time to collision T​T​CTTC Minimum time to collision with a human agent at any point in time in the trajectory, should all robots and humans move in a linear trajectory. SHT - mm [0,∞)[0,\infty) [28]
Aggregated Time A​TAT Time taken for a subset of cooperative agents to meet their goals. SHT Cooperative agents’ set tt [0,∞)[0,\infty) [10]
TABLE I: Suggested hand-crafted metrics for the evaluation of social navigation systems. The first tranche in the table are traditional navigation metrics, included to ensure that social navigation systems do not regress on traditional navigation performance; the second tranche concern aspects of the quality and socialness of navigation. Citations refer to either papers or challenges defining the term, or R@G for metrics from an unpublished Robotics at Google[91] robot deployment.

VI-D Recommendations for Metric Usage and Development

The proposed compromise combines a shared set of conventional navigation measures, iteratively validated human surveys, and further research on learned proxies. These sources of evidence address different questions; inexpensive proxy scores do not by themselves validate human social experience.

VI-D 1 Recommendations for Hand-crafted Metrics

Table I specifies a recommended collection of success, collision, failure, trajectory, and social-proxy measures, together with required parameters, units, ranges, and references. Report context and thresholds because a good value is task dependent. Across repeated trials, distributions and outliers should accompany averages, especially for safety-related outcomes.

VI-D 2 Recommendations for Survey Development

Questionnaires should be iteratively evaluated for test–retest reliability, construct validity, and bias. Survey length and repetition trade information against fatigue and response burden. Use question-appropriate response options, consider nonapplicable answers, balance experimental conditions, and apply suitable statistical analyses before claiming statistical significance. Reliability statistics support instrument assessment but do not alone prove that a survey measures the intended social construct.

VI-D 3 Recommendations for Future Metric Development

Proposed research directions include time-to-collision, motion smoothness, and visibility-based proxies, and datasets pairing algorithmic measurements with human ratings. Correlation and held-out evaluation can test when a proxy predicts surveyed responses. Learned reward functions are another possibility, but the paper presents these as research directions, not already validated replacements for human evaluation.

VI-E Metric Guidelines

The eight metric guidelines call for robust conventional navigation, simulation checks before deployment, explicitly reported context-dependent parameters, cautious use of learned scores for iteration, validated surveys for social claims, consistent experimental design, iterative analysis, and detailed reporting. The recommended evidence includes several metrics and temporal scopes plus distributions, rather than a single favorable average.

VII Social Navigation Scenarios

Scenario definitions connect naturalistic observations with controlled experiments. They can identify encounters in field data, elicit behavior in deployments, support regression tests, curate datasets, and guide imitation learning. These uses need different outcome labels: an interrupted frontal approach may be valid observational data even when it would be a failed training demonstration. Coverage of naturally occurring variants should be separated from criteria for success.

Social Navigation Scenario Card
Scenario Metadata
Scenario Name Frontal Approach
Scenario Description A robot and a human approach head-on in a passable space.
Scientific Purpose Low-density pedestrian scenario applicable indoors and outdoors.
Scenario Definition
Geometric Layout A space wide enough for the robot and human can pass each other.
Intended Robot Task The robot navigates from one side of the space to the other
Intended Human Behavior The human navigates in the opposite direction of the robot.
Scenario Usage Guide
Success Metrics Success Rate, (No) Collisions
Quality Metrics Comfort, Politeness
Ideal Outcome Robot goes around human in a socially acceptable manner.
Failure Modes 1. Robot collides with human
2. Robot fails to exit in time limit
Labeling Criteria 1. Robot and human face each other
2. Robot and human move towards each other at start of episode
3. Sufficient clearance exists for robot and human to pass each other
TABLE II: Scenario Card for Frontal Approach

VII-A Scenario Design Methodology

Scenario design has three stages: define its research context, robot task, human behavior, and success measures; evaluate commonality with existing scenarios, flexibility, and fitness to purpose; and communicate a reproducible specification. Pilot studies should check whether the setup actually elicits and measures the intended behavior. Avoid prescribing a preferred solution so narrowly that alternative or unexpected interactions disappear from the data.

Scenario Name Scenario Description Physical Env. Geom. Layout Scientific Purpose Robot Role Robot Task Human Behavior Ideal Outcome Related Scenarios Cited In
Hallway Scenarios
Frontal Approach A pedestrian and robot approach head-on. Generic Passable Space Pedestrian Interaction Any Navigate A to B Navigate B to A Robot / humans pass Ped. Obstruct [30, 6, 10]
Pedestrian Overtaking A pedestrian overtakes a moving robot. Generic Passable Space Pedestrian Interaction Any Navigate A to B Navigate A to B Human passes robot Down Path [45]
Robot Overtaking A robot overtakes a moving pedestrian. Generic Passable Space Pedestrian Interaction Any Navigate A to B Navigate A to B Robot passes human [6, 10]
Intersection (No Gesture) A robot and a human cross at an intersection. Indoor Intersection Pedestrian Interaction Any Navigate A to B Cross Navigate Both pass no collision [89, 6, 10, 109]
Intersection (Gest. Wait) A robot is told to wait at an intersection. Indoor Intersection Pedestrian Interaction Servant Navigate A to B Cross Navigate Human goes then robot Gesture Proceed [30]
Blind Corner A robot and human meet at a blind corner. Indoor Corner Pedestrian Interaction Any Navigate A to B Navigate B to A No collision / obstruction [30, 31]
Doorway Scenarios
Narrow Doorway A robot and human pass a narrow doorway. Indoor Room & Door Pedestrian Interaction Any Navigate A to B Navigate B to A No collision / obstruction Narrow Arch [30]
Entering Room A robot enters a room occupied by a human Indoor Room & Door Pedestrian Interaction Any Navigate out to in Navigate in to out Robot allows human exit Entering Elevator R@G
Exiting Room A robot exits a room while a person enters. Indoor Room & Door Pedestrian Interaction Any Navigate in to out Navigate out to in Robot exits first Exiting Elevator R@G
Interpersonal Scenarios
Joining a Group A robot joins a group of robots or people. Generic Open Space Group Interaction Any Navigate to group Continue conversing Robot joins group Leaving a Group [89, 6]
Following A robot follows a person. Generic Walking Space Joint Navigation Servant Follow human Lead robot Robot follows person Accompany Peer [6]
Leading A robot leads a person. Generic Walking Space Joint Navigation Leader Lead human Follow robot Robot leads person Tour Guide [6]
Crowd Scenarios
Crowd Navigation A robot navigates through a crowd. Generic Passable Space Crowd Navigation Any Navigate thru Mill about No collision / obstruction Robot Crowding Various
Parallel Traffic Crowd moves parallel to the robot. Generic Passable Space Crowd Navigation Any Navigate A to B Mill from A to B No collision / obstruction Circular Crossing [10]
Perpendicular Traffic Crowd moves perpendicular to robot. Generic Intersection Crowd Navigation Any Cross Navigate Mill from A to B No collision / obstruction Plaza Crossing [10]
Specialized Scenarios
Object Handover A robot hands an object to a human. Generic Passable Space Interactive Navigation Servant Deliver object Receive object Human takes object Robot Courier [89]
Crash Cart Robot delivering a medical product. Indoor Passable Space Interactive Navigation Leader Deliver object Receive object Delivery of medicine Food Delivery This paper
TABLE III: Example social navigation scenarios. For illustrations of the geometric layout, see Figure 7. Closely related scenarios are listed in the second-to-last column. Citations refer to either papers or challenges defining the scenario, or R@G for scenarios from an unpublished Robotics at Google[91] deployment, developed according to the protocol in [30] .

VII-B Social Navigation Scenario Cards

A scenario card combines metadata, a definition, and a usage guide. The goal is enough precision to recognize and reproduce an encounter while allowing meaningful behavioral variation. Realism, scalability, and repeatability are useful design goals, but complex crowd encounters may require different levels of detail from a simple passing interaction.

VII-B 1 Scenario Metadata

Metadata identifies the scenario’s category, unique name, brief description, and scientific purpose. Location, pedestrian density, and high-level tasks help readers determine coverage and avoid redundant scenarios. A name such as Frontal Approach should refer to an explicit definition rather than an informal label.

VII-B 2 Scenario Definition

The definition records relevant geometric features, robot numbers and intended tasks, and intended human behavior. It should distinguish essential conditions from permitted variation. For example, head-on approach can still describe an encounter when a pedestrian stops or changes direction; success criteria belong separately in the usage guide.

VII-B 3 Scenario Usage Guide

The usage guide supplies labeling rules, outcome measures, desirable behavior, failure modes, and participant scripts. These are separate from the scenario definition so unsuccessful or unexpected encounters remain recognizable in field data. Context can change the expected behavior and scoring. For repeatable experiments, describe participants’ roles and permitted variations explicitly.

Original Figure 7 with all 25 labeled panels in the source arrangement.
Fig. 7: Geometric layout and intended human and robot behavior for example social navigation scenarios from Table III. The blue circles, arrows, dotted lines, and dotted circles represent the robot, its direction of motion, its intended path and its intended destination, respectively; red figures represent the corresponding items for humans. Grey backgrounds represent obstructions, while green figures represent signals or gestures emitted by an agent such as a gesture to stop or go ahead. Note sample paths and gestures are provided as examples to make the graphics clear; the actual scenario card definition should be flexible enough to capture a range of behaviors.

VII-C Example Social Navigation Scenarios

The examples cover low-density pedestrian encounters, dense crowds, and tasks involving interaction. Passing, overtaking, crossing, blind corners, and doorways stress different navigation skills; group joining, handover, and monitoring also constrain the robot’s task. Tables II–III and Figure 7 retain the scenario card and geometric examples. They are a useful coverage set, not an exhaustive taxonomy or a required suite for every application.

VII-D Scenario Guidelines

Guidelines N1–N8 ask researchers to state context, robot tasks, intended human behavior, and success measures; cover commonly studied encounters; permit realistic variants; verify that scenarios elicit the relevant behavior; and communicate them using scenario cards. The key is reproducible meaning with sufficient flexibility, rather than merely reusing a familiar scenario name.

VIII Social Navigation Benchmarks

A benchmark combines scenarios with metrics so policies can be compared. Its meaning still depends on which social-navigation factors it tests. The review separates protocols, executable environments, and public challenges, then argues for human-grounded social measures alongside conventional navigation regression tests.

VIII-A Expanding the Factors for Benchmark Analysis

Benchmark descriptions need implementation and access information beyond the shared experimental taxonomy. The following factors expose whether another group can run a comparable evaluation on its own hardware or simulator.

Simulation Platform

State the simulator and setup procedure. A ready-to-run implementation on an accessible platform lowers the effort needed to reproduce the benchmark.

Associated Dataset

Identify any reference dataset and how it is used for comparison. A benchmark may include data, but a dataset alone is not the executable evaluation defined here.

Provided Baselines

Provide baseline policies and their configurations so new results have interpretable comparison points.

Challenge Leaderboard

A challenge may add a public leaderboard. This supports shared comparison only for the task and measures actually specified by that benchmark.

Downloadability

Downloadable evaluation software supports replication and independent inspection of the procedure.

Most Recent Update

Report maintenance recency and compatibility. The source’s resource descriptions are a 2023 snapshot, so its judgments about available versions should not be read as a current software audit.

Robot Hardware Platform

Document supported robot morphologies and customization. Comparisons are more useful when they can represent the embodiment intended for deployment.

Human Behavior Authoring Methods

State how pedestrians and other agents are generated. Reactive models, authored behavior, and replayed trajectories provide different evidence about human–robot interaction.

Benchmark ArenaBench CrowdBot DynaBarn gym-collision-avoidance HuNavSim iGibson Soc-NavBench Sea-NavBench Social Navigation Protocol
Factors for Analysis
Benchmark Classification Benchmark Challenge Benchmark Benchmark Benchmark Challenge Benchmark Challenge Protocol
Benchmark Context and Scope Dynamic Obstacle Benchmark Crowd Simulation Benchmark Dynamic Obstacle Benchmark Collision Avoidance Benchmark Human Simulation Benchmark Social Navigation Benchmark Social Navigation Benchmark Social Navigation Benchmark Human-Robot Expt. Design
Physical Environment Indoor Indoor Synthetic Synthetic Indoor Indoor Indoor & Outdoor Indoor & Outdoor Principally Indoor
Intended Human User Type Synthetic Pedestrian Synthetic Pedestrian Varied Human Motion Synthetic Pedestrian Varied Resp to Robot Synthetic Pedestrian Synthetic Pedestrian Synthetic Pedestrian Human Coworkers
Supported Robot Tasks Navigation Navigation Navigation Navigation Navigation Navigation Navigation Navigation Navigation
Social Scenarios Evaluated 3 Worlds, 5/10 Peds Basic Crowd Scenarios 60 Crowd Scenarios Multi-agent Scenarios House, Cafe, Warehouse 15 House Scenes 5 Curated Environments TBD 6 Social Nav Scenarios
Coverage of Corner Cases Diversity, Random Not Tested Diversity, Random Not Tested Not Tested Not Tested Not Tested TBD Not Specified
Simulation Platform Flatland, Gazebo Unity Gazebo Custom Gazebo iGibson SocNavBench SEAN 2.0 None
Benchmarking Dataset None CrowdBot None None None None UCY & ETH UCY & ETH None
Human Behavior Authoring Pedsim UMANS Multiple Algorithms Baseline Policies Soc. Force, Behav. Tree ORCA Replay, Planned Replay, Soc Force Scripted
Human Simulation Fidelity Walking Humans Walking Humans Moving Cylinders Moving Cylinders Walking Humans Moving Humans Walking Humans Walking Humans Real Humans
Supported Robot Embodiments Jackal, Burger, Robotino Pepper, Wheelchair, CuyBot, Qolo Custom Robots, ClearPath Jackal Cylinders ROS Gazebo-Compatible 8 real, 2 Mujoco Simulated Mobile Robot Fetch, Jackal, Turtlebot, Warthog Human-Scale Robots
Communication Modalities None None None None None None None None Human Gestures
Challenge Leaderboard None None None None None 2021 None 2022 None
Benchmark Last Updated 2022 2021 2023 2022 2023 2021 2022 2022 2022
Guidelines for Benchmarks
B1: Evaluate Social Behavior Yes Yes Yes Yes Yes Yes Yes Yes Yes
B2: Quantitative Metrics Provided Many Many Succ. Rate Succ. Rate, Time2Goal Many Succ. Rate, PSC Many Many No
B3: Baseline Policies Provided SOA Nav, RL Policies No SOA Nav, RL Policies SOA Social, Worst-Case No SOA RL SOA Social, Worst-Case SOA Social, Worst-Case No
B4: Scalable, Repeatable Simulated, Download Simulated, Download Simulated, Download Simulated, Download Simulated, Download Simulated, Download Simulated, Download Simulated, Download Setup Req., Phys. Eval
B5: Eval Grounded in Human Data No No Demo. pipeline No No No No SEAN-EP extension Yes
B6: Use Validated Instruments No No No No No No No No Validation in Process
TABLE IV: Characteristics of existing social navigation benchmarks

VIII-B Existing Social Navigation Benchmarks

The review requires a system that runs agents, defined scenarios, and evaluation metrics. Protocols specify how to construct studies; benchmark environments add executable tests and often baselines; challenges add a shared forum for results. Reference datasets are optional. These classes should not be conflated when judging accessibility or coverage.

VIII-B 1 Benchmarking Protocols

The Social Navigation Protocol defines controlled interactions for collecting expert demonstrations and evaluating policies with relatively low variability. Successive studies refined the questionnaires through reliability analysis. Its flexibility comes with manual experimental setup: the reviewed protocol does not provide a ready-made simulated environment.

Original Figure 8 with all 9 labeled panels in the source arrangement.
Fig. 8: Commonly used social benchmarks. Benchmarks range from abstract tests of dynamic obstacle avoidance to simulated interactions with moving humans of varying degrees of fidelity to protocols for setting up physical experiments in well-specified scenarios.

VIII-B 2 Social Navigation Benchmarks

ArenaBench supplies several robot models, classical and learned baselines, and varied motion metrics. DynaBARN stresses dynamic-obstacle models; gym-collision-avoidance evaluates policy-controlled agents; HuNavSim adds diverse pedestrian attitudes; SocNavBench replays real pedestrian trajectories. In the reviewed versions, these scalable environments largely rely on computed proxies rather than ratings from people experiencing the robot. Their quantitative scores therefore cover different constructs and are not interchangeable social-quality rankings.

VIII-B 3 Social Navigation Challenges

CrowdBot targets dense crowds and several robot morphologies. The 2021 iGibson challenge compared navigation and personal-space performance in household simulation; SEANavBench combined scalable simulation with scenario variation and a web infrastructure that could collect human feedback. The article describes the challenges’ status at publication: these are historical descriptions, and human ratings or physical-robot tests were absent from the specific public evaluations discussed.

VIII-C Strengths and Limitations of Existing Benchmarks

The recurring tradeoff is scalability versus grounded evidence about people. Automated suites often omit human ratings; human-centered protocols need more manual setup; few resources cover consequential edge cases well. These limitations motivate combining resources rather than claiming that one existing benchmark fully tests social navigation.

VIII-D Properties of a Good Social Navigation Benchmark

Guidelines B1–B6 call for social-behavior measures, quantitative results on several dimensions, comparison baselines, efficient repeatable trials, human-grounded evaluation, and validated measurement instruments. Include weak baselines and, when feasible, strong demonstrations or policies; neither constitutes a guaranteed mathematical performance bound. The suggested thirty physical trials is a rule of thumb, while sample size should follow the variability and statistical question.

The proposed improvements are more human ratings, shared validated questionnaires, consistent quantitative metrics, and conventional-navigation corner-case tests. Learned perception scores could scale evaluation, but their validity may not transfer across contexts. A common reporting language should expose these limits rather than erase them.

IX Social Navigation Datasets

Dataset evaluation adds sampling coverage, sensor viewpoints, annotations, behavior generation, and privacy to the common taxonomy. The review distinguishes recordings of pedestrians, demonstrations by robots, synthetic scenes with human ratings, and autonomous interaction logs. Each supports different learning and evaluation claims.

IX-A Expanding the Factors for Dataset Analysis

A useful dataset description explains both what was observed and how that distribution was produced. Hardware, collection policy, annotation granularity, and scenario sampling can be as important as raw dataset size.

Robot Hardware Platform

Embodiment and expressive features may change people’s reactions. Multiple robot types can broaden coverage, but the dataset should identify which morphologies actually generated its interactions.

Sensors

Record control commands alongside sensor observations when possible. Otherwise, a motion trace may not reveal the teleoperator’s or policy’s intended action.

Robot Behavior Authoring Methods

Human trajectories can support imitation when human-like behavior is the objective; robot demonstrations are preferable when robot behavior should differ. Teleoperation protocols should explain operator location and visibility, prevent avoidable interference with the interaction, and use diverse operators to reduce dependence on one style.

Data Collected

Match dataset breadth to collection resources. Unconstrained recording captures natural variation, while targeted events improve coverage of selected scenarios. Each declared scenario needs adequate examples; broad labels with sparse coverage do not establish reliable deployment generalization.

Annotations

Specify annotation taxonomy and granularity, from episode labels to temporal segments, trajectories, bounding boxes, pose, or gaze. Existing activity-recognition resources offer examples, but the paper notes the absence of a settled social-interaction taxonomy.

Privacy and Fairness

The authors recommend deciding early how human recordings will be anonymized and governed. The discussion does not itself provide a complete fairness protocol or evidence that anonymization removes all dataset biases.

IX-B Existing Social Navigation Datasets

JRDB, THÖR, and LCAS support perception and trajectory analysis; SCAND supplies teleoperated socially compliant robot demonstrations, whereas MuSoHu records people’s egocentric navigation with a sensor helmet. ETH/UCY, EIFPD, SDD, EFL, WILDTRACK, and related collections vary in viewpoint, agents, and annotation. Pedestrian trajectories can be replayed but do not thereby become reactive robot interactions.

SocNav1/2 provide rated synthetic interactions; DynaBARN supplies synthetic environments; CrowdBot and SG-LSTM include robot-perspective crowd data. SACSoN reports seventy-five hours, fifty-eight kilometers, and over four thousand interactions from autonomous collection with rescue support. Its deliberately interaction-rich collection objective differs from the socially compliant policy later learned from those data. Table V retains the detailed comparison; collection purpose and policy matter when choosing training examples.

Benchmark Dataset THOR SCAND ETH/UCY Trajnet++ EIFPD SocNav1 SocNav2
Dataset Context and Scope Three predefined human-robot social role-play activities. Socially compliant navigation in-the-wid Human trajectories recorded from a bird’s eye view vantage point Human trajectories recorded from a bird’s eye view vantage point Human trajectories recorded from a bird’s eye view vantage point Scenarios with interactions labeled with a social score Short sequences with interactions labeled with social and holistic scores
Environment Curated indoor environment: two rooms with arranged furniture and motion caption system Indoor and Outdoor campus-scale environment Outdoor, fixed environment Indoor and Outdoor, fixed environment Outdoor, fixed environment Indoor, abstract Indoor, abstract
Data Collected 60 minutes of motion tracking across 600 human trajectories 522 minutes, consisting of 138 trajectories of teleoperation data from 4 demonstrators Bird’s eye view frames, with annotated human trajectories across time on 5 scenes >>200K human trajectories across dozens type scenes with high density crowds Bird’s eye view frames, with annotated human trajectories across time 9280 static scenario descriptions with social scores 53600 dynamic scenario descriptions with social and all-encompassing scores
Scenarios Visiting areas, carrying boxes, inspecting targets Goal-oriented social navigation Pedestrian navigation Pedestrian navigation Pedestrian navigation Evaluating robot disturbance Evaluating robot trajectories
Robot Platform Linde CitiTruck robot (W 1.56m x L 0.55x x H 1.17m) Boston Dynamic Spot, ClearPath Jackal N/A N/A N/A Turtlebot-sized robot Turtlebot-sized robot
Robot Behavior Programmed to follow a predefined path in a socially unaware manner Human teleoperation in a socially compliant manner N/A N/A N/A Static placement Teleoperation and policy
Human behavior Follow a pre-defined path, and solving tasks in presence of other humans Demonstrators teleoperate the robot in an open environment with other humans Open world navigation Open world navigation Open world navigation Static placement Randomized simulated trajectories
Sensors Stationary Velodyne 3D LiDAR, Qualisys Oqus 7+ motion tracking system, Tobii Pro Glasses for gaze tracking Velodyne 3D LiDAR, Azure RGB, Odometry, Joystick Stationary RGB camera overlooking pedestrians Stationary RGB camera overlooking pedestrians Camera fixed overhead 23 meters from the floor Overhead view of robots and humans Robot and human poses, with 53600 short videos
Tasks and Metrics N/A N/A Human trajectory prediction Human trajectory prediction Human trajectory prediction Acceptability of robot disturbance of humans Acceptability of robot movement around humans
Benchmark Dataset SDD EFL LCAS WILDTRACK JRDB Crowd-Bot Dyna-BARN SACSoN SG-LSTM
Dataset Context and Scope Human trajectories recorded from a bird’s eye view vantage point using a drone Human trajectories recorded from the perspective of a human Online human detection from 3D lidar scans Multi-camera detection and tracking of moving humans Dataset of social interactions in indoor and outdoor environments for solving perceptual tasks Outdoor pedestrian tracking around a personal mobility robot Diverse set of moving agent scenarios Autonomous policy interacting with humans Curated interaction, movements and formation of pedestrian groups
Environment Outdoor environment, focusing on diverse social navigation scenarios Outdoor scenes such as parks, malls and a university campus Outdoor environment Outdoor environment Indoor and outdoor environment Crowded outdoor scenes Indoor, abstract Indoor environment Outdoor university campus environment
Data Collected Bird’s eye view frames with annotations of pedestrians, bikes, cars, etc in 100 different scenes with annotations of social interactions RGBD frames recorded from an egocentric perspective 49 minutes of 3D lidar scans Multi-camera synchronized frames at 10fps 60,000 annotated frames of humans, recorded from an egocentric robot view 250K frames / 200 minutes from an egocentric POV Moving polynomial agent scenarios 75 hours of visual navigation with 4000 human-robot interactions collected from robot POV Color and depth frames, pedestrian and group bounding boxes
Scenarios Real-world navigation with social interactions Egocentric real-world navigation Real-world navigation in crowded environments Third-person view open-world pedestrian navigation in crowded environments Navigation in a campus-scale crowded environment Real-world navigation in a crowded outdoor environment Multiple moving agents Real-world navigation in a crowded indoor environment Paths, green spaces, study spaces, cafes, gatherings, weather events
Robot Platform 3DR Solo drone N/A Pioneer 3-AT mobile robot N/A JackRabbot mobile manipulator Qolo personal mobility robot N/A iRobot Roomba GO1 Edu robot
Robot Behavior N/A N/A Human teleoperation N/A Teleoperated Both shared-control and reactive control N/A Policy-controlled Unobtrusive data collection
Human Behavior Performing navigation activities such as walking, driving, and biking in a socially compliant manner Socially compliant navigation in the open world Open-world navigation Open-world navigation Open-world navigation Open-world navigation Navigation among moving obstacles Open-world navigation Many types of pedestrian groups
Sensors RGB camera GoPro Hero 3 stereo cameras with 100mm baseline Velodyne VLP-16 3D LiDAR three GoPro Hero 4 and four GoPro Hero 3 360 RGB cameras, Velodyne LiDARs, Sick LiDARs, microphone, RGBD camera, fisheye camera, wheel encoders Point clouds, RGBD, people trackers, pose, contact forces N/A Spherical RGBD, fisheye RGB, 2D LIDAR, odometry, bumper RGB-D cameras
Task and Metrics Social activity recognition, planning and trajectory prediction Trajectory prediction Online human detection and tracking Trajectory prediction, person tracking and re-identification Benchmark and metrics for 2D and 3D Person detection and tracking Benchmark for crowd navigation Dataset of scenarios for benchmarking dynamic navigation Learning dataset of autonomous policy interacting with humans. Group size, walking speeds, proximity, cohesiveness, interactions.
TABLE V: Characteristics of existing social navigation datasets
Original Figure 9 with all 14 labeled panels in the source arrangement.
Fig. 9: Illustration of various social navigation datasets. See the text and Table V for details.

IX-C Guidelines for Datasets

Guidelines D1–D8 balance broad scope with adequate sampling, favor robot data when robot-specific behavior is needed, encourage embodiment diversity, retain generation commands, formalize labels, and address privacy during design. Good coverage reduces some distribution gaps; it cannot guarantee that no out-of-distribution event will occur after deployment.

X Simulation-based Evaluation

A social-navigation simulator must represent interactions between at least two agents. The reviewed systems range from simplified collision avoidance to detailed environments with animated pedestrians. The section compares their fidelity and behavior assumptions, then proposes a common metrics interface. Simulation supports controlled evaluation, while its human models remain an important limitation.

X-A Expanding the Factors for Simulator Analysis

Simulator descriptions need to separate geometric and visual detail, physical constraints, behavior, and interoperability. A realistic-looking scene does not establish realistic social responses.

Abstraction Level

The comparison includes systems capable of individual human–robot encounters, rather than only aggregate crowd flows.

Simulation Focus

State whether the simulator targets crowds, particular human–robot interactions, or dynamic-obstacle avoidance. This focus shapes which algorithmic claims its tests can support.

Simulation Platform

Identify the underlying simulation engine or standalone implementation, since it constrains integration and reuse.

Agent Representation

Specify whether humans and robots are distinct agent classes or share one abstract representation.

Scene Representation

Scenes may use planar geometry, modeled three-dimensional assets, or scans of real places. Describe both representation and intended environment types.

Scene Visual Fidelity

Visual fidelity ranges from planar displays to detailed rendering. It matters for perception policies but is distinct from correct interaction behavior.

Physics Simulation Fidelity

Distinguish kinematic motion from force, mass, and full physics simulation. Different approximations constrain the kinds of failures that can be observed.

Robot Simulation Fidelity

Robots may be points, cylinders, or detailed articulated models. Report embodiment and dynamical limitations when transferring simulation results.

Pedestrian Simulation Fidelity

Pedestrian simulation can include only position, or extend to body shape, gait, and behavioral variability. Each added feature addresses a different source of realism.

Pedestrian Visual Fidelity

Visual agents range from simple discs to animated meshes. The table groups detailed human meshes together because photorealism lacks an objective threshold here.

Pedestrian Reactivity

Recorded pedestrians follow fixed trajectories; reactive models adjust to other agents. Higher-level controllers can add attitudes, styles, and social activities. These choices directly affect what a robot policy learns.

Simulation Interoperability

Describe interfaces such as Gym and ROS that permit integration with other training, control, and evaluation systems.

X-B Existing Social Navigation Simulators

CrowdNav and gym-collision-avoidance use simple agents for collision avoidance; DynaBARN varies obstacle motion; CrowdBot and MengeROS target crowds. HuNavSim, InHuS, and IMHuS add authored pedestrian attitudes or higher-level behaviors. PedSimROS integrates social-force and group behavior models with robot software.

iGibson provides interactive household scenes, SEAN 2.0 detailed social situations, SocialGym 2.0 multi-agent policies with motion constraints, and SocNavBench recorded pedestrian replay. Table VI and Figure 10 retain the systems’ detailed contrasts. These are the paper’s descriptions of contemporary versions, not a current compatibility or quality ranking.

Original Figure 10 with all 13 labeled panels in the source arrangement.
Fig. 10: Visual description of select simulators. See Section X-A for details.
Simulator Crowd-Bot CrowdNav Dyna-BARN gym-collision-avoidance Hu-Nav-Sim iGibson InHuS IMHuS Menge-ROS PedSimROS SEAN 2.0 SocialGym 2.0 Soc-Nav-Bench
Sim Focus Crowd Sim Crowd Sim Dyn. Obstacles Collision Avoid. Human Sim Social Nav Human Sim Human Sim Crowd Sim Crowd Sim Social Nav Social Nav Social Nav
Sim Platform Gazebo Crowd-Nav Gazebo gym-collision-avoidance Gazebo iGibson MORSE & Stage Gazebo Menge-ROS Gazebo SEAN 2.0 SocialGym 2.0 Soc-Nav-Bench
Agent Repr. Ped. & Robot Ped. & Robot Ped. & Robot Robots Ped. & Robot Ped. & Robot Ped. & Robot Ped. & Robot Ped. & Robot Ped. & Robot Ped. & Robot Ped. & Robot Ped. & Robot
Scene Repr. Model 3D Geom. 2D Geom. 2D Geom. 2D Model 3D Scanned Mesh Geom. 2D Geom. 2D Geom. 2D Model 3D Model 3D Geom. 2D Scanned Mesh
Scene Fidelity Realist. 3D 2D Abstr. 3D 2D Abstr. 3D Abstr. 3D Realist. 3D Abstr. 3D 2D Abstr. 3D Realist. 3D 2D Realist. 3D
Physical Fidelity Physics Model Kinematic Kinematic Kinematic Kinematic Force & Mass Kinematic Kinematic Kinematic Physics Model Physics Model Kino-dynamic Kinematic
Robot Fidelity Robot Dyn. Disc Robot Shape Disc Robot Dyn. Robot Shape Robot Shape Robot Shape Disc Robot Shape Robot Dyn. Kino-dynamic Robot Shape
Ped. Sim Fidelity UMANS Move w/o Gait Poly-nomials Policy-Based Gait & Attitude Move w/o Gait Gait & Attitude Gait & Attitude Move w/o Gait Gait & Activity Move w/o Gait Move w/o Gait Gait & Pose
Ped. Viz Fidelity Detail’d Disc Cylind. Disc Detail’d Detail’d Detail’d Detail’d Disc Sensors Detail’d Polyg. Detail’d
Ped. Reaction UMANS ORCA Policy-Based Policy-Based SFM & Attitude ORCA React & Attitude ORCA, Attitude SFM & ORCA SFM SFM SFM Replay Only
Sim Interop Unity Interf. Gym ROS bag Gym ROS Interf. Gym ROS Interf. ROS Interf. ROS Interf. ROS Interf. ROS & Unity Gym, ROS ROS Interface
TABLE VI: Characteristics of existing social navigation simulations. See Section X-A for details.

X-C Analysis of Simulation Platforms

The analysis asks which simulator differences are scientifically meaningful and which could be shared infrastructure. Focus, pedestrian reactivity, agent policies, environment geometry, and metric definitions all constrain cross-system comparisons.

X-C 1 Simulation Focus

An algorithm trained with cylinders and one trained with gait-aware pedestrians may solve different problems. Specialized simulators remain useful, but a common interface could permit shared features and more informative cross-system evaluation.

X-C 2 Common Platforms

Common engines, scene representations, and pedestrian models create opportunities to reuse implementations. Reuse can reduce software fragmentation without making all simulators equivalent.

X-C 3 Pedestrian Reactivity

Replay preserves observed trajectories but loses some body-motion detail and cannot react when a robot deviates from the recorded situation. Reactive models such as social forces and ORCA address that counterfactual response, yet remain imperfect human models. Neither approach alone reproduces human interaction; comparing policies requires disclosing this choice.

X-C 4 Multi-Agent Policies

When other agents’ policies are unknown or learned simultaneously, fixed pedestrian models may be inadequate. Multi-agent learning offers another route, but the review does not show that jointly learned behavior automatically matches people.

X-C 5 Environments

Constrained geometry and dense traffic create many consequential encounters, including corridors, hallways, malls, and airports. Simulator environments should represent the constraints relevant to the intended task rather than relying only on open-space tests.

X-C 6 Metrics

Simulation can reset conditions and expose ground truth for consistent metric calculation. It also supports learned scores and human ratings of simulated episodes. Repeatability strengthens comparisons within those assumptions; it does not establish ecological validity by itself.

Original figure
Fig. 11: Proposed social navigation simulation metrics API. A wide diversity of simulators and robot platforms exist, many of them supporting one or more platforms APIs such as OpenAPI Gym or ROS. We propose to define a unified API that specifies the inputs needed to generate our recommended social navigation metrics, specified as either Gym observations or ROS messages. A unified metrics API with implementations for Gym and ROS will output a single output representation, enabling post-processing tools to generate visualizations, analytics and logging with uniform code. To take advantage of these tools, simulator and robot developers only need to contribute bridge code to output the required Gym or ROS data; dataset developers only need output the single output representation.

X-D Towards a Unified API for Social Navigation Simulation

The proposed interface standardizes inputs, metric computations, and output records across robots, datasets, and simulators. Shared computation would reduce discrepancies caused by separate implementations. This is an interface proposal and preliminary implementation effort, rather than an experimentally demonstrated universal evaluation solution.

X-D 1 Design of the high-level simulator metric API

Inputs include agent poses over time, robot observations and actions, resulting trajectories, and static or dynamic obstacle geometry. Shared libraries should compute step-level and episode-level metrics, while a defined output schema supports analysis, visualization, and learning tools. Extensibility is necessary because different research questions require different observations and metrics.

X-D 2 Implementation of Social Navigation API

The authors describe planned JSON schemas, C++ and Python metric libraries, and example bridges for ROS and GymCollisionAvoidance at SocialNavAPI. Other simulator integrations are reported as ongoing work. The edition preserves that publication-time status and does not assert that every proposed bridge is now implemented.

X-E Guidelines for Simulators

Guidelines S1–S9 encourage standard interfaces and metrics, extensible software, systematic data recording, benchmark construction, human labeling, common robot models, detailed pedestrians, and several behavior-generation methods. Developers should validate and refine each simulator for its intended use. More visual or behavioral features do not remove the need to test fidelity.

XI Conclusions

The paper connects eight behavioral principles to concrete evaluation resources and reporting choices. Its central contribution is a shared framework for asking what a benchmark measures and where its conclusions apply. Human-grounded evidence, contextual validation, diverse metrics, and conventional navigation reliability remain complementary requirements; the proposed API addresses implementation consistency rather than resolving all social constructs.

ACKNOWLEDGMENT

An earlier symposium excerpt focused on the human–robot interaction aspects of the work. The present article expands that discussion into the broader evaluation framework.

Contributions

The contributions statement records symposium participation and responsibilities across definitions, metrics, evaluation, scenarios, datasets, benchmarks, simulation, illustrations, and manuscript editing. Claudia Pérez-D’Arpino and Anthony Francis co-led the paper; the broad author group supplied and synthesized expertise rather than running a single shared experimental benchmark.

Concept demonstration

Open image ↗