Overview
This long review organizes social navigation around explicit behavioral principles, research questions, evaluation metrics, scenarios, benchmarks, datasets, and simulators. It proposes reporting and interface guidelines rather than a new navigation policy or a universally validated social-performance score. The comparisons describe the field as reviewed in 2023.
At a glance
Social navigation evaluation must assess both navigation outcomes and how people experience a robot’s behavior. The paper proposes eight principles, a taxonomy of experimental conditions, guidelines for evaluation resources, and a common metrics-interface design. Its aim is more comparable studies while preserving task and cultural context; it does not demonstrate that a single benchmark can measure every aspect of socially appropriate behavior.
I Introduction
Discussions at the 2022 Social Navigation Symposium motivated this synthesis. Principles describe desired behavior; guidelines specify practical choices for measuring it and building reusable evaluation resources. The article connects metrics to scenarios, benchmarks, datasets, and simulators. It offers a framework for principled comparisons across varied research questions rather than one mandatory experimental protocol.
II Related Work
The review builds on prior evaluation surveys and analyses of social-navigation challenges. Its contribution is a shared definition, expanded principles, research lifecycle, resource taxonomy, scenario cards, and interface proposal. These recommendations extend existing work; the paper is not a comprehensive survey of every navigation algorithm or an empirical leaderboard of the proposed principles.
III Towards a Definition of Social Navigation
The term social navigation covers dynamic-obstacle avoidance, social convention, and intentional communication. The authors separate these meanings before defining behavior that accommodates others while pursuing the robot’s own task. This distinction determines which evidence is needed: collision avoidance alone cannot establish comfort or politeness.
III-A What is a Social Robot?
A socially navigating robot acts around humans or robots while adjusting its behavior so their experience is preserved or improved. Expressive hardware or interactive features alone do not guarantee beneficial interaction. The definition concerns accommodation of other agents’ needs, feelings, capabilities, and conventions; how those effects are measured remains an experimental question.
III-B Principles of Social Navigation
The eight principles are safety, comfort, legibility, politeness, social competency, understanding other agents, proactivity, and contextual appropriateness. Safety includes damage and induced harm beyond direct collision. Comfort concerns annoyance and stress; legibility makes goals understandable; politeness concerns consideration in motion and communication; social competency follows local conventions. Understanding others supports prediction and accommodation, while proactivity resolves emerging conflicts or deadlocks rather than simply waiting.
Context changes how these objectives matter: culture, individual abilities and histories, geometry, operational setting, task urgency, and interpersonal relationships can all alter appropriate behavior. The principles overlap and can conflict. Illustrated personal-space distances are contextual examples, not universal thresholds. The article recommends describing intended contexts precisely enough for readers to assess transfer, without claiming a complete computational definition of context.
IV Research Methodologies of Social Navigation
Evaluation methods should follow the scientific question. Algorithmic comparisons may use measurable navigation outcomes; questions about human perceptions require evidence about human responses. A benchmark’s scope therefore depends on whether it studies performance, mechanisms, generalization, perceived social quality, behavior change, or failures found in deployment.
IV-A Research Questions of Social Navigation
The paper distinguishes comparisons against baselines, component ablations, and environmental generalization from human ratings, observed behavior changes, and exploratory issue discovery. Simulation can make ablations practical, but claims about comfort or politeness often need validated human studies. Live deployments are especially useful for discovering the distribution of encounter types and failures. The recommendation is to combine suitable navigation tests and human evidence, while documenting what each study can actually establish.
IV-B Types of Social Navigation Studies
The review distinguishes uncontrolled field studies, partly controlled robot deployments, laboratory experiments, repeatable social-navigation scenarios, and staged interactions. Natural settings reveal unanticipated behavior but make individual encounters hard to reproduce. Familiarity and participant instructions can change reactions in deployments and laboratories. Staged gatherings increase interaction density but remain staged. Ecological validity and construct validity require evidence, not simply choosing one of these study labels.
IV-C Lifecycle of Social Navigation Research
The proposed lifecycle links data collection, issue discovery, laboratory experiments, scenario development, benchmarking, and public challenges. Frequent or high-risk failures found in deployments should inform reproducible scenarios, while targeted experiments should feed findings back into evaluation resources. Controlled scenarios cannot replace observing new real-world conditions. Shared datasets and clearly defined success criteria help the wider community benefit from costly deployments.
IV-D Guidelines for Real-world Studies
Three real-study guidelines are to preserve safety through monitoring or protective controls, respect participant consent and privacy, and define a clear scientific objective. Ablated policies may behave unexpectedly and may be better studied in simulation when adequate controls are absent. These are the paper’s experimental-design recommendations; institution-specific review and legal requirements depend on the study context.
V A Taxonomy of Social Navigation
The taxonomy provides common terms for comparing metrics, scenarios, benchmarks, datasets, and simulators. It supports explicit descriptions of what an evaluation resource measures and what conditions it represents, rather than treating resources with different assumptions as interchangeable.
V-A A Taxonomy for Analysis
Metrics quantify behavior; scenarios specify interactions; benchmarks combine protocols and metrics; public challenges add submission and comparison mechanisms. Datasets supply observations and annotations, while simulators generate controlled interactions. The proposed shared interface targets incompatible simulator outputs. Each resource also needs factors specific to its purpose, such as sampling coverage for datasets or fidelity for simulators.
V-B Factors Common to Social Navigation
The seventeen shared factors cover context, physical environment, human-user population, human behavior, robot task and role, scenarios and their coverage, robot embodiment, sensors, actuators, communication, collected data, metrics, human and robot behavior-generation methods, and simulation versus reality. Descriptions should disclose whether observations are noisy robot-view measurements or privileged ground truth, whether humans react to the robot, and how test cases and rare events are sampled.
Robot familiarity, participant instructions, morphology, visibility, sensing delay, and noncooperative behavior can all change outcomes. A shared vocabulary does not require identical robots or one universal setting. Dataset annotations/privacy, benchmark baselines/accessibility, and simulation fidelity require additional reporting beyond these common factors.
VI Social Navigation Metrics
Social navigation involves several partly overlapping constructs, so a scalar navigation-efficiency score cannot capture its full quality. Even safety can refer to physical harm, perceived threat, or social disruption. The paper recommends connecting each metric to a defined construct and context, and treating combinations of metrics as design choices that require justification.
VI-A Taxonomy of Existing Social Navigation Metrics
Metrics are classified along three independent axes: how they are obtained, what variable they model, and their temporal scope. Describing all three makes it easier to distinguish a human perception measurement from a reproducible proxy or a time-local control signal.
VI-A 1 Taxonomy based on their nature
Surveyed measurements include questionnaires and sensor-based indicators of human responses. Algorithmic metrics are hand-designed or learned from human-labeled data. Hand-designed measures are objective in their computation, while their interpretation may be subjective or incomplete. Learned scores can be cheap to apply but inherit their training data and assumptions. Human studies are expensive and variable, yet remain important evidence for claims about human experience.
VI-A 2 Taxonomy based on the variable being modeled
Non-social measures assess task success, efficiency, or energy; social measures target comfort, acceptance, trust, or related constructs; all-encompassing measures attempt both. A reliable task-efficiency number still does not provide social-performance evidence. The authors regard comprehensive human-aligned scores as desirable but note that few had been proposed and none resolves all contextual issues.
VI-A 3 Taxonomy based on temporal scope
Step-wise metrics score individual instants; task-wise metrics score a whole episode. Averaging time steps can conceal brief consequential events or reversals, so episode measures may need temporal structure or event weighting. For reinforcement learning, an immediate score is useful only if its accumulated value matches the intended task outcome; delayed episode feedback can be more faithful to that objective.
VI-B The Challenges of Measuring Social Navigation
Measurement has expanded from whether navigation succeeds, to how efficiently it succeeds, to its effects on others. Four difficulties remain central: changing environments, participant adaptation, changes in deployed robot behavior, and limits of the metrics themselves. These complicate reproducibility and comparisons across studies.
VI-B 1 The Challenges of Dynamic Environments
Lighting, weather, pedestrian behavior, and robot battery state can alter measured performance during deployment. Controlling every factor may be impractical, but reports should identify relevant changes and limits of comparability rather than attributing all variation to the navigation policy.
VI-B 2 Challenges Based on How Robots Change People
People adapt to robot signals and behavior across encounters, changing fluency, conflict rates, and acceptance even if the policy stays fixed. Student or robotics-expert participant pools may be unrepresentative of the public. Studies should report prior exposure and examine novelty and long-term adaptation rather than treating all encounters as independent experiences of equally unfamiliar users.
VI-B 3 Challenges Based on How People Change Robots
Experienced users may implicitly compensate for a robot’s weaknesses, making its deployment appear safer or more acceptable than encounters with visitors. Anecdotes about collision-free development teams illustrate this confound; they are not quantified performance results. Long-term acceptance can also remain high despite faults, so task statistics and overall attitudes need separate interpretation.
VI-B 4 The Limitations of Social Metrics Themselves
Human ratings depend on culture, goals, setting, and familiarity, while realistic studies and large participant pools are expensive. Choosing a small set of observable signals can omit gaze, gestures, or facial expressions. Combining metrics requires context-sensitive weights that remain an open problem, and heterogeneous sensing or communication capabilities can make a standardized score unfair across robot embodiments.
VI-C Assessment of Existing Social Navigation Metrics
The authors favor ratings from people experiencing an interaction, while acknowledging scaling, disruption, and reproducibility limits. They find no convincing universal procedure for ranking one metric as better than another. Reports should identify both a metric’s original validation context and the context of its new use. Learned comprehensive scores and narrowly diagnostic metrics each have limited applicability.
| Metric | Short | Description | Class. | Parameters | Unit | Range | Cited | |
| Success metrics | Success | Binary variable describing whether the robot reaches the goal. When averaged, it is referred as Success Rate (SR). | NHT | - | boolean | [15] | ||
| Collision | Number of collisions in the trajectory. When averaged it is referred to as Collision Rate (CR). | NHT | Collisions to terminate episode | collision | [21] | |||
| Wall Collisions | Number of collisions against walls. | NHT | - | collision | R@G | |||
| Agent Collisions | Number of collisions against humans or robots. | NHT | - | collision | R@G | |||
| Human Collisions | Number of collisions against humans. Also called H-collisions [88]. | NHT | - | collision | R@G [88] | |||
| Timeout before reaching goal | Binary variable accounting for failures caused by a timeout. | NHT | Time threshold | timeout | R@G | |||
| Failure to progress | Number of failures caused by not decreasing the distance to the goal for a given period of time. | NHT | Distance & time thresholds | failure | R@G | |||
| Stalled time | Time where the magnitude of the speed of the robot falls within a given threshold. | NHT | Distance & time thresholds | [89] | ||||
| Time to reach goal | Time between task assignment and completion. | NHT | - | [90, 21] | ||||
| Path length | Length of the trajectory. | NHT | - | [90, 21] | ||||
| Success weighted by path length | Success weighted using normalized inverse path length, i.e., weighted using path length divided by the max of the min distance and path length [15]. | NHT | - | success | [15] | |||
| Quality and social metrics | Velocity-based features | , , | Minimum, average and maximum linear velocity on a trajectory. | SHT | - | [21] | ||
| Linear acceleration based features | , , | Minimum, average and maximum linear acceleration on a trajectory. | SHT | - | [21] | |||
| Movement jerk | , , | Minimum, average and maximum linear jerk (i.e., the second-order derivative of the linear speed). | SHT | - | [21] | |||
| Clearing distance | , | Minimum and average distance to obstacles in a trajectory. | SHT | - | [21] | |||
| Space compliance | Ratio of the trajectory with the minimum distance to a human under a given threshold. If the threshold is , it is referred to as Personal Space Compliance () [26]. | SHT | Distance threshold | [71] | ||||
| Minimum distance to human | Minimum distance to a human in a given trajectory. | SHT | - | |||||
| Minimum time to collision | Minimum time to collision with a human agent at any point in time in the trajectory, should all robots and humans move in a linear trajectory. | SHT | - | [28] | ||||
| Aggregated Time | Time taken for a subset of cooperative agents to meet their goals. | SHT | Cooperative agents’ set | [10] |
VI-D Recommendations for Metric Usage and Development
The proposed compromise combines a shared set of conventional navigation measures, iteratively validated human surveys, and further research on learned proxies. These sources of evidence address different questions; inexpensive proxy scores do not by themselves validate human social experience.
VI-D 1 Recommendations for Hand-crafted Metrics
Table I specifies a recommended collection of success, collision, failure, trajectory, and social-proxy measures, together with required parameters, units, ranges, and references. Report context and thresholds because a good value is task dependent. Across repeated trials, distributions and outliers should accompany averages, especially for safety-related outcomes.
VI-D 2 Recommendations for Survey Development
Questionnaires should be iteratively evaluated for test–retest reliability, construct validity, and bias. Survey length and repetition trade information against fatigue and response burden. Use question-appropriate response options, consider nonapplicable answers, balance experimental conditions, and apply suitable statistical analyses before claiming statistical significance. Reliability statistics support instrument assessment but do not alone prove that a survey measures the intended social construct.
VI-D 3 Recommendations for Future Metric Development
Proposed research directions include time-to-collision, motion smoothness, and visibility-based proxies, and datasets pairing algorithmic measurements with human ratings. Correlation and held-out evaluation can test when a proxy predicts surveyed responses. Learned reward functions are another possibility, but the paper presents these as research directions, not already validated replacements for human evaluation.
VI-E Metric Guidelines
The eight metric guidelines call for robust conventional navigation, simulation checks before deployment, explicitly reported context-dependent parameters, cautious use of learned scores for iteration, validated surveys for social claims, consistent experimental design, iterative analysis, and detailed reporting. The recommended evidence includes several metrics and temporal scopes plus distributions, rather than a single favorable average.
VII Social Navigation Scenarios
Scenario definitions connect naturalistic observations with controlled experiments. They can identify encounters in field data, elicit behavior in deployments, support regression tests, curate datasets, and guide imitation learning. These uses need different outcome labels: an interrupted frontal approach may be valid observational data even when it would be a failed training demonstration. Coverage of naturally occurring variants should be separated from criteria for success.
| Social Navigation Scenario Card | |
| Scenario Metadata | |
| Scenario Name | Frontal Approach |
| Scenario Description | A robot and a human approach head-on in a passable space. |
| Scientific Purpose | Low-density pedestrian scenario applicable indoors and outdoors. |
| Scenario Definition | |
| Geometric Layout | A space wide enough for the robot and human can pass each other. |
| Intended Robot Task | The robot navigates from one side of the space to the other |
| Intended Human Behavior | The human navigates in the opposite direction of the robot. |
| Scenario Usage Guide | |
| Success Metrics | Success Rate, (No) Collisions |
| Quality Metrics | Comfort, Politeness |
| Ideal Outcome | Robot goes around human in a socially acceptable manner. |
| Failure Modes | 1. Robot collides with human |
| 2. Robot fails to exit in time limit | |
| Labeling Criteria | 1. Robot and human face each other |
| 2. Robot and human move towards each other at start of episode | |
| 3. Sufficient clearance exists for robot and human to pass each other | |
VII-A Scenario Design Methodology
Scenario design has three stages: define its research context, robot task, human behavior, and success measures; evaluate commonality with existing scenarios, flexibility, and fitness to purpose; and communicate a reproducible specification. Pilot studies should check whether the setup actually elicits and measures the intended behavior. Avoid prescribing a preferred solution so narrowly that alternative or unexpected interactions disappear from the data.
| Scenario Name | Scenario Description | Physical Env. | Geom. Layout | Scientific Purpose | Robot Role | Robot Task | Human Behavior | Ideal Outcome | Related Scenarios | Cited In |
| Hallway Scenarios | ||||||||||
| Frontal Approach | A pedestrian and robot approach head-on. | Generic | Passable Space | Pedestrian Interaction | Any | Navigate A to B | Navigate B to A | Robot / humans pass | Ped. Obstruct | [30, 6, 10] |
| Pedestrian Overtaking | A pedestrian overtakes a moving robot. | Generic | Passable Space | Pedestrian Interaction | Any | Navigate A to B | Navigate A to B | Human passes robot | Down Path | [45] |
| Robot Overtaking | A robot overtakes a moving pedestrian. | Generic | Passable Space | Pedestrian Interaction | Any | Navigate A to B | Navigate A to B | Robot passes human | [6, 10] | |
| Intersection (No Gesture) | A robot and a human cross at an intersection. | Indoor | Intersection | Pedestrian Interaction | Any | Navigate A to B | Cross Navigate | Both pass no collision | [89, 6, 10, 109] | |
| Intersection (Gest. Wait) | A robot is told to wait at an intersection. | Indoor | Intersection | Pedestrian Interaction | Servant | Navigate A to B | Cross Navigate | Human goes then robot | Gesture Proceed | [30] |
| Blind Corner | A robot and human meet at a blind corner. | Indoor | Corner | Pedestrian Interaction | Any | Navigate A to B | Navigate B to A | No collision / obstruction | [30, 31] | |
| Doorway Scenarios | ||||||||||
| Narrow Doorway | A robot and human pass a narrow doorway. | Indoor | Room & Door | Pedestrian Interaction | Any | Navigate A to B | Navigate B to A | No collision / obstruction | Narrow Arch | [30] |
| Entering Room | A robot enters a room occupied by a human | Indoor | Room & Door | Pedestrian Interaction | Any | Navigate out to in | Navigate in to out | Robot allows human exit | Entering Elevator | R@G |
| Exiting Room | A robot exits a room while a person enters. | Indoor | Room & Door | Pedestrian Interaction | Any | Navigate in to out | Navigate out to in | Robot exits first | Exiting Elevator | R@G |
| Interpersonal Scenarios | ||||||||||
| Joining a Group | A robot joins a group of robots or people. | Generic | Open Space | Group Interaction | Any | Navigate to group | Continue conversing | Robot joins group | Leaving a Group | [89, 6] |
| Following | A robot follows a person. | Generic | Walking Space | Joint Navigation | Servant | Follow human | Lead robot | Robot follows person | Accompany Peer | [6] |
| Leading | A robot leads a person. | Generic | Walking Space | Joint Navigation | Leader | Lead human | Follow robot | Robot leads person | Tour Guide | [6] |
| Crowd Scenarios | ||||||||||
| Crowd Navigation | A robot navigates through a crowd. | Generic | Passable Space | Crowd Navigation | Any | Navigate thru | Mill about | No collision / obstruction | Robot Crowding | Various |
| Parallel Traffic | Crowd moves parallel to the robot. | Generic | Passable Space | Crowd Navigation | Any | Navigate A to B | Mill from A to B | No collision / obstruction | Circular Crossing | [10] |
| Perpendicular Traffic | Crowd moves perpendicular to robot. | Generic | Intersection | Crowd Navigation | Any | Cross Navigate | Mill from A to B | No collision / obstruction | Plaza Crossing | [10] |
| Specialized Scenarios | ||||||||||
| Object Handover | A robot hands an object to a human. | Generic | Passable Space | Interactive Navigation | Servant | Deliver object | Receive object | Human takes object | Robot Courier | [89] |
| Crash Cart | Robot delivering a medical product. | Indoor | Passable Space | Interactive Navigation | Leader | Deliver object | Receive object | Delivery of medicine | Food Delivery | This paper |
VII-B Social Navigation Scenario Cards
A scenario card combines metadata, a definition, and a usage guide. The goal is enough precision to recognize and reproduce an encounter while allowing meaningful behavioral variation. Realism, scalability, and repeatability are useful design goals, but complex crowd encounters may require different levels of detail from a simple passing interaction.
VII-B 1 Scenario Metadata
Metadata identifies the scenario’s category, unique name, brief description, and scientific purpose. Location, pedestrian density, and high-level tasks help readers determine coverage and avoid redundant scenarios. A name such as Frontal Approach should refer to an explicit definition rather than an informal label.
VII-B 2 Scenario Definition
The definition records relevant geometric features, robot numbers and intended tasks, and intended human behavior. It should distinguish essential conditions from permitted variation. For example, head-on approach can still describe an encounter when a pedestrian stops or changes direction; success criteria belong separately in the usage guide.
VII-B 3 Scenario Usage Guide
The usage guide supplies labeling rules, outcome measures, desirable behavior, failure modes, and participant scripts. These are separate from the scenario definition so unsuccessful or unexpected encounters remain recognizable in field data. Context can change the expected behavior and scoring. For repeatable experiments, describe participants’ roles and permitted variations explicitly.
VII-C Example Social Navigation Scenarios
The examples cover low-density pedestrian encounters, dense crowds, and tasks involving interaction. Passing, overtaking, crossing, blind corners, and doorways stress different navigation skills; group joining, handover, and monitoring also constrain the robot’s task. Tables II–III and Figure 7 retain the scenario card and geometric examples. They are a useful coverage set, not an exhaustive taxonomy or a required suite for every application.
VII-D Scenario Guidelines
Guidelines N1–N8 ask researchers to state context, robot tasks, intended human behavior, and success measures; cover commonly studied encounters; permit realistic variants; verify that scenarios elicit the relevant behavior; and communicate them using scenario cards. The key is reproducible meaning with sufficient flexibility, rather than merely reusing a familiar scenario name.
VIII Social Navigation Benchmarks
A benchmark combines scenarios with metrics so policies can be compared. Its meaning still depends on which social-navigation factors it tests. The review separates protocols, executable environments, and public challenges, then argues for human-grounded social measures alongside conventional navigation regression tests.
VIII-A Expanding the Factors for Benchmark Analysis
Benchmark descriptions need implementation and access information beyond the shared experimental taxonomy. The following factors expose whether another group can run a comparable evaluation on its own hardware or simulator.
Simulation Platform
State the simulator and setup procedure. A ready-to-run implementation on an accessible platform lowers the effort needed to reproduce the benchmark.
Associated Dataset
Identify any reference dataset and how it is used for comparison. A benchmark may include data, but a dataset alone is not the executable evaluation defined here.
Provided Baselines
Provide baseline policies and their configurations so new results have interpretable comparison points.
Challenge Leaderboard
A challenge may add a public leaderboard. This supports shared comparison only for the task and measures actually specified by that benchmark.
Downloadability
Downloadable evaluation software supports replication and independent inspection of the procedure.
Most Recent Update
Report maintenance recency and compatibility. The source’s resource descriptions are a 2023 snapshot, so its judgments about available versions should not be read as a current software audit.
Robot Hardware Platform
Document supported robot morphologies and customization. Comparisons are more useful when they can represent the embodiment intended for deployment.
Human Behavior Authoring Methods
State how pedestrians and other agents are generated. Reactive models, authored behavior, and replayed trajectories provide different evidence about human–robot interaction.
| Benchmark | ArenaBench | CrowdBot | DynaBarn | gym-collision-avoidance | HuNavSim | iGibson | Soc-NavBench | Sea-NavBench | Social Navigation Protocol |
| Factors for Analysis | |||||||||
| Benchmark Classification | Benchmark | Challenge | Benchmark | Benchmark | Benchmark | Challenge | Benchmark | Challenge | Protocol |
| Benchmark Context and Scope | Dynamic Obstacle Benchmark | Crowd Simulation Benchmark | Dynamic Obstacle Benchmark | Collision Avoidance Benchmark | Human Simulation Benchmark | Social Navigation Benchmark | Social Navigation Benchmark | Social Navigation Benchmark | Human-Robot Expt. Design |
| Physical Environment | Indoor | Indoor | Synthetic | Synthetic | Indoor | Indoor | Indoor & Outdoor | Indoor & Outdoor | Principally Indoor |
| Intended Human User Type | Synthetic Pedestrian | Synthetic Pedestrian | Varied Human Motion | Synthetic Pedestrian | Varied Resp to Robot | Synthetic Pedestrian | Synthetic Pedestrian | Synthetic Pedestrian | Human Coworkers |
| Supported Robot Tasks | Navigation | Navigation | Navigation | Navigation | Navigation | Navigation | Navigation | Navigation | Navigation |
| Social Scenarios Evaluated | 3 Worlds, 5/10 Peds | Basic Crowd Scenarios | 60 Crowd Scenarios | Multi-agent Scenarios | House, Cafe, Warehouse | 15 House Scenes | 5 Curated Environments | TBD | 6 Social Nav Scenarios |
| Coverage of Corner Cases | Diversity, Random | Not Tested | Diversity, Random | Not Tested | Not Tested | Not Tested | Not Tested | TBD | Not Specified |
| Simulation Platform | Flatland, Gazebo | Unity | Gazebo | Custom | Gazebo | iGibson | SocNavBench | SEAN 2.0 | None |
| Benchmarking Dataset | None | CrowdBot | None | None | None | None | UCY & ETH | UCY & ETH | None |
| Human Behavior Authoring | Pedsim | UMANS | Multiple Algorithms | Baseline Policies | Soc. Force, Behav. Tree | ORCA | Replay, Planned | Replay, Soc Force | Scripted |
| Human Simulation Fidelity | Walking Humans | Walking Humans | Moving Cylinders | Moving Cylinders | Walking Humans | Moving Humans | Walking Humans | Walking Humans | Real Humans |
| Supported Robot Embodiments | Jackal, Burger, Robotino | Pepper, Wheelchair, CuyBot, Qolo | Custom Robots, ClearPath Jackal | Cylinders | ROS Gazebo-Compatible | 8 real, 2 Mujoco | Simulated Mobile Robot | Fetch, Jackal, Turtlebot, Warthog | Human-Scale Robots |
| Communication Modalities | None | None | None | None | None | None | None | None | Human Gestures |
| Challenge Leaderboard | None | None | None | None | None | 2021 | None | 2022 | None |
| Benchmark Last Updated | 2022 | 2021 | 2023 | 2022 | 2023 | 2021 | 2022 | 2022 | 2022 |
| Guidelines for Benchmarks | |||||||||
| B1: Evaluate Social Behavior | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| B2: Quantitative Metrics Provided | Many | Many | Succ. Rate | Succ. Rate, Time2Goal | Many | Succ. Rate, PSC | Many | Many | No |
| B3: Baseline Policies Provided | SOA Nav, RL Policies | No | SOA Nav, RL Policies | SOA Social, Worst-Case | No | SOA RL | SOA Social, Worst-Case | SOA Social, Worst-Case | No |
| B4: Scalable, Repeatable | Simulated, Download | Simulated, Download | Simulated, Download | Simulated, Download | Simulated, Download | Simulated, Download | Simulated, Download | Simulated, Download | Setup Req., Phys. Eval |
| B5: Eval Grounded in Human Data | No | No | Demo. pipeline | No | No | No | No | SEAN-EP extension | Yes |
| B6: Use Validated Instruments | No | No | No | No | No | No | No | No | Validation in Process |
VIII-B Existing Social Navigation Benchmarks
The review requires a system that runs agents, defined scenarios, and evaluation metrics. Protocols specify how to construct studies; benchmark environments add executable tests and often baselines; challenges add a shared forum for results. Reference datasets are optional. These classes should not be conflated when judging accessibility or coverage.
VIII-B 1 Benchmarking Protocols
The Social Navigation Protocol defines controlled interactions for collecting expert demonstrations and evaluating policies with relatively low variability. Successive studies refined the questionnaires through reliability analysis. Its flexibility comes with manual experimental setup: the reviewed protocol does not provide a ready-made simulated environment.
VIII-B 2 Social Navigation Benchmarks
ArenaBench supplies several robot models, classical and learned baselines, and varied motion metrics. DynaBARN stresses dynamic-obstacle models; gym-collision-avoidance evaluates policy-controlled agents; HuNavSim adds diverse pedestrian attitudes; SocNavBench replays real pedestrian trajectories. In the reviewed versions, these scalable environments largely rely on computed proxies rather than ratings from people experiencing the robot. Their quantitative scores therefore cover different constructs and are not interchangeable social-quality rankings.
VIII-B 3 Social Navigation Challenges
CrowdBot targets dense crowds and several robot morphologies. The 2021 iGibson challenge compared navigation and personal-space performance in household simulation; SEANavBench combined scalable simulation with scenario variation and a web infrastructure that could collect human feedback. The article describes the challenges’ status at publication: these are historical descriptions, and human ratings or physical-robot tests were absent from the specific public evaluations discussed.
VIII-C Strengths and Limitations of Existing Benchmarks
The recurring tradeoff is scalability versus grounded evidence about people. Automated suites often omit human ratings; human-centered protocols need more manual setup; few resources cover consequential edge cases well. These limitations motivate combining resources rather than claiming that one existing benchmark fully tests social navigation.
VIII-D Properties of a Good Social Navigation Benchmark
Guidelines B1–B6 call for social-behavior measures, quantitative results on several dimensions, comparison baselines, efficient repeatable trials, human-grounded evaluation, and validated measurement instruments. Include weak baselines and, when feasible, strong demonstrations or policies; neither constitutes a guaranteed mathematical performance bound. The suggested thirty physical trials is a rule of thumb, while sample size should follow the variability and statistical question.
The proposed improvements are more human ratings, shared validated questionnaires, consistent quantitative metrics, and conventional-navigation corner-case tests. Learned perception scores could scale evaluation, but their validity may not transfer across contexts. A common reporting language should expose these limits rather than erase them.
IX Social Navigation Datasets
Dataset evaluation adds sampling coverage, sensor viewpoints, annotations, behavior generation, and privacy to the common taxonomy. The review distinguishes recordings of pedestrians, demonstrations by robots, synthetic scenes with human ratings, and autonomous interaction logs. Each supports different learning and evaluation claims.
IX-A Expanding the Factors for Dataset Analysis
A useful dataset description explains both what was observed and how that distribution was produced. Hardware, collection policy, annotation granularity, and scenario sampling can be as important as raw dataset size.
Robot Hardware Platform
Embodiment and expressive features may change people’s reactions. Multiple robot types can broaden coverage, but the dataset should identify which morphologies actually generated its interactions.
Sensors
Record control commands alongside sensor observations when possible. Otherwise, a motion trace may not reveal the teleoperator’s or policy’s intended action.
Robot Behavior Authoring Methods
Human trajectories can support imitation when human-like behavior is the objective; robot demonstrations are preferable when robot behavior should differ. Teleoperation protocols should explain operator location and visibility, prevent avoidable interference with the interaction, and use diverse operators to reduce dependence on one style.
Data Collected
Match dataset breadth to collection resources. Unconstrained recording captures natural variation, while targeted events improve coverage of selected scenarios. Each declared scenario needs adequate examples; broad labels with sparse coverage do not establish reliable deployment generalization.
Annotations
Specify annotation taxonomy and granularity, from episode labels to temporal segments, trajectories, bounding boxes, pose, or gaze. Existing activity-recognition resources offer examples, but the paper notes the absence of a settled social-interaction taxonomy.
Privacy and Fairness
The authors recommend deciding early how human recordings will be anonymized and governed. The discussion does not itself provide a complete fairness protocol or evidence that anonymization removes all dataset biases.
IX-B Existing Social Navigation Datasets
JRDB, THÖR, and LCAS support perception and trajectory analysis; SCAND supplies teleoperated socially compliant robot demonstrations, whereas MuSoHu records people’s egocentric navigation with a sensor helmet. ETH/UCY, EIFPD, SDD, EFL, WILDTRACK, and related collections vary in viewpoint, agents, and annotation. Pedestrian trajectories can be replayed but do not thereby become reactive robot interactions.
SocNav1/2 provide rated synthetic interactions; DynaBARN supplies synthetic environments; CrowdBot and SG-LSTM include robot-perspective crowd data. SACSoN reports seventy-five hours, fifty-eight kilometers, and over four thousand interactions from autonomous collection with rescue support. Its deliberately interaction-rich collection objective differs from the socially compliant policy later learned from those data. Table V retains the detailed comparison; collection purpose and policy matter when choosing training examples.
| Benchmark Dataset | THOR | SCAND | ETH/UCY | Trajnet++ | EIFPD | SocNav1 | SocNav2 |
| Dataset Context and Scope | Three predefined human-robot social role-play activities. | Socially compliant navigation in-the-wid | Human trajectories recorded from a bird’s eye view vantage point | Human trajectories recorded from a bird’s eye view vantage point | Human trajectories recorded from a bird’s eye view vantage point | Scenarios with interactions labeled with a social score | Short sequences with interactions labeled with social and holistic scores |
| Environment | Curated indoor environment: two rooms with arranged furniture and motion caption system | Indoor and Outdoor campus-scale environment | Outdoor, fixed environment | Indoor and Outdoor, fixed environment | Outdoor, fixed environment | Indoor, abstract | Indoor, abstract |
| Data Collected | 60 minutes of motion tracking across 600 human trajectories | 522 minutes, consisting of 138 trajectories of teleoperation data from 4 demonstrators | Bird’s eye view frames, with annotated human trajectories across time on 5 scenes | 200K human trajectories across dozens type scenes with high density crowds | Bird’s eye view frames, with annotated human trajectories across time | 9280 static scenario descriptions with social scores | 53600 dynamic scenario descriptions with social and all-encompassing scores |
| Scenarios | Visiting areas, carrying boxes, inspecting targets | Goal-oriented social navigation | Pedestrian navigation | Pedestrian navigation | Pedestrian navigation | Evaluating robot disturbance | Evaluating robot trajectories |
| Robot Platform | Linde CitiTruck robot (W 1.56m x L 0.55x x H 1.17m) | Boston Dynamic Spot, ClearPath Jackal | N/A | N/A | N/A | Turtlebot-sized robot | Turtlebot-sized robot |
| Robot Behavior | Programmed to follow a predefined path in a socially unaware manner | Human teleoperation in a socially compliant manner | N/A | N/A | N/A | Static placement | Teleoperation and policy |
| Human behavior | Follow a pre-defined path, and solving tasks in presence of other humans | Demonstrators teleoperate the robot in an open environment with other humans | Open world navigation | Open world navigation | Open world navigation | Static placement | Randomized simulated trajectories |
| Sensors | Stationary Velodyne 3D LiDAR, Qualisys Oqus 7+ motion tracking system, Tobii Pro Glasses for gaze tracking | Velodyne 3D LiDAR, Azure RGB, Odometry, Joystick | Stationary RGB camera overlooking pedestrians | Stationary RGB camera overlooking pedestrians | Camera fixed overhead 23 meters from the floor | Overhead view of robots and humans | Robot and human poses, with 53600 short videos |
| Tasks and Metrics | N/A | N/A | Human trajectory prediction | Human trajectory prediction | Human trajectory prediction | Acceptability of robot disturbance of humans | Acceptability of robot movement around humans |
| Benchmark Dataset | SDD | EFL | LCAS | WILDTRACK | JRDB | Crowd-Bot | Dyna-BARN | SACSoN | SG-LSTM |
| Dataset Context and Scope | Human trajectories recorded from a bird’s eye view vantage point using a drone | Human trajectories recorded from the perspective of a human | Online human detection from 3D lidar scans | Multi-camera detection and tracking of moving humans | Dataset of social interactions in indoor and outdoor environments for solving perceptual tasks | Outdoor pedestrian tracking around a personal mobility robot | Diverse set of moving agent scenarios | Autonomous policy interacting with humans | Curated interaction, movements and formation of pedestrian groups |
| Environment | Outdoor environment, focusing on diverse social navigation scenarios | Outdoor scenes such as parks, malls and a university campus | Outdoor environment | Outdoor environment | Indoor and outdoor environment | Crowded outdoor scenes | Indoor, abstract | Indoor environment | Outdoor university campus environment |
| Data Collected | Bird’s eye view frames with annotations of pedestrians, bikes, cars, etc in 100 different scenes with annotations of social interactions | RGBD frames recorded from an egocentric perspective | 49 minutes of 3D lidar scans | Multi-camera synchronized frames at 10fps | 60,000 annotated frames of humans, recorded from an egocentric robot view | 250K frames / 200 minutes from an egocentric POV | Moving polynomial agent scenarios | 75 hours of visual navigation with 4000 human-robot interactions collected from robot POV | Color and depth frames, pedestrian and group bounding boxes |
| Scenarios | Real-world navigation with social interactions | Egocentric real-world navigation | Real-world navigation in crowded environments | Third-person view open-world pedestrian navigation in crowded environments | Navigation in a campus-scale crowded environment | Real-world navigation in a crowded outdoor environment | Multiple moving agents | Real-world navigation in a crowded indoor environment | Paths, green spaces, study spaces, cafes, gatherings, weather events |
| Robot Platform | 3DR Solo drone | N/A | Pioneer 3-AT mobile robot | N/A | JackRabbot mobile manipulator | Qolo personal mobility robot | N/A | iRobot Roomba | GO1 Edu robot |
| Robot Behavior | N/A | N/A | Human teleoperation | N/A | Teleoperated | Both shared-control and reactive control | N/A | Policy-controlled | Unobtrusive data collection |
| Human Behavior | Performing navigation activities such as walking, driving, and biking in a socially compliant manner | Socially compliant navigation in the open world | Open-world navigation | Open-world navigation | Open-world navigation | Open-world navigation | Navigation among moving obstacles | Open-world navigation | Many types of pedestrian groups |
| Sensors | RGB camera | GoPro Hero 3 stereo cameras with 100mm baseline | Velodyne VLP-16 3D LiDAR | three GoPro Hero 4 and four GoPro Hero 3 | 360 RGB cameras, Velodyne LiDARs, Sick LiDARs, microphone, RGBD camera, fisheye camera, wheel encoders | Point clouds, RGBD, people trackers, pose, contact forces | N/A | Spherical RGBD, fisheye RGB, 2D LIDAR, odometry, bumper | RGB-D cameras |
| Task and Metrics | Social activity recognition, planning and trajectory prediction | Trajectory prediction | Online human detection and tracking | Trajectory prediction, person tracking and re-identification | Benchmark and metrics for 2D and 3D Person detection and tracking | Benchmark for crowd navigation | Dataset of scenarios for benchmarking dynamic navigation | Learning dataset of autonomous policy interacting with humans. | Group size, walking speeds, proximity, cohesiveness, interactions. |
IX-C Guidelines for Datasets
Guidelines D1–D8 balance broad scope with adequate sampling, favor robot data when robot-specific behavior is needed, encourage embodiment diversity, retain generation commands, formalize labels, and address privacy during design. Good coverage reduces some distribution gaps; it cannot guarantee that no out-of-distribution event will occur after deployment.
X Simulation-based Evaluation
A social-navigation simulator must represent interactions between at least two agents. The reviewed systems range from simplified collision avoidance to detailed environments with animated pedestrians. The section compares their fidelity and behavior assumptions, then proposes a common metrics interface. Simulation supports controlled evaluation, while its human models remain an important limitation.
X-A Expanding the Factors for Simulator Analysis
Simulator descriptions need to separate geometric and visual detail, physical constraints, behavior, and interoperability. A realistic-looking scene does not establish realistic social responses.
Abstraction Level
The comparison includes systems capable of individual human–robot encounters, rather than only aggregate crowd flows.
Simulation Focus
State whether the simulator targets crowds, particular human–robot interactions, or dynamic-obstacle avoidance. This focus shapes which algorithmic claims its tests can support.
Simulation Platform
Identify the underlying simulation engine or standalone implementation, since it constrains integration and reuse.
Agent Representation
Specify whether humans and robots are distinct agent classes or share one abstract representation.
Scene Representation
Scenes may use planar geometry, modeled three-dimensional assets, or scans of real places. Describe both representation and intended environment types.
Scene Visual Fidelity
Visual fidelity ranges from planar displays to detailed rendering. It matters for perception policies but is distinct from correct interaction behavior.
Physics Simulation Fidelity
Distinguish kinematic motion from force, mass, and full physics simulation. Different approximations constrain the kinds of failures that can be observed.
Robot Simulation Fidelity
Robots may be points, cylinders, or detailed articulated models. Report embodiment and dynamical limitations when transferring simulation results.
Pedestrian Simulation Fidelity
Pedestrian simulation can include only position, or extend to body shape, gait, and behavioral variability. Each added feature addresses a different source of realism.
Pedestrian Visual Fidelity
Visual agents range from simple discs to animated meshes. The table groups detailed human meshes together because photorealism lacks an objective threshold here.
Pedestrian Reactivity
Recorded pedestrians follow fixed trajectories; reactive models adjust to other agents. Higher-level controllers can add attitudes, styles, and social activities. These choices directly affect what a robot policy learns.
Simulation Interoperability
Describe interfaces such as Gym and ROS that permit integration with other training, control, and evaluation systems.
X-B Existing Social Navigation Simulators
CrowdNav and gym-collision-avoidance use simple agents for collision avoidance; DynaBARN varies obstacle motion; CrowdBot and MengeROS target crowds. HuNavSim, InHuS, and IMHuS add authored pedestrian attitudes or higher-level behaviors. PedSimROS integrates social-force and group behavior models with robot software.
iGibson provides interactive household scenes, SEAN 2.0 detailed social situations, SocialGym 2.0 multi-agent policies with motion constraints, and SocNavBench recorded pedestrian replay. Table VI and Figure 10 retain the systems’ detailed contrasts. These are the paper’s descriptions of contemporary versions, not a current compatibility or quality ranking.
| Simulator | Crowd-Bot | CrowdNav | Dyna-BARN | gym-collision-avoidance | Hu-Nav-Sim | iGibson | InHuS | IMHuS | Menge-ROS | PedSimROS | SEAN 2.0 | SocialGym 2.0 | Soc-Nav-Bench |
| Sim Focus | Crowd Sim | Crowd Sim | Dyn. Obstacles | Collision Avoid. | Human Sim | Social Nav | Human Sim | Human Sim | Crowd Sim | Crowd Sim | Social Nav | Social Nav | Social Nav |
| Sim Platform | Gazebo | Crowd-Nav | Gazebo | gym-collision-avoidance | Gazebo | iGibson | MORSE & Stage | Gazebo | Menge-ROS | Gazebo | SEAN 2.0 | SocialGym 2.0 | Soc-Nav-Bench |
| Agent Repr. | Ped. & Robot | Ped. & Robot | Ped. & Robot | Robots | Ped. & Robot | Ped. & Robot | Ped. & Robot | Ped. & Robot | Ped. & Robot | Ped. & Robot | Ped. & Robot | Ped. & Robot | Ped. & Robot |
| Scene Repr. | Model 3D | Geom. 2D | Geom. 2D | Geom. 2D | Model 3D | Scanned Mesh | Geom. 2D | Geom. 2D | Geom. 2D | Model 3D | Model 3D | Geom. 2D | Scanned Mesh |
| Scene Fidelity | Realist. 3D | 2D | Abstr. 3D | 2D | Abstr. 3D | Abstr. 3D | Realist. 3D | Abstr. 3D | 2D | Abstr. 3D | Realist. 3D | 2D | Realist. 3D |
| Physical Fidelity | Physics Model | Kinematic | Kinematic | Kinematic | Kinematic | Force & Mass | Kinematic | Kinematic | Kinematic | Physics Model | Physics Model | Kino-dynamic | Kinematic |
| Robot Fidelity | Robot Dyn. | Disc | Robot Shape | Disc | Robot Dyn. | Robot Shape | Robot Shape | Robot Shape | Disc | Robot Shape | Robot Dyn. | Kino-dynamic | Robot Shape |
| Ped. Sim Fidelity | UMANS | Move w/o Gait | Poly-nomials | Policy-Based | Gait & Attitude | Move w/o Gait | Gait & Attitude | Gait & Attitude | Move w/o Gait | Gait & Activity | Move w/o Gait | Move w/o Gait | Gait & Pose |
| Ped. Viz Fidelity | Detail’d | Disc | Cylind. | Disc | Detail’d | Detail’d | Detail’d | Detail’d | Disc | Sensors | Detail’d | Polyg. | Detail’d |
| Ped. Reaction | UMANS | ORCA | Policy-Based | Policy-Based | SFM & Attitude | ORCA | React & Attitude | ORCA, Attitude | SFM & ORCA | SFM | SFM | SFM | Replay Only |
| Sim Interop | Unity Interf. | Gym | ROS bag | Gym | ROS Interf. | Gym | ROS Interf. | ROS Interf. | ROS Interf. | ROS Interf. | ROS & Unity | Gym, ROS | ROS Interface |
X-C Analysis of Simulation Platforms
The analysis asks which simulator differences are scientifically meaningful and which could be shared infrastructure. Focus, pedestrian reactivity, agent policies, environment geometry, and metric definitions all constrain cross-system comparisons.
X-C 1 Simulation Focus
An algorithm trained with cylinders and one trained with gait-aware pedestrians may solve different problems. Specialized simulators remain useful, but a common interface could permit shared features and more informative cross-system evaluation.
X-C 2 Common Platforms
Common engines, scene representations, and pedestrian models create opportunities to reuse implementations. Reuse can reduce software fragmentation without making all simulators equivalent.
X-C 3 Pedestrian Reactivity
Replay preserves observed trajectories but loses some body-motion detail and cannot react when a robot deviates from the recorded situation. Reactive models such as social forces and ORCA address that counterfactual response, yet remain imperfect human models. Neither approach alone reproduces human interaction; comparing policies requires disclosing this choice.
X-C 4 Multi-Agent Policies
When other agents’ policies are unknown or learned simultaneously, fixed pedestrian models may be inadequate. Multi-agent learning offers another route, but the review does not show that jointly learned behavior automatically matches people.
X-C 5 Environments
Constrained geometry and dense traffic create many consequential encounters, including corridors, hallways, malls, and airports. Simulator environments should represent the constraints relevant to the intended task rather than relying only on open-space tests.
X-C 6 Metrics
Simulation can reset conditions and expose ground truth for consistent metric calculation. It also supports learned scores and human ratings of simulated episodes. Repeatability strengthens comparisons within those assumptions; it does not establish ecological validity by itself.
X-D Towards a Unified API for Social Navigation Simulation
The proposed interface standardizes inputs, metric computations, and output records across robots, datasets, and simulators. Shared computation would reduce discrepancies caused by separate implementations. This is an interface proposal and preliminary implementation effort, rather than an experimentally demonstrated universal evaluation solution.
X-D 1 Design of the high-level simulator metric API
Inputs include agent poses over time, robot observations and actions, resulting trajectories, and static or dynamic obstacle geometry. Shared libraries should compute step-level and episode-level metrics, while a defined output schema supports analysis, visualization, and learning tools. Extensibility is necessary because different research questions require different observations and metrics.
X-D 2 Implementation of Social Navigation API
The authors describe planned JSON schemas, C++ and Python metric libraries, and example bridges for ROS and GymCollisionAvoidance at SocialNavAPI. Other simulator integrations are reported as ongoing work. The edition preserves that publication-time status and does not assert that every proposed bridge is now implemented.
X-E Guidelines for Simulators
Guidelines S1–S9 encourage standard interfaces and metrics, extensible software, systematic data recording, benchmark construction, human labeling, common robot models, detailed pedestrians, and several behavior-generation methods. Developers should validate and refine each simulator for its intended use. More visual or behavioral features do not remove the need to test fidelity.
XI Conclusions
The paper connects eight behavioral principles to concrete evaluation resources and reporting choices. Its central contribution is a shared framework for asking what a benchmark measures and where its conclusions apply. Human-grounded evidence, contextual validation, diverse metrics, and conventional navigation reliability remain complementary requirements; the proposed API addresses implementation consistency rather than resolving all social constructs.
ACKNOWLEDGMENT
An earlier symposium excerpt focused on the human–robot interaction aspects of the work. The present article expands that discussion into the broader evaluation framework.
Contributions
The contributions statement records symposium participation and responsibilities across definitions, metrics, evaluation, scenarios, datasets, benchmarks, simulation, illustrations, and manuscript editing. Claudia Pérez-D’Arpino and Anthony Francis co-led the paper; the broad author group supplied and synthesized expertise rather than running a single shared experimental benchmark.