Overview
This edition follows the September 2024 survey version. Its domain comparisons, deployment judgments, and open problems describe the literature available then. The supplementary material supplies precise taxonomy definitions and detailed paper-by-paper classifications; the original publication placeholders are not treated as bibliographic facts.
At a glance
The survey asks where deep reinforcement learning has actually worked on physical robots and what enabled those results. It compares competencies, problem formulations, learning methods, and a proposed deployment-maturity scale. Strong demonstrations exist, particularly in quadruped control, but generalization, safe and efficient learning, long-horizon composition, and rigorous evaluation remain uneven across domains. The article synthesizes evidence rather than presenting a new algorithm or controlled cross-domain experiment.
1 Introduction
Physical interaction is costly, imperfectly simulated, and sometimes difficult to recover from, making robotic reinforcement learning different from games. The reviewed successes range from locomotion controllers to racing and manipulation, with sharply different deployment conditions. The survey aims to identify transferable techniques while distinguishing a laboratory demonstration from sustained operational use. Its claims about companies and deployed capabilities are publication-time reports, not a current market survey.
2 Why Another Survey on RL for Robotics?
Unlike a general algorithm survey, this review prioritizes papers with at least some physical-robot evidence and compares multiple competencies. A common taxonomy connects application, formulation, solution, and reported maturity. This selection highlights demonstrated successes and their limitations; it is not a complete census of simulation research or an unbiased estimate of how often reinforcement-learning projects succeed. Related comparisons include lessons from physical robot learning and a manipulation-focused review.
3 Taxonomy
Four axes organize the evidence: the competency learned, the control problem supplied to learning, the algorithm and supporting techniques, and the demonstrated level of physical-world use. Separating these axes prevents attributing an entire robotic system’s success to the learned component when substantial perception, planning, or control is supplied elsewhere.
3.1 Robot Competencies Learned with DRL
Single-robot competencies include locomotion, navigation, stationary manipulation, and mobile manipulation. Multi-agent competencies cover interaction with humans or other robots. Classification follows what reinforcement learning actually controls: an interaction policy may rely on pre-existing motor skills, and a navigation system may jointly learn movement or assume a low-level controller. The categories overlap in system requirements but isolate the learned capability for comparison.
3.2 Problem Formulation
The formulation records action abstraction, observation representation, and reward density. Actions range from motor commands to task-space controls or extended skills; observations range from estimated states to rich sensor inputs. Partial observability and decentralized multi-agent settings introduce memory and coordination requirements. These engineering choices can simplify learning substantially, so algorithm names alone do not explain performance.
3.3 Solution Approach
The solution taxonomy covers simulator use and adaptation, learned dynamics, expert data, policy optimization, and neural representation. Zero-shot transfer, real-world fine-tuning, and direct real-world learning involve different data and modeling assumptions. The appendix defines these labels so methods can be compared at the level of their whole training and deployment pipeline.
3.4 Level of Real-World Success
The authors propose six maturity levels: simulation only; limited laboratory testing; diverse laboratory testing; confined operational testing; diverse representative operational testing; and commercial deployment. Only Levels 1–5 count as physical-world success in this review. Ratings are subjective judgments from reported experiments, sometimes a single trial, and concern each method’s target problem rather than task difficulty. They are not formal certification, independent replication, or proof of reliability.
4 Competency-Specific Review
Each competency review connects representative applications to the techniques that made them feasible, then identifies unresolved problems. Overview and supplementary tables retain individual classifications. Differences between domains reflect hardware, dynamics, data, and task scope as well as algorithm choice.
4.1 Locomotion
Locomotion concerns motor control given higher-level motion goals. The review separates quadrupeds, freely moving three-dimensional bipeds, and quadrotor flight because their dynamics and failure recovery differ. The strongest evidence is for specialized control skills rather than unrestricted navigation or complete task autonomy.
Caption and linked table references
| Quadruped | 28 , 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54 |
| Biped | 27 , 55, 56, 57, 58, 59, 60, 61, 62, 63 |
| Flight | 64 , 65, 66, 67, 68 |
4.1.1 Quadruped Locomotion
Quadruped control is the most mature reviewed locomotion domain. Common recipes combine simulation, actuator modeling, dynamics randomization, and carefully chosen interfaces to lower-level controllers. Challenging terrain adds privileged teacher information, observation history or state estimation, curricula, and exteroceptive sensing. Visual representation learning and sensor-noise modeling help transfer perception-based control.
Specialized policies, hierarchical selection, gait conditioning, motion imitation, or unified rewards can produce multiple skills. Precise footholds may still benefit from trajectory optimization. Mature examples mainly use zero-shot transfer and on-policy optimization; direct physical learning and adaptation have narrower validation. Reported commercial integrations do not imply that every jumping, recovery, or parkour skill has equivalent operational maturity.
4.1.2 Biped Locomotion
Biped control needs balance under underactuated dynamics, making recovery and repeated real-world trials harder. Reviewed systems use reference motions or model-based control, periodic rewards, and recurrent or history-based policies. Most transfer from simulation without physical fine-tuning. The survey distinguishes humanoids from leg-only bipeds and excludes boom-constrained planar systems; industrial demonstrations without technical evidence are not assumed to use reinforcement learning.
4.1.3 Quadrotor Flight Control
Quadrotor control benefits from modeled dynamics, randomization, suitable thrust/body-rate actions, and adaptation to disturbances. Some learned controllers are more robust than conventional feedback baselines near hover, yet carefully designed optimization-based controllers can retain lower tracking error. The eighteen-second training example concerns a specific waypoint controller. Joint optimization of planning and control is a stronger motivation for racing than a universal claim that reinforcement learning dominates tracking control.
4.1.4 Trends and Open Challenges in Locomotion
Across locomotion, transferable techniques include adaptation, privileged learning, and randomized simulation. Quadrupeds benefit from more forgiving dynamics and accessible standardized hardware; bipeds and drones face more costly failure modes. Efficient real-world learning, recovery, reliable long-horizon behavior, and integrating motor skills with downstream tasks remain open even in comparatively mature domains.
4.2 Navigation
Navigation chooses routes and collision-avoidance behavior, often relying on an effective locomotion controller. The review separates wheeled, legged, and aerial systems, with social and multi-robot navigation treated later. Learned policies may replace the whole navigation pipeline or only exploration, planning, or tracking components.
Caption and linked table references
| Wheeled | 73 , 74, 75, 76, 78, 81, 82, 85, 88, 89, 90, 91, 92, 93 |
| Legged | 20 , 83, 86, 87, 94, 95, 96, 97, 98, 99, 100 |
| Aerial | 7 , 21, 101, 102, 103 |
4.2.1 Wheeled Navigation
For wheeled robots, early laser-to-control policies did not consistently outperform classical stacks. Stronger physical results often replace a particular planner or exploration module. Visual navigation adds semantic interpretation, exploration, and memory; excellent simulator scores do not establish robust transfer, and both end-to-end and modular methods face visual and dynamical domain gaps.
Off-road work uses learned models, demonstrations, or learned visual control to handle terrain-dependent dynamics. Full-size autonomous-driving evidence is more limited and constrained. The cited hundred-vehicle cruise-control field test wraps learned commands in manually specified limits; it demonstrates a useful subsystem and traffic effect, not unrestricted autonomous driving.
4.2.2 Legged Navigation
Abstracting leg dynamics behind a locomotion controller enables fast kinematic simulation and easier navigation learning, but limits exploitation of agile skills. Joint navigation–locomotion methods can traverse gaps, obstacles, and confined spaces; hierarchical alternatives reuse learned motor policies. A kilometer-scale wheeled-legged demonstration shows operational promise, while long horizons, sparse rewards, and low-level dynamics still make training costly. Classical waypoints and feasible-path guidance remain useful supporting structure.
4.2.3 Aerial Navigation
Aerial navigation must manage fragile hardware, sensing constraints, and tight control margins. Domain randomization and real-data dynamics models support transfer from simulation. Champion-level racing combines residual perception/dynamics models with direct optimization of a long-horizon racing objective. Its tested operational envelope remains narrower than human robustness, and hybrid reinforcement learning with predictive control is discussed as a possible improvement.
4.2.4 Trends and Challenges in Navigation
The strongest physical navigation evidence often comes from modular systems, even when end-to-end policies perform well in simulation. Choosing which components to learn remains open. Joint navigation and locomotion enable agility but complicate stable long-horizon learning. Generalization, explainability, and safety evidence remain especially demanding for outdoor drones and urban vehicles; using real data to improve simulation helps without automatically providing guarantees.
4.3 Manipulation
Manipulation adds contact mechanics, object variation, costly exploration, reward design, and difficult resets. The review separates pick-and-place, contact-rich interaction, in-hand motion, and non-prehensile control. The real-world evidence is strongest where tasks and object families can be constrained in advance, while open-world manipulation requires broader generalization and skill composition.
Caption and linked table references
| Pick-and-place | Grasping | 108 , 109, 110, 111, 112 |
| End-to-end Pick-and-place | 54 , 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125 | |
| Contact-rich | Assembly | 126 , 127, 128, 129, 130 |
| Articulated Objects | 122 , 131, 132, 133 | |
| Deformable Objects | 134 , 135, 136, 137 | |
| In-hand | — | 138 , 139, 140, 141, 142, 143 144 |
| Non-prehensile | — | 109 , 118, 145, 146, 147 |
4.3.1 Pick-and-place
Industrial pick-and-place often combines learned perception or grasp selection with analytical motion planning. General end-to-end rearrangement of arbitrary objects remains harder. The review distinguishes the grasping subproblem from learning the complete pick-and-place sequence, since successes at one do not establish the other.
4.3.1.1 Grasping
Open-loop grasp selection can be treated as a bandit or classification problem and trained with large self-supervised datasets of grasp outcomes. Such constrained actions simplify control and have supported operational applications. Closed-loop finger or end-effector control is a richer sequential problem with less mature deployment evidence. Both physical data and simulated training contribute; visual transfer remains a central consideration.
4.3.1.2 End-to-end Pick-and-place
End-to-end pick-and-place must cover many objects, placements, and task objectives. Hierarchies, motion primitives, demonstrations, and pretrained models supply structure that can shorten learning horizons. Reviewed milestones also address multitask learning, reward acquisition, resets, and sample efficiency. Foundation models and larger datasets are promising directions in this 2024 account, rather than evidence that general open-world manipulation is already solved.
4.3.2 Contact-rich Manipulation
Assembly, articulated objects, and deformable materials demand explicit attention to contact and dynamics. The review asks whether learning improves robustness or reduces engineering within these tasks, while tracking how much object knowledge, grasp setup, and task structure is supplied.
4.3.2.1 Assembly
Assembly policies use simulation, demonstrations, direct physical learning, or residual corrections to a reference trajectory. Some comparisons find better perturbation robustness than integrator-designed solutions, but broad industrial deployment is not established. Many studies begin with an already grasped part; complete detection-to-grasp-to-insertion pipelines are therefore a more demanding and less common demonstration.
4.3.2.2 Articulated Objects
Articulated-object tasks such as opening drawers have mostly served as constrained demonstrations or algorithm testbeds. The review finds limited evidence of deployment-ready general manipulation of such objects.
4.3.2.3 Deformable Objects
Cloth handling and assistive dressing expose inaccurate soft-material models. Simulation transfer and simplified actions such as picking or flinging make selected tasks tractable. The demonstrated successes typically constrain task families and materials; open-world contact-rich manipulation still combines contact-control difficulty with object and goal generalization.
4.3.3 In-hand Manipulation
In-hand policies have progressed from rotating known objects to broader object and axis variation, using vision, tactile sensing, or proprioception. Simulation randomization, privileged teachers, and dense rewards are common. Reorientation alone is not the same as achieving a downstream tool-use or assembly configuration; integrating dexterity with other manipulation stages remains a major gap.
4.3.4 Non-prehensile Manipulation
Pushing and other non-grasping actions are useful when grasps are unavailable or insufficient. Privileged geometry and teacher–student transfer support reorienting varied objects. Combining these skills with grasping, in-hand control, and deliberate environmental contact is an unresolved step toward general manipulation.
4.3.5 Trends and Open Challenges in Manipulation
Constrained grasping, assembly, and dexterous tasks benefit from enumerable training conditions and designed rewards. Broader manipulation needs better simulation, autonomous data collection, adaptation, reward learning, and structured priors. Most demonstrations remain short horizon and use task-specific action abstractions; composing them into one capable agent is a separate challenge from improving any single skill.
4.4 Mobile Manipulation
Mobile manipulation requires coordinated base, arm, and sometimes leg control under partial observability. The review separates whole-body tracking, short interactive skills, and long-horizon tasks. Simply combining a navigation policy with an arm controller does not resolve coupling, perception, or task planning.
Caption and linked table references
| WBC | 154 ,155,156,157 |
| Short-Horizon | 160 ,161,162,163,164,165,166,167,168,169,170,171 |
| Long-Horizon | 159 ,172,173 |
4.4.1 Learning Whole-Body Control
Whole-body policies often map proprioception and a desired link pose to coordinated motor commands. Learning can help with hard-to-model contacts or limited online computation, while analytical planners may still control part of the body. Higher-level perception and goal selection are needed to turn accurate pose tracking into interaction with surrounding objects; hierarchical methods begin to connect these roles.
4.4.2 Short-horizon Interactive Tasks
Grasping, kicking, tracking, door opening, and interactive navigation demonstrate short sensorimotor skills. Hierarchical control, teacher–student learning, behavior trees, and randomized simulation reduce training difficulty. These tasks add rich observations and uncertain contact dynamics, but successful short episodes do not by themselves establish memory or planning over extended household activities.
4.4.3 Long-horizon Interactive Tasks
Long tasks compound exploration difficulty with incomplete observations and the need for memory or explicit mapping. Demonstrations, scripted policies, and simulation can bootstrap learning; planning or hierarchical policies can chain shorter skills. The surveyed long-horizon evidence remains sparse, and reliable integration is necessary before individual skills become useful household assistance.
4.4.4 Trends and Open Challenges in Mobile Manipulation
Large workspaces make safe physical exploration costly, encouraging simulation, demonstrations, and behavioral priors. Long tasks need memory of a scene that changes through manipulation, not only a navigation map. Morphologies range from wheeled arms to humanoids and legs used as manipulators. Joint commands, task-space actions, learned low-level controllers, and factored actions each impose different assumptions; the survey finds no universal action-space choice.
4.5 Human-Robot Interaction
Human–robot interaction here means interacting with people during execution, rather than merely using human demonstrations in training. The review separates collaborative physical tasks, non-collaborative sharing of space, and shared autonomy. Human behavior is difficult to simulate and expensive to sample, limiting the diversity and maturity of demonstrations.
Caption and linked table references
| Collaborative pHRI | 175 , 174, 176, 182 |
| Non-collaborative pHRI | 177 , 178, 179, 180, 181 |
| Shared Autonomy | 183 , 184, 185 |
4.5.1 Collaborative pHRI
Collaborative examples include packaging and human-to-robot handover. Recurrent learning and behavior trees support task coordination, while handover policies can train against human-motion models built from precollected data. Such models reduce the need for continuous human participation, but their realism and coverage constrain what simulation transfer establishes about actual collaboration.
4.5.2 Non-collaborative pHRI
Social-navigation policies use simulated crowds, designed rewards, recurrent observation histories, or human-motion models; residual learning can improve a pretrained policy through physical interactions. These methods mainly handle people as other actors in a shared space, so simple models can suffice for selected demonstrations without establishing richer social understanding. Human-aware manipulation also uses action transformations to impose constraints. Performance depends on the particular crowd, perception, and constraint assumptions.
4.5.3 Shared Autonomy
Shared-autonomy policies combine a person’s input with task rewards or constraints, often minimizing corrections. A demonstrated drone-perching method chooses a valuable action close to the user’s command; other residual-control examples are simulation studies and should not all be counted as physical deployments. Offline language-conditioned tabletop learning offers another interface. The common issue is helping a user succeed while respecting their intended control.
4.5.4 Trends and Open Challenges in HRI
Human interaction makes experience collection costly and introduces history dependence, uncertain intentions, and departures from ideal rationality. Existing approaches use simple tasks, pretrained policies, recorded or synthesized human motion, and sometimes hardcoded human models. More realistic human models and safe, sample-efficient physical learning remain complementary needs; success against one model is not broad validation of interaction with people.
4.6 Multi-Robot Interaction
Multiple robots introduce decentralized information and coupled decisions. General interaction can be modeled as a partially observable stochastic game; fully cooperative cases admit decentralized partially observable decision processes. The surveyed physical demonstrations focus on collision avoidance, cooperative manipulation, and soccer, usually with small teams or tightly specified settings.
Caption and linked table references
| Multi-Robot Collision Avoidance | 186 , 187, 189, 190, 191 |
| Multi-Robot Loco-Manipulation | 192 |
| Robot Soccer | 193 |
4.6.1 Multi-Agent Collision Avoidance
Collision avoidance has used state-vector policies initialized by reciprocal-avoidance solutions, as well as sensor-based policies trained with curricula and dense rewards. Hybrid controllers can address gaps in learned behavior. A factory-like multi-robot demonstration combines imitation, environment sampling, and penalties for blocking others. These examples establish feasibility under their sensing and interaction assumptions, while realistic scale and unpredictable surrounding agents remain difficult.
4.6.2 Multi-Agent Loco-Manipulation
Hierarchical simulation-to-real learning lets multiple quadrupeds coordinate locomotion to manipulate an object. The reviewed example has two robots push a heavy block toward a target pose. This demonstrates a useful coupled skill but does not establish coordination across large fleets, heterogeneous bodies, or many simultaneous manipulation objectives.
4.6.3 Robot Soccer
Many robot-soccer studies train individual locomotion or ball-control skills before placing robots together, rather than learning coordination. The reviewed biped soccer work uses self-play in a one-on-one setting to develop walking, turning, kicking, recovery, and opponent-aware interception. Its competitive setting also means the subsequent broad claim that all physical multi-robot learning is cooperative should not be read literally.
4.6.4 Trends and Open Challenges in Multi-Robot Interaction
Progress requires scaling team size, tolerating physical noise and moving obstacles, integrating existing control stacks, and generalizing beyond training configurations. Much of the reviewed evidence concerns homogeneous, cooperative teams, with the preceding competitive soccer example an exception. Communication and network effects, stability, heterogeneous embodiments, and mixed cooperative or adversarial behavior remain underexplored in the physical evidence.
5 General Trends and Open Challenges
The most mature reported systems often use zero-shot simulation transfer, manageable dynamics, dense rewards, and stable on-policy optimization. Constrained manipulation also benefits from enumerable conditions, while open-world tasks, human interaction, and larger teams strain simulation and data collection. These are patterns in a selected literature review, not causal evidence that one recipe dominates all tasks.
Improving Stability and Sample-Efficiency in RL Algorithms.
On-policy methods are comparatively dependable in many reviewed control systems but require substantial experience, even in simulation. Off-policy and offline methods reuse data, yet distribution shift and unstable value estimates can hinder deployment and online fine-tuning. Hybrid model-based and model-free learning may improve efficiency. Complex long-horizon tasks and multi-agent scaling still expose both stability and sample-cost limitations.
Real-World Learning.
Physical learning needs safe experience, recoverable failures, resets, and useful rewards. Many settings depend on human interventions, scripts, or demonstrations, although the review also includes autonomous collection systems. Robust hardware and better reuse of offline experience can help. Adaptation may update perception or residual dynamics rather than the entire policy; deciding which modules to freeze or learn is an important system choice.
Learning for Long-Horizon Robotic Tasks.
Most reviewed successes are short skills. Longer tasks require discovering useful behaviors, selecting them, and ensuring their composition remains reliable. Manually defined skills and rewards simplify the problem but limit flexibility; goal-conditioned or unsupervised learning, hierarchical control, planning, and end-to-end learning offer different compromises. A collection of successful skills is not yet a dependable long-horizon agent.
Designing Principled Approaches for RL Systems.
Reward design, action abstraction, and the division between analytical and learned components strongly affect results. Expert structure can make learning tractable but can also constrain performance. The review calls for principled, controlled comparisons of these choices, including shaping rewards and hybrid architectures, rather than treating an algorithm label as a complete method.
Benchmarking Real-World Success.
The proposed maturity ratings rely on authors’ reported experiments, with varied environments, metrics, and trial counts. Shared hardware is useful but insufficient for reproducible evaluation. Representative tasks, repeatable environments, and scalable measures of open-world robustness are needed before deployment claims become reliably comparable.
Leveraging Foundation Models.
Large robot datasets and pretrained models may supply representations, language-conditioned behavior, or help with reward and simulation design. DrEureka illustrates language-model assistance with simulation-to-real training. This 2024 discussion is a research outlook: it does not establish that foundation models already deliver general physical competence or remove the need for careful task evaluation.
6 Conclusion
Deep reinforcement learning has enabled important physical control and interaction capabilities, but their maturity varies with task scope and supporting engineering. Continued progress needs stable and efficient algorithms, better physical-learning procedures, skill composition, and reproducible system-level evaluation. The survey’s taxonomy makes those differences visible without turning laboratory success into a general deployment guarantee.
Acknowledgements
The authors acknowledge colleagues’ feedback, research support, and permission to reproduce robot-football images. Peter Stone discloses a compensated chief-scientist role at Sony AI, reviewed and approved under his university’s outside-activity policy. These disclosures are retained as publication context.
Appendix A Term Definition
The terminology appendix defines the labels used throughout the survey and comparison tables. They describe problem formulations and training pipelines rather than proposing a new learning algorithm. A paper can occupy multiple categories when its system combines components.
A.1 Problem Formulation
The formulation taxonomy records action abstraction, observation dimensionality, and reward density. These choices change the exploration difficulty, control requirements, and information supplied to learning. The labels should therefore be considered alongside performance and maturity claims.
A.1.1 Action Space
Low-level actions directly control joints or motors, preserving fine control but increasing exploration difficulty, action dimensionality, and effective horizon. Mid-level actions specify task-space motion and rely on an external controller, trading implementation knowledge for easier learning. High-level actions invoke extended skills, shortening exploration horizons while requiring a useful skill set whose design and transfer may themselves be difficult.
A.1.2 Observation Space
Low-dimensional observations provide compact state, proprioception, object positions, or task variables. High-dimensional observations include images, laser scans, and point clouds. This distinction describes the policy’s input rather than guaranteeing what information is observable or how difficult perception is.
A.1.3 Reward Function
Sparse rewards provide little feedback outside successes or failures. Dense rewards supply frequent feedback, either naturally through continuous tracking objectives or through added shaping terms. Shaping can accelerate exploration in long tasks, but the reward function is part of the supplied problem design and need not express only the final objective.
A.2 Solution Approach
Solution labels describe simulator use, learned dynamics, external expertise, optimization, and neural representation. The tables allow combined labels because a complete robot-learning pipeline can mix planning with learned policies, multiple observation encoders, and several action abstractions.
A.2.1 Simulator Usage
Zero-shot transfer trains in simulation and deploys without further physical learning. Few-shot transfer adds limited physical fine-tuning after simulation. The no-simulator category learns from real-world experience or data. These labels distinguish training assumptions rather than measuring robustness or total engineering effort.
A.2.2 Model Learning
Model-free methods learn policies or values without an explicit learned dynamics model. Model-based methods learn predictions for planning or imagined experience. The survey also counts learned residual or actuator dynamics as model learning, even when the final policy uses model-free optimization; the categories therefore describe the whole system rather than mutually exclusive algorithm families.
A.2.3 Expert Usage
External expertise includes human demonstrations, trajectory planners, oracle actions, and other offline or online guidance. Such assistance can speed learning but supplies information unavailable to a method starting from random behavior. The tables record its use so comparisons can account for that advantage.
A.2.4 Policy Optimization
Planning uses a learned model to choose actions through an online control or search problem. Offline optimization learns from fixed data without new interactions. On-policy learning updates from current-policy experience; off-policy learning can also reuse earlier or other-policy transitions. These definitions distinguish how data enter optimization, not whether a system has a broader simulation or expert-data stage.
A.2.5 Policy/Model Representation
Fully connected networks commonly process compact vectors; convolutional networks exploit local structure in images or other ordered sensor data. Recurrent networks maintain state across time, while transformers use attention over tokens and can fuse modalities. The source calls recurrent networks bidirectional, but bidirectionality is an optional architecture choice, not a defining property or guaranteed access to future observations in a deployed controller.
Appendix B Additional Tables
Six original tables classify every reviewed competency by formulation and solution approach. Colors encode the authors’ reported maturity judgments; they are not independent reliability measurements. Asterisks indicate methods that jointly use elements from multiple columns, such as a hierarchical action space or combined convolutional and recurrent networks. The full tables remain available for checking specific methods and assumptions.
Caption and linked table references
| Action Space | |||
| Application | Low-Level | Mid-Level | High-Level |
| Locomotion | 27 , 28, 29, 30, 31∗, 32∗, 33, 36∗, 37, 38, 40, 41, 42, 43, 44, 45, 46, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 61, 62, 63, 64, 65, 68 | 31 ∗, 32∗, 34, 35, 47, 66, 67 | 36 ∗, 60 |
| Navigation | 20 , 90, 96∗, 97∗, 99, 100, | 7 , 21, 73, 74, 75, 78, 83, 85, 88, 89, 91, 92, 93, 94, 95∗, 98, 101, 102, 103 | 76 , 81, 82, 86, 87, 95∗, 96∗, 97∗ |
| Manipulation | 113 , 122, 127, 131, 138, 139, 140, 141, 142, 143, 144 | 54 , 110, 111, 112, 114, 115, 116, 117, 118, 119, 120, 121, 124, 125, 126, 128, 129, 130, 132, 133, 134, 137, 145, 146, 147 | 108 , 109, 123, 135, 136 |
| MoMa | 155 , 156, 157, 165, 169, 171, 160, 164, 161, 168 | 154 , 166, 162, 163, 173, 159, 172, 167 | 170 |
| HRI | 177 , 178, 179, 180, 181, 182 | 175 , 176, 183, 184, 185 | 174 |
| Multi-Robot Interaction | 186 , 187, 189, 190, 192, 193 | 191 | |
Caption and linked table references
Caption and linked table references
| Simulator Usage | |||
| Application | Zero-shot Sim-to-Real | Few-shot Sim-to-Real | No Simulator |
| Locomotion | 27 , 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 55, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68 | 43 , 48, 56 | 53 , 54 |
| Navigation | 20 , 21, 73, 74, 75, 76, 78, 81, 82, 83, 85, 86, 87, 93, 94, 95, 96, 97, 98, 99, 100, 101, 103 | 7 , 102 | 88 , 90, 91, 92 |
| Manipulation | 108 , 111, 123, 130, 133, 134, 135, 137, 138, 139, 141, 142, 143, 144, 145, 146, 147 | 116 , 131 | 54 , 109, 110, 112, 113, 114, 115, 117, 118, 119, 120, 121, 122, 124, 125, 126, 127, 128, 129, 132, 136, 140 |
| MoMa | 155 , 156, 154, 157, 165, 169, 171, 164, 166, 163, 167, 161, 168, 173, 159 | 160 , 172 | 162 , 170 |
| HRI | 174 , 175, 176, 177, 178, 179, 181 | 184 | 180 , 183, 182 |
| Multi-Robot Interaction | 186 , 187, 189, 190, 191, 192, 193 | ||
Caption and linked table references
| Model Learning | Expert Usage | |||
| Application | with Model Learning | No Model Learning | No Expert | with Expert |
| Locomotion | 29 , 31, 34, 35, 36, 41, 51, 52, 54, 56 | 27 , 28, 30, 32, 33, 37, 38, 40, 42, 43, 44, 45, 46, 47, 48, 49, 50, 53, 55, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68 | 28 , 29, 31, 32, 33, 34, 35, 37, 38, 41, 42, 43, 44, 45, 47, 49, 50, 52, 53, 54, 56, 57, 59, 60, 61, 62, 65, 66, 67, 68 | 27 , 30, 36, 40, 46, 48, 51, 55, 58, 63, 64 |
| Navigation | 7 , 20, 74*, 88, 90, 95, 96, 97, 98, 102 | 21 , 73, 74∗ 75, 76, 78, 81, 82, 83, 85, 86, 87, 89, 91, 92, 93, 94, 99, 100, 101, 103 | 7 , 20, 21, 73, 74, 75, 78, 81, 82, 83, 85, 86, 87, 88, 93, 94, 95, 96, 98, 99, 100, 101, 102, 103 | 76 , 89, 90, 91, 92, 97 |
| Manipulation | 54 , 113, 118, 140, 144 | 108 , 109, 110, 111, 112, 114, 115, 116, 117, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 141, 142, 143, 145, 146, 147 | 54 , 108, 109, 110, 111, 114, 115, 116, 118, 119, 123, 125, 130, 131, 133, 135, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147 | 112 , 113, 117, 120, 121, 122, 124, 126, 127, 128, 129, 132, 134, 136 |
| MoMa | 155 , 156, 154, 157, 165, 169, 171, 160, 164, 166, 162, 163, 170, 167, 161, 168, 173, 159, 172 | 155 , 156, 154, 165, 169, 171, 160, 164, 166, 162, 163, 167, 161, 168, 173 | 157 , 170, 159, 172 | |
| HRI | 175 , 176, 183 | 174 , 177, 178, 179, 180, 181, 182, 184, 185 | 175 , 176, 179, 181, 182, 184, 185 | 174 , 177, 178, 180, 183 |
| Multi-Robot Interaction | 186 , 187, 189, 190, 191, 192, 193 | 189 , 190, 192 | 186 , 187, 191, 193 | |
Caption and linked table references
| Policy Optimization | ||||
| Application | Planning | Offline | Off-Policy | On-Policy |
| Locomotion | 32 , 36∗, 48, 53, 54, 68 | 27 , 28, 29, 30, 31, 32, 33, 34, 35, 36∗, 37, 38, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 52, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67 | ||
| Navigation | 74 ∗, 76∗, 88, 90, 102, 103∗ | 76 ∗, 89, 91∗, | 73 , 74∗, 75, 78, 91∗, 92, 94, 101 | 7 , 20, 21, 81, 82, 83, 85, 86, 87, 93, 95, 96, 97, 98, 99, 100, 103∗ |
| Manipulation | 118 , 140 | 108 , 115, 121, 122, 124, 129 | 54 , 109, 110, 111, 112, 114, 116, 117, 119, 120, 123, 125, 126, 127, 128, 132, 133, 134, 135, 136, 137, 143, 145, 146 | 113 , 130, 131, 138, 139, 141, 142, 144, 147 |
| MoMa | 160 ∗, 166, 162, 163, 173, 172 | 155 , 156, 154, 157, 165, 169, 171, 160∗, 164, 170, 167, 161, 168, 159 | ||
| HRI | 183 | 180 ∗ | 174 , 175, 176, 180∗, 181, 182, 184 | 177 , 178, 179, 185 |
| Multi-Robot Interaction | 193 | 186 , 187, 189, 190, 191, 192 | ||
Caption and linked table references
| Policy/Model Representation | ||||
| Application | MLP Only | CNN | RNN | Transformer |
| Locomotion | 28 , 29, 30, 31, 32, 38, 40, 41, 46, 47, 48, 51, 52, 53, 56, 58, 60, 64, 65, 66, 68 | 27 , 33, 34, 36, 42, 43, 44∗, 45, 49∗, 54∗, 63, 67 | 35 , 37, 44∗, 49, 50, 54∗, 55, 57, 59, 61 | 62 |
| Navigation | 7 , 20, 21, 73, 74∗, 75, 76, 90, 93, 99, 100, 103§ | 78 , 81, 82, 83∗, 85, 86∗, 87∗, 89, 91, 92, 94, 96, 97, 98∗, 101, 102∗ | 74 ∗, 83∗, 85, 86∗, 87∗, 95, 98∗, 102∗ | 74 ∗, |
| Manipulation | 123 , 129, 131, 132, 138, 139, 140, 143, 144∗, 145 | 54 ∗, 108, 109, 110, 111, 112, 113, 114, 115, 117, 118∗, 119, 120, 121, 122, 125, 126, 127, 128, 133, 134, 135, 136, 137, 142, 146 | 54 ∗, 118∗, 130, 144∗ | 116 , 124, 141, 147 |
| MoMa | 155 , 156, 154, 157, 171, 160, 164, 163, 168 | 165 , 169∗, 166, 162, 170, 167∗, 161, 173, 159, 172∗ | 169 ∗, 167∗, 172∗ | |
| HRI | 177 , 181, 182, 184, 185 | 175 , 176, 179, 180, 183∗ | 174 , 178 | 183 ∗ |
| Multi-Robot Interaction | 186 , 189, 192, 193 | 187 , 190, 191 | ||