Literature
Display
← All papers

ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback

Sirui Chen · Chen Wang · Kaden Nguyen · Li Fei-Fei · C. Karen Liu

· ICRA 2025 ↗

Robotics4 min text · CondensedOriginal paper ↗
Contents

At a glance

ARCap collects manipulation demonstrations without a physical robot while showing a retargeted virtual robot in an augmented-reality headset. Visual cues and vibration warn about camera visibility, speed limits, and collisions with a scanned scene. A 20-person study and robot experiments support better demonstration usability than the tested no-feedback system. The hardware results cover two end effectors and selected tasks, rather than guaranteeing that arbitrary human demonstrations will transfer safely to robots.

I Introduction

Portable capture reduces dependence on robot hardware but can record motions that violate robot geometry, speed limits, or contact constraints. ARCap makes some of those failures visible during collection, so users can adjust or discard demonstrations immediately. Its central contribution is feedback about the embodiment gap, not a new policy-learning algorithm.

Refer to caption
Fig. 1: ARCap System Overview. (a) Collect human hand motion data. (b) Provide real-time AR feedback, visualizing a virtual robot retargeted to the human hand in AR display. (c) Rollout robot policies trained with the collected data.

II Related work

Teleoperation offers direct robot feedback but requires robot hardware during collection. Prior portable systems reduce that requirement while leaving more of the embodiment mismatch to the user and later processing. ARCap adds live retargeting, visibility cues, and collision feedback; its evaluated comparison is with DexCap without those feedback channels.

III Method

The system combines a portable sensing interface, robot-specific retargeting, and diffusion-policy training. It supports different end effectors when an appropriate retargeting procedure and robot model are provided.

III-A ARCap System Design

Feedback addresses three practical mismatches: the robot may move more slowly than a human, occupy more space, or require a different grasp configuration. The virtual robot and warning signals help users anticipate these problems during collection.

Real-time visibility checking

A rectangular overlay shows the depth camera’s narrower field of view within the headset display. Users must keep the manipulated scene inside that frame; the interface warns about visibility but does not autonomously correct the camera viewpoint.

Real-time retargeting

A virtual robot is fixed in the world frame and retargeted to the user’s hand. Joint-speed limits create a visible lag when the user moves too quickly, and the display flashes yellow. This makes invalid motion apparent, but users still need to interpret and respond to the mismatch.

Refer to caption
Fig. 2: Visualization of AR Feedback. (a) Normal data recording: the red frame indicates visible region of the RGB-D camera. (b) Collision warning: when the virtual robot collides with the environment, the controller on the human gloves vibrates, and the frame blinks blue. (c) Fast motion warning: when the user moves faster than the robot’s speed limits, the frame turns yellow. (d) Users can check if target objects are within camera’s view during data collection.

Real-time collision checking

Collision checks compare the virtual robot with a pre-scanned static scene and trigger vibration and a flashing frame. This is not a guarantee for dynamic or incompletely reconstructed obstacles. For the Leap hand, inverse kinematics matches wrist and fingertips with posture regularization; for the parallel gripper, thumb/index position and wrist orientation define a grasp pose and binary opening command. Gripper state changes are limited to once per second. The portable rig combines Quest 3, a mounted D435 camera, motion gloves, controllers, and a backpack laptop. Fixed printed mounts simplify repeat calibration, while every new embodiment still needs suitable retargeting.

Refer to caption
Fig. 3: Cross-Embodiment Data Collection. (a) ARCap can collect data for parallel-jaw grippers by guiding the user to form their hands into a gripper-like shape. If the user changes the hand gesture, the retargeting error will be large. (b) For a multi-finger dexterous hand, ARCap retargets the robot’s fingertips to match the human fingertips, with the robot’s wrist orientation determined by the orientation of the controller mounted on the user’s gloves.
Refer to caption
Fig. 4: ARCap System Layout. The user wears an AR headset and motion capture gloves, with controllers mounted on the gloves for tracking the 6D pose of the palms. Data is stored on a laptop carried in the backpack.

III-B Imitation learning

The captured observations and retargeted actions are processed into robot imitation-learning trajectories.

III-B 1 Data processing

Recorded signals include colored point clouds, virtual-robot joint angles, and headset and robot poses. Processing transforms them into a common world frame, removes background points, and superimposes the visible virtual-robot geometry to reduce the human-to-robot observation gap.

III-B 2 Training and testing

A point-cloud encoder supplies visual features to a diffusion policy together with current joint angles. Predicted actions are target arm and hand joint angles or a binary gripper command. At deployment, the camera-to-robot transform is calibrated by aligning virtual and real robot bases in the headset.

Refer to caption
Fig. 5: AR-based Camera Calibration. When calibrating the camera, users align the virtual robot’s base with the actual robot’s base.
Refer to caption
Fig. 6: Experiment Tasks. (a) Pick-and-place in a cluttered scene with a dexterous robot hand. (b) Single-stage Lego assembly with a parallel-jaw gripper. (c) Long-horizon assembly of three Lego blocks.

IV Experiments

Experiments test demonstration replay and visibility, cluttered manipulation, two different end effectors, and a three-stage assembly task. Replay quality and learned-policy success are separate outcomes.

Refer to caption
Fig. 7: Setups for Real Robot Evaluation. During evaluation, we mounted the headset on a tripod and connected its camera to the robot workstation. The trained policy takes point cloud observations from the camera attached to the headset and outputs actions to control the robots.
Refer to caption
Fig. 8: Results of User Study. (a) We invited 20 users with different backgrounds to collect data with DexCap [38] and ARCap. (b)-(e) Survey results of user experience. (f)-(g) On-robot evaluation results with collected data.

IV-A Experiment setup

The robot setup uses Franka Panda arms with either a Leap hand or a compliant Fin-ray parallel gripper. The collection interface runs on Quest 3 and a laptop; policy training and evaluation use a workstation with an RTX 3090. During policy execution, the headset camera is fixed rather than worn by an operator.

IV-B User study

Twenty participants collect demonstrations with ARCap and DexCap for cluttered ball pick-and-place and single-block assembly. Half report no prior data-collection or robot-learning experience, and none previously used either system. Each task has three initial states and three collected trajectories per state. The study measures robot replay completion and whether the scene remains visible, alongside user surveys. Reported improvements in these measures do not by themselves establish learned-policy generalization.

IV-C Manipulation in cluttered environment

Table I reports expert-data pick-and-place success of 70% for ARCap versus 25% for DexCap after equal 30-minute collection periods, evaluated over 20 trials. That is a 45-percentage-point difference; the prose’s 35% improvement does not match the table. Policies trained on pooled first-time-user data achieve 60% versus 0% across three designated initial states. No collisions were observed in the reported ARCap-policy tests, which is a finite-sample observation rather than a safety guarantee.

Cluttered pick and place Expert User
DexCap [38] 0.25 0
ARCap 0.7 0.6
Long-horizon lego assembly Stage 1 Stage 2 Stage 3 Full
DexCap [38] 0.5 0.15 0.05 0.0
ARCap 0.7 0.8 0.85 0.4
TABLE I: Success Rate of Autonomous Policy

IV-D Long horizon manipulation with different embodiments

With one-hour datasets, isolated assembly-stage success is 70%, 80%, and 85% for ARCap, versus 50%, 15%, and 5% for DexCap. Full three-stage autonomous assembly succeeds in 40% versus 0%. Isolated stages use human resets, so their rates should not be confused with full-task success. The prose’s approximately 51% improvement is not the full-task difference; the original table is retained to make the comparison explicit.

V Conclusions and Future Work

Feedback during portable capture can improve the usefulness of demonstrations for the tested robot embodiments and tasks. Current limits include retargeting quality, static-scene reconstruction, user interpretation, and the restricted hardware and task evaluation. Torso capture for mobile or humanoid robots and proactive language-model coaching are future directions, not demonstrated capabilities.

Concept demonstration

Open image ↗