At a glance
ARCap collects manipulation demonstrations without a physical robot while showing a retargeted virtual robot in an augmented-reality headset. Visual cues and vibration warn about camera visibility, speed limits, and collisions with a scanned scene. A 20-person study and robot experiments support better demonstration usability than the tested no-feedback system. The hardware results cover two end effectors and selected tasks, rather than guaranteeing that arbitrary human demonstrations will transfer safely to robots.
I Introduction
Portable capture reduces dependence on robot hardware but can record motions that violate robot geometry, speed limits, or contact constraints. ARCap makes some of those failures visible during collection, so users can adjust or discard demonstrations immediately. Its central contribution is feedback about the embodiment gap, not a new policy-learning algorithm.
II Related work
Teleoperation offers direct robot feedback but requires robot hardware during collection. Prior portable systems reduce that requirement while leaving more of the embodiment mismatch to the user and later processing. ARCap adds live retargeting, visibility cues, and collision feedback; its evaluated comparison is with DexCap without those feedback channels.
III Method
The system combines a portable sensing interface, robot-specific retargeting, and diffusion-policy training. It supports different end effectors when an appropriate retargeting procedure and robot model are provided.
III-A ARCap System Design
Feedback addresses three practical mismatches: the robot may move more slowly than a human, occupy more space, or require a different grasp configuration. The virtual robot and warning signals help users anticipate these problems during collection.
Real-time visibility checking
A rectangular overlay shows the depth camera’s narrower field of view within the headset display. Users must keep the manipulated scene inside that frame; the interface warns about visibility but does not autonomously correct the camera viewpoint.
Real-time retargeting
A virtual robot is fixed in the world frame and retargeted to the user’s hand. Joint-speed limits create a visible lag when the user moves too quickly, and the display flashes yellow. This makes invalid motion apparent, but users still need to interpret and respond to the mismatch.
Real-time collision checking
Collision checks compare the virtual robot with a pre-scanned static scene and trigger vibration and a flashing frame. This is not a guarantee for dynamic or incompletely reconstructed obstacles. For the Leap hand, inverse kinematics matches wrist and fingertips with posture regularization; for the parallel gripper, thumb/index position and wrist orientation define a grasp pose and binary opening command. Gripper state changes are limited to once per second. The portable rig combines Quest 3, a mounted D435 camera, motion gloves, controllers, and a backpack laptop. Fixed printed mounts simplify repeat calibration, while every new embodiment still needs suitable retargeting.
III-B Imitation learning
The captured observations and retargeted actions are processed into robot imitation-learning trajectories.
III-B 1 Data processing
Recorded signals include colored point clouds, virtual-robot joint angles, and headset and robot poses. Processing transforms them into a common world frame, removes background points, and superimposes the visible virtual-robot geometry to reduce the human-to-robot observation gap.
III-B 2 Training and testing
A point-cloud encoder supplies visual features to a diffusion policy together with current joint angles. Predicted actions are target arm and hand joint angles or a binary gripper command. At deployment, the camera-to-robot transform is calibrated by aligning virtual and real robot bases in the headset.
IV Experiments
Experiments test demonstration replay and visibility, cluttered manipulation, two different end effectors, and a three-stage assembly task. Replay quality and learned-policy success are separate outcomes.
IV-A Experiment setup
The robot setup uses Franka Panda arms with either a Leap hand or a compliant Fin-ray parallel gripper. The collection interface runs on Quest 3 and a laptop; policy training and evaluation use a workstation with an RTX 3090. During policy execution, the headset camera is fixed rather than worn by an operator.
IV-B User study
Twenty participants collect demonstrations with ARCap and DexCap for cluttered ball pick-and-place and single-block assembly. Half report no prior data-collection or robot-learning experience, and none previously used either system. Each task has three initial states and three collected trajectories per state. The study measures robot replay completion and whether the scene remains visible, alongside user surveys. Reported improvements in these measures do not by themselves establish learned-policy generalization.
IV-C Manipulation in cluttered environment
Table I reports expert-data pick-and-place success of 70% for ARCap versus 25% for DexCap after equal 30-minute collection periods, evaluated over 20 trials. That is a 45-percentage-point difference; the prose’s 35% improvement does not match the table. Policies trained on pooled first-time-user data achieve 60% versus 0% across three designated initial states. No collisions were observed in the reported ARCap-policy tests, which is a finite-sample observation rather than a safety guarantee.
IV-D Long horizon manipulation with different embodiments
With one-hour datasets, isolated assembly-stage success is 70%, 80%, and 85% for ARCap, versus 50%, 15%, and 5% for DexCap. Full three-stage autonomous assembly succeeds in 40% versus 0%. Isolated stages use human resets, so their rates should not be confused with full-task success. The prose’s approximately 51% improvement is not the full-task difference; the original table is retained to make the comparison explicit.
V Conclusions and Future Work
Feedback during portable capture can improve the usefulness of demonstrations for the tested robot embodiments and tasks. Current limits include retargeting quality, static-scene reconstruction, user interpretation, and the restricted hardware and task evaluation. Torso capture for mobile or humanoid robots and proactive language-model coaching are future directions, not demonstrated capabilities.