Literature
Display
← All papers

BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities

Yunfan Jiang · Ruohan Zhang · Josiah Wong · Chen Wang · Yanjie Ze · Hang Yin · Cem Gokmen · Shuran Song · Jiajun Wu · Li Fei-Fei

Preprint posted · CoRL 2025 (accepted) ↗

Robotics15 min text · CondensedOriginal paper ↗
Contents

Overview

Figure 1 shows the five evaluated household activities and the body capabilities they emphasize: bimanual coordination, navigation, and reachability.

Original PDF illustration S0.F1
Figure 1: Everyday household activities enabled by BEHAVIOR Robot Suite (BRS), showcasing its three core capabilities: bimanual coordination (B), stable and accurate navigation (N), and extensive end-effector reachability (R).

At a glance

BEHAVIOR Robot Suite combines the Galaxea R1 mobile dual-arm robot, the low-cost JoyLo teleoperation interface, and WB-VIMA, a policy that predicts base, torso, and arm actions in sequence. Demonstrations cover five household activities lasting roughly one to three and a half minutes. The paper reports 88% average sub-task success and about 58% average full-task success, with 93% on the best task. These are distinct metrics: failed sub-tasks can be reset before evaluating later sub-tasks. The work demonstrates coordinated whole-body imitation in the tested environments, while broad scene transfer and general household autonomy remain open.

1 Introduction

The BEHAVIOR-1K benchmark [8] motivates three requirements: using both arms together, navigating precisely while manipulating, and reaching objects from floor to overhead heights. Figure 2 shows multiple modes in object height, rather than one tabletop workspace. Hardware that expands these capabilities also enlarges the action space and complicates demonstration collection. BRS addresses both with JoyLo and a hierarchical action model. Table 1 retains the authors’ comparison with other systems; its capability entries refer to demonstrated learned policies, and its dollar symbols describe interface cost, not the full robot.

  BRS (Ours)   Mobile ALOHA [13]   TidyBot++ [14]   ACE [15]   BiDex [16]   HomeRobot [10]   TTT [17]   TeleMoMa [18]   RoboCopilot [19]   Open-TeleVision [20]   TRILL [21]   GR00T N1 [22]   ALOHA [23]   GELLO [24]   HATO [25]   FACTR [26]
Mobile manipulation Humanoid manipulation Stationary manipulation
Simultaneous control of
arms, torso, and mobile base
✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Interface Bimanual control ✓ ✓ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Torso control ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Single-operator mobile base control ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗
Untethered mobile base control ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗
Haptic feedback ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓
Cost(a) $ $ $ $ $ $ $ $ $ N.A. N.A. $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $
Capabilities(b) Omnidirectional navigation ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Bimanual coordination ✓ ✓ ✗ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓
Ground-level reach ✓ ✗ ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Comfortable overhead reach(c) ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Operation in confined spaces ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Coordinated whole-body manipulation
involving hip, waist, and mobile base
✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
Learning-based method ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ N.A. ✓ ✓
Algorithm Novel algorithm ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ N.A. ✗ ✓
Autoregressive whole-
body action prediction
✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ N.A. ✗ ✗
Sensory observation modality Colored
point cloud
RGB RGB RGB RGB Depth +
semantic seg.
RGB-D RGB-D RGB RGB RGB RGB RGB N.A. RGB-D +
tactile
RGB
Policy model backbone(d) XF XF UNet UNet XF RNN N.A. MLP/RNN UNet XF RNN XF XF N.A. UNet XF
   Open-source everything(e) ✓ ✓ ✓ Hardware
+ teleop.
Teleop. ✓ ✗ Teleop. ✗ ✓ ✓ Weights
+ finetuning
✓ ✓ ✓ ✓
  • (a)

    Interface hardware cost. $: $0–$500; $$: $500–$1000; $$$: $1000+.

  • (b)

    We consider robot capabilities that are demonstrated by learned autonomous policies.

  • (c)

    Following Panero and Zelnik [27], we use 182.9 cm182.9\text{\,}\mathrm{cm} (72 in72\text{\,}\mathrm{i}\mathrm{n}) as the maximum height for comfortable overhead reach.

  • (d)

    Neural network architecture of the policy backbone. “XF” stands for Transformer.

  • (e)

    Everything includes the interface hardware, teleoperation code, algorithm code, and documentation.

Table 1: Comparison of recent real-robot frameworks. BRS is comprehensive, integrating a unique whole-body control interface JoyLo and a novel algorithm WB-VIMA for learning whole-body visuomotor policies, demonstrating several unprecedented robotic capabilities.
Original PDF illustration S1.F2
Figure 2: Ecological distributions of task-relevant objects in daily household activities. Multiple distinct modes appear in the vertical distance distribution, located at 0.09 m0.09\text{\,}\mathrm{m}, 0.49 m0.49\text{\,}\mathrm{m}, 0.94 m0.94\text{\,}\mathrm{m}, and 1.43 m1.43\text{\,}\mathrm{m}, representing heights at which objects are typically found.
Original PDF illustration S1.F3
Figure 3: BRS hardware system. Left: The R1 robot with two 6-DoF arms and a 4-DoF torso mounted on an omnidirectional mobile base. Right: The JoyLo system, consisting of compact, off-the-shelf Nintendo Joy-Con controllers mounted at the ends of two kinematic-twin arms. Joy-Con serves as the interface for controlling the grippers, torso, and mobile base.

2 JoyLo: Joy-Con on Low-Cost Kinematic-Twin Arms

JoyLo pairs two kinematic leader arms with Nintendo Joy-Con controls for the torso, grippers, and mobile base. Its concrete implementation targets the R1 robot; adaptation to another embodiment is a design possibility rather than an evaluated transfer result.

Efficient Whole-Body Control

The operator physically moves miniature arms to command the corresponding robot joints. The left thumbstick controls base velocity, the right controls waist and hip, arrow keys adjust height, and triggers operate the grippers. This permits one operator to coordinate the entire body. Kinematic matching constrains arm commands to the robot’s joint structure, but does not by itself guarantee collision avoidance or safe interaction with the environment.

Rich User Feedback

Bilateral feedback regularizes leader-arm motion using robot joint positions and velocities, providing resistance when motion lags or contact occurs without additional force sensors. The source gives τ=𝐊𝐩​(𝐪robot−𝐪JoyLo)+𝐊𝐝​(𝐪˙robot−𝐪˙JoyLo)−𝐊\tau=\mathbf{K_{p}}\left(\mathbf{q}_{\text{robot}}-\mathbf{q}_{\text{JoyLo}}\right)+\mathbf{K_{d}}(\dot{\mathbf{q}}_{\text{robot}}-\dot{\mathbf{q}}_{\text{JoyLo}})-\mathbf{K}. Its final K term is described as damping but is not expanded into a complete velocity-dependent implementation law. This feedback mechanism should be distinguished from a calibrated measurement of environmental contact force.

Low Cost and Easy Accessibility

The interface uses printed links, low-cost Dynamixel motors, and Joy-Cons. Its bill of materials totals approximately $499.50 at the reported prices. That estimate excludes the R1 robot, cameras, workstation, printer, and assembly labor. Replaceable components and a Python controller support practical maintenance and data collection.

3 WB-VIMA: Whole-Body VIsuoMotor Attention Policy

WB-VIMA separates observation encoding from action generation. A transformer combines recent visual and proprioceptive observations; three diffusion heads then decode body components in kinematic order.

Autoregressive Whole-Body Action Decoding

Small base or torso errors can displace the hands substantially; the paper illustrates that a 0.17 rad knee motion can shift an end-effector by about 0.14 m. WB-VIMA first predicts three base-velocity channels, conditions four torso-joint channels on that prediction, then conditions twelve arm joints and two gripper channels on both. Equation 1 preserves the three denoising updates for a 21-channel action trajectory. Autoregression is across body components, with trajectory chunks predicted by each diffusion head. It supplies learned coordination, not an analytic guarantee that downstream joints cancel every upstream error.

Original PDF illustration S3.F4
Figure 4: WB-VIMA architecture. It autoregressively decodes whole-body actions by leveraging the hierarchical interdependencies within the embodiment space, and dynamically aggregates multi-modal observations using self-attention.
Equation 1
𝐚basek−1∼𝒩⁡(μk​(𝐚basek,ϵbase​(𝐚basek|𝐄a,k)),σk2​I),𝐚torsok−1∼𝒩⁡(μk​(𝐚torsok,ϵtorso​(𝐚torsok|𝐚base0,𝐄a,k)),σk2​I),𝐚armsk−1∼𝒩⁡(μk​(𝐚armsk,ϵarms​(𝐚armsk|𝐚torso0,𝐚base0,𝐄a,k)),σk2​I).\begin{split}\mathbf{a}^{k-1}_{\text{base}}&\sim\mathcal{N}\left(\mu_{k}\left(\mathbf{a}^{k}_{\text{base}},\epsilon_{\text{base}}\left(\mathbf{a}^{k}_{\text{base}}|\mathbf{E}^{a},k\right)\right),\sigma_{k}^{2}I\right),\\ \mathbf{a}^{k-1}_{\text{torso}}&\sim\mathcal{N}\left(\mu_{k}\left(\mathbf{a}^{k}_{\text{torso}},\epsilon_{\text{torso}}\left(\mathbf{a}^{k}_{\text{torso}}|\mathbf{a}^{0}_{\text{base}},\mathbf{E}^{a},k\right)\right),\sigma_{k}^{2}I\right),\\ \mathbf{a}^{k-1}_{\text{arms}}&\sim\mathcal{N}\left(\mu_{k}\left(\mathbf{a}^{k}_{\text{arms}},\epsilon_{\text{arms}}\left(\mathbf{a}^{k}_{\text{arms}}|\mathbf{a}^{0}_{\text{torso}},\mathbf{a}^{0}_{\text{base}},\mathbf{E}^{a},k\right)\right),\sigma_{k}^{2}I\right).\end{split} (1)

Multi-Modal Observation Attention

A PointNet [68] encodes colored point clouds and an MLP encodes joint/base state. For each observed time step, visual, proprioceptive, and action-readout tokens enter a causal attention sequence: the sequence S of point-cloud, proprioceptive, and action tokens over the observation window (displayed below). The last readout token conditions the action heads. The intended benefit is to use scene evidence alongside robot state rather than reproduce a sequence based mostly on proprioception.

Equation𝐒=[𝐄t−To+1pcd,𝐄t−To+1prop,𝐄t−To+1a,…,𝐄tpcd,𝐄tprop,𝐄ta]∈ℝ3​To×E\mathbf{S}=[\mathbf{E}^{\text{pcd}}_{t-T_{o}+1},\mathbf{E}^{\text{prop}}_{t-T_{o}+1},\mathbf{E}^{\text{a}}_{t-T_{o}+1},\ldots,\mathbf{E}^{\text{pcd}}_{t},\mathbf{E}^{\text{prop}}_{t},\mathbf{E}^{\text{a}}_{t}]\in\mathbb{R}^{3T_{o}\times E}

Training and Deployment

Training minimizes the noise-prediction loss ℒ=MSE(ϵk,ϵθ(⋅|k))\mathcal{L}=MSE(\epsilon^{k},\epsilon_{\theta}(\cdot|k)) for each diffusion head and sums the three losses, following diffusion-policy methods [65]. The observation backbone runs once per inference call. Deployment reports 0.02 s effective latency on RTX 4090 hardware; recorded demonstrations and new policy actions run at 10 Hz, while the low-level controller repeats commands at 100 Hz. Asynchronous execution and discarded stale chunk actions help achieve that effective latency.

4 Experiments

Experiments evaluate task success, component ablations, teleoperation usability/data quality, and the contribution of base/torso motion. The physical task study and the simulator user study are separate evaluations.

Experiment Settings

There are 100 shelf, 103 toilet, 98 clothing, 138 post-party, and 122 trash demonstrations, totaling 561. Tasks randomize at least two factors among starting pose, object placement, instances, and distractors. Policies generally receive 15 runs per task. Sub-task evaluation resets after failures before continuing, whereas entire-task success requires completing the full sequence. Appendix Table A.XIII reports only eight RGB-DP clothing trials, an exception to the general fifteen-run description.

DP3, RGB diffusion policy, and ACT [23] are compared with WB-VIMA and human teleoperation. Recorded safety violations mean collisions or motor power loss under excessive force. Human rows have different trial counts from policies, so isolated human-versus-policy percentages are not matched-sample evidence of general superiority.

BRS enables various household activities, on which WB-VIMA consistently outperforms baseline methods ( 𝒬 ​ 𝟏 ).

The appendix gives full-task successes of 6/15 for post-party cleanup, 8/15 for toilet cleaning, 8/15 for trash, 14/15 for shelves, and 8/15 for clothing: 44/75 overall, or 58.7%. DP3 completes three shelf trials and no other full tasks; RGB-DP completes two shelf trials, and ACT none. WB-VIMA’s stronger sub-task results do not eliminate compounding failures. Its 93% peak belongs to the two-stage shelf task, not every household activity.

WB-VIMA records one safety violation in trash disposal and zero in the other four tasks. This is an empirical count over limited trials, not proof that point-cloud features inherently enforce safety. Baselines differ in color input, observation fusion, and action structure, so their overall gap cannot isolate one architectural choice. The paper’s headline 13×/21× full-task multipliers are not reproduced directly by the appendix’s reported macro-averages; the explicit task counts are retained rather than treating those multipliers as exact.

Original PDF illustration S4.F5
Figure 5: Evaluation results for five household tasks. Left: Initial randomization. Middle: Success rates over 15 runs (“ET” = entire task, “ST” = sub-task). Right: Number of safety violations.
Original figure
# Safety
Violations
Ours No W.B.
Action Decoding
No Multi-
Modal Obs. Attn.
Put Items
onto Shelves
0 0 0
Lay Clothes Out 0 0 4
Figure 6: Real-world ablation results for “put items onto shelves” and “lay clothes out.”

Synergistic whole-body action prediction and multi-modal feature extraction are key to WB-VIMA’s strong performance ( 𝒬 ​ 𝟐 ).

Removing hierarchical action decoding reduces shelf full-task success from 14/15 to 6/15 and clothing from 8/15 to 2/15. Removing multimodal attention reduces them to 2/15 and 0/15, with four clothing collisions. The shelf difference from 93% to 40% is about 53 percentage points. Figure 6 and the appendix retain all sub-task values and failure counts.

The simulated wiping study averages five runs of 100 rollouts and reports standard-deviation bars. Adding attention and hierarchical decoding progressively improves success. Its inputs and motion constraints differ from physical deployment: it uses a target position instead of camera observations and locks the base and two torso joints. This is a controlled coordination ablation, not another full visual mobile-manipulation benchmark.

Original PDF illustration S4.F7
Figure 7: Simulation ablation results for “wiping table.” The robot must wipe toward the goal using whole-body motions while maintaining continuous hand contact. Results are averaged over five runs with 100 rollouts each; error bars indicate standard deviation.

JoyLo is an efficient, user-friendly interface that provides high-quality data for policy learning ( 𝒬 ​ 𝟑 ).

Ten participants compare JoyLo, VR controllers, and Apple Vision Pro in an OmniGibson post-party task. Interface order is randomized and each receives ten minutes of practice. JoyLo yields the highest task success, shortest reported times, and strongest open-loop replay results; the paper reports five times VR’s task success and 23% shorter median completion time. None finishes the entire task with Apple Vision Pro. All participants prefer JoyLo afterward, despite 70% initially favoring an IK-based option.

These results concern the implemented control mappings and a small simulator study. Replay success measures execution of recorded trajectories, not independently measured policy-learning gains. “Singularity” is measured using Jacobian conditioning for IK interfaces but excessive joint velocity for JoyLo, so the compared ratios use different operational definitions. Meta Quest 3 provides depth perception for both JoyLo and VR conditions.

Original figure
Figure 8: User study results. “S.R.” is success rate. “ET Comp. Time” and “ST Comp. Time” refer to entire and sub-task completion times.

Coordinated torso and mobile base movements enhance maneuverability beyond stationary arms ( 𝒬 ​ 𝟒 ).

Figure 9 shows the robot moving its hip/base to open a door and backing its base to pull a dishwasher. Locking those components prevents opening and increases arm effort. These examples support the utility of whole-body motion for articulated objects. Additional recovery videos are qualitative examples rather than a quantified disturbance-robustness study.

5 Related Work

Related work covers household benchmarks and robotic systems, inexpensive robots and teleoperation devices, and planning/learning for whole-body manipulation. BRS combines leader-arm control, independent torso/base input, bilateral feedback, and a learned action hierarchy. Its comparison with Mobile ALOHA [13], DP3 [70], and earlier multimodal policies motivates those choices. The contribution is an integrated implementation and evaluated policy design, not a claim that prior systems lack all whole-body coordination.

Original PDF illustration S5.F9
Figure 9: Coordinated torso and mobile base movements enhance maneuverability. WB-VIMA policies use the hip and mobile base to open a door and dishwasher; if the torso or mobile base is locked, opening fails and arm joint effort surges, risking hardware damage.

6 Conclusion

The combination of suitable hardware, coordinated demonstrations, visual/state attention, and structured action generation improves the tested household tasks. The full-task success rate and limitations remain central to judging readiness: successful sub-tasks and individual demonstrations do not establish reliable general home assistance.

7 Limitations

The four limitations below concern partial observability, compounding errors, perception quality, and the scope of the training distribution. Figure 10 preserves the detailed trash-task failure flows.

Original figure
Figure 10: Failure modes in the “take trash outside” task. Left: Failure analysis during data collection using JoyLo. Right: Failure analysis during autonomous WB-VIMA policy rollouts. “S” indicates number of successful trials. “F” indicates number of failed trials.

Mismatched camera field of view between robot and operator.

Operators view the robot from outside while policies receive onboard cameras. The operator must keep targets visible to those cameras and avoid appearing in their view; otherwise demonstrations may rely on information unavailable to the policy. Active perception or a shared robot-view interface is proposed as future work.

Compounding errors in long-horizon, multi-stage tasks.

Errors accumulate across long sequences even when individual sub-task success is high. Human correction data or model-based task planning may help, but are not evaluated remedies here. Figure 10’s human trash flow reports 85 successes out of 127, whereas Table A.XI reports 96/127; these source results disagree and should not be merged into a single resolved success estimate. The policy flow’s eight successes out of fifteen agrees with the table.

Imperfect point cloud observations.

Stereo point clouds degrade with lighting changes and reflective surfaces. Daytime training need not generalize to night. Improving stereo with FoundationStereo [154] is suggested, not tested in the reported system.

Robot-specific training data.

Training data comes only from the R1 embodiment and the available task scenes. Cross-robot transfer, broad scene-level generalization, pretrained vision-language-action models, and synthetic/human data are future directions. The paper does not establish zero-shot transfer to unfamiliar homes or embodiments.

Acknowledgments

The authors acknowledge technical collaborators, Galaxea hardware support, study participants, and Stanford facilities, with support from Stanford HAI, Schmidt Futures, NSF, and ONR programs.

Appendix A Robot Hardware Details

The hardware appendix records the concrete platform, measured capabilities, sensor setup, and workstation communication required to reproduce this implementation.

A.1 Robot Platform

R1 has two six-joint arms with parallel grippers, a four-joint torso, and an omnidirectional wheeled base. The authors’ tests report 5 kg payload per arm, ground-to-2 m vertical reach, and up to 2.06 m horizontal reach with the torso. Base limits include 1.5 m/s translation, 3 rad/s yaw, and 30 mm ground clearance. These are reported platform capabilities, not the velocities or loads used in every demonstration.

A.2 Hardware Specifications

Figure A.1 retains all arm, torso, and base dimension drawings and labels. Tables A.I–A.III retain joint, mobility, and camera specifications.

Refer to caption
(a)
Original PDF illustration A1.F1.sf2
(b)
Original PDF illustration A1.F1.sf3
(c)
Figure A.1: Robot diagrams. (a): Each arm has six DoFs and a parallel jaw gripper. (b): The torso features four revolute joints for waist rotation, hip bending, and knee-like motions. (c): The wheeled, omnidirectional mobile base is equipped with three steering motors and three wheel motors.

A.2.1 Arms

Each arm has 923 mm full reach and 128 mm width. Joint impedance gains are Kp = [140, 200, 120, 20, 20, 20] and Kd = [10, 50, 5, 1, 1, 0.4]. Grippers span 0–100 mm with a reported 100 N rated force; target width is converted into motor current.

A.2.2 Torso

The torso combines waist yaw, hip pitch, and two knee-like joints. Fully extended dimensions are 340 mm wide and 1,223 mm high excluding the head. Table A.I retains each angular range and the 108 N·m rated/304 N·m maximum motor torque specifications.

Table A.I: Torso motor specifications.
Parameter Value
Waist Joint Range (Yaw) ±\pm 3.05 rad3.05\text{\,}\mathrm{rad} (175 °175\text{\,}\mathrm{\SIUnitSymbolDegree})
Hip Joint Range (Pitch) −2.09 rad-2.09\text{\,}\mathrm{rad} (−120 °-120\text{\,}\mathrm{\SIUnitSymbolDegree}) ∼\sim 1.83 rad1.83\text{\,}\mathrm{rad} (105 °105\text{\,}\mathrm{\SIUnitSymbolDegree})
Knee Joint 1 Range −2.79 rad-2.79\text{\,}\mathrm{rad} (−160 °-160\text{\,}\mathrm{\SIUnitSymbolDegree}) ∼\sim 2.53 rad2.53\text{\,}\mathrm{rad} (145 °145\text{\,}\mathrm{\SIUnitSymbolDegree})
Knee Joint 2 Range −1.13 rad-1.13\text{\,}\mathrm{rad} (−65 °-65\text{\,}\mathrm{\SIUnitSymbolDegree}) ∼\sim 1.83 rad1.83\text{\,}\mathrm{rad} (105 °105\text{\,}\mathrm{\SIUnitSymbolDegree})
Rated Motor Torque 108 N m108\text{\,}\mathrm{N}\text{\,}\mathrm{m}
Maximum Motor Torque 304 N m304\text{\,}\mathrm{N}\text{\,}\mathrm{m}

A.2.3 Mobile Base

Three wheel motors and three steering motors provide planar x/y velocity and yaw control. Table A.II retains distinct acceleration limits: 2.5 m/s² forward, 1.0 m/s² lateral, and 1.0 rad/s² yaw. The broad prose claim of 2.5 m/s² horizontal acceleration should not be applied to every direction.

Table A.II: Mobile base specifications.
Parameter Value
Forward Velocity Limit ±\pm 1.5 m s−11.5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}
Lateral Velocity Limit ±\pm 1.5 m s−11.5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}
Yaw Rotation Velocity Limit ±\pm 3 rad s−13\text{\,}\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}
Forward Acceleration Limit ±\pm 2.5 m s−22.5\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-2}
Lateral Acceleration Limit ±\pm 1.0 m s−21.0\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-2}
Yaw Rotation Acceleration Limit ±\pm 1.0 rad s−21.0\text{\,}\mathrm{rad}\text{\,}{\mathrm{s}}^{-2}

A.3 Onboard Sensors and Computing

A head ZED 2 and two wrist ZED-Mini cameras stream RGB-D data at 60 Hz; a rear RealSense T265 supplies odometry at 200 Hz. Forward kinematics updates camera poses at 500 Hz. The source fuses camera point clouds as 𝐏ego-centric=⋃c​a​m​e​r​aall cameras𝐏c​a​m​e​r​a​(𝐑c​a​m​e​r​a)⊺+(𝐭c​a​m​e​r​a)⊺\mathbf{P}^{\text{ego-centric}}=\bigcup_{camera}^{\text{all cameras}}\mathbf{P}^{camera}\left(\mathbf{R}^{camera}\right)^{\intercal}+\left(\mathbf{t}^{camera}\right)^{\intercal}, then crops spatially and uses farthest-point sampling. Although the prose names a base-to-camera transform, the displayed row-vector operation must be implemented with the appropriate camera-to-base convention. A Jetson Orin handles onboard sensing and processing.

Table A.III: Configurations for the ZED RGB-D cameras and RealSense T265 tracking camera.
Parameter Value
RGB-D Cameras
Frequency 60 Hz60\text{\,}\mathrm{Hz}
Image Resolution 1344×\times376
ZED Depth Mode PERFORMANCE
Head Camera Min Depth 0.2
Head Camera Max Depth 3
Wrist Camera Min Depth 0.1
Wrist Camera Max Depth 1
Tracking Camera
Odometry Frequency 200 Hz200\text{\,}\mathrm{Hz}
Refer to caption
Figure A.2: Visualization of the fused, ego-centric colored point clouds. Left: The colored point cloud observation, aligned with the robot’s coordinate frame. Right: The robot’s orientation and its surrounding environment.

A.4 Communication Scheme

ROS camera/state nodes communicate with a workstation master over a local network. The workstation subscribes to observations, runs policy/control logic, and sends commands through ROS topics; onboard sensing does not imply onboard policy inference.

Appendix B JoyLo Details

JoyLo’s implementation details include printable hardware, communication rates, impedance control, and recorded data fields.

B.1 Hardware Components

Two arms take a reported thirteen printer-hours and 317 g of PLA. Table A.IV retains every component, quantity, supplier, and price. Sixteen Dynamixel motors account for $382.40, Joy-Cons $70, and the USB converter $32.10. Motor states are read at roughly 400–500 Hz and Joy-Cons communicate over Bluetooth at 66 Hz.

Refer to caption
Figure A.3: Individual JoyLo links.
Table A.IV: JoyLo bill of materials.
Item No. Part Name Description Quantity Unit Price ($) Total Price ($) Supplier
1 Dynamixel XL330-M288-T JoyLo arm joint motors 16 23.90 382.40 Dynamixel
2 Nintendo Joy-Con JoyLo hand-held controllers 1 70 70 Nintendo
3 Dynamixel U2D2 USB communication converter for controlling Dynamixel motors 1 32.10 32.10 Dynamixel
4 5V DC Power Supply Power supply for Dynamixel motors 1 <10 <10 Various
5 3D Printer PLA Filament PLA filament for 3D printing JoyLo arm links 1 ∼\sim5 ∼\sim5 Various
Total Cost: ∼\sim$499.5

B.2 Controller Implementation

The Python interface sends torso/arm joint targets, gripper widths, and base velocities at 100 Hz. Bilateral leader-arm control uses current-based impedance with proportional gains 0.5 and derivative gains 0.01 for each of six joints. Coupled motors increase torque at load-bearing leader-arm joints. Pseudocode 1 is preserved exactly; its right-gripper command repeats the left-gripper target variable, which should be checked before reuse rather than silently treated as a correct independent command.

B.3 Data Collection

Data is recorded at 10 Hz while control runs at 100 Hz. Joy-Con buttons start, pause, save, or discard a recording. Saved observations include color, depth, point clouds, joints, odometry, and commands. The full controller listing below retains original indentation and all source text.

from brs_ctrl.robot_interface import R1Interface

# instantiate the controller
robot = R1Interface(...)
# send a control command
robot.control(
    # the torso and arms commands are target joint positions
    arm_cmd={
        "left": left_arm_target_q,
        "right": right_arm_target_q,
    },
    gripper_cmd={
        "left": left_gripper_target_width,
        "right": left_gripper_target_width,
    },
    torso_cmd=torso_target_q,
    # the mobile base commands are target velocities
    base_cmd=mobile_base_target_velocity,
)
Pseudocode 1: Python interface for the R1 robot controller.

Appendix C Model Architectures, Policy Training, and Deployment Details

This appendix supplies the observation/action architecture, baseline differences, optimizer schedules, and asynchronous deployment details.

C.1 Preliminaries

The formulation describes a state-to-action policy and trains it by imitation through diffusion noise prediction.

Problem Formulation

The paper introduces an MDP with states, actions, transition dynamics, initial-state distribution, and reward. The implemented policy training is behavior cloning of demonstrations; an MDP notation or reward symbol does not imply reinforcement-learning optimization in these experiments.

Denoising Diffusion for Policy Learning

A DDPM progressively corrupts data with Gaussian noise and learns a reverse process conditioned on observations. Equation A.1 preserves the reverse Gaussian update and scheduled variance. Each action head predicts the noise needed to denoise an action trajectory, following prior diffusion policies.

Equation A.1
xk−1∼𝒩⁡(μk​(xk,ϵθ​(xk,k)),σk2​I),x^{k-1}\sim\mathcal{N}\left(\mu_{k}\left(x^{k},\epsilon_{\theta}\left(x^{k},k\right)\right),\sigma_{k}^{2}I\right), (A.1)

C.2 WB-VIMA Architecture

WB-VIMA uses 256-dimensional observation features, a short causal observation history, and compact diffusion heads for eight future action steps.

C.2.1 Observation Encoder

Each of 4,096 points contains RGB plus XYZ. Colors are divided by 255, and coordinates by task-specific spatial limits. PointNet uses two hidden layers of width 256, GELU, and a 256-dimensional output. The proprioception MLP receives 21 values, uses three hidden layers of width 256 and ReLU, and emits 256 features. Table A.V retains the full configuration.

Table A.V: Hyperparameters for PointNet and the proprioception MLP.
Hyperparameter Value Hyperparameter Value
PointNet Prop. MLP
NpcdN_{\text{pcd}} 4096 Input Dim 21
Hidden Dim 256 Hidden Dim 256
Hidden Depth 2 Hidden Depth 3
Output Dim 256 Output Dim 256
Activation GELU Activation ReLU

C.2.2 Multi-Modal Observation Attention

The transformer has width 256, two layers, eight heads, dropout 0.1, and GEGLU activation. Observation window To = 2 is shared across methods. Action-readout tokens attend to preceding observations; the last readout becomes the diffusion condition. Table A.VI preserves these settings.

Table A.VI: Hyperparameters for the transformer decoder used in multi-modal observation attention.
Hyperparameter Value
Embed Size 256
Num Layers 2
Num Heads 8
Dropout Rate 0.1
Activation GEGLU [173]

C.2.3 Autoregressive Whole-Body Action Decoding

Three separate UNets predict base, torso, and arm/gripper chunks in that order, using action horizon Ta = 8. Higher-body heads receive completed lower-body predictions. The observation backbone runs once, while denoising iterates in the smaller heads. Table A.VII reports widths [64, 128], kernel size 2, five GroupNorm groups, and diffusion embedding size 8. Five groups do not evenly divide the listed widths, so the actual normalization implementation should be checked rather than assuming that table is a complete executable specification.

Table A.VII: Hyperparameters for the UNet models used for denoising.
Hyperparameter Value
Hidden Dim [64,128]
Kernel Size 2
GroupNorm Num Groups 5
Diffusion Step Embd Dim 8

C.3 Baselines Architectures

DP3 shares the PointNet/proprioception encoders but omits RGB channels and fuses features with a two-layer 512-unit MLP before flat 21-channel diffusion. RGB-DP uses pretrained ResNet-18 visual features with a 512-dimensional output. ACT follows its recommended hyperparameters. Thus color, fusion architecture, and action factorization differ across baselines; the targeted ablations provide more direct evidence for individual WB-VIMA components.

C.4 Policy Training Details

Data splits are 90% training and 10% validation. Methods train for equal steps and use the final checkpoint. AdamW uses learning rate 7×10⁻⁴, weight decay 0.1, 1,000 warm-up steps, cosine decay over 300,000 steps, and minimum rate 5×10⁻⁶. Training uses 100 DDPM noise steps; inference uses 16 DDIM steps. Multi-GPU training employs RTX A5000, RTX 4090, and A40 devices. Table A.VIII retains all schedule values.

Table A.VIII: Training hyperparameters.
Hyperparameter Value
Learning Rate 7×10−47\times 10^{-4}
Weight Decay 0.1
Learning Rate Warm Up Steps 1000
Learning Rate Cosine Decay Steps 300,000
Minimal Learning Rate 5×10−65\times 10^{-6}

C.5 Policies Deployment Details

Policy inference runs continuously on a workstation while the controller executes an existing action chunk. When switching chunks, early actions that became stale during inference are discarded. This addresses effective execution delay; the reported 0.02 s figure should not be interpreted as a complete breakdown of all sensor/network/compute latency.

Appendix D Task Definition and Evaluation Details

The final appendix defines each task, its randomization and demonstrations, the numerical evaluations, the constrained simulation ablation, and the user-study protocol.

D.1 Task Definition

Post-party cleanup has six stages: navigate to dishwasher, open it, move to the table, collect bowls, return, and load/close; 138 demonstrations average 210 s. Toilet cleaning has six stages from sponge pickup and lid opening through wiping, closing, and flushing; 103 demonstrations average 120 s. Trash disposal has four stages from bag pickup through door opening and outside disposal; 122 demonstrations average 130 s.

Shelving has two stages, floor-box pickup and placement on an available shelf; 100 demonstrations average 60 s. Clothing has four stages, wardrobe opening, jacket retrieval, laying it on a sofa bed, and closing the wardrobe; 98 demonstrations average 120 s. All randomize initial position, with task-specific changes to objects, placements, free shelf space, contents, or distractors. These randomizations are narrower than entirely new household layouts.

D.2 Policy Evaluation Results

Tables A.IX–A.XIII retain every success percentage, numerator/denominator, baseline, ablation row, and safety count. Human denominators vary across stages, and RGB-DP clothing has eight trials. Sub-task resets mean those columns are not conditional survival probabilities whose product must reproduce entire-task success.

D.3 Simulation Ablation Details

The OmniGibson wiping ablation uses 100,000 cuRobo-generated trajectories. The base and first two torso joints are locked, and the target hand position replaces camera inputs while proprioception remains. The robot must reach the target while maintaining table contact. These simplifications isolate control/fusion choices but limit inference about visual navigation or unconstrained whole-body behavior.

D.4 User Study Details

Ten participants test all three interfaces in randomized order with ten minutes of practice per interface. JoyLo and VR users wear a Meta Quest 3 for depth perception. Rollouts are manually segmented and labeled with the GUI in Figure A.5; Figure A.6 preserves the complete illustrated successful sequence. IK interfaces use a Jacobian-condition threshold to identify singularities, while JoyLo uses excessive joint speed, with threshold values not specified here.

The five survey questions ask about prior collection experience, expected and observed usability, whether physically holding the leader arms helped, and whether torso/base thumbsticks improved control. Figure A.4 retains the reported demographics and responses, including unanimous post-study preference for JoyLo and 90% reporting helpful Joy-Con control. Preferences and the simulator performance are evidence about this study population and mappings, not universal rankings of every teleoperation implementation.

Original figure
Figure A.4: Participant demographics and questionnaire results.
Refer to caption
Figure A.5: GUI for annotating user study rollouts.
Refer to caption
Figure A.6: Successful task completion by a participant. The robot navigates to a dishwasher and opens it, moves to a table to collect teacups, returns to the dishwasher, places the teacups inside, and closes it.
Table A.IX: Numerical evaluation results for the task “clean house after a wild party.” Success rates are shown as percentages. Values in parentheses indicate the number of successful trials out of the total trials.
ET ST-1 ST-2 ST-3 ST-4 ST-5 ST-6 Safety Violations
Human Teleop. 68%
(50/73)
100%
(73/73)
93%
(69/74)
100%
(69/69)
89%
(64/72)
94%
(60/64)
88%
(53/60)
N/A
Ours 40%
(6/15)
100%
(15/15)
80%
(12/15)
80%
(12/15)
73%
(11/15)
93%
(14/15)
93%
(14/15)
0
DP3 [70] 0%
(0/15)
80%
(12/15)
7%
(1/15)
27%
(4 / 15)
7%
(1/15)
33%
(5/15)
40%
(6/15)
13
RGB-DP [65] 0%
(0/15)
93%
(14/15)
0%
(0/15)
0%
(0/15)
7%
(1/15)
7%
(1/15)
20%
(3/15)
2
ACT [23] 0%
(0/15)
80%
(12/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
2
Table A.X: Numerical evaluation results for the task “clean the toilet.” Success rates are shown as percentages. Values in parentheses indicate the number of successful trials out of the total trials.
ET ST-1 ST-2 ST-3 ST-4 ST-5 ST-6 Safety Violations
Human Teleop. 61%
(100/164)
91%
(150/164)
72%
(106/148)
99%
(104/105)
100%
(103/103)
98%
(102/104)
98%
(100/102)
N/A
Ours 53%
(8/15)
100%
(15/15)
80%
(12/15)
100%
(15/15)
100%
(15/15)
100%
(15/15)
73%
(11/15)
0
DP3 [70] 0%
(0/15)
100%
(15/15)
47%
(7/15)
93%
(14/15)
0%
(0/15)
13%
(2/15)
0%
(0/15)
0
RGB-DP [65] 0%
(0/15)
93%
(14/15)
13%
(2/15)
7%
(1/15)
7%
(1/15)
0%
(0/15)
20%
(3/15)
2
ACT [23] 0%
(0/15)
20%
(3/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0
Table A.XI: Numerical evaluation results for the task “take trash outside.” Success rates are shown as percentages. Values in parentheses indicate the number of successful trials out of the total trials.
ET ST-1 ST-2 ST-3 ST-4 Safety Violations
Human Teleop. 76%
(96/127)
91%
(116/128)
100%
(124/124)
85%
(106/125)
100%
(115/115)
N/A
Ours 53%
(8/15)
80%
(12/15)
100%
(15/15)
87%
(13/15)
87%
(13/15)
1
DP3 [70] 0%
(0/15)
60%
(9/15)
53%
(8/15)
20%
(3/15)
7%
(1/15)
9
RGB-DP [65] 0%
(0/15)
20%
(3/15)
7%
(1/15)
7%
(1/15)
7%
(1/15)
3
ACT [23] 0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
5
Table A.XII: Numerical evaluation results for the task “put items onto shelf.” Success rates are shown as percentages. Values in parentheses indicate the number of successful trials out of the total trials.
ET ST-1 ST-2 Safety Violations
Human Teleop. 89%
(93/104)
90%
(94/104)
100%
(93/93)
N/A
Ours 93%
(14/15)
93%
(14/15)
100%
(15/15)
0
DP3 [70] 20%
(3/15)
27%
(4/15)
47%
(7/15)
0
RGB-DP [65] 13%
(2/15)
20%
(3/15)
40%
(6/15)
0
ACT [23] 0%
(0/15)
0%
(0/15)
33%
(5/15)
1
Ours w/o W.B. Action Denoising 40%
(6/15)
40%
(6/15)
60%
(9/15)
0
Ours w/o Multi-Modal Obs. Attn. 13%
(2/15)
33%
(5/15)
40%
(6/15)
0
Table A.XIII: Numerical evaluation results for the task “lay clothes out.” Success rates are shown as percentages. Values in parentheses indicate the number of successful trials out of the total trials.
ET ST-1 ST-2 ST-3 ST-4 Safety Violations
Human Teleop. 50%
(54/108)
56%
(60/108)
93%
(56/60)
96%
(54/56)
100%
(54/54)
N/A
Ours 53%
(8/15)
87%
(13/15)
93%
(14/15)
80%
(12/15)
60%
(9/15)
0
DP3 [70] 0%
(0/15)
13%
(2/15)
13%
(2/15)
27%
(4/15)
27%
(4/15)
7
RGB-DP [65] 0%
(0/8)
13%
(1/8)
25%
(2/8)
13%
(1/8)
13%
(1/8)
3
ACT [23] 0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
0%
(0/15)
1
Ours w/o W.B. Action Denoising 13%
(2/15)
33%
(5/15)
73%
(11/15)
73%
(11/15)
67%
(10/15)
0
Ours w/o Multi-Modal Obs. Attn. 0%
(0/15)
33%
(5/15)
40%
(6/15)
47%
(7/15)
13%
(2/15)
4
Concept demonstration

Open image ↗