Literature
Display
← All papers

MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulation

Chengshu Li · Mengdi Xu · Arpit Bahety · Hang Yin · Yunfan Jiang · Huang Huang · Josiah Wong · Sujay Garlanka · Cem Gokmen · Ruohan Zhang · Weiyu Liu · Jiajun Wu · Roberto Martín-Martín · Li Fei-Fei

· ICLR 2026 ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

The reviewed arXiv v4 is dated 25 February 2026 and marked ICLR 2026. The fourteen authors are affiliated with Stanford, UT Austin and Amazon; Chengshu Li, Mengdi Xu, Arpit Bahety and Hang Yin contributed equally. All four appendices are retained.

At a glance

MoMaGen generates new demonstrations from one annotated source demonstration per task. It adapts the mobile base and camera as well as the arms, enforcing reachability and manipulation visibility while favoring visibility and compact posture during navigation.

Experiments generate data for four household tasks and train policies for cup picking and table tidying. On a matched real cup-picking setup, simulation pretraining followed by forty real training demonstrations yields 60% success for π₀ versus 0% without that simulation stage. The evidence concerns this small adaptation experiment, not general zero-shot real-world deployment. Project resources accompany the paper.

1 Introduction

Replaying a human demonstration in a new object arrangement can place a mobile robot too far from the next object or leave it outside the camera view. A feasible arm motion alone therefore does not provide suitable whole-body training data.

MoMaGen samples and plans new base/camera configurations, preserving object-relative contact motions from the source. Simulation validates generated trajectories before behavior cloning. This reduces repeated teleoperation, but still requires source annotation, task setup, privileged scene geometry and substantial generation compute.

Complete original Figure 1.
Figure 1: (left) MoMaGen uses a single human-collected demonstration to generate a large set of demonstrations, formulating data generation as a constrained optimization problem. (top-left) shows a human-collected demo for cleaning a pan with a scrub. (top-right) shows three novel object configurations with aggressive object pose randomization and additional distractors/obstacles. MoMaGen can generate novel trajectories in these diverse scenarios. (bottom-left) shows three robot base poses and (bottom-right) shows two arm trajectories for picking up the pan.

2 Related Works

The MimicGen family expands seed demonstrations through simulation, complementing offline image augmentation, trial-and-error collection and scripted experts. SkillMimicGen adds motion planning; DexMimicGen supports bimanual timing; other approaches address dynamics and geometry.

Mobile imitation learning must coordinate navigation, manipulation and observation, making demonstrations costly to collect. Table 1 compares constraint choices. MoMaGen extends the two evaluated baselines with source-base replay, then tests its additional base sampling and visibility controls against those extensions.

Methods Bimanual Mobile Obstacles Base Active Hard Soft
Random. Perception Constraints Constraints
MimicGen (Mandlekar et al., 2023b) ✗ ✓ ✗ ✗ ✗ Succ N/A
SkillMimicGen (Garrett et al., 2024) ✗ ✗ ✓ ✗ ✗ Succ, Kin, C-Free N/A
DexMimicGen (Jiang et al., 2025b) ✓ ✗ ✗ ✗ ✗ Succ, Temp N/A
DemoGen (Xue et al., 2025) ✓ ✗ ✓ ✗ ✗ Kin, C-Free N/A
PhysicsGen (Yang et al., 2025) ✓ ✗ ✗ ✗ ✗ Kin, C-Free, Dyn Trac
MoMaGen (Ours) ✓ ✓ ✓ ✓ ✓ Succ, Kin, C-Free, Temp, Vis Vis, Ret
Table 1: Comparison of different automated data generation methods and the constraints they enforce. “Succ”: task success; “Kin”: kinematic feasibility; “C-Free”: collision-free execution; “Temp”: temporal constraints for bimanual coordination; “Dyn”: system dynamics; “Trac”: target trajectory tracking; “Vis”: visibility of task-relevant objects in the robot’s camera view; “Ret”: retraction of robot torso and arm to a compact configuration before navigation.

3 Problem Formulation: Automated Data Generation as Constrained Optimization

A task is an MDP. A source trajectory supplies state/action sequences; successful generated trajectories become behavior-cloning data. Source demonstrations are segmented into object-centered subtasks with associated object and end-effector trajectories. The original dataset, learning-objective and trajectory expressions are retained below.

Free-space segments can be replanned under kinematic and collision constraints. Contact-rich segments retain the original end-effector motion relative to the relevant object. Equation 1 expresses dynamics, kinematics, collisions, visibility, transformed contact motion and eventual task success as hard conditions, with user-selected soft costs in the objective.

This is a unifying formulation; the implementation uses sampling, inverse kinematics, planning and replay rather than solving one globally optimal trajectory problem. The source equation uses the same end-effector symbol on both sides of its transform relation, although the prose distinguishes source and generated trajectories; the original notation remains unchanged.

Equations 1
Original Equation 1: constrained demonstration generation, preserving all six constraints and the optimization objective.

Original source/generated data, initial-state distribution, constraints, behavior-cloning objective and object/end-effector trajectories:

𝒟s​r​c={dj=(s0j,a0j,…,sTs​r​cj)}j=0Ns​r​c\mathcal{D}_{src}=\{d^{j}=(s_{0}^{j},a_{0}^{j},\ldots,s_{T_{src}}^{j})\}_{j=0}^{N_{src}}

s0∼Ds_{0}\sim D

𝒟={d}Ng​e​n\mathcal{D}=\{d\}^{N_{gen}}

{𝒢i}\{\mathcal{G}_{i}\}

arg⁡minθ​𝔼(s,a)∼𝒟​[−log⁡πθ​(a|s)]\arg\min_{\theta}\mathbb{E}_{(s,a)\sim\mathcal{D}}[-\log\pi_{\theta}(a|s)]

Si​(oi),i∈[N]S_{i}(o_{i}),i\in[N]

τi={𝐓WEk}k=0Ki,i∈[N],k∈[Ki]\tau_{i}=\{\mathbf{T}^{E_{k}}_{W}\}_{k=0}^{K_{i}},i\in[N],k\in[K_{i}]

4 MoMaGen

MoMaGen specializes the general constraints for mobile bimanual tasks, then generates subtasks through conditional base/camera sampling and arm execution.

4.1 Constraints for Bimanual Mobile Manipulation

Reachability is hard: a sampled base pose must permit the required end-effector trajectory. Manipulation visibility is also hard: the camera must see task-relevant objects, using head or torso articulation as needed.

Navigation visibility is soft, allowing temporary loss of sight while biasing the camera toward the target. Retraction is another preference: after manipulation, arms and torso attempt a compact posture for safer subsequent navigation. These application-specific choices are motivated by policy training, not proved human-optimal behavior.

4.2 Automated Demonstration Generation for Bimanual Mobile Manipulation

One 1–3-minute demonstration per task is segmented into single-arm or synchronized bimanual subtasks. Annotation specifies target and held objects, the step before contact, the end step and the desired post-manipulation retraction.

For each subtask, generation aborts if the required held object is missing, transforms the contact trajectory into the new target-object frame, and checks visibility and arm inverse kinematics at the current pose. If needed, it samples another base/camera pose and plans the transition while favoring visibility.

The arm then plans to pregrasp, replays the transformed contact trajectory in task space and attempts retraction. cuRobo supplies inverse kinematics and motion planning. Fast feasibility checks precede expensive planning, and separate torso/arm subspaces simplify sampling.

Algorithm 1 retains all original operations and nesting. A maximum sampling budget, exact soft-cost weights and a global-optimality guarantee are not specified in this version; generation can fail through infeasibility, planning or simulation execution.

Original Figure 2, retaining all native raster pixels.
Figure 2: MoMaGen method. Given a single source demonstration, as well as annotations for object-centric subtasks for each end-effector, MoMaGen first randomizes scene configuration, and transforms the end-effector poses from the source demo to the new objects’ frame of reference. For each subtask, it tries to sample a valid base pose that satisfies reachability and visibility constraints. Once found, it plans a base and torso trajectory to reach the desired base and head camera pose while trying to look at the target object during navigation. Once arrived, it plans an arm trajectory to the pregrasp pose and uses task space control for replay, before retracting back to a tucked, neutral pose.
Algorithm 1 MoMaGen 
Input: original demo, new initial state s0s_{0}
Output: generated demo
1: for each segment do
2:      Get current 𝐓base\mathbf{T}^{\mathrm{base}} , 𝐓cam\mathbf{T}^{\mathrm{cam}} , qtorsoq^{\mathrm{torso}} , qarmq^{\mathrm{arm}}
3:      if held object not in hand then abort
4:      Compute transformed end-effector pose 𝐓eef\mathbf{T}^{\mathrm{eef}} using new target object pose
5:      Check visibility of target object with 𝐓cam\mathbf{T}^{\mathrm{cam}}
6:      Solve IK for arm trajectory {qtarm}\{q^{\mathrm{arm}}_{t}\} with current 𝐓base\mathbf{T}^{\mathrm{base}} , 𝐓cam\mathbf{T}^{\mathrm{cam}}
7:      while not visible or no IK exists do
8:          Sample new base pose 𝐓base\mathbf{T}^{\mathrm{base}}
9:          Sample new camera pose 𝐓cam\mathbf{T}^{\mathrm{cam}}
10:          Solve IK for arm {qtarm}\{q^{\mathrm{arm}}_{t}\} and torso {qttorso}\{q^{\mathrm{torso}}_{t}\} with sampled 𝐓base\mathbf{T}^{\mathrm{base}} , 𝐓cam\mathbf{T}^{\mathrm{cam}}
11:          Plan motion for {qttorso}\{q^{\mathrm{torso}}_{t}\} from current 𝐓base\mathbf{T}^{\mathrm{base}} to sampled 𝐓base\mathbf{T}^{\mathrm{base}} , 𝐓cam\mathbf{T}^{\mathrm{cam}} w/ soft visibility
12:      Plan motion for {qtarm}\{q^{\mathrm{arm}}_{t}\} from previous 𝐓eef\mathbf{T}^{\mathrm{eef}} to pregrasp 𝐓eef\mathbf{T}^{\mathrm{eef}}
13:      Control end-effector in task space to follow transformed 𝐓eef\mathbf{T}^{\mathrm{eef}}
14:      Attempt retraction

5 Experiments and Results

Generation experiments cover four household tasks and three randomization levels. Separate policy and hardware experiments test whether the resulting observations/actions provide useful imitation-learning data.

Original Figure 3, retaining all native raster pixels.
Figure 3: Task visualization. Our multi-step tasks include long-range navigation, sequential and coordinated bimanual manipulation, requiring pick-and-place and contact-rich motion.

5.1 Task Setup

The BEHAVIOR-1K/OmniGibson tasks are Pick Cup, Tidy Table (counter to sink), Put Dishes Away (independent two-arm plate stacking) and Clean Frying Pan (coordinated scrubbing). They span navigation, grasping and contact-rich manipulation.

Each task uses one human demonstration, with base movement accounting for about 45% of its 1–3-minute duration. This is one demonstration per task, not a single trajectory shared across all four tasks.

5.2 Domain Randomization Schemes

D0 perturbs relevant objects by ±15 cm and ±15° on their original furniture. D1 permits placement anywhere on that furniture with unrestricted orientation. D2 adds tabletop manipulation obstacles and floor navigation obstacles to D1. These perturbations retain the task and furniture layout; they do not establish arbitrary new-room generalization.

5.3 Data Generation Comparison

The compared SkillMimicGen and DexMimicGen extensions replay the source base trajectory. They generate data under D0 but report zero success under D1/D2 because replayed base poses cannot reach many moved objects. MoMaGen samples new base poses, widening object, end-effector and joint coverage in the fifty-trajectory visualizations.

Across the four tasks, MoMaGen’s D0 generation success averages about 64% (the text rounds to 63%). Rates fall from 0.86/0.80/0.38/0.51 in D0 to 0.47/0.22/0.07/0.16 in D2. The baselines are better on simple D0 cup picking and tie on dishes; they still generate some D0 pan-cleaning data despite prose saying they “fail” on harder tasks.

Visibility constraints increase target visibility during navigation: for example, D0 Tidy Table rises to 0.86 versus about 0.40 for the baselines. Removing either hard or soft visibility weakens the visibility distribution. The reported ratio characterizes generated data, not an unconditional visibility guarantee for every subsequent learned rollout.

A separate demonstration transfers an R1 Pick Cup trajectory to TIAGo through end-effector-space replay. Appendix B.2 describes this as proof of concept, with gripper/workspace limitations and no broad quantitative cross-embodiment benchmark.

Original Figure 4, retaining all native raster pixels.
Figure 4: Generated data diversity analysis for Tidy Table task (50 trajectories, subsampled). Given the same object randomization (D0) (a), compared to SkillMimicGen, MoMaGen samples diverse base poses (b), and as a result, diverse end-effector poses (c) and joint positions (d). MoMaGen is also the only method that can generate data for D1 randomization (red) for even greater diversity.
Original Figure 5, retaining all native raster pixels.
Figure 5: Object visibility analysis for MoMaGen and ablations. The x-axis is the % of frames where the target object is visible during navigation, and the y-axis is the trajectory count (out of 1000). MoMaGen significantly outperforms ablations thanks to both hard and soft visibility constraints.
Methods Pick Cup Tidy Table Put Dishes Away Clean Frying Pan
D0 MoMaGen 0.86 0.80 0.38 0.51
SkillMimicGen 1.00 0.69 0.38 0.40
DexMimicGen 1.00 0.72 0.38 0.35
MoMaGen w/o soft vis. const. 0.88 0.78 0.50 0.46
MoMaGen w/o hard vis. const. 0.97 0.59 0.29 0.24
MoMaGen w/o vis. const. 0.97 0.74 0.29 0.36
D1 MoMaGen 0.60 0.64 0.34 0.20
MoMaGen w/o vis. const. 0.66 0.48 0.23 0.13
D2 MoMaGen 0.47 0.22 0.07 0.16
MoMaGen w/o vis. const. 0.50 0.16 0.05 0.12
Table 2: Data generation success rates comparison. For simpler tasks (Pick Cup), ablations and baselines achieve higher data gen success rates because of fewer constraints and less motion planning stochasticity. However, for more complex tasks (the other three), enforcing hard visibility constraints helps position the robot torso to a suitable configuration that facilitates downstream manipulation, leading to higher success rates. The baselines suffer from zero success rates and hence are omitted for D1/D2 because the objects are beyond the reachability of replayed base poses from the source demo.
Methods Pick Cup Tidy Table Put Dishes Away Clean Frying Pan
D0 MoMaGen 1.00 0.86 0.79 0.69
SkillMimicGen 1.00 0.40 0.71 0.65
DexMimicGen 1.00 0.39 0.71 0.67
MoMaGen w/o soft vis. const. 1.00 0.63 0.62 0.56
MoMaGen w/o hard vis. const. 0.98 0.63 0.68 0.55
MoMaGen w/o vis. const. 0.90 0.46 0.40 0.35
D1 MoMaGen 0.93 0.89 0.78 0.80
MoMaGen w/o vis. const. 0.71 0.46 0.40 0.43
D2 MoMaGen 0.94 0.79 0.75 0.81
MoMaGen w/o vis. const. 0.73 0.48 0.40 0.44
Table 3: Task-relevant object visibility comparison. Our hard and soft visibility constraints are exceedingly effective in keeping the object in view during navigation, achieving over 75%75\% visibility even for aggressively randomized object pose (D1) and obstacles/occluders (D2). We omit baselines for D1/D2 due to zero data generation success rates.

5.4 Policy Learning with Generated Demonstrations

Generation uses privileged object poses and geometry. Deployment instead uses learned policies: WB-VIMA trains from scratch on proprioception and fused colored point clouds using simulated ground-truth depth; pretrained π₀ is fine-tuned with rank-32 LoRA on proprioception and three RGB views.

The main policy comparison uses 1,000 demonstrations for Pick Cup D0/D1 and Tidy Table D0, with twenty unseen evaluation episodes. MoMaGen matches baselines on easy cup picking and improves table tidying. For D1 cup picking, WB-VIMA succeeds on 25% of trials while baselines trained only on D0 fail, so this comparison changes training coverage as well as generation method.

π₀ also benefits from the generated data on these tasks. Removing visibility constraints reduces WB-VIMA cup success from 0.75 to 0.45–0.65 and table-tidying success from 0.40 to at most 0.05. This supports the constraints for these short-history policies, without isolating visibility from every associated trajectory change.

π₀ scaling uses 500, 1,000 or 2,000 demonstrations and 50k optimization steps. The trend is broadly positive, especially for D1, but Tidy Table dips at 1,000 before recovering. Repeated-training-seed uncertainty is not reported.

Real-world Pick Cup uses simulation pretraining followed by forty real training demonstrations: WB-VIMA reaches 10% and π₀ 60%, versus 0% without task-specific simulation pretraining. Ten additional real demonstrations are held out for validation. This is adaptation to a matched setup; no zero-shot transfer result or hardware evaluation trial count is supplied.

Original Figure 6, retaining all native raster pixels.
Figure 6: Comparison between MoMaGen and other data generation methods on WB-VIMA’s performances in (a) and (b), performances of WB-VIMA and π0\pi_{0} trained with MoMaGen data in (c) and visibility ablations in (d). The success rate is averaged over 20 unseen evaluation episodes. Policies trained on MoMaGen data consistently perform better than those trained on others’ data.
Original Figure 7, retaining all native raster pixels.
Figure 7: Data Scaling of π0\pi_{0}

5.5 MoMaGen Failure and Computational Cost Analysis

Failures are categorized as base sampling, base/arm inverse kinematics or trajectory optimization, other arm-planning failures, and simulation instability. Across randomization levels, the paper attributes about 35% to simulation instability, 40% to arm planning and 26% to base-related planning; these rounded averages are descriptive, not identified causal effects.

D2 obstacles increase navigation-related failures. The additional task-step plot shows later navigation/placement bottlenecks for dishes. Percentages are conditional on failed episodes and should not be read as whole-dataset failure probabilities.

Execution in simulation dominates wall time: base planning averages about eighteen seconds versus about one hundred seconds for execution. Ring-shaped base sampling varies greatly with feasible-space size. The component-percentage chart includes successful episodes only; it excludes the aggregate cost of failed attempts. Informed sampling and better simulation/control are proposed improvements.

Complete original Figure 8.
Figure 8: Failure Analysis
Complete original Figure 9.
Figure 9: Compute costs (wall-clock time) for MoMaGen data generation components.
Complete original Figure 10.
Figure 10: Compute cost of each component as a percentage of the episode

6 Conclusions and Limitations

The experiments support generating more varied mobile-manipulation data by adapting base placement and camera view. Effective learned policies still require suitable sensory inputs, training and real-world adaptation.

Generation assumes full scene geometry and object poses and consumes substantial GPU time. The main benchmark alternates navigation and manipulation. Although the conclusion leaves coupled whole-body manipulation to future work, Appendix B.1 adds a chair-pushing proof of concept; it does not provide a comparable quantitative whole-body benchmark. Perception-based generation is proposed, not demonstrated.

Reproducibility Statement

The authors provide a code submission and project-site resources for simulation tasks, generation and policy training. Appendix C retains the reported architecture and training settings; some implementation details and evaluation uncertainty remain unspecified in the paper.

Acknowledgments

Support is acknowledged from Stanford HAI, two ONR MURI grants, a UT-CNS Catalyst Grant and the Amazon Personal Robotics Group Faculty Award.

Appendix A Additional Experimental Results

The additional experiment describes task-specific simulation pretraining followed by real-world adaptation.

A.1 Sim-to-real

Hardware is a Galaxea R1 with an onboard ZED 2 and a ZED Mini above the left gripper. Refined depth is downsampled to 4,096 points. A white table and 3D-printed green cup approximate the simulated Pick Cup D0 setup.

Fifty real demonstrations are collected: forty train and ten validate. WB-VIMA starts from either random weights or 1.8M steps on 1,000 synthetic demonstrations, then trains 35k real-data steps. Simulation pretraining reduces validation loss to about three versus six and yields 10% versus 0% hardware success.

Both π₀ variants start from foundation-model weights; only one receives the extra simulation stage before real-data fine-tuning. Its 60% versus 0% success supports this adaptation sequence, not a comparison against an unpretrained foundation model. Trial counts, uncertainty and additional hardware tasks are not reported.

Complete original Figure 11.
Figure 11: Real world setup for Pick Cup (a) and validation loss curve of WB-VIMA (b).

Appendix B Additional Data Generation Details

The appendix extends generation qualitatively, visualizes perturbations/diversity and reports generation cost and failure locations.

B.1 Can MoMaGen generalize across different families of mobile manipulation tasks?

Three additional task families are illustrated through proof-of-concept generated demonstrations and website videos. No aggregate success rates or learned-policy evaluation are given for these extensions.

Get Bottle from Fridge (Articulated Object Manipulation).

For Get Bottle from Fridge, the robot opens the door and then retrieves the initially occluded bottle. The relevant target switches from fridge to bottle after opening, making visibility constraints depend on the current subtask.

Pour Cat Food (Pouring and Particle Interaction).

For Pour Cat Food, the robot carries a tin of rigid particles to a bowl and tilts it to transfer the particles. The example requires coordinated orientation and sustained container control; it is not a general fluid-simulation result.

Push Chair (Whole-Body Control).

For Push Chair, base motion and end-effector contact direction are coupled while pushing the chair beneath a table. This demonstrates a whole-body contact example beyond the alternating main benchmark.

B.2 Can MoMaGen achieve cross-embodiment data generation?

An R1 source demonstration produces TIAGo Pick Cup trajectories despite different torso and arm workspaces. Task-space planning/replay avoids direct joint-trajectory copying.

Transfer still depends on embodiment feasibility. Larger grippers can collide in confined spaces, and bulkier arms can make R1 trajectories unreachable or self-colliding. One demonstrated transfer does not establish arbitrary robot interchangeability.

B.3 Visualizations of Domain Randomization Schemes

Figure 12 retains the source, D0, D1 and D2 configurations for all four tasks. Blue marks relevant objects, green their allowed placement range and red obstacles/distractors. The range is constrained to the same furniture even in D1/D2.

Original Figure 12, retaining all native raster pixels.
Figure 12: Visualization of Domain Randomization. Blue represents the task-relevant objects. Green represents the approximate randomization range for these objects (for D1 and D2, the range is anywhere on the same furniture). Red represents the obstacles/distractor objects.

B.4 Visualizations of Data Diversity

Figures 13–15 extend the fifty-trajectory diversity visualizations to cup picking, dishes and pan cleaning. The plots show broader sampled state/action coverage with new base poses and D1 object placement; PCA projections alone do not quantify the full trajectory distribution.

B.5 Compute Resources

Generation runs on one NVIDIA TITAN RTX. A successful demonstration costs 0.1–1.3 GPU-hours, from cup picking to dishes. Cost depends on subtask count/length, scene complexity and success rate, so generating thousands of demonstrations remains substantial computation.

Original Figure 13, retaining all native raster pixels.
Figure 13: Generated data diversity analysis for Pick Cup task (50 trajectories, subsampled).
Original Figure 14, retaining all native raster pixels.
Figure 14: Generated data diversity analysis for Put Dishes Away task (50 trajectories, subsampled).
Original Figure 15, retaining all native raster pixels.
Figure 15: Generated data diversity analysis for Clean Frying Pan task (50 trajectories, subsampled).

B.6 Failure Step Analysis

Figure 16 preserves failure shares by task step. Dishes fail more often during navigation and placement than initial picking. These distributions describe where failed generation stops, rather than independent per-step hazard estimates.

Complete original Figure 16.
Figure 16: Distribution of failures across task-level steps within each multi-step task.

Appendix C Additional Policy Training Details

Both policy pipelines split generated demonstrations 90% for training and 10% for validation. The following sections retain preprocessing, architecture, task-specific crops and compute settings.

C.1 Data Cleaning

To remove teleoperation hesitation, the preprocessing discards a five-step interval whenever every absolute joint-position difference between its endpoints is below 10⁻³. This endpoint criterion is the reported rule; it need not imply that every intermediate state is identical. WB-VIMA has a two-step history and π₀ uses the current state.

C.2 WB-VIMA Training Details

WB-VIMA combines proprioception and egocentric colored point clouds in a task-specific policy trained from scratch.

Policy Architecture.

Inputs comprise three base-velocity values, four torso joints, six joints per arm and one width per gripper, totaling twenty-one. Three simulated RGB-D cameras are fused, cropped around the robot and farthest-point sampled to 4,096 points. Two observations form the history; Figure 17 retains all twelve example frames.

Hyperparameters.

Table 4 retains the full PointNet, transformer, diffusion-head and optimizer settings, including task-specific point-cloud crops. The 37.1M-parameter policy uses four transformer layers, eight heads, sixteen inference denoising steps and AdamW with a cosine schedule.

The table prints proprioceptive MLP hidden dimension two and hidden depth 256, apparently reversed relative to conventional usage; the values are preserved. Its batch is 128 per GPU on two GPUs, whereas the compute paragraph simply calls it batch 128. Hyperparameters are selected by grid search.

Complete original Figure 17.
Figure 17: Visualization of ego-centric point cloud for Pick Cup and Tidy Table, used for WB-VIMA training. We fuse the colored point-cloud from three RGB-D cameras, crop with a robot-centric bounding box, and then downsample to 4096 points via farthest point sampling.
Hyperparameter Value
Number of points in point cloud 4096
PointNet hidden dim 256
PointNet hidden depth 2
PointNet output dim 256
PointNet activation GELU
Proprioceptive MLP input dim 21
Proprioceptive MLP hidden dim 2
Proprioceptive MLP hidden depth 256
Proprioceptive MLP output dim 256
Proprioceptive MLP activation ReLU
Transformer embedding size 512
Transformer layers 4
Transformer heads 8
Transformer dropout rate 0.1
Transformer activation GEGLU
Action dim 21
Unet down dims [128, 256]
Unet kernel size 5
Unet number of groups 8
Diffusion step embedding dim 256
Diffusion noise scheduler DDIM
Number of training steps 100
Beta schedule squaredcos_cap_v2
Number of denoise steps per inference 16
Learning rate 1e-4
Learning rate scheduler Cosine decay
Learning rate warmup steps 100000
Learning rate cosine steps 1300000
Optimizer AdamW
Batch size per GPU 128
Number of GPUs in parallel 2
Pick Cup (D0) pointcloud clip range x: [0.0, 2.3], y: [-0.5, 0.5], z: [0.7, 2.0]
Pick Cup (D1) pointcloud clip range x: [0.0, 2.7], y: [-1.0, 1.0], z: [0.7, 2.0]
Tidy Table (D0) pointcloud clip range x: [0.0, 2.3], y: [-1.5, 1.5], z: [0.7, 1.5]
Model size 37.1M
Table 4: Hyperparameters for WB-VIMA.

C.3 π 0 Training Details

π₀ provides a separate pretrained RGB-based policy, adapted through LoRA rather than training WB-VIMA’s point-cloud architecture.

Policy Architecture.

Rank-32 LoRA fine-tuning runs for 50k steps with batch 64. Three camera images and the same proprioceptive signals predict a fifty-step target-joint-position chunk.

Hyperparameters.

RGB inputs are resized to 224×224. Actions and proprioception are normalized by first/99th quantiles and padded to the expected 32 dimensions. A PaliGemma backbone accompanies a 300M-parameter action expert; Table 5 retains all reported settings.

The table describes a 3.3B-parameter model, ten flow-matching inference steps, learning rate 2.5×10⁻⁵, 1,000 warmup steps and 30,000 cosine steps. The text trains for 50k steps without explaining the remaining schedule behavior.

Hyperparameter Value
Proprioceptive MLP input dim 32
Proprioceptive MLP output dim 1024
Flow Matching MLP input dim 32
Flow Matching MLP hidden dim 2048
Flow Matching MLP hidden depth 2
Flow Matching MLP output dim 1024
Flow Matching MLP activation swish
PaliGemma embedding size 2048
PaliGemma number of layers 18
PaliGemma number of heads 18
PaliGemma heads dimension 256
PaliGemma MLP dimension 16384
Action expert embedding size 1024
Action expert MLP dimension 4096
Number of flow matching steps per inference 10
Learning rate 2.5e-5
Learning rate scheduler Cosine decay
Learning rate warmup steps 1000
Learning rate cosine steps 30000
Optimizer AdamW
Batch size 64
Number of GPUs in parallel 4
Model size 3.3B
Table 5: Hyperparameters for π0\pi_{0}.

C.4 Compute Resources

WB-VIMA takes about forty hours for one million steps on two 24-GB RTX 3090 GPUs. π₀ takes about seven hours for 50k steps on four H200 GPUs. These training costs are additional to demonstration generation.

Appendix D Use of Large Language Models

The authors state that ChatGPT was used only for grammar, clarity and concision, not research ideation, experimental design or analysis.

Concept demonstration

Open image ↗