Literature
Display
← All papers

RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation

Shihan Wu · Xuecheng Liu · Shaoxuan Xie · Pengwei Wang · Xinghang Li · Bowen Yang · Zhe Li · Kai Zhu · Hongyu Wu · Yiheng Liu · Zhaoye Long · Runtian Xu · Yue Wang · Chong Liu · Dihan Wang · Ziqiang Ni · Xiang Yang · You Liu · Ruoxuan Feng · Lei Zhang · Denghang Huang · Chenghao Jin · Anlan Yin · Xinlong Wang · Zhenguo Sun · Junkai Zhao · Mengfei Du · Mingyu Cao · Xiansheng Chen · Hongyang Cheng · Xiaojie Zhang · Yankai Fu · Ning Chen · Cheng Chi · Sixiang Chen · Huaihai Lyu · Xiaoshuai Hao · Yequan Wang · Bo Lei · Dong Liu · Xi Yang · Yance Jiao · Tengfei Pan · Yunyan Zhang · Songjing Wang · Ziqian Zhang · Xu Liu · Ji Zhang · Caowei Meng · Zhizheng Zhang · Jiyang Gao · Song Wang · Xiaokun Leng · Zhiqiang Xie · Zhenzhen Zhou · Peng Huang · Wu Yang · Yandong Guo · Yichao Zhu · Suibing Zheng · Hao Cheng · Xinmin Ding · Yang Yue · Huanqian Wang · Chi Chen · Jingrui Pang · YuXi Qian · Haoran Geng · Lianli Gao · Haiyuan Li · Bin Fang · Gao Huang · Yaodong Yang · Hao Dong · He Wang · Hang Zhao · Yadong Mu · Di Hu · Hao Zhao · Tiejun Huang · Shanghang Zhang · Yonghua Lin · Zhongyuan Wang · Guocai Yao

Preprint posted · arXiv (preprint) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

RoboCOIN combines teleoperated bimanual demonstrations, annotations at three temporal scales, and the CoRobot data-processing interface. The authors report an open release; the overview retains its platform and task montage.

Complete original PDF figure S0.F1.
Fig. 1: Overview of our RoboCOIN dataset. RoboCOIN is a large-scale, multi-embodiment dataset explicitly built for bimanual robotic manipulation research. It supports 15 distinct robotic platforms, with 180K+ high-quality manipulation demonstrations spanning 421 structured bimanual tasks and 16 diverse interactive scenarios. The dataset is fully open to the research community, equipped with a ready-to-use one-click installation toolchain, and can be directly deployed via the simple command “pip install robocoin”.

At a glance

RoboCOIN reports over 180,000 demonstrations from 15 platforms, covering 421 tasks, 16 scenarios, 39 action types and 432 object types. Its hierarchy links trajectory-level scene/task descriptions, segment-level subtasks and frame-level motion labels. CoRobot adds expert-rule trajectory filtering, assisted annotation and common hardware/data interfaces. Experiments support useful dataset post-training, enriched language inputs and phase-aware filtering in selected tasks; they do not establish universal cross-robot transfer or fully autonomous hierarchical reasoning.

I Introduction

Bimanual learning must handle both coordination between arms and differences in robot morphology, sensors and hands. RoboCOIN addresses these with diverse teleoperated data and explicit task structure, while CoRobot standardizes processing and reuse. This edition follows all 22 pages of arXiv v3, dated 21 May 2026; the catalog retains the original 2025 posting date.

The three experimental claims should be separated: extra post-training improves later adaptation to two simulated target robots; hierarchical language conditioning improves selected real-world tasks; expert trajectory filters improve two G1 tasks. The headline percentage gains are relative changes from their respective baselines, not percentage-point improvements or one combined score.

II Related Work

Datasets. Open X-Embodiment, RoboMIND and single-platform bimanual datasets provide different coverage and annotation structures. RoboCOIN emphasizes multi-platform bimanual collection plus temporal hierarchy. Table II is the paper’s historical comparison, not a continually updated inventory.

Policies. ACT and Diffusion Policy motivate learned action generation, while generalist VLAs use larger multimodal backbones. Experiments here focus on π0 and GR00T N1.5. Broad claims that other policies lack task understanding should be read as motivation, rather than conclusions from direct reasoning evaluations of every listed model.

Complete original PDF figure S2.F2.
Fig. 2: Data collection platforms of our RoboCOIN. (a) Dual Robot (e.g., AgileX Cobot Magic). A bimanual robot with two 6-DoF arms and parallel grippers, capable of performing complex bimanual tasks. (b) Half-Humanoid Robot (e.g., Realman RMC-AIDA-L). Left: wearable teleoperation device for human demonstration, right: bimanual robot with parallel grippers. (c) Humanoid Robot (e.g., Unitree G1edu-u3). Left: Wearable motion capture device for human demonstration. Right: Humanoid robot with advanced dexterous manipulation capabilities. Head and wrist cameras provide multi-view visual observations.
TABLE I: Robotic platforms used in RoboCOIN dataset collection.
Type Name Arm DoF Camera Configuration Gripper Type Teleoperation Method
Dual-Arm Agilex Cobot Magic [41] 2×\times6 Head + Wrist Parallel Gripper Isomorphic Arm
Agilex Split ALOHA [41] 2×\times6 Head + Wrist Parallel Gripper Isomorphic Arm
Galaxea R1 Lite [16] 2×\times6 Head + Wrist Parallel Gripper Isomorphic Arm
Half-Humanoid Realman RMC-AIDA-L [44] 2×\times7 Head + Wrist Parallel Gripper Exoskeleton
Agibot G1 [1] 2×\times7 Head + Wrist + Back Parallel Gripper Virtual Reality
AI2 AlphaBot 2 [42] 2×\times7 Head + Wrist + Chest Dexterous Hand Virtual Reality / Isomorphic Arm
AI2 AlphaBot 1s [42] 2×\times7 Head + Wrist + Chest Dexterous Hand Virtual Reality / Isomorphic Arm
Galbot G1 [17] 2×\times7 Head + Wrist Parallel Gripper Motion Capture
Tianqing A2 [49] 2×\times7 Head + Wrist + Back Parallel Gripper Virtual Reality
Realman Rs-02 [44] 2×\times7 Head + Wrist Parallel Gripper Exoskeleton
Realman Rs-01 [44] 2×\times7 Head + Wrist Parallel Gripper Exoskeleton
Airbot MMK2 [2] 2×\times6 Head + Wrist + Third-Person Dexterous Hand Virtual Reality
Leju Kuavo 4 LB [43] 2×\times7 Head + Wrist Dexterous Hand Virtual Reality
Humanoid Leju Kuavo 4 Pro [43] 2×\times7 Head + Wrist Dexterous Hand Virtual Reality
Unitree G1edu-u3 [50] 2×\times7 Head + Wrist Dexterous Hand Motion Capture / Exoskeleton
Complete original PDF figure S2.F3.
Fig. 3: Overview statistics of the RoboCOIN dataset. RoboCOIN incorporates (a) 15 distinct robotic platforms, including bimanual, half-humanoid, and humanoid robots with grippers and dexterous hands; (b) 16 diverse environments such as residential, commercial, and working scenes; (c) 39 action types categorized by collaboration levels; and (d) 432 object types spanning rigid, articulated, and deformable categories.

III RoboCOIN Dataset

The dataset joins hardware diversity with synchronized annotations and a two-axis task taxonomy. Its demonstrations are human-teleoperated, rather than autonomous rollouts across all advertised scenarios.

Complete original PDF figure S3.F4.
Fig. 4: Hierarchical capability pyramid of RoboCOIN. Trajectory-level Annotations define the global concepts; Segment-level Annotations decompose the task into executable subtasks; and Frame-level Annotations provide dense low-level details such as motion trajectories and gripper states. All annotations are temporally synchronized to form a cohesive data structure.

III-A Data Collection and Storage

Table I preserves all 15 robot configurations: dual-arm, half-humanoid and humanoid systems with parallel grippers or dexterous hands. Collection uses isomorphic leader–follower arms, exoskeletons, virtual reality or motion capture, with camera placement varying by platform.

Joint states use radians, grippers are normalized to [0,1], and end-effector positions/Euler angles use meters/radians. Coordinates are right-handed with x forward, y left and z up; timestamp alignment synchronizes observations and kinematics. This common representation eases ingestion, but does not by itself make heterogeneous action spaces interchangeable.

III-B Statistics and Taxonomy

The reported demonstration mix is 33.2% dual-arm, 49.0% half-humanoid and 17.8% humanoid; scenarios are 47.1% residential, 18.6% commercial and 34.3% working settings. Low/high collaboration accounts for 58.9%/41.1%, and rigid/hinged/deformable objects for 83.9%/4.2%/11.9%. These uneven distributions matter when interpreting coverage.

Collaboration distinguishes largely sequential from closely synchronized arm use; object flexibility forms the second task axis. Figure 3 retains all detailed distributions. Its two humanoid platform labels, 8.1% and 7.6%, do not sum to the stated 17.8% category share, so the illustrated percentages are retained without silently reconciling them.

III-C Hierarchical Capability Pyramid

Trajectory level: describe the scene, objects and overall objective. Segment level: align atomic subtasks to time, allowing overlaps between arms and marking exceptions such as failed grasps. Frame level: describe end-effector motion and gripper/hand states. Figure 4 shows their temporal alignment on a peach-and-basket task. The hierarchy supplies information for learning; its presence alone does not demonstrate that a policy reasons correctly at every level.

TABLE II: Qualitative comparison of existing real-world datasets for robot manipulation. All data is drawn from the original paper or the RoboMIND paper. N/A indicates that the dataset does not provide explicit annotations for the corresponding attribute. † Not a dataset in itself, but an aggregation of existing datasets.
Dataset Arm Embodiments Trajectories Tasks Actions Dexterous Annotation Collection Method
Pinto and Gupta [38] Dual 1 50k N/A 1 ✗ No Scripted
RoboNet [12] Single 1 162k N/A N/A ✗ No Scripted
MT-Opt [23] Single 1 800k 12 1 ✗ No Scripted
BridgeData [13] Single 1 7.2k 71 4 ✗ No Human Teleoperation
BC-Z [20] Single 1 26k 100 3 ✗ No Human Teleoperation
RH20T [15] Single 1 13k 140 33 ✗ No Human Teleoperation
RoboSet [4] Single 1 98k 38 6 ✗ No 30% Human / 70% Scripted
BridgeData V2 [51] Single 1 60k N/A 13 ✗ No 85% Human / 15% Scripted
DROID [24] Single 1 76k N/A 86 ✗ No Human Teleoperation
Open X-Embodiment† [36] Single+Dual 22 1.4M 160k 217 ✗ No Dataset Aggregation
RoboMIND [53] Single+Dual 4 107k 479 38 ✓ Flat Human Teleoperation
AgiBot World Beta [6] Dual 1 1M 217 87 ✗ Flat Human Teleoperation
Open Galaxea [22] Dual 1 50k 150 58 ✓ Flat Human Teleoperation
RoboCOIN (Ours) Dual 15 180K+ 421 39 ✓ Hierarchical Human Teleoperation

III-D Comparison with Existing Datasets

Table II compares embodiment counts, trajectory/task/action counts, dexterous hardware and annotation type. RoboCOIN’s distinguishing combination is broad bimanual hardware coverage and hierarchical annotations. The prose describes Open X-Embodiment as single-arm in Related Work, while the table correctly labels it “Single+Dual”; the claim should be limited to comparatively sparse bimanual coverage, not complete absence.

IV CoRobot Data Processing Pipeline

CoRobot combines rule-based quality assessment, assisted hierarchical annotation and a common software interface. The components address dataset preparation and deployment engineering rather than constituting a new end-to-end robot policy.

Complete original PDF figure S4.F5.
Fig. 5: Overview of the CoRobot data processing pipeline. (a) Robot Trajectory Markup Language (RTML) for automated trajectory validation. (b) Annotation Pipeline for generating rich and hierarchical task descriptions. (c) An out-of-the-box Unified Robotic Interface for unified robot control and multi-embodiment data management.

IV-A Robot Trajectory Markup Language Evaluation

RTML encodes expert constraints in YAML. Global rules bound workspace, velocity, acceleration and duration; stage-specific rules add overrides, orientation tolerances and timing limits for actions such as approach, grasp and place. An evaluator produces diagnostic reports and a quality score.

Filtering therefore means compliance with selected rules. It is not an independently verified label that every rejected trajectory is physically invalid or that every accepted one is safe and successful. Choosing constraints requires task and embodiment knowledge.

IV-B Annotation Pipeline

Object detectors such as Grounding DINO feed locations to a language model for scene descriptions. Gripper events and task progress propose segment boundaries, which humans adjust and label. Cascaded pose windows produce frame-level motion descriptors, then thresholds turn continuous values into text. Human review makes the overall process semi-automatic; the appendix reports residual scene-description errors and no manual verification of frame labels.

IV-C Unified Robotic Interface

Built on LeRobot, the interface wraps vendor SDKs and ROS/ROS 2 support, normalizes units and coordinates, and groups reusable data subsets by robot, task and environment. The source advertises installation through pip install robocoin and links its CoRobot repository. This describes the supplied interface design; no installation or hardware compatibility test is part of the paper evidence reviewed here.

Complete original PDF figure S4.F6.
Fig. 6: Cross-embodiment transfer results on RoboTwin 2.0. (a) Our selected robotic platforms and bimanual tasks in the RoboTwin 2.0 environment. (b) Performance comparison on ARX-X5. (c) Performance comparison on Franka Emika Panda. N/A indicates that the trajectory generation failed on this task.

V Experiments and Analysis

Three questions organize evaluation: whether RoboCOIN helps target-robot adaptation, whether hierarchical inputs improve task execution, and whether RTML filtering improves policy training. The experimental subsets are much smaller than the full collection.

Q1. How does the RoboCOIN dataset enhance the multi-embodiment capability of VLA models? Q2. Does the hierarchical capability pyramid improve VLA models, and in what aspects? Q3. To what extent does the RTML contribute to improving data quality and model performance?

V-A Experiment Setup

Simulation: use RoboTwin 2.0 on ARX-X5 and Franka Panda, absent from RoboCOIN collection. Of 16 intended tasks, trajectory generation succeeds for 14/11 respectively; missing tasks are marked N/A. Target-task fine-tuning uses 50 demonstrations per task, so this is adaptation to new embodiments rather than zero-shot execution.

Hardware: Realman RMC-AIDA-L evaluates the hierarchy, and G1edu-u3 evaluates filtering. π0 uses LoRA, AdamW, learning rate 2.5×10⁻⁵, batch 32 and 30,000 steps. GR00T N1.5 updates diffusion/projector components with learning rate 10⁻⁴, batch 24 and 10,000 steps. The appendix clarifies that its two filtering tasks have 298/997 source trajectories, despite the main text’s general statement of 50 per task.

Success conditions include lifting a pot clear, placing both shoes in a box, closing a peach-containing drawer, and completing a bowl handover/placement. The supplied manuscript does not clearly state evaluation trial counts, repeated training seeds or confidence intervals, limiting assessment of small differences.

Complete original PDF figure S5.F7.
Fig. 7: Results of Realman RMC-AIDA-L over the π0\pi_{0} baseline. It follows an multi-dimensional task design (a), with the in-distribution (b) and out-of-distribution results (c) showing the success rates with and without HAI.

V-B Impact of Multi-Embodiment Adaptability

Post-train π0 on a random 10,000-demonstration RoboCOIN subset, then fine-tune on target RoboTwin tasks. Reported mean success rises from 17.3% to 30.4% on ARX-X5 and 8.5% to 26.6% on Panda: gains of 13.1/18.1 percentage points, or 75.7%/212.9% relative.

High-collaboration tasks show larger gains in these tests, but absolute success remains modest. Dual-shoe placement rises from zero to only 9%/3%; this is evidence of some successful executions, not reliable task mastery. The comparison establishes the value of this additional post-training recipe, without separately isolating data diversity from extra training or evaluating all 15 collection robots.

Complete original PDF figure S5.F8.
Fig. 8: Case study of the impact of RTML on data quality and model performance.

V-C Impact of Hierarchical Capability Pyramid

Keep the π0 architecture/learned parameters unchanged for the hierarchy integration and enrich its language input with task structure, manually defined subtasks, rule-based transitions and historical motion labels. Appendix E makes clear that high-level flow comes from human demonstrations; the system is not discovering the hierarchy autonomously.

The authors report in-distribution success of 28%→43% and four Pass Bowl appearance/lighting/background variants of 22.5%→57.5%. Figure 7’s six individual hierarchy bars are 20,70,90,70,0,30%, whose unweighted mean is 46.7%, not the plotted/reported 43%; no weighting rule is given. Peach Box remains 0% for both methods, qualifying the statement that every task improves.

Pass Bowl’s in-distribution 40%→70% and color-change 20%→50% examples illustrate possible benefits. Figure 8 depicts hierarchy case studies, although its caption calls them RTML effects. Selected visual successes do not establish causal contributions of each hierarchy level separately.

Complete original PDF figure S5.F9.
Fig. 9: Impact of RTML on data quality and model performance. (a) RTML filtering effect on two tasks. (b) Failure cause analysis from phase-wise and metric-wise perspectives. (c) Model performance improvement of Unitree G1edu-u3 + GR00T N1.5 under 3 fine-tuning settings.

V-D Impact of RTML

RTML retains 204/298 Grape Plate and 634/997 Bread Bowl trajectories: 838/1,295 overall, or 35.3% rejected by the rules. Grasp phases account for 52.7% of flagged phase failures, and velocity constraints for 46.2% of metric failures. These diagnose rule violations in the two tasks, not the failure distribution of the whole dataset.

GR00T trained on raw, globally filtered and globally-plus-phase-filtered data reports rounded average success of 72%,75%,88%. The last difference is 16 percentage points or 22.2% relative to the reported raw average. Exact task bars are 75/70%,75/75%,90/85%; their unweighted means are 72.5%,75%,87.5%. Filtering helps in this comparison, while data volume and selection change together.

VI Conclusion, Limitations and Future Work

RoboCOIN contributes a multi-platform bimanual resource and supporting processing conventions. Teleoperator biases, costly subjective annotation and task-specific expert constraints limit scale and generalization. Current tests support selected adaptation and conditioning/filtering benefits; more standardized splits, uncertainty reporting and broader deployment tests would be needed for stronger claims.

Appendix A Action and Object Taxonomy

Table III preserves all 39 canonical verbs, synonyms and definitions across general manipulation, object-state changes, object-relation changes and task-specific actions. Synonyms support instruction variation, while distinctions such as picking versus holding give annotators a common vocabulary. These conventions structure data; they do not ensure every real action has a uniquely unambiguous verbal label.

TABLE III: Robotic Manipulation Action Glossary
  Category   Verb   Synonyms   Definition
  General Manipulation   pick   grab, take, get   Grasp and lift an object from its position
  grasp   grip, clutch, hold   Firmly hold an object without positional change
  place   put, set, position   Put an object onto a surface or target location
  pass   hand over, give   Transfer an object between two hands/end-effectors
  rotate   twist, turn, revolve   Change the angular orientation of an object
  push   slide, shove   Move an object by applying forward force
  pull   drag, tug   Draw an object toward the agent
  lift   raise, hoist   Move an object upward to a higher position
  store   stow, deposit   Place an object inside a closed/semi-enclosed container
  Object State Change   press   depress, push down   Apply continuous force to deform/activate an object
  click   tap, press briefly   Apply momentary force to actuate a button/switch
  open   unfold, unzip   Switch an object from closed to accessible state
  unzip   –   Open a zippered fastener
  unfold   spread out, expand   Flatten a folded or compact object
  close   shut, zip   Switch an object from open to closed state
  fold   tuck, collapse   Bend an object into a compact/layered form
  unscrew   loosen, twist off   Release a lid/cap by counterclockwise rotation
  Object Relation Change   stack   pile, heap   Arrange objects vertically in layers
  arrange   line up, organize   Place objects in an orderly spatial pattern
  insert   put in, slot   Place one object into another structure
  plug in   connect, attach   Insert an electrical plug into a power socket
  remove   pull out, extract   Take an object out of a space/connection
  unplug   disconnect   Remove an electrical plug from a socket
  assemble   construct, build   Combine parts into a functional unit
  stick   paste, attach   Affix an object to a surface via adhesion
  swap   exchange, switch   Exchange positions of two objects
  Task-Specific Actions   wipe   scrub, clean   Clean a surface via rubbing motion
  erase   remove marks   Eliminate traces or residues from a surface
  rinse   wash, flush   Clean an object using flowing water
  turn on   activate, start   Power on an electrical/mechanical device
  turn off   deactivate, stop   Power off an electrical/mechanical device
  pour   transfer, spill   Move liquid from one container to another
  scoop   ladle, dig out   Lift material using a spoon or scooping tool
  stir   mix, blend   Homogenize liquid/particulate matter
  cut   chop, slice, split   Separate an object using a sharp tool
  play   perform   Operate a musical instrument
  make   build, create   Construct an object or prepare food
  spell   recite letters   Form words by sequencing individual characters
  write   inscribe, jot   Produce characters on a surface

Appendix B RTML Specification

Listing 1 preserves the complete 125-line RTML example over two PDF pages. Global linear-speed maximum/mean limits are 0.5/0.3 m/s and maximum acceleration is 12 m/s². Six local stages move the bowl, grasp/place long bread, grasp/place round bread, and end. Stage rules specify active-arm workspace, speed statistics, idle-arm motion, orientation deviation/variance and duration.

For example, moving the bowl confines the right arm to [0.10,−0.40,0.10]–[0.25,−0.20,0.30] m, while bread grasps use 2–8 seconds. These example thresholds are task-specific specifications, not universal robot safety limits. The listing’s exact source text is also retained for selection/copying.

Appendix C Annotation Pipeline

Scene annotation: detections and normalized boxes become language descriptions; the appendix illustrates DeepSeek R1, while the main bibliography cites a DeepSeek V3 report. Segmentation: stationary intervals, gripper transitions and task completion propose boundaries, with humans checking boundaries and labels in real time. Frame annotation: displacement, a derived “velocity,” and acceleration windows feed discrete labels for both arms.

The four source equations are preserved exactly. The printed velocity is ‖mt−mt−1‖/n after mt=xt−xt−n; it is not standard speed and becomes zero for constant linear motion. Neither that expression nor the acceleration explicitly converts frames to elapsed seconds. Implementation and threshold units therefore require care before interpreting these labels as physical velocity/acceleration.

The example labels speed below 0.1 stationary, 0.1–0.5 slow, and above 0.5 fast. Human checking reports 84.2% scene-annotation accuracy, with 11.2% missed objects,4.4% localization errors and0.2% extra objects. Frame labels are not manually verified; accurate pose inputs alone do not validate the derivative scheme or semantic thresholds.

Complete original PDF figure A3.F10.
Fig. 10: Annotation Pipeline Design. (a) Trajectory-Level Annotation Pipeline. (b) Segment-Level Annotation Pipeline. (c) Frame-Level Annotation Pipeline.
Equations
mt=xt−xt−n,m_{t}=x_{t}-x_{t-n},
vt=‖mt−mt−1‖2n.v_{t}=\frac{\|m_{t}-m_{t-1}\|_{2}}{n}.
at=vt−vt−nn.a_{t}=\frac{v_{t}-v_{t-n}}{n}.
Vel={Stationary,v<vel_slow_thresh,Slow,vel_slow_thresh≤v≤vel_fast_thresh,Fast,v>vel_fast_thresh.\text{Vel}=\begin{cases}\text{Stationary},&v<\text{vel\_slow\_thresh},\\ \text{Slow},&\text{vel\_slow\_thresh}\leq v\leq\text{vel\_fast\_thresh},\\ \text{Fast},&v>\text{vel\_fast\_thresh}.\end{cases}
TABLE IV: Simulation task settings.
Task Trajectories Example Instruction
Blocks Ranking RGB 50 Arrange the red block, the green block, and the blue block from left to right in a row.
Grab Roller 50 Take hold of the roller using your arms.
Handover Block 50 Pass the red block from the left to the right.
Handover Mic 50 Pick the microphone and transfer it to the other arm.
Hanging Mug 50 Use the left arm to grab the mug, rotate it, set it down, then hang the mug onto the rack.
Lift Pot 50 Hold the pot firmly, then lift.
Place Bread Basket 50 Pick up the first bread and put it in the basket.
Place Bread Skillet 50 Grab the bread and place it inside the skillet.
Place Burger Fries 50 Use both arms to move the hamburg and the frenchfries to the tray.
Place Can Basket 50 Place the can in the basket and then lift the basket.
Place Can Plasticbox 50 Use both arms to move the left can and right can into the plasticbox.
Place Dual Shoes 50 Put two shoes, tips to the left, into the shoebox.
Put Object Cabinet 50 Open the cabinet’s drawer and place the object inside it.
Stack Blocks Two 50 Shift the red block and the green block to the center and stack the green block on the red block.
Stack Bowls Two 50 Lift the first bowl, put it on the table, and stack the second bowl above.
TABLE V: Real-world task settings.
Task Trajectories Example Instruction
Block Basket 50 Put the block into the basket.
Peach Drawer 50 Put the peach in the drawer and close it.
Towel Basket 50 Put the towel into the basket.
Pass Bowl 50 Pass the bowl from the left to the right.
Peach Box 50 Open the box and put the peach in it.
Fold Towel 50 Fold the towel.
Grape Plate 298 Pick the grapes and place them on the plate.
Bread Bowl 997 Pick the bread and put it in the bowl.

Appendix D Experimental Task Setups

The simulation uses clean scenes, 0.8 m inter-arm spacing and 50 demonstrations per available task. Most objects are rigid; the appendix’s “deformable” examples are limited articulated mechanisms, not broad cloth-like simulation. Table IV lists 15 tasks although the suite and Figure 11 show 16, omitting Scan Object from the table.

Hardware includes six hierarchy tasks with 50 demonstrations each and two filtering tasks with 298/997 trajectories. Figure 11 retains every simulation and hardware sequence; Figure 12 retains original/strong/weak lighting. Four OOD Pass Bowl variants additionally change background or object color. These controlled variations are narrower than arbitrary environment transfer.

Complete original PDF figure A4.F11.
Fig. 11: Example task setups for (a) simulation environment, and (b) real-world environment.
Complete original PDF figure A4.F12.
Fig. 12: Out-of-distribution lighting variations in the real-world task setup, including (Left) original lighting, (Middle) strong lighting, and (Right) weak lighting.
Complete original PDF figure A4.F13.
Fig. 13: Hierarchical Capability Pyramid Integration. (a) Vision-Language-Action (VLA) Baseline Model Implementation. The model directly maps multi-modal inputs (state, observation, instruction) to primitive actions without explicit hierarchical knowledge integration. (b) Hierarchical Integration Implementation. The framework incorporates hierarchical annotations into a structured decision-making pipeline, enhancing action generation with multi-level task knowledge and motion constraints.

Appendix E Hierarchical Capability Pyramid Integration

The baseline maps state, image and instruction to actions. Hierarchical integration adds a human-demonstrated overall flow, manually predefined atomic subtasks, rule-based switching based on gripper/task progress, and thresholded descriptors from past poses. It changes conditioning without redesigning the VLA architecture or adding a learned planning module.

The appendix explicitly presents this as a proof of concept. Natural-language motion constraints can guide a policy but do not enforce hard kinematic feasibility; the method’s benefits should be interpreted alongside the human-provided priors and the formula limitations above.

Listing 1: RTML Example ”pull bowl storage bread”
Complete original RTML Listing1, part1of2; source lines1–72Complete original RTML Listing1, part2of2; source lines73–125
Selectable source YAML
# RTML V1.0
task:
  id: "pull_bowl_storage_bread"

  # Global constraints
  global_constraints:
    velocity:
      linear:
        max: 0.5          # m/s
        mean_max: 0.3     # m/s
    acceleration:
      linear:
        max: 12.0          # m/s2

  # Local stage constraints
  stages:
    - id: "move_bowl_right"
      match_subtask: "Move the pink bowl to the center of table with right hand"
      constraints:
        workspace:
          right:
            min: [0.10, -0.40, 0.10]
            max: [0.25, -0.20, 0.30]
        velocity:
          linear:
            mean_max: 0.10
            std_max: 0.08
        idle_arm:
          arm: "left"
          velocity_linear_mean_max: 0.05
        temporal:
          duration_min: 2.0
          duration_max: 6.0

    - id: "grasp_long_bread_left"
      match_subtask: "Grasp the long bread with left hand"
      constraints:
        workspace:
          left:
            min: [0.05, -0.05, -0.05]
            max: [0.25, 0.35, 0.20]
        orientation:
          left:
            angular_mean_deviation_max: 0.8
            std_max: [0.5, 0.5, 0.8]
            angular_variance_max: 0.15
        velocity:
          linear:
            mean_max: 0.12
            std_max: 0.10
        idle_arm:
          arm: "right"
          velocity_linear_mean_max: 0.05
        temporal:
          duration_min: 2.0
          duration_max: 8.0

    - id: "place_long_bread_in_bowl"
      match_subtask: "Place the long bread in pink bowl with left hand"
      constraints:
        workspace:
          left:
            min: [0.05, -0.05, -0.05]
            max: [0.25, 0.35, 0.20]
        velocity:
          linear:
            mean_max: 0.15
            std_max: 0.15
        idle_arm:
          arm: "right"
          velocity_linear_mean_max: 0.05
        temporal:
          duration_min: 1.0
          duration_max: 4.0

    - id: "grasp_round_bread_left"
      match_subtask: "Grasp the round bread with left hand"
      constraints:
        workspace:
          left:
            min: [0.05, 0.00, -0.05]
            max: [0.25, 0.35, 0.20]
        orientation:
          left:
            angular_mean_deviation_max: 0.5
            std_max: [0.5, 0.5, 0.5]
            angular_variance_max: 0.15
        velocity:
          linear:
            mean_max: 0.12
            std_max: 0.10
        idle_arm:
          arm: "right"
          velocity_linear_mean_max: 0.05
        temporal:
          duration_min: 2.0
          duration_max: 8.0

    - id: "place_round_bread_in_bowl"
      match_subtask: "Place the round bread in pink bowl with left hand"
      constraints:
        workspace:
          left:
            min: [0.05, 0.00, -0.05]
            max: [0.25, 0.35, 0.20]
        velocity:
          linear:
            mean_max: 0.15
            std_max: 0.15
        idle_arm:
          arm: "right"
          velocity_linear_mean_max: 0.05
        temporal:
          duration_min: 1.0
          duration_max: 4.0

    - id: "End"
      match_subtask: "End"
      constraints:
        velocity:
          linear:
            mean_max: 0.12
            std_max: 0.12
        temporal:
          duration_max: 6.0
Concept demonstration

Open image ↗