Overview
RoboCOIN combines teleoperated bimanual demonstrations, annotations at three temporal scales, and the CoRobot data-processing interface. The authors report an open release; the overview retains its platform and task montage.
At a glance
RoboCOIN reports over 180,000 demonstrations from 15 platforms, covering 421 tasks, 16 scenarios, 39 action types and 432 object types. Its hierarchy links trajectory-level scene/task descriptions, segment-level subtasks and frame-level motion labels. CoRobot adds expert-rule trajectory filtering, assisted annotation and common hardware/data interfaces. Experiments support useful dataset post-training, enriched language inputs and phase-aware filtering in selected tasks; they do not establish universal cross-robot transfer or fully autonomous hierarchical reasoning.
I Introduction
Bimanual learning must handle both coordination between arms and differences in robot morphology, sensors and hands. RoboCOIN addresses these with diverse teleoperated data and explicit task structure, while CoRobot standardizes processing and reuse. This edition follows all 22 pages of arXiv v3, dated 21 May 2026; the catalog retains the original 2025 posting date.
The three experimental claims should be separated: extra post-training improves later adaptation to two simulated target robots; hierarchical language conditioning improves selected real-world tasks; expert trajectory filters improve two G1 tasks. The headline percentage gains are relative changes from their respective baselines, not percentage-point improvements or one combined score.
II Related Work
Datasets. Open X-Embodiment, RoboMIND and single-platform bimanual datasets provide different coverage and annotation structures. RoboCOIN emphasizes multi-platform bimanual collection plus temporal hierarchy. Table II is the paper’s historical comparison, not a continually updated inventory.
Policies. ACT and Diffusion Policy motivate learned action generation, while generalist VLAs use larger multimodal backbones. Experiments here focus on π0 and GR00T N1.5. Broad claims that other policies lack task understanding should be read as motivation, rather than conclusions from direct reasoning evaluations of every listed model.
| Type | Name | Arm DoF | Camera Configuration | Gripper Type | Teleoperation Method |
| Dual-Arm | Agilex Cobot Magic [41] | 26 | Head + Wrist | Parallel Gripper | Isomorphic Arm |
| Agilex Split ALOHA [41] | 26 | Head + Wrist | Parallel Gripper | Isomorphic Arm | |
| Galaxea R1 Lite [16] | 26 | Head + Wrist | Parallel Gripper | Isomorphic Arm | |
| Half-Humanoid | Realman RMC-AIDA-L [44] | 27 | Head + Wrist | Parallel Gripper | Exoskeleton |
| Agibot G1 [1] | 27 | Head + Wrist + Back | Parallel Gripper | Virtual Reality | |
| AI2 AlphaBot 2 [42] | 27 | Head + Wrist + Chest | Dexterous Hand | Virtual Reality / Isomorphic Arm | |
| AI2 AlphaBot 1s [42] | 27 | Head + Wrist + Chest | Dexterous Hand | Virtual Reality / Isomorphic Arm | |
| Galbot G1 [17] | 27 | Head + Wrist | Parallel Gripper | Motion Capture | |
| Tianqing A2 [49] | 27 | Head + Wrist + Back | Parallel Gripper | Virtual Reality | |
| Realman Rs-02 [44] | 27 | Head + Wrist | Parallel Gripper | Exoskeleton | |
| Realman Rs-01 [44] | 27 | Head + Wrist | Parallel Gripper | Exoskeleton | |
| Airbot MMK2 [2] | 26 | Head + Wrist + Third-Person | Dexterous Hand | Virtual Reality | |
| Leju Kuavo 4 LB [43] | 27 | Head + Wrist | Dexterous Hand | Virtual Reality | |
| Humanoid | Leju Kuavo 4 Pro [43] | 27 | Head + Wrist | Dexterous Hand | Virtual Reality |
| Unitree G1edu-u3 [50] | 27 | Head + Wrist | Dexterous Hand | Motion Capture / Exoskeleton |
III RoboCOIN Dataset
The dataset joins hardware diversity with synchronized annotations and a two-axis task taxonomy. Its demonstrations are human-teleoperated, rather than autonomous rollouts across all advertised scenarios.
III-A Data Collection and Storage
Table I preserves all 15 robot configurations: dual-arm, half-humanoid and humanoid systems with parallel grippers or dexterous hands. Collection uses isomorphic leader–follower arms, exoskeletons, virtual reality or motion capture, with camera placement varying by platform.
Joint states use radians, grippers are normalized to [0,1], and end-effector positions/Euler angles use meters/radians. Coordinates are right-handed with x forward, y left and z up; timestamp alignment synchronizes observations and kinematics. This common representation eases ingestion, but does not by itself make heterogeneous action spaces interchangeable.
III-B Statistics and Taxonomy
The reported demonstration mix is 33.2% dual-arm, 49.0% half-humanoid and 17.8% humanoid; scenarios are 47.1% residential, 18.6% commercial and 34.3% working settings. Low/high collaboration accounts for 58.9%/41.1%, and rigid/hinged/deformable objects for 83.9%/4.2%/11.9%. These uneven distributions matter when interpreting coverage.
Collaboration distinguishes largely sequential from closely synchronized arm use; object flexibility forms the second task axis. Figure 3 retains all detailed distributions. Its two humanoid platform labels, 8.1% and 7.6%, do not sum to the stated 17.8% category share, so the illustrated percentages are retained without silently reconciling them.
III-C Hierarchical Capability Pyramid
Trajectory level: describe the scene, objects and overall objective. Segment level: align atomic subtasks to time, allowing overlaps between arms and marking exceptions such as failed grasps. Frame level: describe end-effector motion and gripper/hand states. Figure 4 shows their temporal alignment on a peach-and-basket task. The hierarchy supplies information for learning; its presence alone does not demonstrate that a policy reasons correctly at every level.
| Dataset | Arm | Embodiments | Trajectories | Tasks | Actions | Dexterous | Annotation | Collection Method |
| Pinto and Gupta [38] | Dual | 1 | 50k | N/A | 1 | ✗ | No | Scripted |
| RoboNet [12] | Single | 1 | 162k | N/A | N/A | ✗ | No | Scripted |
| MT-Opt [23] | Single | 1 | 800k | 12 | 1 | ✗ | No | Scripted |
| BridgeData [13] | Single | 1 | 7.2k | 71 | 4 | ✗ | No | Human Teleoperation |
| BC-Z [20] | Single | 1 | 26k | 100 | 3 | ✗ | No | Human Teleoperation |
| RH20T [15] | Single | 1 | 13k | 140 | 33 | ✗ | No | Human Teleoperation |
| RoboSet [4] | Single | 1 | 98k | 38 | 6 | ✗ | No | 30% Human / 70% Scripted |
| BridgeData V2 [51] | Single | 1 | 60k | N/A | 13 | ✗ | No | 85% Human / 15% Scripted |
| DROID [24] | Single | 1 | 76k | N/A | 86 | ✗ | No | Human Teleoperation |
| Open X-Embodiment† [36] | Single+Dual | 22 | 1.4M | 160k | 217 | ✗ | No | Dataset Aggregation |
| RoboMIND [53] | Single+Dual | 4 | 107k | 479 | 38 | ✓ | Flat | Human Teleoperation |
| AgiBot World Beta [6] | Dual | 1 | 1M | 217 | 87 | ✗ | Flat | Human Teleoperation |
| Open Galaxea [22] | Dual | 1 | 50k | 150 | 58 | ✓ | Flat | Human Teleoperation |
| RoboCOIN (Ours) | Dual | 15 | 180K+ | 421 | 39 | ✓ | Hierarchical | Human Teleoperation |
III-D Comparison with Existing Datasets
Table II compares embodiment counts, trajectory/task/action counts, dexterous hardware and annotation type. RoboCOIN’s distinguishing combination is broad bimanual hardware coverage and hierarchical annotations. The prose describes Open X-Embodiment as single-arm in Related Work, while the table correctly labels it “Single+Dual”; the claim should be limited to comparatively sparse bimanual coverage, not complete absence.
IV CoRobot Data Processing Pipeline
CoRobot combines rule-based quality assessment, assisted hierarchical annotation and a common software interface. The components address dataset preparation and deployment engineering rather than constituting a new end-to-end robot policy.
IV-A Robot Trajectory Markup Language Evaluation
RTML encodes expert constraints in YAML. Global rules bound workspace, velocity, acceleration and duration; stage-specific rules add overrides, orientation tolerances and timing limits for actions such as approach, grasp and place. An evaluator produces diagnostic reports and a quality score.
Filtering therefore means compliance with selected rules. It is not an independently verified label that every rejected trajectory is physically invalid or that every accepted one is safe and successful. Choosing constraints requires task and embodiment knowledge.
IV-B Annotation Pipeline
Object detectors such as Grounding DINO feed locations to a language model for scene descriptions. Gripper events and task progress propose segment boundaries, which humans adjust and label. Cascaded pose windows produce frame-level motion descriptors, then thresholds turn continuous values into text. Human review makes the overall process semi-automatic; the appendix reports residual scene-description errors and no manual verification of frame labels.
IV-C Unified Robotic Interface
Built on LeRobot, the interface wraps vendor SDKs and ROS/ROS 2 support, normalizes units and coordinates, and groups reusable data subsets by robot, task and environment. The source advertises installation through pip install robocoin and links its CoRobot repository. This describes the supplied interface design; no installation or hardware compatibility test is part of the paper evidence reviewed here.
V Experiments and Analysis
Three questions organize evaluation: whether RoboCOIN helps target-robot adaptation, whether hierarchical inputs improve task execution, and whether RTML filtering improves policy training. The experimental subsets are much smaller than the full collection.
V-A Experiment Setup
Simulation: use RoboTwin 2.0 on ARX-X5 and Franka Panda, absent from RoboCOIN collection. Of 16 intended tasks, trajectory generation succeeds for 14/11 respectively; missing tasks are marked N/A. Target-task fine-tuning uses 50 demonstrations per task, so this is adaptation to new embodiments rather than zero-shot execution.
Hardware: Realman RMC-AIDA-L evaluates the hierarchy, and G1edu-u3 evaluates filtering. π0 uses LoRA, AdamW, learning rate 2.5×10⁻⁵, batch 32 and 30,000 steps. GR00T N1.5 updates diffusion/projector components with learning rate 10⁻⁴, batch 24 and 10,000 steps. The appendix clarifies that its two filtering tasks have 298/997 source trajectories, despite the main text’s general statement of 50 per task.
Success conditions include lifting a pot clear, placing both shoes in a box, closing a peach-containing drawer, and completing a bowl handover/placement. The supplied manuscript does not clearly state evaluation trial counts, repeated training seeds or confidence intervals, limiting assessment of small differences.
V-B Impact of Multi-Embodiment Adaptability
Post-train π0 on a random 10,000-demonstration RoboCOIN subset, then fine-tune on target RoboTwin tasks. Reported mean success rises from 17.3% to 30.4% on ARX-X5 and 8.5% to 26.6% on Panda: gains of 13.1/18.1 percentage points, or 75.7%/212.9% relative.
High-collaboration tasks show larger gains in these tests, but absolute success remains modest. Dual-shoe placement rises from zero to only 9%/3%; this is evidence of some successful executions, not reliable task mastery. The comparison establishes the value of this additional post-training recipe, without separately isolating data diversity from extra training or evaluating all 15 collection robots.
V-C Impact of Hierarchical Capability Pyramid
Keep the π0 architecture/learned parameters unchanged for the hierarchy integration and enrich its language input with task structure, manually defined subtasks, rule-based transitions and historical motion labels. Appendix E makes clear that high-level flow comes from human demonstrations; the system is not discovering the hierarchy autonomously.
The authors report in-distribution success of 28%→43% and four Pass Bowl appearance/lighting/background variants of 22.5%→57.5%. Figure 7’s six individual hierarchy bars are 20,70,90,70,0,30%, whose unweighted mean is 46.7%, not the plotted/reported 43%; no weighting rule is given. Peach Box remains 0% for both methods, qualifying the statement that every task improves.
Pass Bowl’s in-distribution 40%→70% and color-change 20%→50% examples illustrate possible benefits. Figure 8 depicts hierarchy case studies, although its caption calls them RTML effects. Selected visual successes do not establish causal contributions of each hierarchy level separately.
V-D Impact of RTML
RTML retains 204/298 Grape Plate and 634/997 Bread Bowl trajectories: 838/1,295 overall, or 35.3% rejected by the rules. Grasp phases account for 52.7% of flagged phase failures, and velocity constraints for 46.2% of metric failures. These diagnose rule violations in the two tasks, not the failure distribution of the whole dataset.
GR00T trained on raw, globally filtered and globally-plus-phase-filtered data reports rounded average success of 72%,75%,88%. The last difference is 16 percentage points or 22.2% relative to the reported raw average. Exact task bars are 75/70%,75/75%,90/85%; their unweighted means are 72.5%,75%,87.5%. Filtering helps in this comparison, while data volume and selection change together.
VI Conclusion, Limitations and Future Work
RoboCOIN contributes a multi-platform bimanual resource and supporting processing conventions. Teleoperator biases, costly subjective annotation and task-specific expert constraints limit scale and generalization. Current tests support selected adaptation and conditioning/filtering benefits; more standardized splits, uncertainty reporting and broader deployment tests would be needed for stronger claims.
Appendix A Action and Object Taxonomy
Table III preserves all 39 canonical verbs, synonyms and definitions across general manipulation, object-state changes, object-relation changes and task-specific actions. Synonyms support instruction variation, while distinctions such as picking versus holding give annotators a common vocabulary. These conventions structure data; they do not ensure every real action has a uniquely unambiguous verbal label.
| Category | Verb | Synonyms | Definition |
| General Manipulation | pick | grab, take, get | Grasp and lift an object from its position |
| grasp | grip, clutch, hold | Firmly hold an object without positional change | |
| place | put, set, position | Put an object onto a surface or target location | |
| pass | hand over, give | Transfer an object between two hands/end-effectors | |
| rotate | twist, turn, revolve | Change the angular orientation of an object | |
| push | slide, shove | Move an object by applying forward force | |
| pull | drag, tug | Draw an object toward the agent | |
| lift | raise, hoist | Move an object upward to a higher position | |
| store | stow, deposit | Place an object inside a closed/semi-enclosed container | |
| Object State Change | press | depress, push down | Apply continuous force to deform/activate an object |
| click | tap, press briefly | Apply momentary force to actuate a button/switch | |
| open | unfold, unzip | Switch an object from closed to accessible state | |
| unzip | – | Open a zippered fastener | |
| unfold | spread out, expand | Flatten a folded or compact object | |
| close | shut, zip | Switch an object from open to closed state | |
| fold | tuck, collapse | Bend an object into a compact/layered form | |
| unscrew | loosen, twist off | Release a lid/cap by counterclockwise rotation | |
| Object Relation Change | stack | pile, heap | Arrange objects vertically in layers |
| arrange | line up, organize | Place objects in an orderly spatial pattern | |
| insert | put in, slot | Place one object into another structure | |
| plug in | connect, attach | Insert an electrical plug into a power socket | |
| remove | pull out, extract | Take an object out of a space/connection | |
| unplug | disconnect | Remove an electrical plug from a socket | |
| assemble | construct, build | Combine parts into a functional unit | |
| stick | paste, attach | Affix an object to a surface via adhesion | |
| swap | exchange, switch | Exchange positions of two objects | |
| Task-Specific Actions | wipe | scrub, clean | Clean a surface via rubbing motion |
| erase | remove marks | Eliminate traces or residues from a surface | |
| rinse | wash, flush | Clean an object using flowing water | |
| turn on | activate, start | Power on an electrical/mechanical device | |
| turn off | deactivate, stop | Power off an electrical/mechanical device | |
| pour | transfer, spill | Move liquid from one container to another | |
| scoop | ladle, dig out | Lift material using a spoon or scooping tool | |
| stir | mix, blend | Homogenize liquid/particulate matter | |
| cut | chop, slice, split | Separate an object using a sharp tool | |
| play | perform | Operate a musical instrument | |
| make | build, create | Construct an object or prepare food | |
| spell | recite letters | Form words by sequencing individual characters | |
| write | inscribe, jot | Produce characters on a surface |
Appendix B RTML Specification
Listing 1 preserves the complete 125-line RTML example over two PDF pages. Global linear-speed maximum/mean limits are 0.5/0.3 m/s and maximum acceleration is 12 m/s². Six local stages move the bowl, grasp/place long bread, grasp/place round bread, and end. Stage rules specify active-arm workspace, speed statistics, idle-arm motion, orientation deviation/variance and duration.
For example, moving the bowl confines the right arm to [0.10,−0.40,0.10]–[0.25,−0.20,0.30] m, while bread grasps use 2–8 seconds. These example thresholds are task-specific specifications, not universal robot safety limits. The listing’s exact source text is also retained for selection/copying.
Appendix C Annotation Pipeline
Scene annotation: detections and normalized boxes become language descriptions; the appendix illustrates DeepSeek R1, while the main bibliography cites a DeepSeek V3 report. Segmentation: stationary intervals, gripper transitions and task completion propose boundaries, with humans checking boundaries and labels in real time. Frame annotation: displacement, a derived “velocity,” and acceleration windows feed discrete labels for both arms.
The four source equations are preserved exactly. The printed velocity is ‖mt−mt−1‖/n after mt=xt−xt−n; it is not standard speed and becomes zero for constant linear motion. Neither that expression nor the acceleration explicitly converts frames to elapsed seconds. Implementation and threshold units therefore require care before interpreting these labels as physical velocity/acceleration.
The example labels speed below 0.1 stationary, 0.1–0.5 slow, and above 0.5 fast. Human checking reports 84.2% scene-annotation accuracy, with 11.2% missed objects,4.4% localization errors and0.2% extra objects. Frame labels are not manually verified; accurate pose inputs alone do not validate the derivative scheme or semantic thresholds.
Equations
| Task | Trajectories | Example Instruction |
| Blocks Ranking RGB | 50 | Arrange the red block, the green block, and the blue block from left to right in a row. |
| Grab Roller | 50 | Take hold of the roller using your arms. |
| Handover Block | 50 | Pass the red block from the left to the right. |
| Handover Mic | 50 | Pick the microphone and transfer it to the other arm. |
| Hanging Mug | 50 | Use the left arm to grab the mug, rotate it, set it down, then hang the mug onto the rack. |
| Lift Pot | 50 | Hold the pot firmly, then lift. |
| Place Bread Basket | 50 | Pick up the first bread and put it in the basket. |
| Place Bread Skillet | 50 | Grab the bread and place it inside the skillet. |
| Place Burger Fries | 50 | Use both arms to move the hamburg and the frenchfries to the tray. |
| Place Can Basket | 50 | Place the can in the basket and then lift the basket. |
| Place Can Plasticbox | 50 | Use both arms to move the left can and right can into the plasticbox. |
| Place Dual Shoes | 50 | Put two shoes, tips to the left, into the shoebox. |
| Put Object Cabinet | 50 | Open the cabinet’s drawer and place the object inside it. |
| Stack Blocks Two | 50 | Shift the red block and the green block to the center and stack the green block on the red block. |
| Stack Bowls Two | 50 | Lift the first bowl, put it on the table, and stack the second bowl above. |
| Task | Trajectories | Example Instruction |
| Block Basket | 50 | Put the block into the basket. |
| Peach Drawer | 50 | Put the peach in the drawer and close it. |
| Towel Basket | 50 | Put the towel into the basket. |
| Pass Bowl | 50 | Pass the bowl from the left to the right. |
| Peach Box | 50 | Open the box and put the peach in it. |
| Fold Towel | 50 | Fold the towel. |
| Grape Plate | 298 | Pick the grapes and place them on the plate. |
| Bread Bowl | 997 | Pick the bread and put it in the bowl. |
Appendix D Experimental Task Setups
The simulation uses clean scenes, 0.8 m inter-arm spacing and 50 demonstrations per available task. Most objects are rigid; the appendix’s “deformable” examples are limited articulated mechanisms, not broad cloth-like simulation. Table IV lists 15 tasks although the suite and Figure 11 show 16, omitting Scan Object from the table.
Hardware includes six hierarchy tasks with 50 demonstrations each and two filtering tasks with 298/997 trajectories. Figure 11 retains every simulation and hardware sequence; Figure 12 retains original/strong/weak lighting. Four OOD Pass Bowl variants additionally change background or object color. These controlled variations are narrower than arbitrary environment transfer.
Appendix E Hierarchical Capability Pyramid Integration
The baseline maps state, image and instruction to actions. Hierarchical integration adds a human-demonstrated overall flow, manually predefined atomic subtasks, rule-based switching based on gripper/task progress, and thresholded descriptors from past poses. It changes conditioning without redesigning the VLA architecture or adding a learned planning module.
The appendix explicitly presents this as a proof of concept. Natural-language motion constraints can guide a policy but do not enforce hard kinematic feasibility; the method’s benefits should be interpreted alongside the human-provided priors and the formula limitations above.
Selectable source YAML
# RTML V1.0
task:
id: "pull_bowl_storage_bread"
# Global constraints
global_constraints:
velocity:
linear:
max: 0.5 # m/s
mean_max: 0.3 # m/s
acceleration:
linear:
max: 12.0 # m/s2
# Local stage constraints
stages:
- id: "move_bowl_right"
match_subtask: "Move the pink bowl to the center of table with right hand"
constraints:
workspace:
right:
min: [0.10, -0.40, 0.10]
max: [0.25, -0.20, 0.30]
velocity:
linear:
mean_max: 0.10
std_max: 0.08
idle_arm:
arm: "left"
velocity_linear_mean_max: 0.05
temporal:
duration_min: 2.0
duration_max: 6.0
- id: "grasp_long_bread_left"
match_subtask: "Grasp the long bread with left hand"
constraints:
workspace:
left:
min: [0.05, -0.05, -0.05]
max: [0.25, 0.35, 0.20]
orientation:
left:
angular_mean_deviation_max: 0.8
std_max: [0.5, 0.5, 0.8]
angular_variance_max: 0.15
velocity:
linear:
mean_max: 0.12
std_max: 0.10
idle_arm:
arm: "right"
velocity_linear_mean_max: 0.05
temporal:
duration_min: 2.0
duration_max: 8.0
- id: "place_long_bread_in_bowl"
match_subtask: "Place the long bread in pink bowl with left hand"
constraints:
workspace:
left:
min: [0.05, -0.05, -0.05]
max: [0.25, 0.35, 0.20]
velocity:
linear:
mean_max: 0.15
std_max: 0.15
idle_arm:
arm: "right"
velocity_linear_mean_max: 0.05
temporal:
duration_min: 1.0
duration_max: 4.0
- id: "grasp_round_bread_left"
match_subtask: "Grasp the round bread with left hand"
constraints:
workspace:
left:
min: [0.05, 0.00, -0.05]
max: [0.25, 0.35, 0.20]
orientation:
left:
angular_mean_deviation_max: 0.5
std_max: [0.5, 0.5, 0.5]
angular_variance_max: 0.15
velocity:
linear:
mean_max: 0.12
std_max: 0.10
idle_arm:
arm: "right"
velocity_linear_mean_max: 0.05
temporal:
duration_min: 2.0
duration_max: 8.0
- id: "place_round_bread_in_bowl"
match_subtask: "Place the round bread in pink bowl with left hand"
constraints:
workspace:
left:
min: [0.05, 0.00, -0.05]
max: [0.25, 0.35, 0.20]
velocity:
linear:
mean_max: 0.15
std_max: 0.15
idle_arm:
arm: "right"
velocity_linear_mean_max: 0.05
temporal:
duration_min: 1.0
duration_max: 4.0
- id: "End"
match_subtask: "End"
constraints:
velocity:
linear:
mean_max: 0.12
std_max: 0.12
temporal:
duration_max: 6.0