Overview
The original report credits the BeingBeyond Team collectively; the twelve individual names in this library entry appear in its contributor appendix, reproduced with roles below. This edition covers all 44 pages of arXiv v1, posted 19 January 2026; the abstract page internally dates the report 20 January. Figure and appendix labels follow the PDF, including its missing HTML cover figure.
At a glance
Being-H0.5 combines human video, robot demonstrations and vision-language data in one physical action interface. Its flow-based action expert shares lower layers while routing upper layers to specialists; additional gating and asynchronous chunking address noisy context and inference delay.
A jointly adapted checkpoint controls five physical platforms. Simulation specialists score 98.9% on LIBERO and 53.9% on RoboCasa; a checkpoint trained on both scores 97.6% and 53.3%. Transfer to an unseen task–robot combination is qualitative and unquantified, and the masked-motion ablation contains an unresolved table/prose contradiction.
1 Introduction
The central proposal is to use human hand interaction as a common source of manipulation priors while reserving semantically aligned action slots for different robot bodies. This separates shared task structure from hardware-specific commands, although mappings and downstream robot demonstrations remain necessary.
UniHand-2.0 combines roughly 16,000 hours of human video, 14,000 hours of robot data and 5,000 equivalent hours of visual-language data, advertised as 400 million samples and 120 billion tokens. UniCraftor adds synchronized human capture; Mixture of Flow, slot-wise adaptation, reliability gating and asynchronous execution provide the model and deployment components.
The author project page is the report’s reference for resources. The paper describes or promises weights, training and deployment code, simulation scripts and a 1,000-GPU-hour recipe; those promises are distinct from a fully specified, reproduced training run. Claims of the largest or first system are historical author claims.
How can robotic intelligence master a new platform with limited data, much like a human learning a minor language?
2 Related Work
Earlier VLAs use discrete autoregression, diffusion/flow action heads or intermediate plans such as language, points and trajectories. Heterogeneous robot datasets expand hardware coverage but introduce inconsistent control formats and units.
Human-video approaches range from representation learning and latent actions to future-video prediction, geometric tracks and reconstructed hand motion. Being-H0.5 extends Being-H0 with a broader human–robot interface, while building on Open X-Embodiment and other robot corpora. The comparison motivates a representation and data recipe rather than proving every previous VLA is confined to one body.
3 UniHand-2.0: A Recipe for Large-Scale Human-Centric Learning
The recipe addresses limited data, narrow hardware coverage, scarce dexterous demonstrations and weak language supervision. It counts embodiments with more than ten hours of data and reports 30 robot types, 134 million human samples and over 400 million total samples.
The scale figures use different accounting units. Table 1 totals 13,817.4 robot hours, rounded to 14,000; visual-language data contributes 5,000 equivalent hours rather than recorded interaction time. Figure 4’s caption assigns 25.6B/45.7B/50.2B tokens to human/robot/vision-language data, while the graphic’s legend assigns 45.7B/50.2B/25.6B. The total is about 121.5B, but that source allocation conflict is unresolved. “35,000 hours” is therefore a mixed, rounded aggregate.
3.1 Human Demonstration Data
Public egocentric sources include Ego 4D, EPIC-KITCHENS and Egocentric-10K. HaWoR estimates MANO hand motion and camera poses; Gemini 2.5 supplies per-second instructions and ten-second intent descriptions. Human motion supervises generation, description and continuation.
Processing diversifies language, filters low-confidence or discontinuous motion using detection/DBA signals, removes non-manipulation clips, and mirrors samples to reduce handedness bias. These are estimated labels, not direct measurements of every contact or force. Filter thresholds, error distributions and a contamination audit are not reported.
| Robot Type | Camera Views | EEF Type | Source | Hours |
| Single-Arm Robots | ||||
| Franka | ego, 3rd2, wrist | Grp/Dex | Real/Sim | 2196.4 |
| Google Robot | ego, 3rd | Grp | Real/Sim | 1195.2 |
| Kuka-iiwa | ego, 3rd | Grp | Real/Sim | 338.2 |
| WidowX | 3rd2, wrist | Grp | Real/Sim | 305.2 |
| Xarm7 Robot | top, 3rd | Grp | Real/Sim | 594.4 |
| UR5E | 3rd, wrist | Grp | Real/Sim | 208.3 |
| Hello Stretch | ego, 3rd | Grp | Real | 270.3 |
| Kinova Jaco | 3rd3 | Grp | Real/Sim | 194.6 |
| Kinova Gen3 | 3rd3 | Grp | Sim | 197.4 |
| Sawyer | 3rd | Grp | Sim | 196.1 |
| Cobotta | 3rd | Grp | Real | 10.6 |
| DLR Sara | 3rd | Grp | Real | 12.1 |
| Willow Garage PR2 | ego | Grp | Real | 11.9 |
| RMC RM65 | 3rd | Dex | Real | 41.3 |
| Dual-Arm Robots | ||||
| RMC Aida L | high, wrist2 | Grp | Real | 325.7 |
| Galaxea R1 Lite | high, wrist2 | Grp | Real | 630.5 |
| Agilex Split ALOHA | high, wrist2 | Grp | Real/Sim | 1099.1 |
| Agilex Cobot Magic | high, wrist2 | Grp | Real | 399.2 |
| Piper | ego, wrist2 | Grp | Real/Sim | 904.5 |
| Portable Education Arm | ||||
| BeingBeyond D1 | ego | Dex | Real | 100.0 |
| SO101 | ego, wrist2 | Grp | Real | 194.4 |
| Half-Humanoid | ||||
| PND AdamU | ego | Dex | Real | 200.0 |
| Agibot-G1 | {ego, wrist}2 | Grp/Dex | Real/Sim | 2391.7 |
| Leju kuavo 4 LB | ego, wrist2 | Grp/Dex | Real | 1198.2 |
| AlphaBot 2 | ego, wrist2 | Grp | Real | 13.8 |
| Airbot MMK2 | 3rd, wrist2 | Grp | Real | 23.1 |
| Galbot G1 | wrist2 | Grp | Sim | 195.8 |
| Tianqing A2 | high, wrist2 | Grp | Real | 19.4 |
| Humanoid | ||||
| Unitree G1edu-u3 | ego, wrist2 | Grp/Dex | Real | 135.7 |
| Tiankung | top | Grp | Real | 214.3 |
| Total: 30 embodiments | 13817.4 | |||
3.2 Robot Manipulation Data
Robot data combines public real/simulated corpora with new Adam-U and D1 demonstrations. The pipeline deduplicates and downsamples frames to 30%, adds procedural simulation and inpainting augmentation, and standardizes 30 embodiments.
The text describes simulation as 26% of the total corpus, but Figure 4’s breakdown applies that fraction to the 13,817-hour robot corpus. It should not be read as 26% of all human, robot and language training. Table 1 retains each platform’s hours, views and end-effector type.
3.3 Visual-Text Understanding Data
Auxiliary data covers general image/video question answering, spatial grounding/affordances, and task decomposition. Examples include LLaVA/FineVision, RefCOCO/RoboPoint and ShareRobot/EO1.5M-QA. The purpose is to retain language and scene reasoning during action learning.
The reported 45.7B visual versus 30M text robot tokens imply about 1,523:1, although the prose rounds this to roughly 1,000:1. Adding language supervision is motivated by that imbalance; the paper does not separately measure preservation of each claimed reasoning capability.
4 UniCraftor: A System for Portable, Extensible, and Affordable Data Collection
UniCraftor uses head-mounted RealSense D435 RGB-D, five tabletop AprilTags for PnP camera poses, and a synchronized foot pedal for demonstrator-marked interaction events. Additional views can move while tags remain visible; the system collects over 200 hours across 43 tabletop tasks.
Post-processing inpaints tags with Grounded-SAM2/DiffuEraser, refines HaWoR hands using multiview depth, and generates Qwen 2.5-VL descriptions followed by human verification. Tactile sensing and mobile capture are planned extensions. Tag-based poses and pedal timing are supervision signals with measurement/human error, despite the report’s “ground truth” wording; calibration accuracy, event latency and system cost are not quantified.
5 Being-H0.5: A Foundational VLA Unifying Cross-Embodiment Control
The model combines a shared multimodal backbone, a physically organized state/action representation and action-specialized transformer computation.
5.1 Model Architecture
Initialized from InternVL3.5, the Mixture-of-Transformers architecture separates understanding and action parameters while sharing attention over a common sequence. Text uses next-token prediction, discrete hand motion uses masked prediction, and continuous control uses rectified flow.
The structure follows BAGEL-style modality specialization. Claimed planning and reasoning functions are not accompanied by a separate reasoning benchmark. Simulation uses a 2B backbone; complete active/total action-expert counts are not specified.
5.1.1 Unified State-Action Space
Fixed-length state and action vectors reserve slots for wrist/end-effector poses, joints, grippers, fingers and base commands. Human MANO wrist/finger parameters occupy the corresponding semantic slots; unused dimensions are zero. Cartesian commands use world-frame relative displacements, rotations use axis-angle, and joint positions use absolute radians.
Raw physical scales are retained with outlier filtering instead of statistical normalization. This is a designed interface, not automatic discovery of kinematic equivalence. Slot dimensions/mappings and filters are not fully tabulated; axis-angle also has representation ambiguities near 180° despite avoiding Euler gimbal lock.
5.1.2 Mixture of Flow
Mixture of Flow shares the action expert’s lower transformer layers and routes upper-layer computation to a sparse subset of specialized experts according to state and instruction. Shared primitives can support transfer while selected paths specialize to hardware or tasks.
The report does not give the expert count, routing top-k, balancing loss or complete parameter budget. Sparse activation reduces computation relative to using every expert, but does not itself prove edge-memory fit or conflict-free learning in shared layers. Orin NX feasibility is asserted without a latency/memory table.
5.2 Pre-Training: Human-Centric Robot Learning
Human traces supply varied interaction examples, robot trajectories anchor executable controls, and visual-language tasks maintain semantic context. All three are serialized through one interface, with losses attached only to available supervised segments.
5.2.1 Unified Sequence Modeling
Embodiment maps place raw state/action values into common slots. Each sequence contains tagged vision, text, state and/or action segments, organized as a query and an answer. Training supervises the answer while conditioning on the query. Equations 1–3 and the retained definitions specify these mappings and modality sets.
Equations 1–3
| (1) |
| (2) |
| (3) |
5.2.2 Human-Centric Multi-Task Objective
Motion generation predicts actions from visual, linguistic and state context. Motion description predicts text from observations and interaction traces; continuation predicts later motion from history. Robot-only and VQA samples instantiate the available modalities.
The joint objective weights text cross-entropy and action losses. This common serialization supports heterogeneous data, but the report does not provide the full mixture schedule or the loss weights needed to reproduce pretraining.
Equations 4–5
| (4) |
| (5) |
5.2.3 Hybrid Human Motion Representation
Each human motion chunk has continuous action targets and discrete tokens from a pretrained motion tokenizer. Flow matching regresses the velocity from Gaussian noise toward the target along a linear path; masked-token cross-entropy reconstructs selected codebook entries.
The intent is to combine precise continuous control with a stable discrete motion prior. Tokenizer/codebook details, masking ratio and loss weights are not specified here. The claimed empirical benefit requires caution because Table 8’s higher-is-better scores favor removing the masked objective.
Equations 6–8
| (6) |
| (7) |
| (8) |
Joint Serialization.
Both target segments attend to the same query, but cannot attend to each other, preventing direct copying between continuous and discrete labels. The 3×3 block gate in Equation 9 enforces that separation. Equation 10 gives both target segments the same positional origin after the query; each retains its own relative positions.
Equations 9–10
| (9) |
| (10) |
5.3 Post-Training: Towards Cross-Embodiment Adaptation
Post-training must adapt control to new hardware while preserving shared representations. The three mechanisms target slot-specific adaptation, uncertain conditioning features and mismatched inference/control timing.
5.3.1 Embodiment-Specific Adaptation
Embodiment-specific adaptation maintains an adapter bank indexed by semantic action slots. Only active-slot adapters update for a given robot; robots with shared components can share adapter subsets. Inactive-slot updates are explicitly zero in Equation 12.
This localizes some specialization, but shared slots and backbone parameters can still receive competing gradients. The equations describe parameter masking, not a guarantee of collision-free actions or universally conflict-free transfer.
Equations 11–12
| (11) |
| (12) |
5.3.2 Manifold-Preserving Gating
MPG modifies state/action suffix features while leaving the visual-language prefix unchanged. It compares projected, layer-normalized features with a pooled noise-free action anchor using random-projection sliced-Wasserstein discrepancy. An exponential gate shrinks as discrepancy rises; training stops gradients through that gate.
The gate scales the feature-dependent residual before a linear projection’s bias is added. A low gate therefore leaves the original features plus an ungated learned offset, rather than discarding unreliable observations. Training anchors use target actions; inference anchors use a previous prediction.
The stated variance comparison is a heuristic: removing bias from the gated term does not universally reduce its squared norm because vector cancellation is possible. The Jacobian approximation ignores gate derivatives and retains an identity path. These formulas do not prove robustness or constrain outputs to a valid physical manifold. Gate temperature, projection count and residual strength are unspecified.
Equations 13–16
| (13) |
| (14) |
| (15) |
| (16) |
5.3.3 Universal Async Chunking
UAC extends training-time real-time chunking by sampling delay according to embodiment control period and expected inference latency. Already committed prefix actions are clean; future postfix actions are noisy and receive the flow loss.
Equations 17–19 define the delay distribution, per-token time and postfix-only objective. Exact platform delay distributions and maxima are not provided. UAC learns continuation under timing constraints; it does not remove the need to budget compute and buffer capacity.
Equations 17–19
| (17) |
| (18) |
| (19) |
6 Infrastructure: Real-Time Cross-Embodiment Deployment
Deployment combines cached visual-language prefix computation, optional MPG refinement and a two-thread action buffer. These are complementary measures for throughput, feature uncertainty and execution timing.
6.1 Manifold-Preserving Refinement
Euler integration transports Gaussian actions through the learned flow, with fewer than ten steps in the reported implementation. Prefix features are cached while action-token embeddings change. The notation here uses H for the broader context, whereas the MPG subsection uses H specifically for its suffix; the intended modified component remains the suffix.
Equations 20
| (20) |
Two-Stage Refinement with MPG.
First generate an ungated baseline action chunk. Then perform one to three refinement rounds, each using the preceding prediction’s pooled zero-noise embedding as the MPG anchor throughout a fresh flow-denoising pass. A bad initial prediction can therefore also provide a bad anchor; iterative refinement is not an independently verified ground-truth reference.
Equations 21
| (21) |
Why MPG Helps in the Low-Step Regime.
The authors report saturation after two or three refinement rounds at fewer than ten Euler steps. No dedicated step-count/latency curve substantiates that statement here. Each refinement adds another flow pass, so fewer integration steps alone does not establish lower total latency.
Equation
6.2 Universal Async Chunking Protocol
The execution protocol commits the portion of the existing buffer expected to run while inference is busy, keeps it fixed during denoising, and writes only the newly predicted suffix back.
Latency Commitment.
The committed prefix must cover the ceiling of inference time divided by the robot’s control period, plus a timing margin. Underestimating it permits the model to revise actions that already execute. Overestimation protects that commitment only within available chunk/buffer capacity and sacrifices responsiveness; it is not a general physical-safety guarantee.
Equation 22
| (22) |
Hard Prefix Locking (denoising-time constraint).
At every Euler step, overwrite prefix positions with their committed buffer values. This enforces agreement with actions that cannot still be changed, while allowing the suffix to evolve.
Equation 23
| (23) |
Hard Stitching (buffer update rule).
After inference, discard the generated prefix and stitch only the suffix onto committed actions. Honoring the timing bound prevents conflicting buffer updates; smoothness at the boundary still depends on the learned continuation and low-level controller.
Equation 24
| (24) |
6.3 Dual-Thread Deployment Architecture
A fixed-frequency consumer sends buffered actions to the robot. An asynchronous producer reads observations, generates a chunk and writes its postfix at the committed offset. The ring buffer is at least twice the chunk length, with a short mutex-protected read/write region.
Underflow holds the last action or invokes a robot-specific fallback. The fallback is not fully defined, and holding a command is not automatically safe under every control mode or contact condition.
Benefits.
The same protocol is described for 10-Hz manipulators and 50-Hz humanoids, using cloud or Orin NX computation. Per-platform delay, buffer size and margin must change. The manuscript gives no measured worst-case latency, jitter, underflow frequency or power/memory results, so its real-time claim rests mainly on task demonstrations.
7 Experiment
Experiments compare physical robot policies, simulation specialists/generalists and component ablations. Figure 8 preserves the five hardware and camera configurations.
7.1 Comparison across Real Robots
The physical evaluation spans ten named tasks across spatial, long-horizon, bimanual and generalization categories. Clear Table runs on two robots, yielding eleven task–embodiment rows in Table 3. A single jointly post-trained checkpoint is compared with independently adapted specialists.
7.1.1 Experimental Setup
Hardware comprises Adam-U, G1 with LinkerBot O6, Franka FR3 with Inspire Hand, BeingBeyond D1 and SO-101. Each task receives 30–60 minutes of demonstrations. Tables 2–3 preserve hardware degrees of freedom, camera layouts, exact instructions and assignments.
Scratch variants omit UniHand-2.0 pretraining while retaining the architecture and post-training pipeline. The π0.5 baseline uses the same per-embodiment data and is evaluated only as a specialist in this implementation. The reported baseline restriction should not be generalized to every possible π0.5 adaptation.
| Embodiment | Structure | Vision | ||||
| Group | Name | DoF | Joints | Hand | Camera | Configuration |
| Upper-body Humanoid | PND Adam-U | 31 | Bimanual+Head+Waist | Dexterous (6DoF) | ZED Mini | Movable (ego), dual-camera |
| Unitree G1 + LinkerBot O6 | 26 | Bimanual | Dexterous (6DoF) | D435 | Fixed (ego) | |
| Single arm + Dexterous hand | FR3 + Inspire Hand | 13 | Single-arm | Dexterous (6DoF) | 2D435 | Fixed (3rd-person) |
| BeingBeyond D1 | 14 | Single-arm | Dexterous (6DoF) | D435 | Movable (ego) | |
| Single arm + Gripper | LeRobot SO-101 | 6 | Single-arm | Gripper | D435 | Fixed (3rd-person) |
| Category | Task | Embodiment | Instruction |
| Spatial | Arrange Flowers | Adam-U | “Arrange the flowers neatly into the vase.” |
| Water Plant | FR3+Inspire | “Use your hand to hold the sprinkling can and water the plants” | |
| Stack Bowls | FR3+Inspire | “Hold the edge of the bowls and stack them on the plate one by one.” | |
| Stack Blocks | D1 | “Stack the blocks following the color order: <ORDER>.” | |
| Long-horizon | Drawer: Open Place Close | FR3+Inspire | “Open the drawer, put <OBJECTS> inside, then close the drawer.” |
| Package: Flip Scan | G1+O6 | “Flip the package then scan the barcode with the scanner.” | |
| Bimanual | Hand-over Box | Adam-U | “Use both hands to pick and pass <OBJECTS> into the box.” |
| Pack two products Close lid | G1+O6 | “Use both hands to put <OBJECTS> into the box, then close the lid.” | |
| Generalization | Wipe Whiteboard | FR3+Inspire | “Use the wiper to clean any marked area of the whiteboard.” |
| Clear Table | Adam-U | “Clear the table: put all objects into the box.” | |
| Clear Table | SO-101 | “Clear the table: put all objects into the plate.” |
Evaluation Protocol.
A common black-box server hides policy identity from the human evaluator, randomizes policies and predefined scene layouts, and records binary success/timeout. The stated default is 20 trials per policy per layout.
The number of layouts, exact category aggregation, success tolerances, timeout and confidence intervals are not fully specified. Blinding reduces model-identity bias but does not by itself guarantee statistical validity or exclude all reset/scoring confounds.
7.1.2 Result Analysis
Figure 10 gives category success rates. The specialist/generalist scores are 75/70 spatial, 60/60 long-horizon, 55/45 bimanual and 75/80 generalization. These support useful joint training, with tradeoffs rather than uniform specialist dominance.
Specialist is strongest, but generalist stays surprisingly close.
The generalist trails by 5 points spatial and 10 bimanual, ties long-horizon and exceeds the specialist by 5 on generalization. Shared skills are a plausible explanation, but the study does not separately identify which overlap or nuisance factor caused each gain.
Clear margin over π 0.5 , especially on long-horizon and bimanual tasks.
π0.5 scores 55/45/40/65 across the four categories. The generalist’s gains are 15/15/5/15 percentage points, so its largest gap is not uniquely bimanual. Without uncertainty estimates these are observed differences, not established statistical significance.
UniHand-2.0 pretraining is crucial for a strong generalist.
Without UniHand pretraining, specialist scores are 50/35/25/60 and generalist scores 35/25/30/50. Pretraining improves all corresponding categories. The scratch generalist actually exceeds the scratch specialist on bimanual tasks, so its degradation is not uniform. This ablation removes the combined human/robot/language recipe, not human data alone.
Embodiment-Level Zero-Shot Transfer.
Adam-U qualitatively transfers package flipping/scanning, drawer interaction and stacking tasks demonstrated on other bodies, without Adam-U demonstrations for those task pairs. Adam-U itself is already represented in training, so this is an unseen task–embodiment pairing, not an entirely unseen robot.
The authors describe nonzero completion but provide no counts, rate or uncertainty, and acknowledge lower precision. These examples suggest compositional transfer; they do not quantify reliable zero-shot deployment or establish a priority claim.
7.2 Comparison on Simulation Benchmarks
Simulation uses 224×224 RGB, language and a 2B backbone. Specialists train separately on LIBERO or RoboCasa; the generalist trains jointly for twice the steps to maintain comparable updates per benchmark. Joint performance is therefore not obtained at the same total training budget.
| Method | L-Spatial | L-Object | L-Goal | L-Long | Average |
| Diffusion Policy [20] | 78.5 | 87.5 | 73.5 | 64.8 | 76.1 |
| OpenVLA [6] | 84.7 | 88.4 | 79.2 | 53.7 | 76.5 |
| SpatialVLA [28] | 88.2 | 89.9 | 78.6 | 55.5 | 78.1 |
| CoT-VLA [132] | 87.5 | 91.6 | 87.6 | 69.0 | 83.9 |
| -Fast [133] | 96.4 | 96.8 | 88.6 | 60.2 | 85.5 |
| GR00T-N1 [9] | 94.4 | 97.6 | 93.0 | 90.6 | 93.9 |
| [8] | 98.0 | 96.8 | 94.4 | 88.4 | 94.4 |
| F1 [134] | 98.2 | 97.8 | 95.4 | 91.3 | 95.7 |
| InternVLA-M1 [54] | 98.0 | 99.0 | 93.8 | 92.6 | 95.9 |
| Discrete Diffusion VLA [31] | 97.2 | 98.6 | 97.4 | 92.0 | 96.3 |
| [34] | 98.8 | 98.2 | 98.0 | 92.4 | 96.9 |
| OpenVLA-OFT [91] | 97.6 | 98.4 | 97.9 | 94.5 | 97.1 |
| X-VLA [11] | 98.2 | 98.6 | 97.8 | 97.6 | 98.1 |
| EO1 [94] | 99.7 | 99.8 | 99.2 | 94.8 | 98.2 |
| Being-H0.5(generalist) | 97.0 | 98.2 | 99.0 | 96.2 | 97.6 |
| Being-H0.5(specialist) | 99.2 | 99.6 | 99.4 | 97.4 | 98.9 |
7.2.1 Results on LIBERO
LIBERO assesses spatial, object, goal and longer task suites under the benchmark’s demonstration and evaluation setup.
Experimental setup.
Inputs include wrist and third-person views. Action chunks have length 8; packed sequences hold 7,680 tokens per GPU with effective batch 128. Specialists train 45,000 steps on four A800 GPUs; generalist training roughly doubles that. Evaluation uses 500 trials per suite, corresponding to 50 per task.
Results and analysis.
The specialist averages 98.9% versus 97.6% for the generalist. Long-suite scores are 97.4% and 96.2%. The specialist leads the listed overall averages but does not beat every row on every suite: X-VLA scores 97.6% on Long, and EO1 is higher on Spatial/Object. Broad state-of-the-art claims are limited to this comparison and date.
| Modality | Method | Pick & Place | Doors/Drawers | Others | Total Avg. |
| 3D (Multi-modal) | |||||
| 3D | 3DA [138] | 0.0 | 2.3 | 13.1 | 5.5 |
| 3D | DP3 [139] | 1.5 | 41.7 | 32.0 | 22.8 |
| 3D | GWM [140] | 14.8 | 54.3 | 49.8 | 39.3 |
| RGB-only (Standard) | |||||
| RGB (256256) | BC [136] | 4.3 | 47.0 | 42.2 | 28.9 |
| RGB (256256) | GR00T-N1 [9] | 18.6 | 50.2 | 39.1 | 36.0 |
| RGB (256256) | [34] | 21.5 | 57.8 | 44.9 | 41.4 |
| RGB (256256) | [8] | 14.0 | 53.1 | 58.5 | 42.4 |
| RGB (224224) | Being-H0.5(generalist) | 40 | 73 | 52 | 53.3 |
| RGB (224224) | Being-H0.5(specialist) | 36 | 71.7 | 57.6 | 53.9 |
7.2.2 Results on RoboCasa
RoboCasa tests 24 kitchen tasks using the Human-50 setting and held-out scene/object appearances.
Experimental setup.
Each task provides 50 demonstrations and receives 50 evaluation trials across five held-out scenes. Observations are 224×224 RGB with natural-language commands, without depth or point clouds. Specialist and joint training budgets follow the LIBERO schedule.
Results and analysis.
Overall success is 53.9% specialist and 53.3% generalist. Pick/place scores 36/40, doors/drawers 71.7/73 and other tasks 57.6/52. The generalist improves two categories and loses ground on the third; nearly half of all trials still fail. Baseline architecture, input modality and pretraining differ, so the comparison does not isolate human-data causality.
Takeaway.
One jointly trained checkpoint covers both simulation benchmarks with a modest aggregate tradeoff after doubled training. The specialist is a comparison point, not a mathematical upper bound: the generalist is better on some categories. These simulation checkpoints are distinct from the physical-robot generalist evaluation.
| Model Configuration | Single-Task LIBERO 5-Shot Success Rate (%) | |||||
| Frozen Components | Initialization | L-Spatial | L-Object | L-Goal | L-Long | Avg. |
| Und + Proj + ViT | Native VLM | 67.8 | 76.8 | 57.2 | 29.8 | 57.9 |
| Human-Centric | 78.6 | 78.9 | 72.6 | 46.2 | 69.0 | |
| Gain () | +10.8 | +2.1 | +15.4 | +16.4 | +11.1 | |
| Und + ViT | Native VLM | 70.6 | 50.0 | 67.0 | 17.4 | 51.3 |
| Human-Centric | 85.0 | 84.2 | 80.0 | 59.0 | 77.1 | |
| Gain () | +14.4 | +34.4 | +13.0 | +41.6 | +25.8 | |
| Und Only | Native VLM | 78.4 | 84.8 | 62.0 | 49.2 | 68.6 |
| Human-Centric | 83.2 | 79.6 | 74.0 | 63.6 | 75.1 | |
| Gain () | +4.8 | -5.2 | +12.0 | +14.4 | +6.5 | |
| Proj + ViT | Native VLM | 87.2 | 86.6 | 80.8 | 58.0 | 78.2 |
| Human-Centric | 86.2 | 86.2 | 84.6 | 66.8 | 81.0 | |
| Gain () | -1.0 | -0.4 | +3.8 | +8.8 | +2.8 | |
| None (Full FT) | Native VLM | 84.2 | 83.6 | 75.0 | 65.8 | 77.2 |
| Human-Centric | 87.2 | 91.4 | 81.6 | 66.8 | 81.8 | |
| Gain () | +3.0 | +7.8 | +6.6 | +1.0 | +4.6 | |
| Model Configuration | Multi-task-suite LIBERO 5-Shot Success Rate (%) | |||||
| Frozen Components | Initialization | L-Spatial | L-Object | L-Goal | L-Long | Avg. |
| Und + Proj + ViT | Native VLM | 61.6 | 77.0 | 68.0 | 36.0 | 60.7 |
| Human-Centric | 80.8 | 81.8 | 74.4 | 52.4 | 72.4 | |
| Gain () | +19.2 | +4.8 | +6.4 | +16.4 | +11.7 | |
| Und + ViT | Native VLM | 78.4 | 84.8 | 71.4 | 38.6 | 68.3 |
| Human-Centric | 83.6 | 82.6 | 80.8 | 59.2 | 76.6 | |
| Gain () | +5.2 | -2.2 | +9.4 | +20.6 | +8.3 | |
| Und Only | Native VLM | 75.8 | 84.2 | 71.2 | 59.8 | 72.8 |
| Human-Centric | 85.2 | 90.6 | 84.2 | 68.4 | 82.1 | |
| Gain () | +9.4 | +6.4 | +13.0 | +8.6 | +9.3 | |
| Proj + ViT | Native VLM | 87.6 | 91.0 | 82.2 | 62.6 | 80.9 |
| Human-Centric | 89.6 | 88.8 | 89.6 | 65.8 | 83.5 | |
| Gain () | +2.0 | -2.2 | +7.4 | +3.2 | +2.6 | |
| None (Full FT) | Native VLM | 87.8 | 94.6 | 82.6 | 71.4 | 84.1 |
| Human-Centric | 88.8 | 90.2 | 85.4 | 76.0 | 85.1 | |
| Gain () | +1.0 | -4.4 | +2.8 | +4.6 | +1.0 | |
7.3 Ablation Study
Ablations examine pretrained weights, trainable components, masked human-motion supervision and runtime consistency. They provide different kinds of evidence and should not be treated as one controlled test of the complete causal story.
7.3.1 How Do Human-Centric Learning Benefit Downstream Adaptation?
LIBERO five-shot adaptation uses 10,000 training steps. Separate-suite and joint-suite runs compare native-VLM initialization with the combined UniHand pretrained weights while varying frozen components; the action expert remains trainable in Tables 6–7. The “single-task” table caption refers to suite-specific training in the protocol.
Impact of Backbone Pretraining.
Freezing understanding plus vision while training the projector/action expert yields the largest separate-suite average gain:51.3→77.1, or 25.8 points; Long improves 17.4→59.0, or 41.6 points. Full tuning narrows the average gain to 4.6 points separate-suite and 1.0 joint-suite, with negative transfer on some Object cells.
Full tuning has the highest listed pretrained averages, 81.8 and 85.1, so freezing the backbone is not demonstrated to be globally optimal. Table 6’s Object gain for the frozen-understanding/vision row is printed+34.4 although 84.2−50.0=34.2; all original values remain visible. The prose’s reference to 50 demonstrations also conflicts with its five-shot framing.
Sensitivity of Action Expert Plasticity.
Figures 11–12 freeze progressively more action-expert layers while keeping the multimodal backbone fixed. Early-layer freezing costs little in several suites; extensive freezing sharply degrades most tasks. The Object curve remains comparatively higher than others when the plain pretrained expert is nearly/all frozen, so “below 20% everywhere” overstates the plot.
Each task receives only 20 rollouts. Figure 11’s legend says “pt + MoE” while its caption calls the component MoF; this original notation is retained. Figure 12 reports relative performance and trainable-parameter ratios, not a separate demonstration of new-task transfer.
7.3.2 How Does Masked Motion Token Prediction Benefit Capturing Behavior Priors?
MWDS is mean cosine similarity between predicted and target wrist displacement from start to finish, evaluated on held-out lab and wild human videos. It tests net direction, not complete trajectory accuracy or task completion; treatment of zero displacement and split sizes is unspecified.
Source inconsistency: Table 8 marks higher as better, but Hybrid scores 0.33/0.20 while removing the masked objective scores 0.35/0.28. Both HTML and PDF show these values. The surrounding claim that removing masking lowers MWDS is unsupported by the displayed table; no values or labels are silently reversed.
| Method | Lab | Wild |
| Hybrid (Ours) | 0.33 | 0.20 |
| w/o | 0.35 | 0.28 |
Equation
7.3.3 Real-Time Efficiency: MPG and UAC
Figure 13 includes separate removal of MPG, UAC and both. Against the full generalist’s 70/60/45/80 scores, removing both gives 50/35/20/55. These are category task rates, not measurements of inference speed or physical-safety guarantees.
Long-horizon drops sharply without UAC-style async consistency.
Long-horizon success falls 60→55 without MPG, 60→40 without UAC and 60→35 without both. This supports a larger role for timing consistency in the tested multi-stage tasks, while the proposed error mechanisms are observational explanations.
Bimanual coordination is brittle without stability mechanisms.
Bimanual success falls 45→40 without MPG and 45→20 without UAC or both. Timing consistency is particularly important here; the aggregate rates alone do not isolate oscillation or camera motion as the cause.
Spatial/generalization are affected but less catastrophically.
Spatial success falls 70→55 without MPG, 70→65 without UAC and 70→50 without both. Generalization falls 80→60 without MPG and 80→55 without both, but remains 80 without UAC. Thus degradation is not universal, and the combined 25-point generalization drop equals the long-horizon/bimanual drops.
8 Conclusion
The report presents a substantial shared-interface and data-engineering approach to human–robot learning, with useful physical multi-platform behavior and strong listed simulation results. Its evidence supports adaptation of a pretrained model to multiple bodies, rather than reliable unrestricted operation on unseen hardware.
Reproduction and interpretation are limited by missing pretraining/routing/MPG parameters, unquantified zero-shot completion, incomplete real-world trial accounting, absent runtime measurements and the masked-motion table contradiction. The combined-data ablation does not isolate the human component, and synthetic labels/scene overlap require further audit.
Appendix A Contributions
Core contributors and model training: Hao Luo, Ye Wang, Wanpeng Zhang and Sipeng Zheng. Infrastructure: Wanpeng Zhang (lead), Ye Wang, Chaoyi Xu, Chi Zhang and Haoqi Yuan.
Human data: Hao Luo (lead), Ziheng Xi, Yiqing Wang and Yicheng Feng. Robot data: Ye Wang and Sipeng Zheng (leads), with Haiweng Xu. Experiments: Wanpeng Zhang, Ye Wang, Hao Luo and Sipeng Zheng. Project leads: Sipeng Zheng and Zongqing Lu.
Appendix B Full Rollouts
Figure 14 retains the complete rollout montage from AppendixB, showing every illustrated real-robot task sequence. These are qualitative examples rather than the raw evaluation trial log.