Literature
Display
← All papers

SkelWAM: A Skeleton-Guided World-Action Model for Zero-Shot Cross-Embodiment Manipulation

Pengjun Niu · Yujia Xie · Rui Peng · Hang Zhao · Ke Liu

Preprint posted · arXiv (preprint) ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

Overview

SkelWAM uses one skeleton definition for robot appearance, numerical state and action targets, then relies on target-specific controllers to realize that intent. This edition retains all seven original figures, both tables and six equations from arXiv v1, including the complete source/target demonstration grids.

[Uncaptioned image]
Fig. 1: SkelWAM overview. The same 25-D centerline/TCP/jaw geometry defines canonical visual observations and future action chunks. Embodiment-specific decoders translate the predicted whole-body motion intent into native controls on LIBERO-Cross10 targets. Franka demonstrations train the simulation policy. In physical experiments, a model trained on JAKA mini2 data is deployed on Feagine A03.
Refer to caption
Fig. 2: SkelWAM world–action model. (a) The shared 25-D body/TCP/jaw state defines external and wrist proxies. (b) Thirty paired Video/Action DiT blocks predict a 32×2532\times 25 skeleton chunk. Starred future-video paths provide supervision only during Stage A. Action attention accesses current-frame keys and values (K/V), which are cached at inference. The deployment heads combine 24-D continuous flow with a jaw event. (c) Native joint IK or PCC decoding prioritizes TCP pose and uses the centerline as posture guidance.

At a glance

A shared 25-dimensional centerline, tool pose and jaw state links canonical images to future robot motion. Training uses one source robot’s demonstrations; target deployment keeps policy weights fixed but requires geometry, camera and controller calibration. On ten selected LIBERO tasks across ten targets, success is 43.3% over 1,000 episodes versus 7.1% for the best evaluated baseline. The JAKA-to-continuum-robot physical study provides qualitative demonstrations on three tasks, without a reported success-rate experiment.

I INTRODUCTION

Changing robot morphology alters both visible appearance and the native commands that can achieve a tool pose. End-effector retargeting supplies a common interaction goal but leaves whole-body posture ambiguous. SkelWAM couples the visual and action interfaces through six centerline samples, tool pose and jaw command. Task learning and model selection use source demonstrations only; deployment still incorporates target-specific calibration and constrained decoding.

II RELATED WORK

Related work addresses multi-robot learning, appearance adaptation, shared actions and predictive visual models. SkelWAM combines explicit geometry on both sides of the policy rather than requiring joint-to-joint correspondence.

II-A Cross-Embodiment Manipulation Policies

Mirage and RoVi-Aug change the appearance presented to a policy; other approaches use physical coordinates, learned action tokens or geometry-aware latents with embodiment-specific decoders. SkelWAM’s shared state expresses body posture and tool interaction in fixed dimensions while leaving native realization to a target controller.

II-B Structured Representations and World–Action Models

Prior structured representations use end-effector tracks, robot–object relations, link geometry, key poses or contact trajectories. World–action models add future visual prediction, sometimes retaining only current-frame conditioning at deployment. SkelWAM uses explicit centerline/tool geometry for its visual proxy, state input and action output, with a decoder for rigid or continuum kinematics.

III METHOD

The pipeline has three parts: canonicalize source/target observations and state, predict a future skeleton chunk, and decode that intent into feasible native motion. Prediction and execution share geometric meaning without sharing native joint coordinates.

III-A Problem Formulation

Source data synchronize external/wrist images, native state/action and instructions. Each target supplies calibrated geometry, cameras and a kinematic controller. Encoders/decoders connect native coordinates to one shared policy; the learned weights and source-fitted normalization remain fixed. Zero-shot here excludes target-task demonstrations and policy updates, while permitting task-independent target calibration.

III-B Canonical Whole-Body Skeleton

Six points sample the arm centerline at equal normalized arc-length intervals, with the sixth at the tool center. Five tool-relative, length-normalized offsets supply 15 body coordinates; task-frame tool position adds three, continuous rotation adds six and jaw command adds one. This separates normalized posture from the spatial interaction goal. Jaw values encode opening/closing, and source statistical normalization is retained on targets.

Equation 1
zt=[p0−p5Lr,…,p4−p5Lr,νℱ​(p5),ρ⁡(R5ℱ),gt]∈ℝ25.z_{t}=\left[\frac{p_{0}-p_{5}}{L_{r}},\ldots,\frac{p_{4}-p_{5}}{L_{r}},\nu_{\mathcal{F}}(p_{5}),\rho({}^{\mathcal{F}}R_{5}),g_{t}\right]\in\mathbb{R}^{25}. (1)

III-C Shared Visual–Action Representation

The native robot is masked and the background filled, then the same skeleton is projected into the external view and a fixed tool-relative wrist proxy. Numerical state, both proxies and future targets share the geometry. Simulation uses segmentation and robot-hidden rendering; physical deployment uses calibrated masks and image filling. Appearance normalization therefore depends on background reconstruction and target camera/tool registration.

Refer to caption
Fig. 3: BBQ-sauce-to-basket on Cross-Object (task 3). The first three columns show one Franka source demonstration and its front/wrist skeleton views. Target columns follow the robot order in Table I and show SkelWAM control replays from recorded states. Rows align initialization, approach, grasp, transport, and the LIBERO goal state across independently selected trials.
Equation 2
otc={Vr​(Itf,Itw,z¯t,𝒞r),z¯t,ℓ},o_{t}^{c}=\{V_{r}(I_{t}^{f},I_{t}^{w},\bar{z}_{t};\mathcal{C}_{r}),\bar{z}_{t},\ell\}, (2)

III-D Skeleton-Guided World–Action Model

Paired 30-layer video/action diffusion transformers initialize from Wan2.2; skeleton-specific projections and jaw heads are newly trained. Action attention sees current-frame visual keys/values and action tokens, excluding future-video tokens. Stage A jointly learns future video and skeleton motion; Stage B freezes video features and trains 24 continuous geometric coordinates plus first jaw-switch occurrence/timing. A chunk contains 32 absolute 25-dimensional states with at most one jaw switch; later switches require replanning. Deployment caches current-frame features and does not sample future video.

Equations 3–4
ℒA=λv​ℒvideo+ℒaction25​D.\mathcal{L}_{A}=\lambda_{v}\mathcal{L}_{\mathrm{video}}+\mathcal{L}_{\mathrm{action}}^{25D}. (3)
ℒB=ℒflow24​D+λg​(ℒpresence+ℒtiming).\mathcal{L}_{B}=\mathcal{L}_{\mathrm{flow}}^{24D}+\lambda_{g}(\mathcal{L}_{\mathrm{presence}}+\mathcal{L}_{\mathrm{timing}}). (4)

III-E Embodiment-Specific Decoding and Feedback

Target decoders prioritize tool pose/jaw commands, subject to coordinate limits and feasibility, with soft centerline tracking and temporal continuity. Rigid arms use joint inverse kinematics; the continuum arm uses piecewise-constant-curvature coordinates. Exact skeleton matching is not required. Policy feedback retains previous body intent while updating measured tool/jaw state; the continuum case also retains intended interaction orientation. The decoder separately uses measured native configuration for feasibility. This distinction between intended and realized body geometry is part of the method.

TABLE I: Zero-shot success on the ten target embodiments of LIBERO-Cross10 (%). Columns follow the four target reporting groups, and rows are grouped by method category. Each target has 100 episodes (ten tasks, ten trials per task).
Family Method Industrial (6-DoF) Redundant (7-DoF) Lightweight Continuum Mean
JAKA mini2 UR5e UR10e Kinova Gen3 xArm7 Rizon4 IIWA ARX-L5 ViperX300 Feagine A03
IL Diffusion Policy [29] 0.0 2.0 3.0 0.0 3.0 0.0 5.0 0.0 0.0 0.0 1.3
VLA π0.5\pi_{0.5} [30] 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.1
OpenVLA-OFT [31] 0.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.2
WAM FastWAM [20] 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.4
LaWAM [21] 0.0 0.0 1.0 4.0 2.0 0.0 0.0 0.0 2.0 0.0 0.9
X-Emb. Mirage [3] 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.4
RoVi-Aug [4] 3.0 15.0 4.0 7.0 13.0 8.0 9.0 10.0 2.0 0.0 7.1
Ours SkelWAM 59.0 13.0 62.0 49.0 19.0 62.0 22.0 56.0 32.0 59.0 43.3

Bold and underlined values indicate the highest and second-highest distinct rates in each column, respectively, including ties. Mean is the equally weighted average over ten targets. Non-SkelWAM methods use their native policy interface with target-specific EEF-IK. Mirage and RoVi-Aug use FastWAM as the underlying policy. Ablations are reported separately.

Original figure
Fig. 4: Success on the four selected task subsets, averaged over ten target embodiments. Each baseline bar is the maximum complete-method average within its family. Error bars show descriptive, episode-level Wilson 95% confidence intervals from binary outcomes (300, 300, 200, and 200 episodes per displayed method in the four panels), without adjustment for maximum selection. Task subsets are defined in the benchmark section.
Equation 5
∑i=04wi​‖fr,i​(xr)−p^i‖22+λq​‖xr−xr,t−1‖22,\sum_{i=0}^{4}w_{i}\|f_{r,i}(x_{r})-\hat{p}_{i}\|_{2}^{2}+\lambda_{q}\|x_{r}-x_{r,t-1}\|_{2}^{2}, (5)

IV EXPERIMENTS

Evaluation compares complete deployed systems on a fixed task/target matrix, then tests component interventions and qualitative physical transfer. Baselines share trial initializations but differ in pretraining, adaptation and interfaces.

IV-A LIBERO-Cross10 Benchmark

LIBERO-Cross10 retains original objects, instructions, resets and success predicates while replacing arms, grippers, installations and cameras. It selects Spatial/Object tasks 0, 2 and 3 and Goal/LIBERO-10 tasks 5 and 8. Franka is the source and excluded from target averages. Targets are three six-joint industrial arms, four seven-joint redundant arms, two lightweight arms and one Feagine A03 continuum arm. These four reporting groups are fixed and non-overlapping; the original table names every target.

IV-B Evaluation Protocol

The 467 successful Franka demonstrations split into 447 training and 20 validation episodes. Two 224×224 views and current state predict 32 future states; future-video supervision samples every fourth step. Stage A trains for 6,000 updates and Stage B for 24,000, selecting Stage-B step 23,000 by source validation. Inference uses 20 flow steps and executes ten predicted states before replanning. Every method receives ten targets × ten tasks × ten paired trials. Baselines keep their own image/action contracts and use end-effector-only target decoders; released pretraining and source adaptation budgets differ. Wilson intervals describe pooled episodes rather than variation across training seeds or independently sampled robot/task populations.

TABLE II: Component interventions on the same 1,000 episodes per variant. Δ\Delta is ablation minus full, in percentage points.
Variant VV AA WW DD SR (%) Δ\Delta (pp)
Full ✓\checkmark ✓\checkmark ✓\checkmark ✓\checkmark 43.3 –
w/o VRV^{R} ×\times ✓\checkmark ✓\checkmark ✓\checkmark 0.0 -43.3
w/o ARA^{R} ✓\checkmark ×\times ✓\checkmark ×\times 0.1 -43.2
w/o WRW^{R} ✓\checkmark ✓\checkmark ×\times ✓\checkmark 43.0 -0.3
w/o DID^{I} ✓\checkmark ✓\checkmark ✓\checkmark ×\times 40.9 -2.4

VV, AA, WW, and DD denote skeleton vision, body-action supervision, wrist input, and the decoder body objective, respectively. RR denotes source retraining, and II denotes an inference-only intervention using the full checkpoint. Removing AA also disables DD, coupling changes in learning and execution.

Refer to caption
Fig. 5: Task-dependent effects of the interventions in Table II. Cells show ablation minus full in percentage points on the four Cross-* subsets. Red indicates a decrease and blue an increase in success.
Refer to caption
Fig. 6: Real-robot task setup. Columns show Toy Cleanup, Ring Placement, and Peg Insertion, while rows show source JAKA mini2 and target Feagine A03. Arrows and translucent poses illustrate the task motions.

JAKA mini2: Data Collection  Feagine A03: Policy Deployment

Refer to caption
Fig. 7: Physical visual–action interface. JAKA mini2 is shown on the left and Feagine A03 on the right. For Toy Cleanup, the five rows show external RGB, mask overlays, skeleton composites, wrist RGB, and wrist illustrations. Lower bands: Peg Insertion and Ring Placement. Each side shows four selected times.
Equation 6
SR=1|ℛ|​|𝒯|​J​∑r∈ℛ∑k∈𝒯∑j=1Jyr,k,j.\mathrm{SR}=\frac{1}{|\mathcal{R}|\,|\mathcal{T}|\,J}\sum_{r\in\mathcal{R}}\sum_{k\in\mathcal{T}}\sum_{j=1}^{J}y_{r,k,j}. (6)

IV-C Zero-Shot Transfer Results

SkelWAM reaches 43.3% success with a pooled Wilson 95% interval of 40.3–46.4%, exceeding the adapted RoVi-Aug baseline by 36.2 percentage points. It leads on nine targets; UR5e instead favors RoVi-Aug, 15% versus 13%. Selected Spatial/Object subsets reach 61.3%/61.0%, versus 15%/18% for Goal/Long. Group means are 44.7% industrial, 38.0% redundant, 44.0% lightweight and 59.0% on the single continuum target. Figure 4 chooses each baseline family’s best complete-method average separately per subset; its intervals are not adjusted for that maximum selection.

IV-D Component Analysis

Retraining with native source images instead of skeleton vision reduces success to 0%; removing body-coordinate supervision together with the decoder body objective gives 0.1%. The latter couples learning and execution and cannot isolate supervision alone. Removing wrist input changes the mean only from 43.3% to 43.0%, but harms Spatial/Object by 6.7/14.7 points and helps Goal/Long by 22.5/8.0. Removing only decoder body guidance, with unchanged policy weights, yields 40.9%, including a Cross-Object decrease and Cross-Long increase. Components therefore have task-dependent effects rather than uniformly positive contributions.

IV-E Real-Robot Deployment

Physical training uses 277 quality-controlled JAKA mini2 demonstrations split 249/28, compiled into 10 Hz skeleton windows. The action expert and last two video blocks train for 6,000 updates with action supervision, selected on source validation. Fixed weights transfer to Feagine A03 for toy cleanup, ring placement and peg insertion using a calibrated continuum decoder. Vision uses region matching, RANSAC, optical flow and color-assisted tool relocking. The full image grids show selected times and processing stages; there are no reported physical trial counts or quantitative success rates.

V DISCUSSION AND LIMITATIONS

Transfer remains limited by reachability, gripper contact geometry, calibration and image filling under occlusion. The harder Goal/Long subsets still have low success. Baseline pretraining/adaptation differences prevent attributing the entire system-level gain to one representation choice, and the action ablation couples two interventions. Physical results illustrate feasibility on three tasks rather than reliable transfer across a quantified deployment distribution.

VI CONCLUSION

A common skeleton representation links perception and action while embodiment-specific decoders handle native feasibility. The reported source-only simulation result is promising for the selected ten-task/ten-target benchmark, with substantial task and morphology variation. The qualitative physical study extends the interface to a continuum robot but does not establish a measured real-world transfer rate.

Concept demonstration

Open image ↗