Overview
SkelWAM uses one skeleton definition for robot appearance, numerical state and action targets, then relies on target-specific controllers to realize that intent. This edition retains all seven original figures, both tables and six equations from arXiv v1, including the complete source/target demonstration grids.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/41/41af8a24aca202d1821c1a3627050c20a72302c2f8dc0d76e74942981ba77b79.webp)
At a glance
A shared 25-dimensional centerline, tool pose and jaw state links canonical images to future robot motion. Training uses one source robot’s demonstrations; target deployment keeps policy weights fixed but requires geometry, camera and controller calibration. On ten selected LIBERO tasks across ten targets, success is 43.3% over 1,000 episodes versus 7.1% for the best evaluated baseline. The JAKA-to-continuum-robot physical study provides qualitative demonstrations on three tasks, without a reported success-rate experiment.
I INTRODUCTION
Changing robot morphology alters both visible appearance and the native commands that can achieve a tool pose. End-effector retargeting supplies a common interaction goal but leaves whole-body posture ambiguous. SkelWAM couples the visual and action interfaces through six centerline samples, tool pose and jaw command. Task learning and model selection use source demonstrations only; deployment still incorporates target-specific calibration and constrained decoding.
II RELATED WORK
Related work addresses multi-robot learning, appearance adaptation, shared actions and predictive visual models. SkelWAM combines explicit geometry on both sides of the policy rather than requiring joint-to-joint correspondence.
II-A Cross-Embodiment Manipulation Policies
Mirage and RoVi-Aug change the appearance presented to a policy; other approaches use physical coordinates, learned action tokens or geometry-aware latents with embodiment-specific decoders. SkelWAM’s shared state expresses body posture and tool interaction in fixed dimensions while leaving native realization to a target controller.
II-B Structured Representations and World–Action Models
Prior structured representations use end-effector tracks, robot–object relations, link geometry, key poses or contact trajectories. World–action models add future visual prediction, sometimes retaining only current-frame conditioning at deployment. SkelWAM uses explicit centerline/tool geometry for its visual proxy, state input and action output, with a decoder for rigid or continuum kinematics.
III METHOD
The pipeline has three parts: canonicalize source/target observations and state, predict a future skeleton chunk, and decode that intent into feasible native motion. Prediction and execution share geometric meaning without sharing native joint coordinates.
III-A Problem Formulation
Source data synchronize external/wrist images, native state/action and instructions. Each target supplies calibrated geometry, cameras and a kinematic controller. Encoders/decoders connect native coordinates to one shared policy; the learned weights and source-fitted normalization remain fixed. Zero-shot here excludes target-task demonstrations and policy updates, while permitting task-independent target calibration.
III-B Canonical Whole-Body Skeleton
Six points sample the arm centerline at equal normalized arc-length intervals, with the sixth at the tool center. Five tool-relative, length-normalized offsets supply 15 body coordinates; task-frame tool position adds three, continuous rotation adds six and jaw command adds one. This separates normalized posture from the spatial interaction goal. Jaw values encode opening/closing, and source statistical normalization is retained on targets.
Equation 1
| (1) |
III-C Shared Visual–Action Representation
The native robot is masked and the background filled, then the same skeleton is projected into the external view and a fixed tool-relative wrist proxy. Numerical state, both proxies and future targets share the geometry. Simulation uses segmentation and robot-hidden rendering; physical deployment uses calibrated masks and image filling. Appearance normalization therefore depends on background reconstruction and target camera/tool registration.
Equation 2
| (2) |
III-D Skeleton-Guided World–Action Model
Paired 30-layer video/action diffusion transformers initialize from Wan2.2; skeleton-specific projections and jaw heads are newly trained. Action attention sees current-frame visual keys/values and action tokens, excluding future-video tokens. Stage A jointly learns future video and skeleton motion; Stage B freezes video features and trains 24 continuous geometric coordinates plus first jaw-switch occurrence/timing. A chunk contains 32 absolute 25-dimensional states with at most one jaw switch; later switches require replanning. Deployment caches current-frame features and does not sample future video.
Equations 3–4
| (3) |
| (4) |
III-E Embodiment-Specific Decoding and Feedback
Target decoders prioritize tool pose/jaw commands, subject to coordinate limits and feasibility, with soft centerline tracking and temporal continuity. Rigid arms use joint inverse kinematics; the continuum arm uses piecewise-constant-curvature coordinates. Exact skeleton matching is not required. Policy feedback retains previous body intent while updating measured tool/jaw state; the continuum case also retains intended interaction orientation. The decoder separately uses measured native configuration for feasibility. This distinction between intended and realized body geometry is part of the method.
| Family | Method | Industrial (6-DoF) | Redundant (7-DoF) | Lightweight | Continuum | Mean | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| JAKA mini2 | UR5e | UR10e | Kinova Gen3 | xArm7 | Rizon4 | IIWA | ARX-L5 | ViperX300 | Feagine A03 | |||
| IL | Diffusion Policy [29] | 0.0 | 2.0 | 3.0 | 0.0 | 3.0 | 0.0 | 5.0 | 0.0 | 0.0 | 0.0 | 1.3 |
| VLA | [30] | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 |
| OpenVLA-OFT [31] | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.2 | |
| WAM | FastWAM [20] | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.4 |
| LaWAM [21] | 0.0 | 0.0 | 1.0 | 4.0 | 2.0 | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 0.9 | |
| X-Emb. | Mirage [3] | 0.0 | 0.0 | 0.0 | 0.0 | 4.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.4 |
| RoVi-Aug [4] | 3.0 | 15.0 | 4.0 | 7.0 | 13.0 | 8.0 | 9.0 | 10.0 | 2.0 | 0.0 | 7.1 | |
| Ours | SkelWAM | 59.0 | 13.0 | 62.0 | 49.0 | 19.0 | 62.0 | 22.0 | 56.0 | 32.0 | 59.0 | 43.3 |
Bold and underlined values indicate the highest and second-highest distinct rates in each column, respectively, including ties. Mean is the equally weighted average over ten targets. Non-SkelWAM methods use their native policy interface with target-specific EEF-IK. Mirage and RoVi-Aug use FastWAM as the underlying policy. Ablations are reported separately.
Equation 5
| (5) |
IV EXPERIMENTS
Evaluation compares complete deployed systems on a fixed task/target matrix, then tests component interventions and qualitative physical transfer. Baselines share trial initializations but differ in pretraining, adaptation and interfaces.
IV-A LIBERO-Cross10 Benchmark
LIBERO-Cross10 retains original objects, instructions, resets and success predicates while replacing arms, grippers, installations and cameras. It selects Spatial/Object tasks 0, 2 and 3 and Goal/LIBERO-10 tasks 5 and 8. Franka is the source and excluded from target averages. Targets are three six-joint industrial arms, four seven-joint redundant arms, two lightweight arms and one Feagine A03 continuum arm. These four reporting groups are fixed and non-overlapping; the original table names every target.
IV-B Evaluation Protocol
The 467 successful Franka demonstrations split into 447 training and 20 validation episodes. Two 224×224 views and current state predict 32 future states; future-video supervision samples every fourth step. Stage A trains for 6,000 updates and Stage B for 24,000, selecting Stage-B step 23,000 by source validation. Inference uses 20 flow steps and executes ten predicted states before replanning. Every method receives ten targets × ten tasks × ten paired trials. Baselines keep their own image/action contracts and use end-effector-only target decoders; released pretraining and source adaptation budgets differ. Wilson intervals describe pooled episodes rather than variation across training seeds or independently sampled robot/task populations.
| Variant | SR (%) | (pp) | ||||
|---|---|---|---|---|---|---|
| Full | 43.3 | – | ||||
| w/o | 0.0 | -43.3 | ||||
| w/o | 0.1 | -43.2 | ||||
| w/o | 43.0 | -0.3 | ||||
| w/o | 40.9 | -2.4 |
, , , and denote skeleton vision, body-action supervision, wrist input, and the decoder body objective, respectively. denotes source retraining, and denotes an inference-only intervention using the full checkpoint. Removing also disables , coupling changes in learning and execution.
Equation 6
| (6) |
IV-C Zero-Shot Transfer Results
SkelWAM reaches 43.3% success with a pooled Wilson 95% interval of 40.3–46.4%, exceeding the adapted RoVi-Aug baseline by 36.2 percentage points. It leads on nine targets; UR5e instead favors RoVi-Aug, 15% versus 13%. Selected Spatial/Object subsets reach 61.3%/61.0%, versus 15%/18% for Goal/Long. Group means are 44.7% industrial, 38.0% redundant, 44.0% lightweight and 59.0% on the single continuum target. Figure 4 chooses each baseline family’s best complete-method average separately per subset; its intervals are not adjusted for that maximum selection.
IV-D Component Analysis
Retraining with native source images instead of skeleton vision reduces success to 0%; removing body-coordinate supervision together with the decoder body objective gives 0.1%. The latter couples learning and execution and cannot isolate supervision alone. Removing wrist input changes the mean only from 43.3% to 43.0%, but harms Spatial/Object by 6.7/14.7 points and helps Goal/Long by 22.5/8.0. Removing only decoder body guidance, with unchanged policy weights, yields 40.9%, including a Cross-Object decrease and Cross-Long increase. Components therefore have task-dependent effects rather than uniformly positive contributions.
IV-E Real-Robot Deployment
Physical training uses 277 quality-controlled JAKA mini2 demonstrations split 249/28, compiled into 10 Hz skeleton windows. The action expert and last two video blocks train for 6,000 updates with action supervision, selected on source validation. Fixed weights transfer to Feagine A03 for toy cleanup, ring placement and peg insertion using a calibrated continuum decoder. Vision uses region matching, RANSAC, optical flow and color-assisted tool relocking. The full image grids show selected times and processing stages; there are no reported physical trial counts or quantitative success rates.
V DISCUSSION AND LIMITATIONS
Transfer remains limited by reachability, gripper contact geometry, calibration and image filling under occlusion. The harder Goal/Long subsets still have low success. Baseline pretraining/adaptation differences prevent attributing the entire system-level gain to one representation choice, and the action ablation couples two interventions. Physical results illustrate feasibility on three tasks rather than reliable transfer across a quantified deployment distribution.
VI CONCLUSION
A common skeleton representation links perception and action while embodiment-specific decoders handle native feasibility. The reported source-only simulation result is promising for the selected ten-task/ten-target benchmark, with substantial task and morphology variation. The qualitative physical study extends the interface to a continuum robot but does not establish a measured real-world transfer rate.
