Literature
Display
← All papers

WM-Craftnet: World Synesthesia Model for Robust and Generalizable Dexterous In-Hand Manipulation

Jie Yin · Zeyuan Zhao · Xiaojing Tan · Yang Liu · Chiyu Wang · Xinyang Gu

Preprint posted · CoRL 2026 (accepted) ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Reading guide

All fifteen pages, nine complete figures, ten complete tables, ten numbered equations and twenty-five references are covered. WM-Craftnet trains a predictive visuotactile recurrent state and feeds a detached projection into PPO. It uses neither imagined trajectories nor policy gradients through the world model. The main evidence concerns a specific 22-joint hand, simulation-trained axis-specific policies, object-set adaptation and limited physical trials. Full original project media accompanies the edition, including failure examples.

At a glance

The World Synesthesia Model combines noisy wrist depth, proprioception, binary contact and previous actions. Reconstruction, prediction and simulator-only auxiliary targets shape a recurrent context for the actor. Pretraining improves nine-object rotation, adaptation reuses that representation for forty-nine new objects, and hardware experiments report 175/200 successful rotations across twenty objects. These results support a useful learned state estimator; they do not establish a general-purpose simulator, a frozen zero-shot forty-nine-object policy or reliable recovery from arbitrary disturbances.

Original mathematical definitions, training assumptions and evaluation protocol

Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor–critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine zz-axis objects serves as a reusable prior for 4949-object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.

1 Introduction

In-hand manipulation changes contact geometry continuously, while depth has missing/noisy pixels and touch alone gives little global shape information. The proposed response is an action-conditioned recurrent representation that predicts across modalities. Its value is tested through clean-depth, recurrence, sensor-head and auxiliary-head controls. Object identifiers are absent from the deployed actor, but privileged simulator labels still contribute to representation learning.

Original mathematical definitions, training assumptions and evaluation protocol

WM-Craftnet learns a World Synesthesia Model with a Dreamer-style recurrent state from proprioception, noisy depth, tactile observations, and recent actions, with reward used as a prediction target during training. The policy is conditioned on this recurrent feature, which captures object state, geometry, and contact evolution. During WSM training, noisy augmented depth is used as input while clean wrist-depth targets supervise reconstruction, encouraging denoised geometry for downstream control. This representation improves robust multi-object rotation, supports unseen-object evaluation, and enables WSM pretraining as a reusable prior for downstream 4949-object control. We summarize our contributions as follows:

Original mathematical definitions, training assumptions and evaluation protocol

We analyze WSM with recurrent-history baselines, auxiliary-head and modality-head ablations, latent-state diagnostics, clean-depth reconstruction, and pretraining, showing that the predictive representation can serve as a reusable prior for 4949-object downstream control.

Original mathematical definitions, training assumptions and evaluation protocol

We conduct quantitative simulation and real-world evaluations, supported by qualitative videos, showing that WSM improves single-policy manipulation across object variations, unseen objects, perturbations, and enables sim-to-real transfer on a human-sized 2222-DoF five-finger dexterous hand.

2 Related Work

The paper connects learned in-hand control, visuotactile perception and predictive world models. The comparison concerns representations and their control interface as much as raw sensors: equal rewards and randomization do not imply identical architectures or training supervision.

Learning-based in-hand manipulation.

Prior in-hand methods learn rotation gaits, adapt through proprioception or specialize to shapes and skill mixtures. Their results motivate richer state inference but do not make every cited benchmark directly comparable. The paper's own baseline reimplementations, retained in the tables, are the controlled experimental comparison. See the cited touch policy and rapid motor adaptation.

Original mathematical definitions, training assumptions and evaluation protocol

Learning-based systems have achieved sim-to-real in-hand control through several complementary strategies. Rapid Motor Adaptation infers object properties online from proprioceptive history for zz-axis rotation [4], whereas DeXtreme couples a domain-randomized reorientation policy with a vision-based pose estimator [3]. Touch Dexterity demonstrates vision-free rotation with distributed binary contact sensing [1], and AnyRotate uses dense tactile feedback for gravity-invariant multi-axis rotation [5]. More complex reorientation can compose pretrained axis-wise skills [8] or route among shape-specialized experts [9]. Other approaches adapt with real pen-spinning trajectories [10], learn joint-wise actuator corrections from hardware data [11], or randomize 3D Gaussians for monocular pose estimation [12]. In adjacent dexterous grasping, Goal-Auxiliary Actor-Critic combines point-cloud control with an auxiliary goal objective [13], while DextrAH-G couples pixel policies with geometric-fabric control priors [14]. These methods address grasp acquisition; WM-Craftnet instead learns recurrent interaction state for sustained in-hand control.

Visuotactile perception for dexterity.

Earlier methods combine touch with visual geometry, reconstruct contact-aware object shape or learn cross-modal features. WM-Craftnet replaces a fixed visual embedding with an action-conditioned recurrent predictive state. It does not explicitly recover a uniquely identifiable physical state; the t-SNE plots are diagnostics of representation structure. Relevant sources include Robot Synesthesia, NeuralFeels and ViTacFormer.

World models for robot control.

The RSSM follows the Dreamer family, but here prediction and policy optimization are separate. The actor uses observed-history state rather than optimizing on model-generated futures. This difference matters when interpreting the phrase world model. The original World Models, Dream to Control, DreamerV3 and visual locomotion references remain available.

3 WM-Craftnet Framework

The pipeline has a multimodal encoder, a recurrent stochastic state-space model and an asymmetric PPO actor–critic. Only the projected deterministic recurrent state crosses from world model to actor, with gradients stopped at that boundary. Simulator object state, clean depth and auxiliary targets provide training supervision without becoming deployed observations.

Original mathematical definitions, training assumptions and evaluation protocol

WM-Craftnet targets partially observed in-hand manipulation on a human-sized 2222-DoF five-finger hand. We define the World Synesthesia Model as a Dreamer-style recurrent latent dynamics model trained with multimodal reconstruction and reward prediction, whose detached deterministic state serves as deployable policy context. The name reflects its cross-modal predictive role: visual geometry, tactile contact, proprioception, and action history jointly constrain the latent interaction state. This predictive state supports object-ID-free adaptation from deployable visuotactile observations and serves as a recurrent state estimate for the actor, as illustrated in Figure 1.

Complete original Figure 1: Overview of WM-Craftnet. A Dreamer-style recurrent state-space model (RSSM) learns a denoised recurrent state from multimodal observations and provides it as predictive context for the manipulation policy.
Figure 1: Overview of WM-Craftnet. A Dreamer-style recurrent state-space model (RSSM) learns a denoised recurrent state from multimodal observations and provides it as predictive context for the manipulation policy.

3.1 Problem Formulation and Observations

Equation 1 lists the actor's stacked proprioception/contact, current depth and previous action; the commanded rotation axis and learned context are additional policy inputs. Equation 2 measures signed incremental rotation projected onto that axis. The actor outputs relative updates for all twenty-two joint targets. Deployment uses neither ground-truth object pose nor object identity, while training rewards, critic targets and auxiliary predictions do use simulator information.

Original mathematical definitions, training assumptions and evaluation protocol

We consider continuous in-hand rotation of a grasped object around a commanded axis 𝐚∈{±x,±y,±z}\mathbf{a}\in\{\pm x,\pm y,\pm z\}. At time tt, we denote the deployable policy observation by

Complete original mathematical display S3.E1
Original selectable mathematical display
Equation 1
𝐨t=(𝐩tstack,𝐜tstack,𝐃t,𝐮t−1),\mathbf{o}_{t}=\big(\mathbf{p}^{\text{stack}}_{t},\mathbf{c}^{\text{stack}}_{t},\mathbf{D}_{t},\mathbf{u}_{t-1}\big), (1)
Original mathematical definitions, training assumptions and evaluation protocol

where 𝐩tstack\mathbf{p}^{\text{stack}}_{t} and 𝐜tstack\mathbf{c}^{\text{stack}}_{t} are short stacks of normalized hand proprioception and binary tactile/contact measurements, 𝐃t\mathbf{D}_{t} is the current noisy wrist-depth stream, and 𝐮t−1\mathbf{u}_{t-1} is the previous joint-target command. Importantly, 𝐨t\mathbf{o}_{t} contains only on-robot sensor streams and control history; it does not include clean simulator depth or any depth reconstructed by the WSM decoder. The actor conditions on 𝐨t\mathbf{o}_{t}, the commanded axis 𝐚\mathbf{a}, and the projected WSM feature ψ⁡(𝐰t)\psi(\mathbf{w}_{t}). The policy outputs a relative 2222-DoF joint target command 𝐮t\mathbf{u}_{t}, which is smoothed and executed by the low-level hand controller.

Original mathematical definitions, training assumptions and evaluation protocol

The objective is signed object rotation around the target axis while maintaining a stable grasp. Let 𝐑to∈S​O​(3)\mathbf{R}^{o}_{t}\in SO(3) denote object orientation. One-step progress is

Complete original mathematical display S3.E2
Original selectable mathematical display
Equation 2
Δ​θt=proj𝐚​(log⁡(𝐑t+1o​(𝐑to)−1)),\Delta\theta_{t}=\mathrm{proj}_{\mathbf{a}}\!\left(\log\!\left(\mathbf{R}^{o}_{t+1}(\mathbf{R}^{o}_{t})^{-1}\right)\right), (2)
Original mathematical definitions, training assumptions and evaluation protocol

where log⁡(⋅)\log(\cdot) maps rotation to axis-angle form and proj𝐚\mathrm{proj}_{\mathbf{a}} projects onto the commanded axis. During simulation training, object pose, velocity, and contact-force fields support reward computation, critic learning, and diagnostics. Table 8 lists the complete observation/action interface.

3.2 Multimodal Predictive State Encoder

Equations 3–4 preserve the exact rollout tuple and multimodal encoder. The world model receives single-step proprioception/contact rather than the actor's stacks. Noisy preprocessed depth is an input; clean crop-only depth is a reconstruction target. Temporally correlated dropout, Gaussian perturbations and small image rotations make denoising useful. Predicting clean geometry requires access to that simulator target during training; it is not self-supervision solely from raw hardware depth.

Original mathematical definitions, training assumptions and evaluation protocol

The encoder maps sensory streams to a latent state for prediction and control. The world-model update receives a single-step proprioceptive vector 𝐩t\mathbf{p}_{t}, tactile/contact vector 𝐜t\mathbf{c}_{t}, noisy normalized depth image 𝐈t\mathbf{I}_{t}, previous action 𝐮t−1\mathbf{u}_{t-1}, and episode-start flag mtm_{t}. The rollout record additionally contains clean depth and reward as training targets; neither is required as a deployment-time input. The actor observes stacked proprioceptive/tactile state, whereas the world model uses unstacked low-dimensional vectors to keep dynamics compact.

Complete original mathematical display S3.E3
Original selectable mathematical display
Equation 3
𝒟={(𝐩t,𝐜t,𝐈t,𝐈tclean,𝐮t−1,rt,mt)}t=1T,\mathcal{D}=\{(\mathbf{p}_{t},\mathbf{c}_{t},\mathbf{I}_{t},\mathbf{I}^{\mathrm{clean}}_{t},\mathbf{u}_{t-1},r_{t},m_{t})\}_{t=1}^{T}, (3)
Complete original mathematical display S3.E4
Original selectable mathematical display
Equation 4
𝐞t=fenc​(𝐩t,𝐜t,𝐈t).\mathbf{e}_{t}=f_{\text{enc}}(\mathbf{p}_{t},\mathbf{c}_{t},\mathbf{I}_{t}). (4)

3.3 World Model for In-Hand Manipulation

Equations 5–7 retain posterior recurrent updating, deterministic actor context and the full loss. The RSSM predicts low-dimensional observations, clean depth and reward, with Dreamer dynamics/representation KL terms. Object pose, value and basis-point-set-to-mesh shape predictions provide additional privileged supervision. Reward is a prediction target, not a current deployed sensor input. The actual actor context is a learned projection of the 512-dimensional deterministic state; the stochastic latent is not directly handed to PPO.

Original mathematical definitions, training assumptions and evaluation protocol

WM-Craftnet uses a Dreamer-style recurrent state-space model (RSSM) as its world model. World-model optimization is decoupled from the PPO gradient path, and the actor receives a detached deterministic latent feature. Let 𝐳t\mathbf{z}_{t} be the stochastic latent state and 𝐡t\mathbf{h}_{t} be the deterministic recurrent state. Given the previous latent state, previous action, and current encoded observation, the posterior update is

Complete original mathematical display S3.E5
Original selectable mathematical display
Equation 5
𝐳t,𝐡t=fwm​(𝐳t−1,𝐡t−1,𝐮t−1,𝐞t,mt).\mathbf{z}_{t},\mathbf{h}_{t}=f_{\text{wm}}\!\left(\mathbf{z}_{t-1},\mathbf{h}_{t-1},\mathbf{u}_{t-1},\mathbf{e}_{t},m_{t}\right). (5)
Complete original mathematical display S3.E6
Original selectable mathematical display
Equation 6
𝐰t=𝐡t,\mathbf{w}_{t}=\mathbf{h}_{t}, (6)
Original mathematical definitions, training assumptions and evaluation protocol

which is detached before entering the policy network. In the current implementation, 512512-dim 𝐡t\mathbf{h}_{t} is projected by a small MLP to a 1616-dim feature before being concatenated with the policy observation.

Original mathematical definitions, training assumptions and evaluation protocol

The world model is trained from a replay buffer populated during PPO rollouts. Each row stores (𝐩t,𝐜t,𝐈t,𝐈tclean,𝐮t−1,rt,mt)(\mathbf{p}_{t},\mathbf{c}_{t},\mathbf{I}_{t},\mathbf{I}^{\mathrm{clean}}_{t},\mathbf{u}_{t-1},r_{t},m_{t}), where 𝐈t\mathbf{I}_{t} is the noisy/augmented depth input and 𝐈tclean\mathbf{I}^{\mathrm{clean}}_{t} is the clean depth target used by the decoder. Training samples contiguous chunks and optimizes the Dreamer-style model objective:

Complete original mathematical display S3.E7
Original selectable mathematical display
Equation 7
ℒwm=ℒimg+ℒprop+λc​ℒtac+λr​ℒreward+λdyn​ℒdyn-KL+λrep​ℒrep-KL+ℒaux.\mathcal{L}_{\text{wm}}=\mathcal{L}_{\text{img}}+\mathcal{L}_{\text{prop}}+\lambda_{c}\mathcal{L}_{\text{tac}}+\lambda_{r}\mathcal{L}_{\text{reward}}+\lambda_{\text{dyn}}\mathcal{L}_{\text{dyn-KL}}+\lambda_{\text{rep}}\mathcal{L}_{\text{rep-KL}}+\mathcal{L}_{\text{aux}}. (7)
Original mathematical definitions, training assumptions and evaluation protocol

Here ℒimg\mathcal{L}_{\text{img}} reconstructs 𝐈tclean\mathbf{I}^{\mathrm{clean}}_{t} from the latent state inferred with noisy depth input, ℒprop\mathcal{L}_{\text{prop}} reconstructs proprioception, ℒreward\mathcal{L}_{\text{reward}} predicts rollout reward, and the KL terms regularize posterior/prior dynamics. Reward supervises the predictive model during training. Together, these targets encourage a geometry- and contact-aware state for deployment with noisy wrist-depth observations. Inspired by recent rotation and grasping systems [15, 18, 13, 17, 14], the full WSM uses depth, proprioception, tactile contact, reward, object pose, value, and object shape auxiliary heads to learn a task-relevant predictive state. During simulation training, the pose and shape heads use simulator labels, with shape represented by a basis-point-set-to-mesh displacement vector. Appendix B.1 analyzes the contribution of these heads.

3.4 Pretrained Synesthetic Prior for Multi-Object Control

Pretraining uses the nine-object z-axis setting. The transferred encoder, RSSM, decoders and reward predictor then continue adapting on new forty-nine-object rollouts, while the downstream actor–critic and task heads learn that distribution. This is a reusable initialization for further learning. The forty-nine-object result is not evaluation of an unchanged nine-object policy on forty-nine unseen objects.

Original mathematical definitions, training assumptions and evaluation protocol

Scaling one dexterous policy from a small object set to many new objects requires inferring how geometry, contact, slip, and action-conditioned motion interact. WM-Craftnet addresses this by making WSM reusable across object distributions. After pretraining on contact-rich data from the nine-object zz-axis setting, the predictive model initializes the 4949-object experiment and continues adapting to new rollouts, while a task-specific controller is learned with reinforcement learning.

3.5 World Model-Conditioned Manipulation Policy

Equations 8–9 give policy conditioning and relative joint-target integration. The actor receives current noisy depth plus proprioception/contact, previous action, axis command and projected recurrent context; its main interface does not replace depth with the WSM's decoded image. The denoised-depth IHR control is a separate baseline. PPO uses the same reward for all methods. After each epoch, contiguous replay chunks update the world model; stop-gradient context prevents actor loss from directly training the model through its recurrent feature.

Complete original mathematical display S3.E8
Original selectable mathematical display
Equation 8
𝐮t=πθ​(𝐨t,𝐚,ψ⁡(𝐰t)),\mathbf{u}_{t}=\pi_{\theta}\!\left(\mathbf{o}_{t},\mathbf{a},\psi(\mathbf{w}_{t})\right), (8)
Original mathematical definitions, training assumptions and evaluation protocol

where 𝐨t\mathbf{o}_{t} is the deployable observation in Eq. (1) and ψ⁡(𝐰t)\psi(\mathbf{w}_{t}) is the projected deterministic WSM feature derived from Eq. (6). Actions are relative joint target updates,

Complete original mathematical display S3.E9
Original selectable mathematical display
Equation 9
𝐪ttarget=𝐪t−1target+α​𝐮t,\mathbf{q}^{\text{target}}_{t}=\mathbf{q}^{\text{target}}_{t-1}+\alpha\mathbf{u}_{t}, (9)
Original mathematical definitions, training assumptions and evaluation protocol

The actor input therefore comprises deployable observations 𝐨t\mathbf{o}_{t} and the WSM feature ψ⁡(𝐰t)\psi(\mathbf{w}_{t}), while the critic additionally receives simulator state during training. After each PPO epoch, replay chunks update the RSSM; a stopped gradient at the actor interface decouples the two optimization paths.

4 Experiments and Evaluations

Four experiment groups address multi-object rotation, the predictive-state mechanism, harder axes and stress conditions. Simulation episode statistics, adaptation training curves, four-object hardware comparisons and the separate twenty-object hardware protocol have different denominators. Qualitative demonstrations and repeated human annotations do not increase the number of independent trials.

4.1 Experimental Setup

The primary platform is Sharpa Wave, with wrist depth and tactile sensing. Simulation evaluates 128 episodes per reported entry unless stated otherwise. Random initial locations, shared rewards and common reset rules standardize the implemented baselines, while model capacity, pretraining and privileged auxiliary supervision remain part of the method being compared.

Hardware and Simulation Setting.

Hardware hand control, camera and tactile streams all run at 10 Hz; simulation control also runs at 10 Hz with 60 Hz physics. Joint-level sinusoidal system identification tunes simulated PD gains before transfer. Randomized dynamics, noise, reset poses, forces and gravity cover residual mismatch. These steps make the result simulation-to-real transfer with calibration, rather than transfer without robot-specific preparation.

Original mathematical definitions, training assumptions and evaluation protocol

We use the same deployable observation/action setting and control frequency in simulation and on hardware. Each reported simulation number averages 128128 evaluation episodes per method under randomized evaluation seeds. To reduce the sim-to-real gap, we combine joint-level system identification with domain randomization over dynamics, sensing, resets, and perturbations. Appendix A provides implementation details, and Appendix A.3 describes system identification and sim-to-real randomization.

Baselines.

Baselines include open-loop replay, proprioceptive Blind RL, tactile Touch Dexterity, depth-only control and the visuotactile In-Hand Rotation implementation. The paper also isolates scratch WSM, sensor heads, recurrence and clean-depth targets. The native tables and selectable copies retain every baseline, including settings where a baseline has lower off-axis motion or object velocity.

Original mathematical definitions, training assumptions and evaluation protocol

For the zz-axis evaluation, we compare Open-loop Replay, Blind RL Policy (no depth, no tactile), Touch Dexterity (w/ tactile) [1], RL Policy (w/ depth, w/o tactile), In-Hand Rotation (w/ depth + tactile) [6], and WM-Craftnet with WSM variants. The xx- and yy-axis evaluations use a compact baseline comparison. All baselines are reimplemented for the 2222-DoF Sharpa Wave Hand with the same reward, randomization range, initialization distribution, and reset strategy.

Metrics.

Simulation confidence intervals describe evaluation episodes, not variation across independent training seeds. Hardware RR is reported in radians over 20 seconds; SR counts trials exceeding a half-spin. Four annotators score the same synchronized recordings. Table 3's RR/SR aggregation is not fully specified: for example denoised IHR corner RR 2.43 rad with 10/10 half-spin success cannot be treated as a simple mean of the same nonnegative thresholded rotation quantities without further convention. The printed values are preserved; this ambiguity is not silently repaired or converted into a significance claim.

Original mathematical definitions, training assumptions and evaluation protocol

For simulation, we report return, episode length, rotation rate (RotR), and, where informative, off-axis motion and angular-velocity variation. Simulation entries are means with 95%95\% confidence intervals over 128128 evaluation episodes unless otherwise noted, and highlighting marks the first three ranks under each metric direction. For hardware, RR reports radians of rotation in a 2020-s episode, and SR is the success rate, where success requires more than a half-spin. Four annotators independently score synchronized videos, and their measurements are averaged.

4.2 Robust Multi-Object z-Axis Rotation

Full pretrained WSM achieves 753.3 return and 1.293 RotR versus 414.3/.742 from scratch. This is strong evidence for the combined pretrained representation, while scratch WSM does not outperform every raw-sensor baseline on rotation rate. Test-time zeroing of all tactile input retains 724.9 return, about 96.2% of the full model; that is robustness to a missing input, not training a tactile-free system. Proprioception-only and proprioception-plus-touch pretraining score 684.5/698.9. All table entries and the nine-object grid remain intact.

Original heading notationzz
Original mathematical definitions, training assumptions and evaluation protocol

We first evaluate whether one policy can rotate multiple everyday objects under randomized starts. The nine-object set beside Table 1 spans diverse shapes, mass-center distributions, textures, and contact affordances, requiring online state and contact inference rather than object-specific gaits. As shown in Table 1, baseline returns range from 236.1236.1 to 386.9386.9. WM-Craftnet trained from scratch improves return to 414.3414.3 and reduces OffAxis and AngVar to 1.2851.285 and 1.7351.735, while its 0.7420.742 RotR remains below some baselines. The pretrained full WSM reaches 753.3753.3 return and 1.2931.293 RotR, with OffAxis 1.2251.225 and AngVar 1.3241.324. Figure 2(a) compares the zz-axis methods; qualitative results are available on the project website11 1 https://wmcraftnet.github.io/.

Complete original Table 1: z z -axis multi-object rotation and WSM analysis on nine objects, including test-time tactile masking and modality controls.
Table 1: zz-axis multi-object rotation and WSM analysis on nine objects, including test-time tactile masking and modality controls.
Accessible original table, complete values and grouped headers
Method Return↑\uparrow EpLen↑\uparrow RotR↑\uparrow OffAxis↓\downarrow AngVar↓\downarrow
Open-loop Replay 306.3 ±\pm 48.3 320.2 ±\pm 35.0 0.332 ±\pm 0.080 1.789 ±\pm 0.295 2.241 ±\pm 0.679
Blind RL Policy (no depth, no tactile) 304.1 ±\pm 3.8 295.8 ±\pm 2.3 0.944 ±\pm 0.004 1.539 ±\pm 0.007 1.810 ±\pm 0.011
Touch Dexterity (w/ tactile)[1] 386.9 ±\pm 3.1 362.5 ±\pm 2.3 1.018 ±\pm 0.004 1.601 ±\pm 0.008 1.764 ±\pm 0.053
RL Policy (w/ depth, w/o tactile) 362.2 ±\pm 4.7 351.0 ±\pm 3.6 0.978 ±\pm 0.004 1.806 ±\pm 0.009 2.222 ±\pm 0.065
In-Hand Rotation (w/ depth + tactile)[6] 236.1 ±\pm 2.7 272.5 ±\pm 2.8 0.882 ±\pm 0.003 1.986 ±\pm 0.009 2.487 ±\pm 0.074
WM-Craftnet (from scratch) 414.3 ±\pm 15.7 264.2 ±\pm 8.8 0.742 ±\pm 0.012 1.285 ±\pm 0.010 1.735 ±\pm 0.052
WM-Craftnet (all-zero tactile) 724.9 ±\pm 4.4 428.8 ±\pm 2.8 1.252 ±\pm 0.004 1.224 ±\pm 0.006 1.388 ±\pm 0.031
WM-Craftnet (75% tactile dropout) 735.2 ±\pm 4.1 429.8 ±\pm 2.7 1.272 ±\pm 0.004 1.224 ±\pm 0.006 1.390 ±\pm 0.031
WM-Craftnet (50% tactile dropout) 737.8 ±\pm 4.4 425.1 ±\pm 3.6 1.276 ±\pm 0.004 1.279 ±\pm 0.006 1.511 ±\pm 0.061
WM-Craftnet (25% tactile dropout) 743.4 ±\pm 4.2 427.9 ±\pm 3.2 1.287 ±\pm 0.004 1.243 ±\pm 0.006 1.434 ±\pm 0.032
WM-Craftnet (WSM pretrained, prop-only) 684.5 ±\pm 4.3 424.0 ±\pm 2.2 1.191 ±\pm 0.003 1.310 ±\pm 0.007 1.234 ±\pm 0.013
WM-Craftnet (WSM pretrained, prop+tac) 698.9 ±\pm 4.3 435.4 ±\pm 1.9 1.192 ±\pm 0.002 1.165 ±\pm 0.006 1.373 ±\pm 0.024
WM-Craftnet (full WSM) 753.3 ±\pm 3.6 435.2 ±\pm 1.9 1.293 ±\pm 0.004 1.225 ±\pm 0.006 1.324 ±\pm 0.027
Original mathematical definitions, training assumptions and evaluation protocol

Figure 2(a) reports training reward for the zz-axis methods and shows the same ordering as the main comparison in Table 1. Table 1 also shows that reducing tactile availability from full input to 25%25\% dropout and all-zero input decreases return from 753.3753.3 to 743.4743.4 and 724.9724.9. The modality controls progress from 684.5684.5 with proprioception alone to 698.9698.9 with proprioception and touch, and 753.3753.3 with full depth–touch input; prop+touch yields lower OffAxis (1.1651.165) but lower RotR than full WSM. Figure 2(b) shows these controls.

Controlled WSM variants.

Clean-depth supervision adds 45.3 return over noisy-depth targets. Removing recurrent input gives 705.4 return, and the encoder–decoder feature gives 667.6. These controls support the full setup on return, but removing recurrent input improves OffAxis 1.225→1.125 and AngVar 1.324→1.185. The result is a tradeoff across metrics, not uniform superiority of recurrence on every stability measure.

Original mathematical definitions, training assumptions and evaluation protocol

Table 2 isolates clean-depth supervision, deterministic recurrence, and the recurrent policy feature. Reconstructing noisy rather than clean depth lowers return from 753.3753.3 to 708.0708.0, while replacing the recurrent policy context with a per-step encoder–decoder feature lowers it to 667.6667.6. Removing ht−1h_{t-1} yields 705.4705.4 return and 1.2821.282 RotR, but improves OffAxis and AngVar to 1.1251.125 and 1.1851.185, indicating a stability–task-performance trade-off. The training curves beside the table show that full WSM reaches the highest final reward.

Complete original Table 2: Controlled WSM variants on the z z -axis benchmark. The left panel shows training reward; the right panel reports evaluation metrics (mean ± \pm 95 % 95\% confidence interval).
Table 2: Controlled WSM variants on the zz-axis benchmark. The left panel shows training reward; the right panel reports evaluation metrics (mean ±\pm 95%95\% confidence interval).
Accessible original table, complete values and grouped headers
Method Return↑\uparrow EpLen↑\uparrow RotR↑\uparrow OffAxis↓\downarrow AngVar↓\downarrow
Noisy-depth supervision 708.0 ±\pm 3.5 431.1 ±\pm 2.0 1.265 ±\pm 0.005 1.299 ±\pm 0.006 1.456 ±\pm 0.045
No ht−1h_{t-1} recurrent input 705.4 ±\pm 4.3 434.3 ±\pm 2.3 1.282 ±\pm 0.005 1.125 ±\pm 0.006 1.185 ±\pm 0.008
Encoder–decoder ztz_{t} policy 667.6 ±\pm 4.7 417.4 ±\pm 2.6 1.254 ±\pm 0.003 1.323 ±\pm 0.006 1.343 ±\pm 0.018
Full WSM 753.3 ±\pm 3.6 435.2 ±\pm 1.9 1.293 ±\pm 0.004 1.225 ±\pm 0.006 1.324 ±\pm 0.027

Sim-to-real experiments.

The four-object robot comparison uses ten trials per object and 20-second rotation measurements. WM-Craftnet reports 10/10 success on duck, cross and corner, and 8/10 on the unseen double-notched block. Denoised-depth IHR improves over raw-depth IHR but remains below the recurrent method in reported rotation. This supports more than a depth-cleaning explanation; differences in representation and training remain, and the small trial counts do not isolate a unique causal mechanism.

Original mathematical definitions, training assumptions and evaluation protocol

We also evaluate real-robot zz-axis rotation on duck, cross block, corner block, and an unseen double-notched block. Table 2 compares a progression from fixed open-loop replay (OlR), to tactile feedback in Touch Dexterity (TD) [1], and then depth–tactile feedback in In-Hand Rotation (IHR) [6]. IHR is sensitive to the shift from simulated depth to noisy real observations. Feeding it WSM-denoised depth improves rotation and success on the three seen objects, confirming the value of geometric denoising, but yields limited rotation and no successful trials on the unseen block. WM-Craftnet conditions the actor on a recurrent WSM feature learned jointly from depth, touch, proprioception, and action history. This temporally informed context supports high RR and SR on both the seen objects and the unseen block.

Complete original Figure 2: Training reward curves on (a) z z -axis baselines, (b) WSM ablation and (c,d) x / y x/y -axis baselines.
Figure 2: Training reward curves on (a) zz-axis baselines, (b) WSM ablation and (c,d) x/yx/y-axis baselines.
Complete original Table 3: Real-robot z z -axis rotation. RR is radians per 20 20 -s episode and SR is success over 10 10 trials.
Table 3: Real-robot zz-axis rotation. RR is radians per 2020-s episode and SR is success over 1010 trials.
Accessible original table, complete values and grouped headers
Method
Duck
Cross
block
Corner
block
Double-notched
block(unseen)
RR SR RR SR RR SR RR SR
OlR 2.32 6/10 3.04 6/10 5.19 7/10 0.23 0/10
TD[1] 1.54 4/10 0.34 2/10 0.22 0/10 0.27 0/10
IHR[6] 1.83 5/10 0.45 1/10 0.36 1/10 0.39 0/10
IHR + WSM-denoised depth 2.76 8/10 1.21 8/10 2.43 10/10 0.80 0/10
Ours 16.179 10/10 8.01 10/10 8.48 10/10 4.32 8/10

4.3 World Synesthesia Model Analysis

Auxiliary-head Table 10 reports 688.1 return without heads,737.0 with value only and 753.3 with all heads. The pose-only head gives lower angular-velocity variation than the full model. Figure 3's t-SNE has object-dependent regions plus substantial overlap; it does not prove object recognition, state identifiability or causal disentanglement. Figure 4 and the original depth videos provide qualitative reconstruction evidence, not an independently measured hardware geometry-error benchmark.

Original mathematical definitions, training assumptions and evaluation protocol

The auxiliary-head study in Table B.1 raises return from 688.1688.1 without auxiliary heads to 737.0737.0 with the value head alone and 753.3753.3 with all heads. Table 2 provides the controlled clean-depth and recurrent-state variants, while the LSTM and GRU history baselines remain in Appendix B.1.

Original mathematical definitions, training assumptions and evaluation protocol

We further examine the learned state with two diagnostics. For the nine zz-axis objects, we project evaluation-time recurrent states 𝐡t\mathbf{h}_{t} with t-SNE to characterize object- and interaction-phase structure; object labels are used only to color the visualization and are not policy inputs. We also decode clean depth from the latent state and compare it with ground-truth and noisy wrist-depth inputs.

Complete original Figure 3: t-SNE visualization of WSM recurrent states 𝐡 t \mathbf{h}_{t} collected from z z -axis rotation rollouts over nine objects.
Figure 3: t-SNE visualization of WSM recurrent states 𝐡t\mathbf{h}_{t} collected from zz-axis rotation rollouts over nine objects.
Complete original Figure 4: Depth reconstruction from a simulated z z -axis duck rotation rollout.
Figure 4: Depth reconstruction from a simulated zz-axis duck rotation rollout.
Complete original Figure 5: Downstream policy learning over 49 49 new objects using the WSM pretrained on nine z z -axis objects.
Figure 5: Downstream policy learning over 4949 new objects using the WSM pretrained on nine zz-axis objects.

Reusable synesthetic prior.

After 3000 epochs, the adapted forty-nine-object policy reaches 9.37±.13 radians per episode versus 3.28 without the prior. These units differ from the rate metric in the main tables. The training fall rate decreases from 6%to.3% over learning; that temporal change is not a controlled 95%fall reduction against the no-prior baseline. A separate five-block prior-scaling diagnostic gives 1.10/1.34/1.46 RotR for scratch/nine-object/forty-nine-object priors. It uses a distinct diagnostic set and should not be pooled with axis-held-out trials.

Original mathematical definitions, training assumptions and evaluation protocol

The same predictive model can be reused beyond the nine-object zz-axis setting. Figure 5 and website videos show one downstream policy trained over 4949 new objects; after 30003000 epochs, each object rotates 9.37±0.13\boldsymbol{9.37\pm 0.13} rad per episode on average, compared with 3.283.28 rad for the no-prior baseline, while the training fall rate drops from 𝟔%\boldsymbol{6\%} to 0.3%\boldsymbol{0.3\%} from start to end. In a separate five-block prior-scaling diagnostic (diagonal, narrow, double-notched, step, and V-cut blocks), RotR improves from scratch / 9-object prior / 49-object prior as 1.10/1.34/1.461.10/1.34/1.46. This diagnostic is distinct from the axis-specific held-out sets below; Appendix B.2 shows the full object set and Figure B.2 visualizes its recurrent-state organization.

4.4 More Difficult Rotation Modes

The paper trains separate world models and policies for x,y and z. On nine y-axis tool-like objects, WSM gives 1.025 RotR versus the best baseline.545, with 215 episode length and.049 fall. On four x-axis objects it achieves 432.8 return and 435.4 episode length, but object velocity.0460 is higher than every baseline in Table 5. The axis command in the architecture therefore does not establish one trained generalist for all axes, nor do high return and low fall imply minimum object motion on every axis.

Original mathematical definitions, training assumptions and evaluation protocol

We next evaluate rotation axes with less passive palm support. yy-axis rotation creates larger moment arms and asymmetric contacts for elongated objects; xx-axis rotation is harder because the hand has limited fingertip workspace and must reallocate contacts while gravity pulls the object away from the palm. All axis-specific policies use the same WM-Craftnet design, reward, randomized starts, and deployable observations, with a separately trained WSM for each axis.

Original mathematical definitions, training assumptions and evaluation protocol

The yy-axis evaluation uses the tool-like object set shown with Table 4, testing whether the policy can rotate objects that tend to lever against the fingers. Table 4 shows that WM-Craftnet improves return from the strongest baseline value of 177.8177.8 to 272.7272.7, nearly doubles the best baseline RotR from 0.5450.545 to 1.0251.025, and achieves the lowest fall rate. The 88.1%88.1\% RotR gain is substantially larger than the 11.7%11.7\% EpLen gain, indicating that the return improvement reflects faster target-axis progress rather than only longer episodes. Its ObjVel ranks second rather than first, so the advantage is specific to controlled rotation and fall reduction, not uniformly lower object motion. Figure 2(d) shows the same training trend, suggesting that the recurrent state helps resolve the changing moment arms and asymmetric contacts.

Complete original Table 4: y y -axis rotation on nine tool-like objects. Metrics are averaged over the nine-object set shown in the adjacent grid and all evaluation trials.
Table 4: yy-axis rotation on nine tool-like objects. Metrics are averaged over the nine-object set shown in the adjacent grid and all evaluation trials.
Accessible original table, complete values and grouped headers
Method Return↑\uparrow EpLen↑\uparrow RotR↑\uparrow Fall↓\downarrow ObjVel↓\downarrow
Open-loop Replay -52.4 ±\pm 5.2 85.8 ±\pm 10.0 0.031 ±\pm 0.063 0.784 ±\pm 0.082 0.0366 ±\pm 0.0042
Blind RL Policy (no depth, no tactile) 177.8 ±\pm 9.2 192.4 ±\pm 7.6 0.545 ±\pm 0.016 0.164 ±\pm 0.002 0.0737 ±\pm 0.0014
Touch Dexterity (w/ tactile)[1] 105.3 ±\pm 5.8 172.0 ±\pm 6.2 0.297 ±\pm 0.030 0.208 ±\pm 0.003 0.0760 ±\pm 0.0021
RL Policy (w/ depth, w/o tactile) 154.8 ±\pm 6.9 178.4 ±\pm 5.4 0.288 ±\pm 0.033 0.103 ±\pm 0.002 0.0909 ±\pm 0.0015
In-Hand Rotation (w/ depth + tactile)[6] 142.8 ±\pm 5.5 162.5 ±\pm 4.8 0.377 ±\pm 0.025 0.063 ±\pm 0.003 0.0787 ±\pm 0.0021
WM-Craftnet 272.7 ±\pm 13.5 215.0 ±\pm 8.7 1.025 ±\pm 0.026 0.049 ±\pm 0.001 0.0682 ±\pm 0.0013
Original mathematical definitions, training assumptions and evaluation protocol

The xx-axis evaluation in Table 5 is a stricter test of contact sequencing: the policy must roll the object within a small effective finger workspace and reallocate contacts before the object leaves the palm-supported configuration.

Complete original Table 5: x x -axis rotation on four objects. Metrics are averaged over the four-object set shown in the adjacent grid and all evaluation trials.
Table 5: xx-axis rotation on four objects. Metrics are averaged over the four-object set shown in the adjacent grid and all evaluation trials.
Accessible original table, complete values and grouped headers
Method Return↑\uparrow EpLen↑\uparrow RotR↑\uparrow Fall↓\downarrow ObjVel↓\downarrow
Open-loop Replay 18.0 ±\pm 6.8 129.4 ±\pm 22.3 0.411 ±\pm 0.054 0.164 ±\pm 0.064 0.0448 ±\pm 0.0019
Blind RL Policy (no depth, no tactile) 44.0 ±\pm 0.8 63.4 ±\pm 1.0 0.867 ±\pm 0.002 0.033 ±\pm 0.001 0.0311 ±\pm 0.0001
Touch Dexterity (w/ tactile)[1] 37.8 ±\pm 0.2 42.7 ±\pm 0.2 1.023 ±\pm 0.002 0.039 ±\pm 0.002 0.0334 ±\pm 0.0001
RL Policy (w/ depth, w/o tactile) 38.2 ±\pm 0.2 43.7 ±\pm 0.4 1.079 ±\pm 0.003 0.033 ±\pm 0.001 0.0323 ±\pm 0.0001
In-Hand Rotation (w/ depth + tactile)[6] 295.1 ±\pm 3.3 365.1 ±\pm 3.1 0.839 ±\pm 0.001 0.103 ±\pm 0.003 0.0433 ±\pm 0.0001
WM-Craftnet 432.8 ±\pm 2.1 435.4 ±\pm 1.8 1.114 ±\pm 0.001 0.020 ±\pm 0.001 0.0460 ±\pm 0.0001
Original mathematical definitions, training assumptions and evaluation protocol

Table 5 and Figure 2(c) show that blind, tactile-only, and depth-only policies stay below 4545 return and 6464 episode length, while open-loop replay lasts longer but still has low return, low RotR, and a high fall rate. WM-Craftnet reaches 432.8432.8 return, 435.4435.4 episode length, the highest RotR, and the lowest fall rate, maintaining grasp while finding contacts that advance the commanded spin.

4.5 Stress Tests in Challenging Interaction States

Held-out tests use four different objects per axis and report zero-shot performance for those axis-specific policies. RotR gains over the best baseline are 8.6/89.2/20.9% for x/y/z, while the gains in return differ. Severe simulated recovery is 14.1% versus the best baseline 6.2%, an increase of 7.9 percentage points with low absolute success. Recovery time 2.56 s is conditional on recovered cases, not an unconditional rescue-time guarantee. Figure 6 deliberately shows successful examples. A separate twenty-object physical z-rotation study reports 175/200 versus 33–53/200 baseline successes; these trials measure broad-object rotation, not a 200-trial disturbance-recovery rate.

Original mathematical definitions, training assumptions and evaluation protocol

Unseen-object tests. We begin with axis-specific zero-shot transfer in simulation. We directly evaluate the xx-, yy-, and zz-axis policies trained on their respective 44-, 99-, and 99-object training sets, without object-specific fine-tuning. For each axis, the held-out evaluation contains four unseen objects: small corner block, narrow block, large step block, and small Lego for xx; double-notched block, large flashlight, narrow block, and V-cut block for yy; and lidded jar, tilted tumbler, handled pot, and pedal bin for zz. Table 6 shows that WM-Craftnet achieves the highest RotR and return in all three axis-specific evaluations. Relative to the strongest non-WSM entry on each metric, the RotR gains are 8.6%8.6\%, 89.2%89.2\%, and 20.9%20.9\% for xx, yy, and zz, respectively; the corresponding return gains are 35.6%35.6\%, 5.4%5.4\%, and 76.7%76.7\%.

Original mathematical definitions, training assumptions and evaluation protocol

Perturbation-recovery test. We next isolate recovery under severe simulated perturbations and measure whether the policy restores a controllable configuration, together with the elapsed time among recovered trials. WM-Craftnet reaches 14.1%14.1\% recovery success, compared with 6.2%6.2\% for the strongest baseline, and has the shortest mean recovery time at 2.562.56 s. This is a 7.97.9-percentage-point absolute gain and a 0.580.58-s reduction in mean recovery time over the strongest baseline on the respective metric. Under difficult starts and in-process force perturbations, the representative successful rollouts in Figure 4.5 show the policy re-centering, uprighting, or transferring the object back into a controllable finger workspace before resuming target-axis rotation. The absolute recovery rate remains limited under these large disturbances.

Complete original Figure 6: Qualitative stress tests in simulation and on hardware. The six rows show representative successful sequences in which WM-Craftnet returns the object to a controllable workspace and resumes rotation; quantitative recovery rates are evaluated in simulation.
Figure 6: Qualitative stress tests in simulation and on hardware. The six rows show representative successful sequences in which WM-Craftnet returns the object to a controllable workspace and resumes rotation; quantitative recovery rates are evaluated in simulation.
Original mathematical definitions, training assumptions and evaluation protocol

Large-scale real-world evaluation. Finally, in a separate large-scale hardware protocol, we test standard zz-axis in-hand rotation on 2020 real objects with 1010 trials per object. The set combines the nine training objects with eleven unseen objects: lidded jar, tilted tumbler, pedal bin, handled pot, diagonal block, narrow block, double-notched block, step block, V-cut block, trash can, and bucket. WM-Craftnet succeeds in 175/200175/200 trials, compared with 33/20033/200–53/20053/200 for the baselines. This hardware result measures rotation success across a broad object set.

Complete original Table 6: Stress tests on axis-specific held-out rotation, simulated z z -axis perturbation recovery, and large-scale real-world z z -axis rotation. The left panel shows four held-out objects per axis. Simulation entries report RotR and return; Real Rot. SR aggregates 200 200 hardware trials over 20 20 objects using the half-spin success criterion.
Table 6: Stress tests on axis-specific held-out rotation, simulated zz-axis perturbation recovery, and large-scale real-world zz-axis rotation. The left panel shows four held-out objects per axis. Simulation entries report RotR and return; Real Rot. SR aggregates 200200 hardware trials over 2020 objects using the half-spin success criterion.
Accessible original table, complete values and grouped headers
Method RotR, Ret. (xx)↑\uparrow RotR, Ret. (yy)↑\uparrow RotR, Ret. (zz)↑\uparrow Sim. Recovery↑\uparrow Sim. Time↓\downarrow (s) Real Rot. SR↑\uparrow
Blind RL Policy 0.852, 134.7 0.300, 135.7 0.665, 277.1 4.7% ±\pm 3.7% 3.62 ±\pm 2.25 41/200
Touch Dexterity [1] 0.923, 35.8 0.174, 78.7 0.605, 242.2 6.2% ±\pm 4.2% 3.33 ±\pm 1.41 33/200
RL Policy (w/o tactile) 0.853, 33.8 0.378, 168.9 0.696, 301.7 5.5% ±\pm 4.0% 3.14 ±\pm 0.79 45/200
In-Hand Rotation [6] 0.732, 246.0 0.309, 193.0 0.468, 156.1 0.0% ±\pm 0.0% N/A 53/200
WM-Craftnet 1.002, 333.6 0.715, 203.5 0.842, 533.1 14.1% ±\pm 6.0% 2.56 ±\pm 0.77 175/200

Tool use behavior.

Screwdriver translation and rotation use the same policy formulation, as detailed in AppendixB.3. The paper offers qualitative tool-style manipulation sequences, without a quantitative tool-use task suite or evidence that one unchanged rotation policy performs every tool behavior.

5 Conclusions and Limitations

Predictive multimodal training provides a useful recurrent state for this manipulation interface, with consistent gains on important return and rotation measures. Limits include short horizons, restricted hand workspace, large offsets outside the recovery basin, drift, jamming and low severe-perturbation recovery. Further work on tools and broader distributions is proposed. The experiments do not demonstrate long-horizon tool completion or universal robustness.

Original mathematical definitions, training assumptions and evaluation protocol

This paper presents WM-Craftnet, a world model-based visuotactile framework for robust dexterous in-hand manipulation. WSM learns Dreamer-style latent dynamics from noisy depth, proprioception, tactile sensing, and actions, reconstructs clean depth targets, and conditions the policy with its deterministic recurrent feature. Depth reconstruction suppresses sensor artifacts while preserving hand-object geometry. The predictive representation also serves as a reusable prior for a 4949-object downstream policy. Experiments show improved multi-object rotation, unseen-object transfer, perturbation recovery, and sim-to-real performance over baseline visuotactile methods.

Original project figures and demonstration recordings

Original project videos, pipeline and learning/latent diagrams, the Sharpa artwork and conference poster are local. Each recording retains its source URL and caption; preview posters are derived frames. The gravity-invariant demonstrations use a closer grasp, modified initial conditions and reward design described separately on the project site. They should not be interpreted as the unchanged palm-up benchmark policy or additional controlled evidence for the tables. Baseline, failure and reconstruction recordings remain alongside successful demonstrations.

WM-Craftnet World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation ↓
Sharpa
Sharpa
Overview of WM-Craftnet pipeline
Overview of WM-Craftnet pipeline
Training reward curves and real robot z-axis evaluation
Training reward curves and real robot z-axis evaluation
t-SNE visualization of WSM recurrent states
t-SNE visualization of WSM recurrent states
49-object downstream rotation with a WSM pretrained on the nine-object z-axis setting.
Real Noisy Wrist Depth
Real Predicted Depth from WSM Latent
Third-Person RGB: duck z-axis rotation
Ground-Truth Wrist Depth
Noisy Depth Input to WSM
Predicted Depth from WSM Latent
Continuous multi-object real-world rollout. The same policy rotates several objects in one uninterrupted run: 0-24 s duck, 24-35 s cylinder, 35-50 s cross block, 50-73 s steamed bun, and 73-94 s an unseen double-notched block. In the unseen-object segment, the policy initially struggles, adapts online through closed-loop sensing , and gradually reaches stable rotation .
OOD-to-duck recovery. An unseen object first drives the policy into an out-of-distribution interaction state. After switching to the duck, WM-Craftnet recovers a controllable grasp and resumes successful z-axis rotation on the physical hand.
Real-world perturbation recovery. 3-5 s and 22-25 s : a human perturbs the corner block's position; the policy adjusts the object back into a controllable state and continues smooth rotation .
Challenging-start recovery. At 0 s , the switch starts in a side-fall state ; the policy recovers from the challenging initial pose and brings the object back into a controllable rotation mode.
Bucket palm-region recovery. During bucket rotation, at 6 s the object pops into the palm region; WM-Craftnet pushes it back toward the comfortable workspace and continues stable rotation .
Cylinder unexpected-contact recovery. Around 6 s and 15 s , unexpected contacts push the cylinder toward the palm; the policy adjusts it back into the comfortable region and keeps the rotation going .
Duck. WM-Craftnet keeps stable finger contacts while rotating a curved, asymmetric object.
Corner block. WM-Craftnet maintains stable z-axis rotation for over one minute without side-fall.
Cross block. Closed-loop visuotactile feedback supports rotation with protruding arms and changing contact patches.
Cylinder. The hand produces smooth z-axis spin on a rounded object with limited tactile landmarks.
Piggy bank. WM-Craftnet preserves controllable rotation on an anisotropic everyday object with direction-dependent contact geometry.
Strawberry. The policy maintains rotation on an anisotropic tapered object whose irregular profile creates shifting contact patches.
Fire hydrant. WM-Craftnet handles protruding side structures and changing contact geometry while sustaining z-axis rotation.
Blue block. The policy keeps the block upright and achieves stable continuous rotation for more than six full turns.
Bulb. Despite an unstable center of mass, closed-loop sensing lets the policy adjust contacts and recover balanced rotation.
Open-loop Replay open-loop The replayed gait has no feedback or adjustment ability ; even with the block near the palm center, it cannot recover a controllable rotation mode .
Touch Dexterity tactile-only feedback The tactile policy can produce rotation, but without explicit geometry the object becomes unstable and frequently tips into side-fall states .
In-Hand Rotation tactile+noisy real depth Real sensor noise quickly pushes the depth observation out of the simulated training distribution, causing an OOD interaction state and loss of stable rotation .
IHR + WSM Depth tactile+denoised depth WSM-reconstructed depth gives a cleaner geometry estimate and slightly improves behavior, but depth denoising alone cannot sustain stable rotation .
Corner block
Duck
Apple
Piggy Bank
Strawberry
Cross block
Coca Can
Bulb
Double-notched Cylinder
Compact-grasp rotation. A single policy rotates a duck, a strawberry, and a cylinder using the closer hand posture.
Successful sim-to-real transfer. The object remains grasped and keeps rotating as palm orientation changes under gravity.
Toothpaste. The policy keeps an elongated object controllable while rotating around its shorter axis under asymmetric leverage.
Screwdriver. Closed-loop visuotactile feedback stabilizes a tool-like object and maintains y-axis rotation despite the handle-shaft geometry.
Corner block. WM-Craftnet sustains real-world y-axis rotation despite sharp edges, intermittent contacts, and changing sideward torques.
Toy Trashcan
Corner block
Bulb
Corner Cylinder
Toothpaste
Coca Can
Screwdriver
Flashlight
Bottle
block
Corner block
Stepped block
Lego Brick
2-3 s : a force toward the palm pushes the cross block out of the comfortable rotation region. 3-6 s : WM-Craftnet adjusts the cross block back into the workspace and continues rotation.
0-16 s : notched block, 16-33 s : narrow block, 33-47 s : double-notched block.
Bottle: thumb-guided re-centering, then y-axis rotation From 0-10 s , the thumb adjusts the bottle closer to the fingers , moving it into a stronger contact region before the hand continues rotation around the y-axis.
Bulb: three-finger uprighting, then z-axis rotation From 0-5 s , the thumb, index finger, and middle finger coordinate to make the bulb closer to an upright pose , enabling smoother rotation around the z-axis.
Coke Can: five-finger stabilization, then z-axis rotation From 0-5 s , all five fingers cooperate to bring the coke can upright , showing closed-loop recovery from a less favorable initial pose before z-axis rotation.
Corner block: workspace-aware adjustment, then z-axis rotation From 0-4 s , the thumb and little finger guide the object toward the palm-finger junction , a comfortable hand workspace where stable contacts can support subsequent z-axis rotation.
Cross block: palm-to-finger transfer, then z-axis rotation From 0-7 s , all five fingers coordinate to move the object from the palm onto the fingers , converting an unstable start into a fingertip-controlled configuration for z-axis rotation.
Piggy Bank: five-finger adjustment, then z-axis rotation From 0-5 s , all five fingers adjust the piggy bank into a comfortable position , establishing stable contacts before continuing rotation around the z-axis.
Goal-conditioned translation. The hand adjusts the screwdriver toward a target position while keeping stable contact along the handle.
Fast screwdriver rotation. The same tool-use setup demonstrates rapid in-hand rotation while preventing the tool from slipping out of the grasp.
Baseline Block, real z-axis Noisy real depth rapidly drives the baseline out of distribution, preventing it from producing a coherent rotation strategy.
Ours Corner block, challenging start WM-Craftnet attempts to adjust from a difficult initial pose, but cannot recover a stable controllable configuration.
Ours Duck, long-horizon drift After completing 5 successful spins, the duck tips toward one side and the policy loses the stable contact mode needed to continue at 29s.
Baseline Bulb, y-axis The bulb starts on the palm; during adjustment, the policy pushes it out of the hand and drops it.
Baseline Bulb, z-axis The object rotates, but the hand uses an unnatural posture and the bulb drifts far from the intended rotation axis.
Baseline Duck, z-axis The baseline can induce rotation, but with an awkward finger gait and substantial off-axis object displacement.
Baseline Notched block, x-axis During x-axis rotation of the notched block, contact becomes unstable and the object drops midway through the rollout.
Baseline block, x-axis The baseline struggles to keep a stable axis-aligned grasp, causing the block to drift and lose controllable rotation.
Baseline Flashlight, y-axis The flashlight enters a stuck contact state, preventing the policy from continuing useful in-hand rotation.
Baseline Toothpaste, y-axis While rotating the toothpaste-like object, the policy fails to explore a useful contact mode and cannot complete rotation.
Ours Bulb, z-axis WM-Craftnet can still fail on difficult bulb manipulation when contact recovery is insufficient and the object falls.
WM-Craftnet CoRL 2026 poster preview
WM-Craftnet CoRL 2026 poster preview

Appendix A Implementation Details

The implementation appendix supplies world-model optimization, deployed versus privileged inputs, PPO/control/reward settings and dynamics calibration. Both original tables remain complete rather than being reduced to selected hyperparameters. Reproduction requires the specified hand, sensing preprocessing, simulator supervision and system-identification procedure.

Complete original Table 7: Compact implementation parameters used in the current WM-Craftnet configuration.
Table 7: Compact implementation parameters used in the current WM-Craftnet configuration.
Accessible original table, complete values and grouped headers
Parameter Value
System
Hand Sharpa Wave, 2222-DoF five-finger hand
Hardware rates hand control / wrist camera / tactile: 1010 Hz
Camera stream wrist-mounted, 640×480640\times 480, fixed by 3D-printed bracket
Simulation rates physics 6060 Hz; observation/control 1010 Hz
World Model
Type Dreamer-style RSSM with multimodal inputs
Deterministic / stochastic / discrete dims 512/32/32512/32/32
Hidden size / recurrent depth 512/1512/1
Actor projection 512→64→32→16512\rightarrow 64\rightarrow 32\rightarrow 16
Replay capacity / warmup 256256 rows / 10,00010{,}000 transitions
Batch size / chunk length / updates 16/32/1016/32/10 per PPO epoch
Optimizer / LR / grad clip Adam / 10−410^{-4} / 10001000
KL free / dyn scale / rep scale 1.0/0.5/0.11.0/0.5/0.1
Depth Input
Policy depth tensor / max depth 96×72/0.496\times 72/0.4 m
Temporal dropout / corr. 0.02/0.950.02/0.95
Random image rotation ±0.3∘\pm 0.3^{\circ}
Policy and Control
Envs / horizon / minibatch / mini-epochs 1024/16/4096/41024/16/4096/4
γ\gamma / GAE τ\tau / LR / clip 0.99/0.95/10−4/0.20.99/0.95/10^{-4}/0.2
Policy MLP / activation [512,256,256][512,256,256] / ELU
DoF / episode / sim dt / control decimation 22/500/0.01667/622/500/0.01667/6
Reward
Spin / max spin / velocity 1.0/2.0/−0.11.0/2.0/-0.1
Torque / work / action / control −0.003/−0.003/−0.002/−1.0-0.003/-0.003/-0.002/-1.0
Finger distance / hand pose / off-axis 0.1/1.5/0.0010.1/1.5/0.001
Object drift / no-spin / fall penalty 100.0/−50.0/−50.0100.0/-50.0/-50.0
Tip force coef. / thresh. / exp. 0.9/2.0/0.010.9/2.0/0.01
Randomization
Object mass train / test [0.2,0.4][0.2,0.4] kg / [0.1,1.4][0.1,1.4] kg
Friction scale / PD scale [0.5,2.0]/[0.8,1.2][0.5,2.0]/[0.8,1.2]
Obs noise / action noise [0,0.05]/[0,0.04][0,0.05]/[0,0.04]
Reset pos. / reset rot. [0.035,0.02,0.0025][0.035,0.02,0.0025] m / 10∘10^{\circ}
External force scale / prob. / decay 0.5/0.25/0.990.5/0.25/0.99
Gravity curriculum 1∘→4∘1^{\circ}\rightarrow 4^{\circ}, step 0.05∘0.05^{\circ}
Complete original Table 8: Deployable and privileged information used by WM-Craftnet. The actor is restricted to deployable signals; simulator-only state is used for reward, critic training, and diagnostics.
Table 8: Deployable and privileged information used by WM-Craftnet. The actor is restricted to deployable signals; simulator-only state is used for reward, critic training, and diagnostics.
Accessible original table, complete values and grouped headers
Signal Used by Description
Hand proprioception Actor, WM normalized 2222-DoF joint state and previous target history
Tactile/contact Actor, WM binary contact stream with threshold, dropout, and latency noise
Depth image Actor, WM noisy wrist depth input; clean depth is used only as WM reconstruction target
Commanded axis Actor goal direction 𝐚∈{±x,±y,±z}\mathbf{a}\in\{\pm x,\pm y,\pm z\}
World-model feature Actor projected deterministic RSSM feature ψ⁡(𝐡t)\psi(\mathbf{h}_{t})
Relative action Controller, WM 2222-DoF joint target increment, smoothed before execution
Object state Reward/critic only pose, velocity, contact force, and fall/reset diagnostics in simulation

A.1 World Model Training Details

A ring replay buffer stores noisy observations, clean depth targets, previous action, reward and reset flags. After 10,000 transition warmup, batches of 16 contiguous 32-step chunks train the model, with 10 updates per PPO epoch. The deterministic/stochastic/discrete dimensions are 512/32/32, and the actor projection is 512→64→32→16. Auxiliary pose/value/shape heads are used in both pretraining and downstream adaptation. Transfer includes continued model updates, not just learning a new actor against a frozen latent state.

Original mathematical definitions, training assumptions and evaluation protocol

Table 7 summarizes the world-model and depth-preprocessing settings. The world-model update receives a single-step proprioceptive vector, tactile/contact vector, noisy preprocessed depth image, previous action, and an episode-start flag; reward is stored only as a prediction target. The RSSM performs an observation update at every control step and exposes only the detached deterministic feature 𝐡t\mathbf{h}_{t}, projected by a small MLP, as actor context.

Original mathematical definitions, training assumptions and evaluation protocol

For pretrained-prior evaluations, we transfer the encoder, multimodal RSSM, depth/proprioceptive decoder, and reward predictor. During downstream 4949-object PPO training, these components continue to adapt with new rollout data, while the actor–critic and task-specific heads are learned for the target distribution. This design transfers predictive physical structure across object sets while learning control for the new task distribution.

A.2 Policy Training Details

Equation 10 preserves all reward terms, including spin, object velocity, contact/distance, torque/work, action/control penalties, pose, drift, off-axis motion and no-spin resets. Reward weights are not interchangeable with measured physical guarantees. Episodes terminate on drop distance, axis deviation, prolonged insufficient spin or horizon, which helps explain the relationship between return, episode length and spin. Relative joint targets are smoothed before the 10 Hz hardware interface. The full parameter table preserves signs and thresholds.

Original mathematical definitions, training assumptions and evaluation protocol

Table 7 lists PPO, control, and reward settings, and Table 8 summarizes deployable and simulation-side signals. The actor outputs a relative target update for all 2222 hand joints. A signal-level filter smooths these targets before execution by the 1010 Hz hardware controller.

Complete original mathematical display A1.E10
Original selectable mathematical display
Equation 10
rt:=\displaystyle r_{t}:= λspin​rtspin+λvel​rtvel+λcontact​rtcontact+λdist​rtdist+λτ​rtτ+λwork​rtwork\displaystyle\lambda_{\text{spin}}r_{t}^{\text{spin}}+\lambda_{\text{vel}}r_{t}^{\text{vel}}+\lambda_{\text{contact}}r_{t}^{\text{contact}}+\lambda_{\text{dist}}r_{t}^{\text{dist}}+\lambda_{\tau}r_{t}^{\tau}+\lambda_{\text{work}}r_{t}^{\text{work}} (10)
+λa​rta+λctrl​rtctrl+λpose​rtpose+rtdrift+rtaxis+rtnospin.\displaystyle+\lambda_{a}r_{t}^{a}+\lambda_{\text{ctrl}}r_{t}^{\text{ctrl}}+\lambda_{\text{pose}}r_{t}^{\text{pose}}+r_{t}^{\text{drift}}+r_{t}^{\text{axis}}+r_{t}^{\text{nospin}}.
Original mathematical definitions, training assumptions and evaluation protocol

The target-axis spin term uses signed incremental rotation projected onto 𝐚\mathbf{a}, clipped by a maximum spin rate. Stability terms penalize object velocity, drift, off-axis rotation, high torque, mechanical work, large actions, and tracking error; contact and pose terms encourage useful fingertip proximity, avoid excessive force or non-tip contacts, and softly favor natural hand postures.

A.3 System Identification and Domain Randomization

The wrist-mounted RealSenseL515 supplies 640×480 depth, preprocessed to 96×72 with.4 m maximum depth. A 1 Hz sinusoid excites each joint for gain tuning; Figure 7 reports average calibrated sim-real tracking error below.2 degrees under that test. Training mass is.2–.4 kg with broader .1–1.4 kg testing, plus friction/gain/noise/reset/force/gravity randomization. Small sinusoidal tracking error does not certify contact dynamics or arbitrary high-speed trajectories.

Original mathematical definitions, training assumptions and evaluation protocol

Real-world evaluation uses the Sharpa Wave hand with tactile sensing and a wrist-mounted depth camera (RealSense L515). The hand controller, wrist camera, and tactile stream run at 1010 Hz, and simulation uses the same policy/control rate with a 6060 Hz physics step. The policy receives the same deployable observation/action interface in both domains.

Original mathematical definitions, training assumptions and evaluation protocol

We identify joint-level dynamics before policy transfer by exciting the real hand with 11 Hz sinusoidal action commands and comparing the measured joint response with simulation. The simulator KpK_{p} and KdK_{d} gains are adjusted to align the amplitude and phase of the simulated trajectories with the hardware response. This calibration provides a nominal dynamics model, while domain randomization covers residual mismatch. Figure 7 visualizes the post-calibration joint tracking fit across the hand. The simulated trajectories closely match the real joint responses under the same sinusoidal commands, with an average sim-real tracking error below 0.2∘0.2^{\circ}.

Complete original Figure 7: Joint-level system identification. Dashed green curves show commanded actions, orange curves show real joint states, and blue curves show calibrated simulation states. Across the evaluated joints, the average sim-real tracking error is below 0.2 ∘ 0.2^{\circ} .
Figure 7: Joint-level system identification. Dashed green curves show commanded actions, orange curves show real joint states, and blue curves show calibrated simulation states. Across the evaluated joints, the average sim-real tracking error is below 0.2∘0.2^{\circ}.
Original mathematical definitions, training assumptions and evaluation protocol

Training randomizes object mass, friction, PD gains, observation/action noise, reset pose, external forces, and gravity direction as in Table 7. Object mass is sampled in [0.2,0.4][0.2,0.4] kg during training and evaluated over a wider test range. Friction and hand-joint gains are randomized to cover contact and actuator uncertainty; depth and tactile observations include dropout, noise, small image rotations, threshold variation, and latency. External forces and a gravity-direction curriculum further encourage perturbation recovery instead of memorizing a single palm-up rotation gait.

Appendix B Additional Experiments

Additional experiments separate recurrent-history baselines from predictive objectives, inspect larger-scale latent structure and show tool-style manipulation. Their evidence has different scope from the standard rotation and recovery tables; the original displays and object grids are preserved in place.

B.1 Additional WSM Mechanism Ablations

Full WSM reaches 753.3 return/1.293 RotR versus 621.0/1.186 for LSTM and 497.8/1.095 for GRU without predictive objectives. The full model's OffAxis 1.225 is nearly the GRU's 1.226, so that metric shows little separation. Table 10 retains all single-head combinations and uncertainty. Value-only is the strongest single head on return; pose-only gives lower AngVar 1.231 than full 1.324. The controls support predictive supervision overall but do not make every auxiliary head improve every metric.

Complete original Table 9: Recurrent-history baselines and Full WSM on the z z -axis benchmark (mean ± \pm 95 % 95\% confidence interval).
Table 9: Recurrent-history baselines and Full WSM on the zz-axis benchmark (mean ±\pm 95%95\% confidence interval).
Accessible original table, complete values and grouped headers
Method Return↑\uparrow EpLen↑\uparrow RotR↑\uparrow OffAxis↓\downarrow AngVar↓\downarrow
LSTM (w/o predictive obj.) 621.0 ±\pm 4.3 424.3 ±\pm 2.7 1.186 ±\pm 0.004 1.297 ±\pm 0.006 1.392 ±\pm 0.028
GRU (w/o predictive obj.) 497.8 ±\pm 4.5 377.1 ±\pm 2.5 1.095 ±\pm 0.003 1.226 ±\pm 0.007 1.357 ±\pm 0.013
Full WSM 753.3 ±\pm 3.6 435.2 ±\pm 1.9 1.293 ±\pm 0.004 1.225 ±\pm 0.006 1.324 ±\pm 0.027
Original mathematical definitions, training assumptions and evaluation protocol

Full WSM reaches 753.3753.3 return and 1.2931.293 RotR, compared with 621.0/1.186621.0/1.186 for LSTM and 497.8/1.095497.8/1.095 for GRU. It also lowers AngVar to 1.3241.324, while its OffAxis 1.2251.225 is comparable to GRU’s 1.2261.226. These results separate the benefit of predictive multimodal training from recurrent memory alone.

Original mathematical definitions, training assumptions and evaluation protocol

Table B.1 measures how simulation-side auxiliary objectives shape WSM. Removing all auxiliary heads yields 688.1688.1 return, while value, pose, and object-shape prediction each improve different aspects of the learned state. The value head produces the largest return gain among the single-head variants, whereas pose prediction gives the lowest angular-velocity variation. Combining all heads achieves the strongest overall return and rotation rate.

Complete original Table 10: Auxiliary-head ablations on the z z -axis benchmark.
Table 10: Auxiliary-head ablations on the zz-axis benchmark.
Accessible original table, complete values and grouped headers
Method Return↑\uparrow EpLen↑\uparrow RotR↑\uparrow OffAxis↓\downarrow AngVar↓\downarrow
No auxiliary heads 688.1 ±\pm 3.7 433.7 ±\pm 2.0 1.239 ±\pm 0.006 1.229 ±\pm 0.005 1.778 ±\pm 0.138
Only value head 737.0 ±\pm 4.9 435.7 ±\pm 2.2 1.291 ±\pm 0.003 1.258 ±\pm 0.007 1.346 ±\pm 0.045
Only pose head 690.0 ±\pm 5.8 432.3 ±\pm 2.5 1.221 ±\pm 0.007 1.228 ±\pm 0.008 1.231 ±\pm 0.011
Only object-shape head 689.2 ±\pm 4.2 428.1 ±\pm 2.2 1.261 ±\pm 0.002 1.255 ±\pm 0.006 1.535 ±\pm 0.032
Full WSM 753.3 ±\pm 3.6 435.2 ±\pm 1.9 1.293 ±\pm 0.004 1.225 ±\pm 0.006 1.324 ±\pm 0.027

B.2 Large-Scale Object Set

Figure 8 contains the entire forty-nine-object training set and the downstream state projection. Colors indicate object identities for visualization; they are not identity labels passed to the actor. Overlapping trajectories are consistent with shared contact phases and geometry-dependent behavior, but t-SNE alone cannot distinguish these explanations or establish a physically complete latent state.

Original mathematical definitions, training assumptions and evaluation protocol

Figure B.2(a) shows the new 4949-object set used for downstream large-scale multi-object policy training after WSM pretraining on the nine zz-axis objects. The set covers substantial variation in size, aspect ratio, curvature, and contact geometry. To examine whether the transferred WSM retains structured interaction states at this larger scale, we project evaluation-time recurrent states from the downstream policy with t-SNE. Figure B.2(b) shows locally coherent trajectories and partially object-dependent regions together with substantial overlap. This pattern is consistent with a representation that captures both object-dependent geometry and interaction phases shared across objects, rather than acting as a strict object-identity classifier.

Complete original Figure 8: Large-scale downstream analysis. (a) The 49 49 -object set used for multi-object policy training. (b) t-SNE projection of evaluation-time WSM recurrent states, with colors denoting object identities; locally coherent trajectories, partially separated regions, and shared regions coexist across the distribution.
Figure 8: Large-scale downstream analysis. (a) The 4949-object set used for multi-object policy training. (b) t-SNE projection of evaluation-time WSM recurrent states, with colors denoting object identities; locally coherent trajectories, partially separated regions, and shared regions coexist across the distribution.

B.3 Screwdriver Tool-Use Manipulation

Figure 9 preserves both goal-conditioned screwdriver translation and elongated-object rotation, including all frames and target labels. The evidence is qualitative. It does not measure successful fastening, force-controlled insertion or multi-step tool-use completion, and the common interface should not be confused with a single universally trained policy.

Complete original Figure 9: Screwdriver tool-use manipulation. (a) Goal-conditioned screwdriver translation, where the policy adjusts the screwdriver to the target position. (b) Screwdriver rotation, where the policy rotates an elongated screwdriver object.
Figure 9: Screwdriver tool-use manipulation. (a) Goal-conditioned screwdriver translation, where the policy adjusts the screwdriver to the target position. (b) Screwdriver rotation, where the policy rotates an elongated screwdriver object.

B.4 Supplementary rollouts and failure cases.

The original project includes depth reconstruction, simulation and physical rotation, perturbations, tool examples, baselines and explicit failures. Failure cases show why successful recovery montages cannot substitute for the quantitative severe-disturbance rate. The recordings are preserved as supplementary observations, without counting scenes, cameras or edited sequences as new independent trials.

Concept demonstration

Open image ↗