Reading guide
All fifteen pages, nine complete figures, ten complete tables, ten numbered equations and twenty-five references are covered. WM-Craftnet trains a predictive visuotactile recurrent state and feeds a detached projection into PPO. It uses neither imagined trajectories nor policy gradients through the world model. The main evidence concerns a specific 22-joint hand, simulation-trained axis-specific policies, object-set adaptation and limited physical trials. Full original project media accompanies the edition, including failure examples.
At a glance
The World Synesthesia Model combines noisy wrist depth, proprioception, binary contact and previous actions. Reconstruction, prediction and simulator-only auxiliary targets shape a recurrent context for the actor. Pretraining improves nine-object rotation, adaptation reuses that representation for forty-nine new objects, and hardware experiments report 175/200 successful rotations across twenty objects. These results support a useful learned state estimator; they do not establish a general-purpose simulator, a frozen zero-shot forty-nine-object policy or reliable recovery from arbitrary disturbances.
Original mathematical definitions, training assumptions and evaluation protocol
Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor–critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine -axis objects serves as a reusable prior for -object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.
1 Introduction
In-hand manipulation changes contact geometry continuously, while depth has missing/noisy pixels and touch alone gives little global shape information. The proposed response is an action-conditioned recurrent representation that predicts across modalities. Its value is tested through clean-depth, recurrence, sensor-head and auxiliary-head controls. Object identifiers are absent from the deployed actor, but privileged simulator labels still contribute to representation learning.
Original mathematical definitions, training assumptions and evaluation protocol
WM-Craftnet learns a World Synesthesia Model with a Dreamer-style recurrent state from proprioception, noisy depth, tactile observations, and recent actions, with reward used as a prediction target during training. The policy is conditioned on this recurrent feature, which captures object state, geometry, and contact evolution. During WSM training, noisy augmented depth is used as input while clean wrist-depth targets supervise reconstruction, encouraging denoised geometry for downstream control. This representation improves robust multi-object rotation, supports unseen-object evaluation, and enables WSM pretraining as a reusable prior for downstream -object control. We summarize our contributions as follows:
Original mathematical definitions, training assumptions and evaluation protocol
We analyze WSM with recurrent-history baselines, auxiliary-head and modality-head ablations, latent-state diagnostics, clean-depth reconstruction, and pretraining, showing that the predictive representation can serve as a reusable prior for -object downstream control.
Original mathematical definitions, training assumptions and evaluation protocol
We conduct quantitative simulation and real-world evaluations, supported by qualitative videos, showing that WSM improves single-policy manipulation across object variations, unseen objects, perturbations, and enables sim-to-real transfer on a human-sized -DoF five-finger dexterous hand.
2 Related Work
The paper connects learned in-hand control, visuotactile perception and predictive world models. The comparison concerns representations and their control interface as much as raw sensors: equal rewards and randomization do not imply identical architectures or training supervision.
Learning-based in-hand manipulation.
Prior in-hand methods learn rotation gaits, adapt through proprioception or specialize to shapes and skill mixtures. Their results motivate richer state inference but do not make every cited benchmark directly comparable. The paper's own baseline reimplementations, retained in the tables, are the controlled experimental comparison. See the cited touch policy and rapid motor adaptation.
Original mathematical definitions, training assumptions and evaluation protocol
Learning-based systems have achieved sim-to-real in-hand control through several complementary strategies. Rapid Motor Adaptation infers object properties online from proprioceptive history for -axis rotation [4], whereas DeXtreme couples a domain-randomized reorientation policy with a vision-based pose estimator [3]. Touch Dexterity demonstrates vision-free rotation with distributed binary contact sensing [1], and AnyRotate uses dense tactile feedback for gravity-invariant multi-axis rotation [5]. More complex reorientation can compose pretrained axis-wise skills [8] or route among shape-specialized experts [9]. Other approaches adapt with real pen-spinning trajectories [10], learn joint-wise actuator corrections from hardware data [11], or randomize 3D Gaussians for monocular pose estimation [12]. In adjacent dexterous grasping, Goal-Auxiliary Actor-Critic combines point-cloud control with an auxiliary goal objective [13], while DextrAH-G couples pixel policies with geometric-fabric control priors [14]. These methods address grasp acquisition; WM-Craftnet instead learns recurrent interaction state for sustained in-hand control.
Visuotactile perception for dexterity.
Earlier methods combine touch with visual geometry, reconstruct contact-aware object shape or learn cross-modal features. WM-Craftnet replaces a fixed visual embedding with an action-conditioned recurrent predictive state. It does not explicitly recover a uniquely identifiable physical state; the t-SNE plots are diagnostics of representation structure. Relevant sources include Robot Synesthesia, NeuralFeels and ViTacFormer.
World models for robot control.
The RSSM follows the Dreamer family, but here prediction and policy optimization are separate. The actor uses observed-history state rather than optimizing on model-generated futures. This difference matters when interpreting the phrase world model. The original World Models, Dream to Control, DreamerV3 and visual locomotion references remain available.
3 WM-Craftnet Framework
The pipeline has a multimodal encoder, a recurrent stochastic state-space model and an asymmetric PPO actor–critic. Only the projected deterministic recurrent state crosses from world model to actor, with gradients stopped at that boundary. Simulator object state, clean depth and auxiliary targets provide training supervision without becoming deployed observations.
Original mathematical definitions, training assumptions and evaluation protocol
WM-Craftnet targets partially observed in-hand manipulation on a human-sized -DoF five-finger hand. We define the World Synesthesia Model as a Dreamer-style recurrent latent dynamics model trained with multimodal reconstruction and reward prediction, whose detached deterministic state serves as deployable policy context. The name reflects its cross-modal predictive role: visual geometry, tactile contact, proprioception, and action history jointly constrain the latent interaction state. This predictive state supports object-ID-free adaptation from deployable visuotactile observations and serves as a recurrent state estimate for the actor, as illustrated in Figure 1.
3.1 Problem Formulation and Observations
Equation 1 lists the actor's stacked proprioception/contact, current depth and previous action; the commanded rotation axis and learned context are additional policy inputs. Equation 2 measures signed incremental rotation projected onto that axis. The actor outputs relative updates for all twenty-two joint targets. Deployment uses neither ground-truth object pose nor object identity, while training rewards, critic targets and auxiliary predictions do use simulator information.
Original mathematical definitions, training assumptions and evaluation protocol
We consider continuous in-hand rotation of a grasped object around a commanded axis . At time , we denote the deployable policy observation by
Original selectable mathematical display
Equation 1
| (1) |
Original mathematical definitions, training assumptions and evaluation protocol
where and are short stacks of normalized hand proprioception and binary tactile/contact measurements, is the current noisy wrist-depth stream, and is the previous joint-target command. Importantly, contains only on-robot sensor streams and control history; it does not include clean simulator depth or any depth reconstructed by the WSM decoder. The actor conditions on , the commanded axis , and the projected WSM feature . The policy outputs a relative -DoF joint target command , which is smoothed and executed by the low-level hand controller.
Original mathematical definitions, training assumptions and evaluation protocol
The objective is signed object rotation around the target axis while maintaining a stable grasp. Let denote object orientation. One-step progress is
Original selectable mathematical display
Equation 2
| (2) |
Original mathematical definitions, training assumptions and evaluation protocol
where maps rotation to axis-angle form and projects onto the commanded axis. During simulation training, object pose, velocity, and contact-force fields support reward computation, critic learning, and diagnostics. Table 8 lists the complete observation/action interface.
3.2 Multimodal Predictive State Encoder
Equations 3–4 preserve the exact rollout tuple and multimodal encoder. The world model receives single-step proprioception/contact rather than the actor's stacks. Noisy preprocessed depth is an input; clean crop-only depth is a reconstruction target. Temporally correlated dropout, Gaussian perturbations and small image rotations make denoising useful. Predicting clean geometry requires access to that simulator target during training; it is not self-supervision solely from raw hardware depth.
Original mathematical definitions, training assumptions and evaluation protocol
The encoder maps sensory streams to a latent state for prediction and control. The world-model update receives a single-step proprioceptive vector , tactile/contact vector , noisy normalized depth image , previous action , and episode-start flag . The rollout record additionally contains clean depth and reward as training targets; neither is required as a deployment-time input. The actor observes stacked proprioceptive/tactile state, whereas the world model uses unstacked low-dimensional vectors to keep dynamics compact.
Original selectable mathematical display
Equation 3
| (3) |
Original selectable mathematical display
Equation 4
| (4) |
3.3 World Model for In-Hand Manipulation
Equations 5–7 retain posterior recurrent updating, deterministic actor context and the full loss. The RSSM predicts low-dimensional observations, clean depth and reward, with Dreamer dynamics/representation KL terms. Object pose, value and basis-point-set-to-mesh shape predictions provide additional privileged supervision. Reward is a prediction target, not a current deployed sensor input. The actual actor context is a learned projection of the 512-dimensional deterministic state; the stochastic latent is not directly handed to PPO.
Original mathematical definitions, training assumptions and evaluation protocol
WM-Craftnet uses a Dreamer-style recurrent state-space model (RSSM) as its world model. World-model optimization is decoupled from the PPO gradient path, and the actor receives a detached deterministic latent feature. Let be the stochastic latent state and be the deterministic recurrent state. Given the previous latent state, previous action, and current encoded observation, the posterior update is
Original selectable mathematical display
Equation 5
| (5) |
Original selectable mathematical display
Equation 6
| (6) |
Original mathematical definitions, training assumptions and evaluation protocol
which is detached before entering the policy network. In the current implementation, -dim is projected by a small MLP to a -dim feature before being concatenated with the policy observation.
Original mathematical definitions, training assumptions and evaluation protocol
The world model is trained from a replay buffer populated during PPO rollouts. Each row stores , where is the noisy/augmented depth input and is the clean depth target used by the decoder. Training samples contiguous chunks and optimizes the Dreamer-style model objective:
Original selectable mathematical display
Equation 7
| (7) |
Original mathematical definitions, training assumptions and evaluation protocol
Here reconstructs from the latent state inferred with noisy depth input, reconstructs proprioception, predicts rollout reward, and the KL terms regularize posterior/prior dynamics. Reward supervises the predictive model during training. Together, these targets encourage a geometry- and contact-aware state for deployment with noisy wrist-depth observations. Inspired by recent rotation and grasping systems [15, 18, 13, 17, 14], the full WSM uses depth, proprioception, tactile contact, reward, object pose, value, and object shape auxiliary heads to learn a task-relevant predictive state. During simulation training, the pose and shape heads use simulator labels, with shape represented by a basis-point-set-to-mesh displacement vector. Appendix B.1 analyzes the contribution of these heads.
3.4 Pretrained Synesthetic Prior for Multi-Object Control
Pretraining uses the nine-object z-axis setting. The transferred encoder, RSSM, decoders and reward predictor then continue adapting on new forty-nine-object rollouts, while the downstream actor–critic and task heads learn that distribution. This is a reusable initialization for further learning. The forty-nine-object result is not evaluation of an unchanged nine-object policy on forty-nine unseen objects.
Original mathematical definitions, training assumptions and evaluation protocol
Scaling one dexterous policy from a small object set to many new objects requires inferring how geometry, contact, slip, and action-conditioned motion interact. WM-Craftnet addresses this by making WSM reusable across object distributions. After pretraining on contact-rich data from the nine-object -axis setting, the predictive model initializes the -object experiment and continues adapting to new rollouts, while a task-specific controller is learned with reinforcement learning.
3.5 World Model-Conditioned Manipulation Policy
Equations 8–9 give policy conditioning and relative joint-target integration. The actor receives current noisy depth plus proprioception/contact, previous action, axis command and projected recurrent context; its main interface does not replace depth with the WSM's decoded image. The denoised-depth IHR control is a separate baseline. PPO uses the same reward for all methods. After each epoch, contiguous replay chunks update the world model; stop-gradient context prevents actor loss from directly training the model through its recurrent feature.
Original selectable mathematical display
Equation 8
| (8) |
Original selectable mathematical display
Equation 9
| (9) |
Original mathematical definitions, training assumptions and evaluation protocol
The actor input therefore comprises deployable observations and the WSM feature , while the critic additionally receives simulator state during training. After each PPO epoch, replay chunks update the RSSM; a stopped gradient at the actor interface decouples the two optimization paths.
4 Experiments and Evaluations
Four experiment groups address multi-object rotation, the predictive-state mechanism, harder axes and stress conditions. Simulation episode statistics, adaptation training curves, four-object hardware comparisons and the separate twenty-object hardware protocol have different denominators. Qualitative demonstrations and repeated human annotations do not increase the number of independent trials.
4.1 Experimental Setup
The primary platform is Sharpa Wave, with wrist depth and tactile sensing. Simulation evaluates 128 episodes per reported entry unless stated otherwise. Random initial locations, shared rewards and common reset rules standardize the implemented baselines, while model capacity, pretraining and privileged auxiliary supervision remain part of the method being compared.
Hardware and Simulation Setting.
Hardware hand control, camera and tactile streams all run at 10 Hz; simulation control also runs at 10 Hz with 60 Hz physics. Joint-level sinusoidal system identification tunes simulated PD gains before transfer. Randomized dynamics, noise, reset poses, forces and gravity cover residual mismatch. These steps make the result simulation-to-real transfer with calibration, rather than transfer without robot-specific preparation.
Original mathematical definitions, training assumptions and evaluation protocol
We use the same deployable observation/action setting and control frequency in simulation and on hardware. Each reported simulation number averages evaluation episodes per method under randomized evaluation seeds. To reduce the sim-to-real gap, we combine joint-level system identification with domain randomization over dynamics, sensing, resets, and perturbations. Appendix A provides implementation details, and Appendix A.3 describes system identification and sim-to-real randomization.
Baselines.
Baselines include open-loop replay, proprioceptive Blind RL, tactile Touch Dexterity, depth-only control and the visuotactile In-Hand Rotation implementation. The paper also isolates scratch WSM, sensor heads, recurrence and clean-depth targets. The native tables and selectable copies retain every baseline, including settings where a baseline has lower off-axis motion or object velocity.
Original mathematical definitions, training assumptions and evaluation protocol
For the -axis evaluation, we compare Open-loop Replay, Blind RL Policy (no depth, no tactile), Touch Dexterity (w/ tactile) [1], RL Policy (w/ depth, w/o tactile), In-Hand Rotation (w/ depth + tactile) [6], and WM-Craftnet with WSM variants. The - and -axis evaluations use a compact baseline comparison. All baselines are reimplemented for the -DoF Sharpa Wave Hand with the same reward, randomization range, initialization distribution, and reset strategy.
Metrics.
Simulation confidence intervals describe evaluation episodes, not variation across independent training seeds. Hardware RR is reported in radians over 20 seconds; SR counts trials exceeding a half-spin. Four annotators score the same synchronized recordings. Table 3's RR/SR aggregation is not fully specified: for example denoised IHR corner RR 2.43 rad with 10/10 half-spin success cannot be treated as a simple mean of the same nonnegative thresholded rotation quantities without further convention. The printed values are preserved; this ambiguity is not silently repaired or converted into a significance claim.
Original mathematical definitions, training assumptions and evaluation protocol
For simulation, we report return, episode length, rotation rate (RotR), and, where informative, off-axis motion and angular-velocity variation. Simulation entries are means with confidence intervals over evaluation episodes unless otherwise noted, and highlighting marks the first three ranks under each metric direction. For hardware, RR reports radians of rotation in a -s episode, and SR is the success rate, where success requires more than a half-spin. Four annotators independently score synchronized videos, and their measurements are averaged.
4.2 Robust Multi-Object z-Axis Rotation
Full pretrained WSM achieves 753.3 return and 1.293 RotR versus 414.3/.742 from scratch. This is strong evidence for the combined pretrained representation, while scratch WSM does not outperform every raw-sensor baseline on rotation rate. Test-time zeroing of all tactile input retains 724.9 return, about 96.2% of the full model; that is robustness to a missing input, not training a tactile-free system. Proprioception-only and proprioception-plus-touch pretraining score 684.5/698.9. All table entries and the nine-object grid remain intact.
Original heading notation
Original mathematical definitions, training assumptions and evaluation protocol
We first evaluate whether one policy can rotate multiple everyday objects under randomized starts. The nine-object set beside Table 1 spans diverse shapes, mass-center distributions, textures, and contact affordances, requiring online state and contact inference rather than object-specific gaits. As shown in Table 1, baseline returns range from to . WM-Craftnet trained from scratch improves return to and reduces OffAxis and AngVar to and , while its RotR remains below some baselines. The pretrained full WSM reaches return and RotR, with OffAxis and AngVar . Figure 2(a) compares the -axis methods; qualitative results are available on the project website11 1 https://wmcraftnet.github.io/.
Accessible original table, complete values and grouped headers
| Method | Return | EpLen | RotR | OffAxis | AngVar |
|---|---|---|---|---|---|
| Open-loop Replay | 306.3 48.3 | 320.2 35.0 | 0.332 0.080 | 1.789 0.295 | 2.241 0.679 |
| Blind RL Policy (no depth, no tactile) | 304.1 3.8 | 295.8 2.3 | 0.944 0.004 | 1.539 0.007 | 1.810 0.011 |
| Touch Dexterity (w/ tactile)[1] | 386.9 3.1 | 362.5 2.3 | 1.018 0.004 | 1.601 0.008 | 1.764 0.053 |
| RL Policy (w/ depth, w/o tactile) | 362.2 4.7 | 351.0 3.6 | 0.978 0.004 | 1.806 0.009 | 2.222 0.065 |
| In-Hand Rotation (w/ depth + tactile)[6] | 236.1 2.7 | 272.5 2.8 | 0.882 0.003 | 1.986 0.009 | 2.487 0.074 |
| WM-Craftnet (from scratch) | 414.3 15.7 | 264.2 8.8 | 0.742 0.012 | 1.285 0.010 | 1.735 0.052 |
| WM-Craftnet (all-zero tactile) | 724.9 4.4 | 428.8 2.8 | 1.252 0.004 | 1.224 0.006 | 1.388 0.031 |
| WM-Craftnet (75% tactile dropout) | 735.2 4.1 | 429.8 2.7 | 1.272 0.004 | 1.224 0.006 | 1.390 0.031 |
| WM-Craftnet (50% tactile dropout) | 737.8 4.4 | 425.1 3.6 | 1.276 0.004 | 1.279 0.006 | 1.511 0.061 |
| WM-Craftnet (25% tactile dropout) | 743.4 4.2 | 427.9 3.2 | 1.287 0.004 | 1.243 0.006 | 1.434 0.032 |
| WM-Craftnet (WSM pretrained, prop-only) | 684.5 4.3 | 424.0 2.2 | 1.191 0.003 | 1.310 0.007 | 1.234 0.013 |
| WM-Craftnet (WSM pretrained, prop+tac) | 698.9 4.3 | 435.4 1.9 | 1.192 0.002 | 1.165 0.006 | 1.373 0.024 |
| WM-Craftnet (full WSM) | 753.3 3.6 | 435.2 1.9 | 1.293 0.004 | 1.225 0.006 | 1.324 0.027 |
Original mathematical definitions, training assumptions and evaluation protocol
Figure 2(a) reports training reward for the -axis methods and shows the same ordering as the main comparison in Table 1. Table 1 also shows that reducing tactile availability from full input to dropout and all-zero input decreases return from to and . The modality controls progress from with proprioception alone to with proprioception and touch, and with full depth–touch input; prop+touch yields lower OffAxis () but lower RotR than full WSM. Figure 2(b) shows these controls.
Controlled WSM variants.
Clean-depth supervision adds 45.3 return over noisy-depth targets. Removing recurrent input gives 705.4 return, and the encoder–decoder feature gives 667.6. These controls support the full setup on return, but removing recurrent input improves OffAxis 1.225→1.125 and AngVar 1.324→1.185. The result is a tradeoff across metrics, not uniform superiority of recurrence on every stability measure.
Original mathematical definitions, training assumptions and evaluation protocol
Table 2 isolates clean-depth supervision, deterministic recurrence, and the recurrent policy feature. Reconstructing noisy rather than clean depth lowers return from to , while replacing the recurrent policy context with a per-step encoder–decoder feature lowers it to . Removing yields return and RotR, but improves OffAxis and AngVar to and , indicating a stability–task-performance trade-off. The training curves beside the table show that full WSM reaches the highest final reward.
Accessible original table, complete values and grouped headers
| Method | Return | EpLen | RotR | OffAxis | AngVar |
|---|---|---|---|---|---|
| Noisy-depth supervision | 708.0 3.5 | 431.1 2.0 | 1.265 0.005 | 1.299 0.006 | 1.456 0.045 |
| No recurrent input | 705.4 4.3 | 434.3 2.3 | 1.282 0.005 | 1.125 0.006 | 1.185 0.008 |
| Encoder–decoder policy | 667.6 4.7 | 417.4 2.6 | 1.254 0.003 | 1.323 0.006 | 1.343 0.018 |
| Full WSM | 753.3 3.6 | 435.2 1.9 | 1.293 0.004 | 1.225 0.006 | 1.324 0.027 |
Sim-to-real experiments.
The four-object robot comparison uses ten trials per object and 20-second rotation measurements. WM-Craftnet reports 10/10 success on duck, cross and corner, and 8/10 on the unseen double-notched block. Denoised-depth IHR improves over raw-depth IHR but remains below the recurrent method in reported rotation. This supports more than a depth-cleaning explanation; differences in representation and training remain, and the small trial counts do not isolate a unique causal mechanism.
Original mathematical definitions, training assumptions and evaluation protocol
We also evaluate real-robot -axis rotation on duck, cross block, corner block, and an unseen double-notched block. Table 2 compares a progression from fixed open-loop replay (OlR), to tactile feedback in Touch Dexterity (TD) [1], and then depth–tactile feedback in In-Hand Rotation (IHR) [6]. IHR is sensitive to the shift from simulated depth to noisy real observations. Feeding it WSM-denoised depth improves rotation and success on the three seen objects, confirming the value of geometric denoising, but yields limited rotation and no successful trials on the unseen block. WM-Craftnet conditions the actor on a recurrent WSM feature learned jointly from depth, touch, proprioception, and action history. This temporally informed context supports high RR and SR on both the seen objects and the unseen block.
Accessible original table, complete values and grouped headers
| Method |
|
|
|
| |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RR | SR | RR | SR | RR | SR | RR | SR | ||||||||
| OlR | 2.32 | 6/10 | 3.04 | 6/10 | 5.19 | 7/10 | 0.23 | 0/10 | |||||||
| TD[1] | 1.54 | 4/10 | 0.34 | 2/10 | 0.22 | 0/10 | 0.27 | 0/10 | |||||||
| IHR[6] | 1.83 | 5/10 | 0.45 | 1/10 | 0.36 | 1/10 | 0.39 | 0/10 | |||||||
| IHR + WSM-denoised depth | 2.76 | 8/10 | 1.21 | 8/10 | 2.43 | 10/10 | 0.80 | 0/10 | |||||||
| Ours | 16.179 | 10/10 | 8.01 | 10/10 | 8.48 | 10/10 | 4.32 | 8/10 | |||||||
4.3 World Synesthesia Model Analysis
Auxiliary-head Table 10 reports 688.1 return without heads,737.0 with value only and 753.3 with all heads. The pose-only head gives lower angular-velocity variation than the full model. Figure 3's t-SNE has object-dependent regions plus substantial overlap; it does not prove object recognition, state identifiability or causal disentanglement. Figure 4 and the original depth videos provide qualitative reconstruction evidence, not an independently measured hardware geometry-error benchmark.
Original mathematical definitions, training assumptions and evaluation protocol
The auxiliary-head study in Table B.1 raises return from without auxiliary heads to with the value head alone and with all heads. Table 2 provides the controlled clean-depth and recurrent-state variants, while the LSTM and GRU history baselines remain in Appendix B.1.
Original mathematical definitions, training assumptions and evaluation protocol
We further examine the learned state with two diagnostics. For the nine -axis objects, we project evaluation-time recurrent states with t-SNE to characterize object- and interaction-phase structure; object labels are used only to color the visualization and are not policy inputs. We also decode clean depth from the latent state and compare it with ground-truth and noisy wrist-depth inputs.
Reusable synesthetic prior.
After 3000 epochs, the adapted forty-nine-object policy reaches 9.37±.13 radians per episode versus 3.28 without the prior. These units differ from the rate metric in the main tables. The training fall rate decreases from 6%to.3% over learning; that temporal change is not a controlled 95%fall reduction against the no-prior baseline. A separate five-block prior-scaling diagnostic gives 1.10/1.34/1.46 RotR for scratch/nine-object/forty-nine-object priors. It uses a distinct diagnostic set and should not be pooled with axis-held-out trials.
Original mathematical definitions, training assumptions and evaluation protocol
The same predictive model can be reused beyond the nine-object -axis setting. Figure 5 and website videos show one downstream policy trained over new objects; after epochs, each object rotates rad per episode on average, compared with rad for the no-prior baseline, while the training fall rate drops from to from start to end. In a separate five-block prior-scaling diagnostic (diagonal, narrow, double-notched, step, and V-cut blocks), RotR improves from scratch / 9-object prior / 49-object prior as . This diagnostic is distinct from the axis-specific held-out sets below; Appendix B.2 shows the full object set and Figure B.2 visualizes its recurrent-state organization.
4.4 More Difficult Rotation Modes
The paper trains separate world models and policies for x,y and z. On nine y-axis tool-like objects, WSM gives 1.025 RotR versus the best baseline.545, with 215 episode length and.049 fall. On four x-axis objects it achieves 432.8 return and 435.4 episode length, but object velocity.0460 is higher than every baseline in Table 5. The axis command in the architecture therefore does not establish one trained generalist for all axes, nor do high return and low fall imply minimum object motion on every axis.
Original mathematical definitions, training assumptions and evaluation protocol
We next evaluate rotation axes with less passive palm support. -axis rotation creates larger moment arms and asymmetric contacts for elongated objects; -axis rotation is harder because the hand has limited fingertip workspace and must reallocate contacts while gravity pulls the object away from the palm. All axis-specific policies use the same WM-Craftnet design, reward, randomized starts, and deployable observations, with a separately trained WSM for each axis.
Original mathematical definitions, training assumptions and evaluation protocol
The -axis evaluation uses the tool-like object set shown with Table 4, testing whether the policy can rotate objects that tend to lever against the fingers. Table 4 shows that WM-Craftnet improves return from the strongest baseline value of to , nearly doubles the best baseline RotR from to , and achieves the lowest fall rate. The RotR gain is substantially larger than the EpLen gain, indicating that the return improvement reflects faster target-axis progress rather than only longer episodes. Its ObjVel ranks second rather than first, so the advantage is specific to controlled rotation and fall reduction, not uniformly lower object motion. Figure 2(d) shows the same training trend, suggesting that the recurrent state helps resolve the changing moment arms and asymmetric contacts.
Accessible original table, complete values and grouped headers
| Method | Return | EpLen | RotR | Fall | ObjVel |
|---|---|---|---|---|---|
| Open-loop Replay | -52.4 5.2 | 85.8 10.0 | 0.031 0.063 | 0.784 0.082 | 0.0366 0.0042 |
| Blind RL Policy (no depth, no tactile) | 177.8 9.2 | 192.4 7.6 | 0.545 0.016 | 0.164 0.002 | 0.0737 0.0014 |
| Touch Dexterity (w/ tactile)[1] | 105.3 5.8 | 172.0 6.2 | 0.297 0.030 | 0.208 0.003 | 0.0760 0.0021 |
| RL Policy (w/ depth, w/o tactile) | 154.8 6.9 | 178.4 5.4 | 0.288 0.033 | 0.103 0.002 | 0.0909 0.0015 |
| In-Hand Rotation (w/ depth + tactile)[6] | 142.8 5.5 | 162.5 4.8 | 0.377 0.025 | 0.063 0.003 | 0.0787 0.0021 |
| WM-Craftnet | 272.7 13.5 | 215.0 8.7 | 1.025 0.026 | 0.049 0.001 | 0.0682 0.0013 |
Original mathematical definitions, training assumptions and evaluation protocol
The -axis evaluation in Table 5 is a stricter test of contact sequencing: the policy must roll the object within a small effective finger workspace and reallocate contacts before the object leaves the palm-supported configuration.
Accessible original table, complete values and grouped headers
| Method | Return | EpLen | RotR | Fall | ObjVel |
|---|---|---|---|---|---|
| Open-loop Replay | 18.0 6.8 | 129.4 22.3 | 0.411 0.054 | 0.164 0.064 | 0.0448 0.0019 |
| Blind RL Policy (no depth, no tactile) | 44.0 0.8 | 63.4 1.0 | 0.867 0.002 | 0.033 0.001 | 0.0311 0.0001 |
| Touch Dexterity (w/ tactile)[1] | 37.8 0.2 | 42.7 0.2 | 1.023 0.002 | 0.039 0.002 | 0.0334 0.0001 |
| RL Policy (w/ depth, w/o tactile) | 38.2 0.2 | 43.7 0.4 | 1.079 0.003 | 0.033 0.001 | 0.0323 0.0001 |
| In-Hand Rotation (w/ depth + tactile)[6] | 295.1 3.3 | 365.1 3.1 | 0.839 0.001 | 0.103 0.003 | 0.0433 0.0001 |
| WM-Craftnet | 432.8 2.1 | 435.4 1.8 | 1.114 0.001 | 0.020 0.001 | 0.0460 0.0001 |
Original mathematical definitions, training assumptions and evaluation protocol
Table 5 and Figure 2(c) show that blind, tactile-only, and depth-only policies stay below return and episode length, while open-loop replay lasts longer but still has low return, low RotR, and a high fall rate. WM-Craftnet reaches return, episode length, the highest RotR, and the lowest fall rate, maintaining grasp while finding contacts that advance the commanded spin.
4.5 Stress Tests in Challenging Interaction States
Held-out tests use four different objects per axis and report zero-shot performance for those axis-specific policies. RotR gains over the best baseline are 8.6/89.2/20.9% for x/y/z, while the gains in return differ. Severe simulated recovery is 14.1% versus the best baseline 6.2%, an increase of 7.9 percentage points with low absolute success. Recovery time 2.56 s is conditional on recovered cases, not an unconditional rescue-time guarantee. Figure 6 deliberately shows successful examples. A separate twenty-object physical z-rotation study reports 175/200 versus 33–53/200 baseline successes; these trials measure broad-object rotation, not a 200-trial disturbance-recovery rate.
Original mathematical definitions, training assumptions and evaluation protocol
Unseen-object tests. We begin with axis-specific zero-shot transfer in simulation. We directly evaluate the -, -, and -axis policies trained on their respective -, -, and -object training sets, without object-specific fine-tuning. For each axis, the held-out evaluation contains four unseen objects: small corner block, narrow block, large step block, and small Lego for ; double-notched block, large flashlight, narrow block, and V-cut block for ; and lidded jar, tilted tumbler, handled pot, and pedal bin for . Table 6 shows that WM-Craftnet achieves the highest RotR and return in all three axis-specific evaluations. Relative to the strongest non-WSM entry on each metric, the RotR gains are , , and for , , and , respectively; the corresponding return gains are , , and .
Original mathematical definitions, training assumptions and evaluation protocol
Perturbation-recovery test. We next isolate recovery under severe simulated perturbations and measure whether the policy restores a controllable configuration, together with the elapsed time among recovered trials. WM-Craftnet reaches recovery success, compared with for the strongest baseline, and has the shortest mean recovery time at s. This is a -percentage-point absolute gain and a -s reduction in mean recovery time over the strongest baseline on the respective metric. Under difficult starts and in-process force perturbations, the representative successful rollouts in Figure 4.5 show the policy re-centering, uprighting, or transferring the object back into a controllable finger workspace before resuming target-axis rotation. The absolute recovery rate remains limited under these large disturbances.
Original mathematical definitions, training assumptions and evaluation protocol
Large-scale real-world evaluation. Finally, in a separate large-scale hardware protocol, we test standard -axis in-hand rotation on real objects with trials per object. The set combines the nine training objects with eleven unseen objects: lidded jar, tilted tumbler, pedal bin, handled pot, diagonal block, narrow block, double-notched block, step block, V-cut block, trash can, and bucket. WM-Craftnet succeeds in trials, compared with – for the baselines. This hardware result measures rotation success across a broad object set.
Accessible original table, complete values and grouped headers
| Method | RotR, Ret. () | RotR, Ret. () | RotR, Ret. () | Sim. Recovery | Sim. Time (s) | Real Rot. SR |
|---|---|---|---|---|---|---|
| Blind RL Policy | 0.852, 134.7 | 0.300, 135.7 | 0.665, 277.1 | 4.7% 3.7% | 3.62 2.25 | 41/200 |
| Touch Dexterity [1] | 0.923, 35.8 | 0.174, 78.7 | 0.605, 242.2 | 6.2% 4.2% | 3.33 1.41 | 33/200 |
| RL Policy (w/o tactile) | 0.853, 33.8 | 0.378, 168.9 | 0.696, 301.7 | 5.5% 4.0% | 3.14 0.79 | 45/200 |
| In-Hand Rotation [6] | 0.732, 246.0 | 0.309, 193.0 | 0.468, 156.1 | 0.0% 0.0% | N/A | 53/200 |
| WM-Craftnet | 1.002, 333.6 | 0.715, 203.5 | 0.842, 533.1 | 14.1% 6.0% | 2.56 0.77 | 175/200 |
Tool use behavior.
Screwdriver translation and rotation use the same policy formulation, as detailed in AppendixB.3. The paper offers qualitative tool-style manipulation sequences, without a quantitative tool-use task suite or evidence that one unchanged rotation policy performs every tool behavior.
5 Conclusions and Limitations
Predictive multimodal training provides a useful recurrent state for this manipulation interface, with consistent gains on important return and rotation measures. Limits include short horizons, restricted hand workspace, large offsets outside the recovery basin, drift, jamming and low severe-perturbation recovery. Further work on tools and broader distributions is proposed. The experiments do not demonstrate long-horizon tool completion or universal robustness.
Original mathematical definitions, training assumptions and evaluation protocol
This paper presents WM-Craftnet, a world model-based visuotactile framework for robust dexterous in-hand manipulation. WSM learns Dreamer-style latent dynamics from noisy depth, proprioception, tactile sensing, and actions, reconstructs clean depth targets, and conditions the policy with its deterministic recurrent feature. Depth reconstruction suppresses sensor artifacts while preserving hand-object geometry. The predictive representation also serves as a reusable prior for a -object downstream policy. Experiments show improved multi-object rotation, unseen-object transfer, perturbation recovery, and sim-to-real performance over baseline visuotactile methods.
Original project figures and demonstration recordings
Original project videos, pipeline and learning/latent diagrams, the Sharpa artwork and conference poster are local. Each recording retains its source URL and caption; preview posters are derived frames. The gravity-invariant demonstrations use a closer grasp, modified initial conditions and reward design described separately on the project site. They should not be interpreted as the unchanged palm-up benchmark policy or additional controlled evidence for the tables. Baseline, failure and reconstruction recordings remain alongside successful demonstrations.




Appendix A Implementation Details
The implementation appendix supplies world-model optimization, deployed versus privileged inputs, PPO/control/reward settings and dynamics calibration. Both original tables remain complete rather than being reduced to selected hyperparameters. Reproduction requires the specified hand, sensing preprocessing, simulator supervision and system-identification procedure.
Accessible original table, complete values and grouped headers
| Parameter | Value |
|---|---|
| System | |
| Hand | Sharpa Wave, -DoF five-finger hand |
| Hardware rates | hand control / wrist camera / tactile: Hz |
| Camera stream | wrist-mounted, , fixed by 3D-printed bracket |
| Simulation rates | physics Hz; observation/control Hz |
| World Model | |
| Type | Dreamer-style RSSM with multimodal inputs |
| Deterministic / stochastic / discrete dims | |
| Hidden size / recurrent depth | |
| Actor projection | |
| Replay capacity / warmup | rows / transitions |
| Batch size / chunk length / updates | per PPO epoch |
| Optimizer / LR / grad clip | Adam / / |
| KL free / dyn scale / rep scale | |
| Depth Input | |
| Policy depth tensor / max depth | m |
| Temporal dropout / corr. | |
| Random image rotation | |
| Policy and Control | |
| Envs / horizon / minibatch / mini-epochs | |
| / GAE / LR / clip | |
| Policy MLP / activation | / ELU |
| DoF / episode / sim dt / control decimation | |
| Reward | |
| Spin / max spin / velocity | |
| Torque / work / action / control | |
| Finger distance / hand pose / off-axis | |
| Object drift / no-spin / fall penalty | |
| Tip force coef. / thresh. / exp. | |
| Randomization | |
| Object mass train / test | kg / kg |
| Friction scale / PD scale | |
| Obs noise / action noise | |
| Reset pos. / reset rot. | m / |
| External force scale / prob. / decay | |
| Gravity curriculum | , step |
Accessible original table, complete values and grouped headers
| Signal | Used by | Description |
|---|---|---|
| Hand proprioception | Actor, WM | normalized -DoF joint state and previous target history |
| Tactile/contact | Actor, WM | binary contact stream with threshold, dropout, and latency noise |
| Depth image | Actor, WM | noisy wrist depth input; clean depth is used only as WM reconstruction target |
| Commanded axis | Actor | goal direction |
| World-model feature | Actor | projected deterministic RSSM feature |
| Relative action | Controller, WM | -DoF joint target increment, smoothed before execution |
| Object state | Reward/critic only | pose, velocity, contact force, and fall/reset diagnostics in simulation |
A.1 World Model Training Details
A ring replay buffer stores noisy observations, clean depth targets, previous action, reward and reset flags. After 10,000 transition warmup, batches of 16 contiguous 32-step chunks train the model, with 10 updates per PPO epoch. The deterministic/stochastic/discrete dimensions are 512/32/32, and the actor projection is 512→64→32→16. Auxiliary pose/value/shape heads are used in both pretraining and downstream adaptation. Transfer includes continued model updates, not just learning a new actor against a frozen latent state.
Original mathematical definitions, training assumptions and evaluation protocol
Table 7 summarizes the world-model and depth-preprocessing settings. The world-model update receives a single-step proprioceptive vector, tactile/contact vector, noisy preprocessed depth image, previous action, and an episode-start flag; reward is stored only as a prediction target. The RSSM performs an observation update at every control step and exposes only the detached deterministic feature , projected by a small MLP, as actor context.
Original mathematical definitions, training assumptions and evaluation protocol
For pretrained-prior evaluations, we transfer the encoder, multimodal RSSM, depth/proprioceptive decoder, and reward predictor. During downstream -object PPO training, these components continue to adapt with new rollout data, while the actor–critic and task-specific heads are learned for the target distribution. This design transfers predictive physical structure across object sets while learning control for the new task distribution.
A.2 Policy Training Details
Equation 10 preserves all reward terms, including spin, object velocity, contact/distance, torque/work, action/control penalties, pose, drift, off-axis motion and no-spin resets. Reward weights are not interchangeable with measured physical guarantees. Episodes terminate on drop distance, axis deviation, prolonged insufficient spin or horizon, which helps explain the relationship between return, episode length and spin. Relative joint targets are smoothed before the 10 Hz hardware interface. The full parameter table preserves signs and thresholds.
Original mathematical definitions, training assumptions and evaluation protocol
Table 7 lists PPO, control, and reward settings, and Table 8 summarizes deployable and simulation-side signals. The actor outputs a relative target update for all hand joints. A signal-level filter smooths these targets before execution by the Hz hardware controller.
Original selectable mathematical display
Equation 10
| (10) | ||||
Original mathematical definitions, training assumptions and evaluation protocol
The target-axis spin term uses signed incremental rotation projected onto , clipped by a maximum spin rate. Stability terms penalize object velocity, drift, off-axis rotation, high torque, mechanical work, large actions, and tracking error; contact and pose terms encourage useful fingertip proximity, avoid excessive force or non-tip contacts, and softly favor natural hand postures.
A.3 System Identification and Domain Randomization
The wrist-mounted RealSenseL515 supplies 640×480 depth, preprocessed to 96×72 with.4 m maximum depth. A 1 Hz sinusoid excites each joint for gain tuning; Figure 7 reports average calibrated sim-real tracking error below.2 degrees under that test. Training mass is.2–.4 kg with broader .1–1.4 kg testing, plus friction/gain/noise/reset/force/gravity randomization. Small sinusoidal tracking error does not certify contact dynamics or arbitrary high-speed trajectories.
Original mathematical definitions, training assumptions and evaluation protocol
Real-world evaluation uses the Sharpa Wave hand with tactile sensing and a wrist-mounted depth camera (RealSense L515). The hand controller, wrist camera, and tactile stream run at Hz, and simulation uses the same policy/control rate with a Hz physics step. The policy receives the same deployable observation/action interface in both domains.
Original mathematical definitions, training assumptions and evaluation protocol
We identify joint-level dynamics before policy transfer by exciting the real hand with Hz sinusoidal action commands and comparing the measured joint response with simulation. The simulator and gains are adjusted to align the amplitude and phase of the simulated trajectories with the hardware response. This calibration provides a nominal dynamics model, while domain randomization covers residual mismatch. Figure 7 visualizes the post-calibration joint tracking fit across the hand. The simulated trajectories closely match the real joint responses under the same sinusoidal commands, with an average sim-real tracking error below .
Original mathematical definitions, training assumptions and evaluation protocol
Training randomizes object mass, friction, PD gains, observation/action noise, reset pose, external forces, and gravity direction as in Table 7. Object mass is sampled in kg during training and evaluated over a wider test range. Friction and hand-joint gains are randomized to cover contact and actuator uncertainty; depth and tactile observations include dropout, noise, small image rotations, threshold variation, and latency. External forces and a gravity-direction curriculum further encourage perturbation recovery instead of memorizing a single palm-up rotation gait.
Appendix B Additional Experiments
Additional experiments separate recurrent-history baselines from predictive objectives, inspect larger-scale latent structure and show tool-style manipulation. Their evidence has different scope from the standard rotation and recovery tables; the original displays and object grids are preserved in place.
B.1 Additional WSM Mechanism Ablations
Full WSM reaches 753.3 return/1.293 RotR versus 621.0/1.186 for LSTM and 497.8/1.095 for GRU without predictive objectives. The full model's OffAxis 1.225 is nearly the GRU's 1.226, so that metric shows little separation. Table 10 retains all single-head combinations and uncertainty. Value-only is the strongest single head on return; pose-only gives lower AngVar 1.231 than full 1.324. The controls support predictive supervision overall but do not make every auxiliary head improve every metric.
Accessible original table, complete values and grouped headers
| Method | Return | EpLen | RotR | OffAxis | AngVar |
|---|---|---|---|---|---|
| LSTM (w/o predictive obj.) | 621.0 4.3 | 424.3 2.7 | 1.186 0.004 | 1.297 0.006 | 1.392 0.028 |
| GRU (w/o predictive obj.) | 497.8 4.5 | 377.1 2.5 | 1.095 0.003 | 1.226 0.007 | 1.357 0.013 |
| Full WSM | 753.3 3.6 | 435.2 1.9 | 1.293 0.004 | 1.225 0.006 | 1.324 0.027 |
Original mathematical definitions, training assumptions and evaluation protocol
Full WSM reaches return and RotR, compared with for LSTM and for GRU. It also lowers AngVar to , while its OffAxis is comparable to GRU’s . These results separate the benefit of predictive multimodal training from recurrent memory alone.
Original mathematical definitions, training assumptions and evaluation protocol
Table B.1 measures how simulation-side auxiliary objectives shape WSM. Removing all auxiliary heads yields return, while value, pose, and object-shape prediction each improve different aspects of the learned state. The value head produces the largest return gain among the single-head variants, whereas pose prediction gives the lowest angular-velocity variation. Combining all heads achieves the strongest overall return and rotation rate.
Accessible original table, complete values and grouped headers
| Method | Return | EpLen | RotR | OffAxis | AngVar |
|---|---|---|---|---|---|
| No auxiliary heads | 688.1 3.7 | 433.7 2.0 | 1.239 0.006 | 1.229 0.005 | 1.778 0.138 |
| Only value head | 737.0 4.9 | 435.7 2.2 | 1.291 0.003 | 1.258 0.007 | 1.346 0.045 |
| Only pose head | 690.0 5.8 | 432.3 2.5 | 1.221 0.007 | 1.228 0.008 | 1.231 0.011 |
| Only object-shape head | 689.2 4.2 | 428.1 2.2 | 1.261 0.002 | 1.255 0.006 | 1.535 0.032 |
| Full WSM | 753.3 3.6 | 435.2 1.9 | 1.293 0.004 | 1.225 0.006 | 1.324 0.027 |
B.2 Large-Scale Object Set
Figure 8 contains the entire forty-nine-object training set and the downstream state projection. Colors indicate object identities for visualization; they are not identity labels passed to the actor. Overlapping trajectories are consistent with shared contact phases and geometry-dependent behavior, but t-SNE alone cannot distinguish these explanations or establish a physically complete latent state.
Original mathematical definitions, training assumptions and evaluation protocol
Figure B.2(a) shows the new -object set used for downstream large-scale multi-object policy training after WSM pretraining on the nine -axis objects. The set covers substantial variation in size, aspect ratio, curvature, and contact geometry. To examine whether the transferred WSM retains structured interaction states at this larger scale, we project evaluation-time recurrent states from the downstream policy with t-SNE. Figure B.2(b) shows locally coherent trajectories and partially object-dependent regions together with substantial overlap. This pattern is consistent with a representation that captures both object-dependent geometry and interaction phases shared across objects, rather than acting as a strict object-identity classifier.
B.3 Screwdriver Tool-Use Manipulation
Figure 9 preserves both goal-conditioned screwdriver translation and elongated-object rotation, including all frames and target labels. The evidence is qualitative. It does not measure successful fastening, force-controlled insertion or multi-step tool-use completion, and the common interface should not be confused with a single universally trained policy.
B.4 Supplementary rollouts and failure cases.
The original project includes depth reconstruction, simulation and physical rotation, perturbations, tool examples, baselines and explicit failures. Failure cases show why successful recovery montages cannot substitute for the quantitative severe-disturbance rate. The recordings are preserved as supplementary observations, without counting scenes, cameras or edited sequences as new independent trials.