Literature
Display
← All papers

OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining

Yuran Wang · Siqiao Huang · Mingleyang Li · Chenhao Zhang · Jiaqi Liang · Weiyang Jin · Yue Chen · Xuemin Chi · Donghao Zhou · Qize Yu · Yu-Kai Wang · Yuhan Rui · Shenzhe Yao · Zhen Yuan · Zhenhao Shen · Kefei Zhu · Zijie Zhu · Ning Gao · Xiaowei Chi · Guanqi He · Shanghang Zhang · Hao Dong · Lin Shao · Hang Zhao

Preprint posted · arXiv (preprint) ↗

World models17 min text · CondensedOriginal paper ↗
Contents

At a glance

OpenWAM makes world–action modeling a configurable research stack, studies the effect of its components, and scales the resulting recipe into OpenWAM-α. The model jointly learns future video and executable actions from 518.5 million frames, representing 6,369 hours of human and robot data.

Its main contribution is a reproducible way to compare coupled design choices. The experiments favor capable video priors, compact visual latents, dedicated action capacity, and joint training. These are empirical results under the tested settings, not universal optimality claims. This edition retains all 43 pages’ figures, tables, equations, references, and appendix protocols from arXiv v1, dated 7 September 2026.

Complete original figure 1
Figure 1 Overview of OpenWAM. OpenWAM-Infra (left) factorizes world–action modeling into composable modules with unified training, deployment, and evaluation; OpenWAM-Study (middle) resolves the design space through controlled questions and distills a pretraining recipe; OpenWAM-α (right), pretrained on 518.5M frames of egocentric and robot data, sustains top-tier performance from simulation to the real world.

1. Introduction

Video generation supplies knowledge of how scenes evolve; robot demonstrations connect that knowledge to controllable actions. Yet a system’s backbone, representation, fusion architecture, data, and inference procedure often change together, making improvements difficult to attribute.

OpenWAM separates three questions: which upstream knowledge to inherit, how world and action prediction should interact, and how embodied pretraining changes generalization. Infra supplies the controlled implementation, Study accumulates an experimental recipe, and α instantiates that recipe at scale.

3. OpenWAM-Infra

One stack covers data ingestion, model construction, training, serving, and evaluation. Changing a component should not require rewriting the trainer or benchmark client. Checkpoints include the information required to rebuild both the model and its action normalization.

3.1 Composable model

A model is the composition C(E, S, M): a frozen visual encoder E, one or more stream backbones S, and a visibility mask M. The encoder maps observations to latents; world, action, and optional understanding streams process them; the mask controls which tokens can exchange information.

Every backbone exposes preparation, per-layer execution, and finalization. Attention can be replaced at a layer boundary without changing the backbone internals. The composition rule itself has no learned parameters.

Complete original figure 2
Figure 2 OpenWAM Model Infra. Top: the three classes of interchangeable modules: the visual encoder E (left); the stream backbones S (middle); and the visibility attention mask M (right). The central Training Utils panel summarizes the utilities of the training runtime (Section 3.2). Bottom: the composition rule C assembles the modules into six architecture variants across the Single-System, Dual-System, and Tri-System families.

Visual encoders, backbones, and attention masks

Reconstruction encoders include Wan2.2-VAE and FLUX.2-VAE; representation encoders include DINOv3 and V-JEPA 2.1, optionally compressed with S-VAE [81] [73] [63] [100]. Five video backbones are supported, from Wan-VACE 1.3B to Wan-I2V 14B, including Cosmos3-Edge-4B. Only four enter the controlled backbone comparison.

Equation 1 partitions visibility between video and action. The clean first video frame attends only to itself; future video tokens can communicate, and action tokens attend bidirectionally within their stream. Cross-stream blocks create isolated, video-sees-action, action-sees-video, or mutual interaction.

Equation
Complete original equation 1

Architecture families

Single-system vanilla shares attention and feed-forward weights across video and action tokens. Its mixture-of-experts variant hard-routes action tokens through a separate feed-forward expert.

Dual systems use a dedicated action transformer. Joint self-attention merges the streams at bridge layers; cross-attention lets actions query video features, optionally stopping action-loss gradients into the video stream. The inverse-dynamics variant trains with teacher-forced video and generates video before actions at inference.

The tri-system variant adds a trainable understanding stream fed by a frozen vision–language model. This stream is a read-only tail: the other streams can attend to it, while it attends only to itself. Six variants become seven experimental rows when cross-attention is tested with and without detached gradients.

3.2 Training runtime

Each sample contains language, a video window, an action chunk, optional proprioception, and a dimension-validity mask. Time zero is noise and time one is clean data. Independent video and action noise levels train both velocity predictions through the flow-matching loss in Equation 2; clean conditioning frames are excluded from video loss.

Independent noise sampling covers the joint timestep plane and supports different inference schedules. It does not prove that all model-generated intermediate states have the same distribution as training examples.

The runtime supports mixed precision, gradient checkpointing, offload, and distributed optimization. Fine-tuning rebuilds a checkpoint’s modules; exact resume restores optimizer, scheduler, and data position and rejects changed normalization. Serving also rejects missing normalization statistics.

Equation
Complete original equation 2

3.3 Deployment runtime

Serving mode and denoising schedule are separate choices. Synchronous serving waits for a fresh action chunk when the buffer empties. Asynchronous serving begins prefetch while d actions remain, skips the delayed prefix of the new prediction, executes the next n actions, and discards any unused tail.

The default lead threshold is n/2, with d

Complete original figure 3
Figure 3 Inference modes and denoising schedules of the deployment runtime. (a) Illustration of the synchronous and asynchronous inference modes. (b) Illustration of the three denoising schedules (variance shift, linear offset, and sync); five denoising steps are drawn for illustration, and circles of the same color denote the timesteps that the two modalities reach at the same denoising step.

Denoising schedules

A schedule traces video and action timesteps from noise to clean data. Synchronized denoising follows the diagonal; variance shift accelerates a leading stream, while linear offset delays the other. Equation 3 preserves both functions. Choosing shift one and offset zero recovers the diagonal.

Each stream advances by its own Euler increment, and checkpointed timestep warps are reused at inference. These choices concern the prediction process within a chunk, independently of whether the robot server prefetches chunks.

Equation
Complete original equation 3

Inference acceleration

Four optimizations cache prompt embeddings, skip video decoding during control, compile fixed-shape denoising under CUDA graphs, and reuse similar velocity predictions for a bounded number of steps. Compilation adds first-request warmup.

Figure 4 reports 149–251 ms across architectures on an RTX 5090, including 175 ms for dual joint self-attention. Reported speedups of 2.01–2.63× compare compilation and velocity caching with a baseline that already caches prompts and skips video decoding. They are chunk-inference measurements, not continuous action frequencies.

Complete original figure 4
Figure 4 Serving latency across architectures. Inference latency with Wan2.2-TI2V-5B as the video backbone on an RTX 5090. The promptembedding cache and video-decode skip are enabled by default; the figure ablates compilation and the DiT velocity cache.

3.4 Evaluation protocol

A benchmark sends images, language, and optional state over a persistent WebSocket connection and receives actions in native physical units. The server crops, resizes, normalizes, and composes up to three views, with a required head view and black-filled missing wrist views. Episode boundaries reset action buffers.

Adapters cover LIBERO, LIBERO-Plus, VLABench, RoboTwin2.0, RoboDojo, RoboCasa365, RoboCasa-GR1, and EBench. Keeping preprocessing and normalization with the checkpoint prevents silent coordinate changes between training and deployment.

Complete original figure 5
Figure 5 Evaluation protocol of OpenWAM-Infra. Benchmarks connect to the policy server as thin clients over WebSocket. The server canonicalizes each observation, maps the proprioceptive state into the model-side action representation (the 80-D unified action space or the benchmark’s native action space), and denormalizes the predicted action chunk back to native physical units before returning actions.

Action-space definition

Single-benchmark models may retain native actions, such as 14-dimensional RoboTwin joints, 20-dimensional bimanual end-effector poses, or 10-dimensional LIBERO actions. Cross-embodiment α uses a shared 80-dimensional layout.

Each of two 34-dimensional arm blocks contains position, six-dimensional rotation, gripper state, and 24 hand coordinates; 12 additional slots are reserved for embodiment-specific channels. Equation 4 normalizes and scatters native coordinates into fixed slots, then gathers and denormalizes predictions. A validity mask removes unused coordinates from action-loss gradients.

Equation
Complete original equation 4

4. OpenWAM-Study

The study sequentially holds the accumulated defaults fixed while varying one design axis. This is a controlled recipe-building exercise, not an exhaustive factorial comparison of every combination.

RoboTwin2.0-Full uses 2,500 clean and 25,000 randomized demonstrations across 50 tasks. Clean2Random instead fine-tunes only on clean demonstrations and evaluates both clean and randomized conditions. Its randomized success measures transfer after fine-tuning, not zero-shot deployment of the pretrained model.

4.1 Inheriting upstream world knowledge

The first question separates a video generator’s learned prior from the latent representation in which prediction occurs. A useful encoder must retain action-relevant information while keeping temporal and channel dimensions manageable.

4.1.1 Generative world priors

With otherwise matched settings, average success increases from 90.14% with Wan-VACE 1.3B to 91.64% with Cosmos-Predict2.5-2B, 92.39% with Wan2.2-TI2V-5B, and 93.79% with Wan2.1-I2V-14B. The 5B model becomes the practical default.

The backbones differ in pretraining data, architecture, and objective as well as size. Figure 6 therefore supports choosing a stronger available prior; it does not isolate a parameter-count scaling law.

Complete original figure 6
Figure 6 WAM Performance with Different Video Backbone Size. With increasing video generation backbone size, performance of the resulting WAM consistently improves.

4.1.2 Visual representation priors

To isolate encoders, this experiment randomly initializes the video transformer instead of inheriting pretrained transformer weights. Non-Wan encoders average four consecutive frames for temporal compression; S-VAE reduces DINOv3’s 768 or V-JEPA’s 1,024 channels to 48.

Figure 7 reports 76.42% and 80.91% for uncompressed DINOv3 and V-JEPA, 84.26% for FLUX, 88.46% for compressed V-JEPA and 90.18% for compressed DINOv3, and 90.30% for Wan-VAE. Compact semantic representations nearly match the reconstruction encoder. Native learned temporal compression versus simple averaging remains a confound.

Complete original figure 7
Figure 7 Visual representation priors. We consider building WAMs with both reconstructive and representation encoders, and include a variant of representation encoders with S-VAE (Zhang et al., 2025a), an adapter that converts highdimensional latents produced by representation encoders into low-dimensional vectors suitable for DiT processing.

4.2 World–action interaction

The second question concerns dedicated action capacity and information exchange. The experiments distinguish architectural separation, attention visibility during training, and denoising order during inference.

4.2.1 Architectural capacity

Table 1 reports 85.50% for single vanilla, 84.63% for single mixture-of-experts, 92.36% for dual joint self-attention, 88.25% for joint cross-attention, 91.85% for detached cross-attention, 87.95% for inverse dynamics, and 92.60% for the tri-system model.

Dual self-attention offers a strong balance of simplicity and accuracy. More parameters do not improve every variant: the single-system expert model is below vanilla, and the tri-system gain over dual is only 0.24 percentage points without uncertainty estimates.

Complete original table 1
Table 1 Architecture Ablation. Averaged success rates (%) on RoboTwin2.0Full. Bold denotes best values.

4.2.2 Training-time information flow

Action-sees-video reaches 92.39%, mutual visibility 92.15%, isolated streams 87.41%, and video-sees-action 87.63%. The large separation follows whether actions can access world features, supporting that connection within this configuration.

Here “from scratch” means without embodied pretraining; the system still inherits the video generator’s prior. It is distinct from the randomly initialized transformer used to isolate visual encoders.

Complete original figure 8
Figure 8 Attention Masking Strategies. We control cross-modality information flow at training time via attention masking.
Complete original table 2
Table 2 Action learning requires access to world information. Success rates (%) on RoboTwin2.0-Full.

4.2.3 Inference-time information flow

Figure 9 compares synchronization with variance-shift and linear-offset schedules on the same trained model. Synchronized joint denoising reaches 93.0%, above the tested action-leading and video-leading alternatives.

This supports joint refinement for the evaluated setup, and helps explain the weak two-stage inverse-dynamics baseline. It does not establish that every planning-based policy or every unequal schedule is inferior.

Complete original figure 9
Figure 9 Inference-time Information Flow via Denoising Schedule. Each curve traces action denoising progress against video denoising progress. Curves above/below the diagonal denoise action/video first, respectively.

4.3 Consolidating knowledge across domains

The third question asks how embodied data changes the inherited prior and preferred world–action interaction. Broad human video offers scene coverage but lacks robot actions; demonstrations provide embodiment-specific control supervision.

4.3.1 Setup and evaluation

All runs hold architecture, backbone, fine-tuning, and inference fixed. A 600-hour pretraining budget compares robot-only RoboCOIN with a 350-hour EgoDex plus 250-hour robot mixture, trained sequentially or jointly. The budget matches total hours, not robot-action exposure.

Clean2Random separates clean-condition performance from randomized transfer. Full-data evaluation additionally checks behavior when downstream fine-tuning already includes both environments.

4.3.2 Pretraining and generalization

Without embodied pretraining, clean/randomized success is 87.0%/14.5%. Robot-only pretraining gives 88.5%/23.8%; sequential human then robot training gives 87.1%/26.5%; joint training gives 87.68%/26.62%.

The main gain is randomized transfer. Robot supervision improves clean execution more, while human-video coverage helps transfer. Joint and sequential mixtures differ by only 0.12 points on randomized conditions; absent uncertainty, joint training’s simpler workflow is a stronger justification than a claimed decisive accuracy gain.

Complete original figure 10
Figure 10 Embodied Pretraining Primarily Improves OOD Generalization. Success rates on RoboTwin2.0-Clean2Random, ordered from lower to higher performance within each evaluation setting.

4.3.3 Pretraining changes preferred information flow

Before embodied pretraining, mutual attention is 0.24 points below action-sees-video. After pretraining, its gains are 0.16 points on RoboTwin-Full and 0.70/0.72 points on clean/randomized Clean2Random.

The recipe therefore changes to mutual visibility. Interpreting this as more useful action-conditioned world prediction is plausible, but the small performance differences do not directly identify the internal mechanism.

Complete original figure 11
Figure 11 Pretraining Changes the Preferred Information Flow. Central markers give the absolute success rate of Action Sees Video; arrows terminate at the matched Mutual result, with horizontal displacement reporting Mutual − Action Sees Video in percentage points. Green and red denote gains and drops, respectively.

4.4 Resulting recipe

Table 3 retains the accumulated recipe: Wan2.2-TI2V-5B with compact latents, dual joint self-attention, synchronized denoising, and one-stage human-plus-robot training with mutual visibility. Each choice is conditional on the preceding defaults and evaluation protocol.

Complete original table 3
Table 3 The recipe accumulated by OpenWAM-Study. Each evidence-backed choice becomes the default for OpenWAM-α.

5. OpenWAM-α

OpenWAM-α combines the selected components in a model trained across heterogeneous data and embodiments. Figure 12 preserves the architecture, data mixture, timestep plane, and unified control layout. The original project overview video is retained below.

Complete original figure 12
Figure 12 Overview of OpenWAM-α. (a) The dual-system architecture: a video-generation DiT and an ActionDiT jointly denoise the future frames and the action chunk through shared attention under the mutual visibility mask, each conditioned on language and proprioception via cross-attention and carrying its own noise timestep. (b) The pretraining mixture: 518.5M frames (6,369 hours) of egocentric and robot data, co-trained in one stage. (c) Timestep sampling: training covers the full joint noise plane, while inference follows the synchronized diagonal. (d) The 80-D unified action space with fixed slot semantics shared across embodiments.
A three-minute tour of the infrastructure, the study and the model.

5.1.1 Architecture

A 5B video transformer and approximately 1B action transformer communicate at all 30 paired layers through shared attention with 24 heads of 128 dimensions. Their residual widths are 3,072 and 1,024. Video uses three-dimensional rotary positions; actions use one-dimensional positions.

A frozen Wan-VAE retains the clean first frame and compresses each subsequent group of four; patches are 1×2×2. Frozen umT5 supplies language context, with an additional projected proprioceptive token. Separate noise-time conditioning and output heads predict video and action velocities.

5.1.2 Training

Video, action, and proprioception projections are trained jointly; language and visual encoders stay frozen. Video and action loss weights are both one, with bell-shaped timestep weighting. Independent warped noise sampling emphasizes high-noise inputs.

The flow shift is five for both streams. Human videos without actions or proprioception contribute only video loss through masking; their visible hand motions are not robot-action annotations.

5.1.3 Deployment

Inference uses ten synchronized Euler steps with the same flow warp and repeatedly pins the clean conditioning frame. The accelerated model takes about 170 ms per action chunk on an RTX 5090 after warmup.

Every reported simulation and real-robot benchmark uses synchronous serving: execution pauses for inference between chunks. The number is not evidence of uninterrupted real-time control under asynchronous deployment.

5.2 Multi-domain pretraining data

Curation reduces 1,333.9 million raw frames, or 14,348 hours, to 518.5 million frames and 6,369 hours. Table 4 retains every source’s raw and retained counts, frame rate, embodiment coverage, and sampling share.

The retained pool includes 1,442 hours of new egocentric video, 1,794 AgiBotWorld-Beta, 686 RoboCOIN, 1,007 DROID, and 1,440 synthetic InternData-A1 hours. It spans 21 robot embodiments plus human video [13] [87] [40] [78].

Complete original table 4
Table 4 The OpenWAM-α pretraining data. #Emb. counts each source’s distinct embodiments. Task Coverage marks the manipulation settings each source spans. Full reports each source’s raw size before processing, while Curated + Sampled reports the data actually used for training, after cleaning (Section 5.2.2) and per-source whole-episode subsampling (Section 5.2.1); Share is each source’s actual per-epoch sample share under proportional sampling.

5.2.1 Mixture and sampling

The human source begins with 71,600 first-person recordings spanning 3,006 tasks, each approximately 0.25–6 minutes long. Whole episodes are subsampled with fixed seeds and source budgets; real-robot budgets are redistributed across contributing subdatasets.

The approximate 30% human, 40% real-robot, and 30% synthetic mixture is a frame/sample ratio, not an hour ratio: sources have different frame rates. Proportional sampling draws each retained example once per epoch.

5.2.2 Data curation

Visual filtering removes unreadable, frozen, blurred, badly exposed, or abruptly discontinuous video. Human recordings also require visible manipulating hands and valid language annotations.

Robot episodes fail integrity checks when commanded and recorded motion disagree excessively or video–signal misalignment exceeds 2%. Leading and trailing idle spans are trimmed using state thresholds and motion below 2 cm/s and 5°/s; pauses within a trajectory are preserved. Median-absolute-deviation rules remove spikes, and visual checks catch sensor motion over frozen views.

Episodes are dropped if more than 70% of frames are removed or at least 90% are frozen. These rules and source budgets jointly determine what actually reaches pretraining.

5.3 Simulation evaluation

The pretrained checkpoint is separately fine-tuned for eight simulation benchmarks spanning single-arm, bimanual, mobile, and dexterous manipulation. RoboTwin-Full and Clean2Random are distinct evaluations of the same benchmark family.

Figure 13 presents representative comparisons; Appendix D preserves all reported baseline rows. Aggregate scores and success rates are not interchangeable, and clean/randomized averages should not be presented as randomized success alone.

Complete original figure 13
Figure 13 Score comparison of OpenWAM-α against representative VLA and WAM baselines across the simulation benchmarks. Within each panel, baselines are ordered by score, and every bar is labeled with its actual value.

5.3.1 Benchmark results

OpenWAM-α achieves 99.3% on LIBERO and 93.6% on RoboTwin-Full, close to reported leaders at 99.4% and 94.1%. It leads the shown EBench aggregate with 49.4% success and 64.7 progress, but loses its tabletop subset to Qwen-RobotManip.

Its RoboTwin-Clean2Random score is 89.4% clean and 48.7% randomized; 69.0% is their average. On RoboDojo, 11.92% success leads the listed world–action models but trails the strongest vision–language–action baseline at 19.34%.

RoboCasa365 is a material exception to the main text’s “marginal gap” description: α averages 38.2% versus the best 57.4%, a 19.2-point gap, and is also below ABot-M0.5 at 40.4%. RoboCasa-GR1 reaches 60.5% versus 64.5%. Table-level evidence supports competitive performance, not uniform leadership.

Complete original figure 14
Figure 14 OpenWAM-α against the best of each family, per benchmark, grouped by embodiment.

LIBERO-Plus robustness

LIBERO-Plus reuses the LIBERO fine-tuned checkpoint. Average success is 69.2%, versus Qwen-RobotManip’s 89.0%. Camera changes score 33.8% and image noise 39.8%, while lighting reaches 97.0%, language 88.0%, and background 87.1%.

The breakdown identifies specific visual robustness gaps; it is more informative than describing all out-of-distribution changes as equally difficult. The full table preserves every perturbation category.

Complete original table 5
Table 5 Evaluation Results on LIBERO-Plus. Bold denotes best values, underline second best.

Data and representation diagnoses

Figure 15 compares roughly 1,100 hours of single-arm pretraining in OpenWAM with over 8,000 in ABot-M0.5 and 5,700 in Being-H0.7. The authors attribute some robustness differences to data diversity and some to pixel-predictive versus semantic representations.

These are cross-model diagnoses with different architectures and data, not isolated causal ablations. The claim that pixel prediction compounds camera/noise errors is a hypothesis consistent with selected comparisons.

Table 4 marks dexterous task coverage for AgiBotWorld-Beta and RoboCOIN. The later adaptation experiment tests a specifically unseen Wuji/Tianji platform and its 9+21-dimensional action layout; it does not establish that pretraining contained no dexterous robot data.

Complete original figure 15
Figure 15 Single-arm pretraining data of ABot-M0.5, Being-H0.7, and OpenWAM-α. The dashed lines indicate that the single-arm data of OpenWAM-α amounts to only a small fraction of what ABot-M0.5 and Being-H0.7 consume.

5.3.2 VLA versus WAM

Figure 16 compares models without embodied pretraining and then larger pretrained systems. The observed pattern favors world–action models on several in-distribution splits and vision–language–action models on several shifted splits.

The paper’s broad “neither paradigm prevails” conclusion is reasonable as an empirical summary. It does not prove inherent superiority on either class of task: data coverage, representation, architecture, and optimization differ together, and the per-category tables contain exceptions.

Complete original figure 16
Figure 16 ID and OOD comparisons between the two paradigms. (a) Fast-WAM versus StarVLA, two models without embodied pretraining, on ID and OOD splits. (b) OpenWAM-α versus the three strongest VLAs on ID splits. (c) OpenWAM-α versus the three strongest VLAs on OOD splits.

5.4 Real-robot evaluation

Figure 17 shows six Franka single-arm tasks, 18 RoboDojo bimanual tasks on ARX X5/Piper/Piper X, and four tasks using a Wuji hand with a Tianji arm. All downstream evaluations use fine-tuned policies and distinguish complete success from partial progress where applicable.

Complete original figure 17
Figure 17 Real-robot experimental setups across three embodiments. Top: the six single-arm tasks on the Franka-Research-3 platform. Bottom left: the three bimanual embodiments of the RoboDojo real-world track (Piper X, Piper, and ARX X5), covering 18 tasks in total. Bottom right: the four dexterous-hand tasks on the Wuji-hand and Tianji-arm platform, each illustrated by key intermediate stages of its execution.

Single-arm and bimanual results

Across 120 single-arm trials, α succeeds in 99 (82.5%), versus LingBot-VA’s 93 (77.5%) and π0.5’s 66 (55%). Against LingBot-VA it wins four tasks, ties hanging the M-shaped object, and loses ring stacking, 60% versus 75%.

On the 18-task bimanual evaluation, aggregate progress/success is 37.6/24.4 versus π0.5’s 22.9/12.8. Several tasks still have zero success, and some individual success rates are lower despite better partial progress. Table 7 retains the exact task-order key needed to interpret its numbered columns.

The retained project rollouts illustrate selected executions. They are demonstrations, not substitutes for the trial counts or evidence of uniform reliability.

Complete original table 6
Table 6 Evaluation Results on Single-Arm Real-Robot Tasks. Bold denotes best values, underline second best.
Complete original table 7
Table 7 Evaluation Results on the Bimanual RoboDojo Real-World Track. Each cell reports Score / SR (%). Bold denotes best values, underline second best. Rows are shaded by embodiment.
cover blocks ARX X5
insert tubes ARX X5
pack and pour fruit ARX X5
store in safe ARX X5
fill pen holder Piper
put objects into basket Piper
stack bowls Piper
stand up bottles Piper
cap pen Piper X
disassemble lego Piper X
pack objects into backpack Piper X
sweep blocks Piper X

Dexterous-hand adaptation

The pretrained model is fine-tuned on a previously unseen 30-dimensional control space: nine end-effector coordinates plus 21 hand degrees of freedom. Table 8 compares four tasks under original and altered conditions. This is adaptation to a new embodiment, not zero-shot dexterous control.

α has higher reported progress and success than π0.5 in every listed task/condition. For example, clothes cleanup reaches 100% in-domain and 85% shifted success, while shifted shuttlecock success remains 4/15, or 26.7%. Small counts and absent confidence intervals limit precision.

Shuttlecock progress uses different shifted object totals, 57 for α and 54 for π0.5. This is an element-weighted score, not a matched per-trial mean. The complete numerators and denominators are retained.

Complete original table 8
Table 8 Evaluation Results on Dexterous-Hand Real-Robot Tasks. Each cell reports Score / SR (%); the OOD column aggregates all variation settings of its task. Bold denotes best values.

6. Conclusions

OpenWAM contributes a modular implementation, a conditional experimental recipe, and a broadly evaluated pretrained baseline. Its strongest practical lesson is to track representation, data coverage, action supervision, and serving protocol together.

The reported evidence favors several components and shows useful transfer, while leaving substantial robustness and long-horizon gaps. The public infrastructure, checkpoint metadata, data recipe, and full tables make those gaps inspectable.

Acknowledgments

The authors thank Nilaksh and Chuning Zhu for discussions and Wuji Technology for compute resources.

A. Limitations and future work

The study emphasizes embodied pretraining rather than post-training algorithms. Architecture exploration centers on attention and hard-routed experts; early fusion and softly routed experts remain open.

UMI-style and DexUMI robot-executable demonstrations are absent from the mixture [21] [88]. They could connect broad scene diversity with action supervision, but this paper does not test that improvement.

Wan-VAE is a practical compromise, not an established optimal encoder. Compact representations that retain control-relevant information and withstand viewpoint, noise, and scene changes remain a central limitation.

B.1 Pretraining configuration

Table 9 retains the complete configuration. Training uses 128 H200 processors for about seven days, 155,862 optimizer steps, one epoch, and batch size 3,072. AdamW uses learning rate 10⁻⁴, cosine decay, 5% warmup, weight decay 0.01, momenta 0.9/0.95, and global gradient clipping at one.

Inputs contain 33 frames with video stride four and window stride one at 384×320 resolution. Training uses bfloat16, ZeRO stage two, multi-view inputs, and ColorJitter(0.2, 0.2, 0.2, 0.0).

Complete original table 9
Table 9 Pretraining configuration for OpenWAM-α.

B.2 Dataset-specific fine-tuning

Table 10 preserves every dataset-specific batch size, training duration, and augmentation setting. Most use batch 256; VLABench uses 196 and RoboCasa365 1,024. LIBERO-Plus receives no additional fine-tuning.

Real single-arm training uses 9,860 steps over ten epochs; bimanual training uses 30,000 steps; dexterous adaptation uses 10,925 steps over five epochs. The appendix does not provide the dexterous demonstration count, so epoch counts alone do not specify dataset size.

Complete original table 10
Table 10 Dataset-specific SFT configuration for OpenWAM-α. Settings not listed are identical to pretraining (Table 9); “–” denotes no image augmentation.

C.1 Single-arm real-robot protocol

The Franka-Research-3 uses a parallel gripper, camera observations, language, and proprioception. Each of six tasks has 100 demonstrations and 20 independent evaluation trials. Success requires stable placement, maintained hanging, or placing the object in the drawer and closing it, depending on the task.

Table 11 retains every original instruction. Figures 18–19 preserve hardware and six-frame task sequences; the original six project rollout videos are also retained.

Complete original figure 18
Figure 18 Single-arm real-robot setup on the Franka-Research-3 platform. The workspace contains the drawer, stacking objects, and hanging fixtures used across the six tasks, while the robot is equipped with an Intel RealSense camera and a Robotiq parallel gripper for closed-loop execution.
Complete original figure 19
Figure 19 Single-arm task execution sequences on the Franka-Research-3 platform. Each row shows one task, with six frames uniformly sampled from the corresponding left-view rollout video.
Complete original table 11
Table 11 Task instructions used in the single-arm real-robot evaluation.
Hang M
Chili into drawer
Hang cup
Jenga
Jenga into drawer
Stack rings

C.2 Dexterous-hand protocol

The four tasks stack three discs by size, collect two to four shuttlecocks, clear three to four clothing items, or unscrew a bottle cap. Table 12 preserves the exact language instructions; Figures 20–21 retain the platform, task stages, and altered scenes.

Each shifted condition changes one factor, such as object identity, placement, lighting, or background, while preserving the objective. Results do not establish robustness to arbitrary simultaneous combinations. Original project videos retain their individual condition labels.

Complete original figure 20
Figure 20 Real-world experimental setup. The figure shows the physical dexterous-hand platform, the workspace, the observation camera, and representative objects for the four manipulation tasks.
Complete original figure 21
Figure 21 Task execution sequences and evaluation conditions. Each row illustrates the main manipulation stages of one task, while the columns show the corresponding ID scene and OOD variations. OOD conditions include changes in object identity, object layout, illumination, and background appearance.
Complete original table 12
Table 12 Task instructions used in the real-world evaluation.
Collect shuttlecocks OOD background
Collect shuttlecocks in-distribution
Collect shuttlecocks OOD layout
Collect shuttlecocks OOD lighting
Twist off bottle cap OOD background
Twist off bottle cap in-distribution
Twist off bottle cap OOD layout
Twist off bottle cap OOD lighting
Twist off bottle cap OOD object
Put away clothes OOD background
Put away clothes in-distribution
Put away clothes OOD layout
Put away clothes OOD lighting
Put away clothes OOD object
Stack toy tower in-distribution
Stack toy tower OOD layout
Stack toy tower OOD lighting

Success, progress, and task criteria

Complete success is the fraction of trials meeting the full objective. Progress divides total completed elements by total available elements across trials, so trials with more objects carry more weight. Both original formulas are preserved.

For tower, shuttlecock, and clothing tasks, partial progress counts placed objects. For the bottle, progress records cap removal; complete success additionally requires maintaining a stable grasp of the bottle or cap. A 100% progress score therefore need not imply 100% successful trials.

Equations
Complete original equation success
Complete original equation progress

C.3 Bimanual real-robot protocol

The study evaluates all 18 tasks on the official RoboDojo real-robot platforms and follows the benchmark’s task definitions. Table 7 includes the six ordered tasks for each embodiment. The three-platform aggregate does not imply that every task generalizes to every robot without adaptation.

D. Complete simulation results

Tables 13–20 retain all simulation results, grouped by model family with the original bold/underline ranking conventions. LIBERO-Plus is retained separately as Table 5.

VLABench reaches 58.9% overall success versus the best 59.1%, but its other metrics and category scores are not uniformly strongest. RoboCasa365 scores 69.7% on atomic tasks, 32.1% on seen composites, and 8.9% on unseen composites. RoboDojo memory success is 9.11% versus the strongest listed 47.44%.

These detailed outcomes qualify the overview charts: high aggregate performance can coexist with weak category transfer, incomplete long-horizon execution, and substantial gaps to a specialist or differently trained baseline.

Complete original table 13
Table 13 Evaluation Results on LIBERO. Bold denotes best values, underline second best.
Complete original table 14
Table 14 Evaluation Results on VLABench. Bold denotes best values, underline second best.
Complete original table 15
Table 15 Evaluation Results on RoboTwin2.0-Clean2Random. Bold denotes best values, underline second best.
Complete original table 16
Table 16 Evaluation Results on RoboTwin2.0-Full. Bold denotes best values, underline second best.
Complete original table 17
Table 17 Evaluation Results on RoboDojo. Bold denotes best values, underline second best.
Complete original table 18
Table 18 Evaluation Results on EBench. Bold denotes best values, underline second best.
Complete original table 19
Table 19 Evaluation Results on RoboCasa365. Bold denotes best values, underline second best.
Complete original table 20
Table 20 Evaluation Results on RoboCasa-GR1. Bold denotes best values, underline second best.
Concept demonstration

Open image ↗