Literature
Display
← All papers

DiWA: Diffusion Policy Adaptation with World Models

Akshay L Chandra · Iman Nematollahi · Chenguang Huang · Tim Welschehold · Wolfram Burgard · Abhinav Valada

Preprint posted · CoRL 2025 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

DiWA adapts separately pretrained robot skills by running policy optimization inside a frozen world model. The supplementary material provides architecture and training settings, the complete algorithm, physical data collection, and additional transfer/regularization experiments.

At a glance

A world model learned from offline play supplies imagined transitions for reinforcement-learning adaptation of diffusion policies. Expert demonstrations initialize each skill and train its reward classifier. The method improves eight CALVIN skills, four selected LIBERO skills, and three physical tabletop skills without collecting new interaction data for the policy-update stage. “Offline” does not remove the earlier play/demonstration collection or the physical evaluations needed to assess checkpoints.

1 Introduction

Diffusion Policy captures multimodal actions but inherits demonstration coverage limits. DPPO applies policy gradients across denoising steps, at the cost of online environment rollouts. DiWA replaces those rollouts with a learned latent simulator. This shifts the bottleneck from repeated task-specific interaction toward world-model coverage, reward accuracy, and simulation exploitation; imagined rollouts are computational work, not cost-free data.

Original figure 1 with complete labels and original mathematical layout.
Figure 1: (a) Standard diffusion policies trained via imitation learning are limited by offline data. (b) DPPO [17] fine-tunes diffusion policies using online interactions, which are expensive and require access to real or simulated environments. (c) DiWA fine-tunes diffusion policies entirely offline through imagined rollouts in a learned world model, enabling safe and efficient policy improvement without additional physical interaction.

2 Related Work

The approach combines imitation initialization, reusable play-trained dynamics, and PPO-style diffusion adaptation. Unlike online model-based methods that continually refresh their dynamics, DiWA freezes its model before policy optimization. Its contribution is the integration of denoising-step optimization with offline imagination, rather than a guarantee that a frozen model predicts every improved policy’s behavior.

3 Problem Formulation

Two datasets are required: task-agnostic play for dynamics and a small task-specific expert set for policy and reward learning. Gaussian noise is progressively denoised into an action or action chunk. The desired objective is real-environment return, but optimization occurs in a latent Markov model with learned dynamics and rewards. Improvement in that surrogate therefore depends on its fidelity along the adapted policy’s trajectories.

Equations 1–2
a¯tk−1∼πθ​(a¯tk−1∣st,a¯tk),fork=K,K−1,…,1,\bar{a}_{t}^{k-1}\sim\pi_{\theta}(\bar{a}_{t}^{k-1}\mid s_{t},\bar{a}_{t}^{k}),\quad\text{for}\quad k=K,K-1,\dots,1, (1)
θ⋆=arg⁡maxθ​𝔼τ∼πθ​[∑t=0∞γt​R​(st,at)].\theta^{\star}=\arg\max_{\theta}\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right]. (2)

4 Offline Adaptation of Diffusion Policy with DiWA

The four stages are world-model training, latent-conditioned behavior cloning, success-classifier training, and policy optimization in imagination. Dynamics, reward estimation, and the pretrained reference policy are fixed during adaptation. Deployment conditions the adapted policy on latents inferred from real observations.

Refer to caption
Figure 2: DiWA framework: (1) A world model is trained on unstructured robot play data to learn latent dynamics. (2) A diffusion policy is pre-trained on expert demonstrations using learned latent representations. (3) A success classifier is trained on expert rollouts to estimate task rewards. (4) The diffusion policy is fine-tuned entirely offline via imagined rollouts within the Dream Diffusion MDP, using policy gradients and classifier-based rewards.

4.1 World Model Learning

A recurrent state-space model combines a deterministic hidden state with categorical stochastic variables. An observation-conditioned posterior trains against a dynamics prior through reconstruction and KL terms, using straight-through gradients. During imagination, the prior advances without new images. Static and wrist-camera encoders follow the LUMOS design. Temporal state helps prediction but does not make long imagined trajectories ground truth.

Equations 3–4
Recurrent state:\displaystyle\text{Recurrent state:} ht\displaystyle h_{t} =fϕ​(s^t−1,at−1)\displaystyle=f_{\phi}(\hat{s}_{t-1},a_{t-1})\quad Representation model:zt\displaystyle\text{Representation model:}\quad z_{t} ∼qϕ​(zt∣ht,xt)\displaystyle\sim q_{\phi}(z_{t}\mid h_{t},x_{t}) (3)
Dynamics predictor:\displaystyle\text{Dynamics predictor:} z^t\displaystyle\hat{z}_{t} ∼pϕ​(z^t∣ht)\displaystyle\sim p_{\phi}(\hat{z}_{t}\mid h_{t})\quad Decoder:x^t\displaystyle\text{Decoder:}\quad\hat{x}_{t} ∼pϕ​(x^t∣s^t),\displaystyle\sim p_{\phi}(\hat{x}_{t}\mid\hat{s}_{t}),
minϕ𝔼qϕ[∑t=1T−logpϕ(xt∣st)+βKL(qϕ(zt∣ht,xt)∥pϕ(zt∣ht))],\min_{\phi}\;\mathbb{E}_{q_{\phi}}\left[\sum_{t=1}^{T}-\log p_{\phi}(x_{t}\mid s_{t})+\beta\,\text{KL}\left(q_{\phi}(z_{t}\mid h_{t},x_{t})\,\|\,p_{\phi}(z_{t}\mid h_{t})\right)\right], (4)

4.2 Pre-training Diffusion Policies

Fifty demonstrations per skill are encoded with the frozen world model. The diffusion policy learns demonstrated action sequences in that representation, aligning the interface used for imitation, imagination, and deployment. Separate policies are trained for each skill; this is not a single language-conditioned generalist policy.

4.3 Latent Reward Estimation from Expert Demonstrations

Annotated successful demonstration frames train a binary latent classifier. Its success probability supplies the reward during imagined action execution. Play is unlabeled, but the entire method is not reward-supervision-free. The main section evaluates the classifier on the next latent state, whereas the appendix writes a current-latent expression; both original conventions are retained and should be checked in implementation.

4.4 Dream Diffusion MDP

The Dream Diffusion MDP nests action denoising within world-model time. Intermediate denoising steps keep the same environment latent and receive zero reward. The final denoised action advances the latent dynamics, yields a classifier reward, and restarts from fresh Gaussian noise. Each Gaussian denoising transition has an explicit likelihood, enabling policy-gradient updates without needing the full marginal action-distribution likelihood.

Equations 5–8
s¯t¯​(t,k)=(zt,a¯tk),a¯t¯​(t,k)=a¯tk−1,R¯t¯​(t,k)={Rψ​(zt,a¯t0),if ​k=1,0,otherwise.\bar{s}_{\bar{t}(t,k)}=(z_{t},\bar{a}_{t}^{k}),\quad\bar{a}_{\bar{t}(t,k)}=\bar{a}_{t}^{k-1},\quad\bar{R}_{\bar{t}(t,k)}=\begin{cases}R_{\psi}(z_{t},\bar{a}_{t}^{0}),&\text{if }k=1,\\ 0,&\text{otherwise}.\end{cases} (5)
P¯​(s¯t¯+1∣s¯t¯,a¯t¯)={δ⁡(zt,a¯tk−1),if ​k>1,Pϕ​(zt+1∣zt,a¯t0)⊗𝒩⁡(0,I),if ​k=1,\bar{P}(\bar{s}_{\bar{t}+1}\mid\bar{s}_{\bar{t}},\bar{a}_{\bar{t}})=\begin{cases}\delta(z_{t},\bar{a}_{t}^{k-1}),&\text{if }k>1,\\ P_{\phi}(z_{t+1}\mid z_{t},\bar{a}_{t}^{0})\otimes\mathcal{N}(0,I),&\text{if }k=1,\end{cases} (6)
π¯θ​(a¯tk−1∣zt,a¯tk)=𝒩⁡(a¯tk−1,μθ​(zt,a¯tk,k),σk2​I),\bar{\pi}_{\theta}(\bar{a}_{t}^{k-1}\mid z_{t},\bar{a}_{t}^{k})=\mathcal{N}\left(\bar{a}_{t}^{k-1};\mu_{\theta}(z_{t},\bar{a}_{t}^{k},k),\sigma_{k}^{2}I\right), (7)
∇θ𝒥¯​(π¯θ)=𝔼π¯θ,P¯​[∑t¯≥0∇θ​log​π¯θ​(a¯t¯∣s¯t¯)​r¯​(s¯t¯,a¯t¯)],\nabla_{\theta}\bar{\mathcal{J}}(\bar{\pi}_{\theta})=\mathbb{E}^{\bar{\pi}_{\theta},\bar{P}}\left[\sum_{\bar{t}\geq 0}\nabla_{\theta}\log\bar{\pi}_{\theta}(\bar{a}_{\bar{t}}\mid\bar{s}_{\bar{t}})\,\bar{r}(\bar{s}_{\bar{t}},\bar{a}_{\bar{t}})\right], (8)

4.5 Fine-tuning within Dream Diffusion MDP

PPO clips likelihood-ratio updates and discounts earlier, noisier denoising steps. A pretrained-policy regularizer is intended to reduce departures that exploit model errors. This restriction trades adaptation against fidelity to demonstrated behavior; it cannot certify correctness outside the model’s support. The published objectives and appendix algorithm are preserved rather than rewritten into a different regularizer.

Equations 9–11
ℒPPO=𝔼(s¯,a¯)π¯θold​[min⁡(ρθ​(s¯,a¯)​A^​(s¯,a¯),clip⁡(ρθ​(s¯,a¯),1−ϵ,1+ϵ)​A^​(s¯,a¯))],\mathcal{L}_{\text{PPO}}=\mathbb{E}^{\bar{\pi}_{\theta_{\text{old}}}}_{(\bar{s},\bar{a})}\left[\min\left(\rho_{\theta}(\bar{s},\bar{a})\hat{A}(\bar{s},\bar{a}),\ \operatorname{clip}(\rho_{\theta}(\bar{s},\bar{a}),1-\epsilon,1+\epsilon)\hat{A}(\bar{s},\bar{a})\right)\right], (9)
A^​(s¯t¯​(t,k),a¯t¯​(t,k))=γdenoisek​(r¯​(s¯t¯,a¯t¯)−V^​(zt)),\hat{A}(\bar{s}_{\bar{t}(t,k)},\bar{a}_{\bar{t}(t,k)})=\gamma_{\text{denoise}}^{k}\left(\bar{r}(\bar{s}_{\bar{t}},\bar{a}_{\bar{t}})-\hat{V}(z_{t})\right), (10)
ℒθ=ℒPPO−αBC​𝔼π¯θold​[∑k=1Klog⁡πθpre​(a¯tk−1∣zt,a¯tk)],\mathcal{L}_{\theta}=\mathcal{L}_{\text{PPO}}-\alpha_{\text{BC}}\ \mathbb{E}^{\bar{\pi}_{\theta_{\text{old}}}}\left[\sum_{k=1}^{K}\log\pi_{\theta_{\text{pre}}}(\bar{a}_{t}^{k-1}\mid z_{t},\bar{a}_{t}^{k})\right], (11)

5 Experimental Evaluation

Experiments separate simulated skill adaptation, latent-model/reward ablations, and physical execution of adapted policies. Online-baseline environment steps, offline play collection, imagined steps, and evaluation rollouts are different costs.

5.1 Simulation Results

CALVIN environment D supplies six hours of play, approximately five hundred thousand transitions, plus fifty demonstrations for each of eight skills. After five million imagined adaptation steps, drawer closing rises from 59.14% to 91.95%, lightbulb switching off from 35.63% to 77.01%, and LED switching on from 48.43% to 86.21%. Table 1 reports three-seed averages and deviations.

The matched latent-input DPPO baseline starts from the same policy and uses ground-truth rewards, while DiWA uses a learned classifier. Matching DiWA takes roughly 118,000–495,000 additional environment steps per skill for latent DPPO, and 134,000–2.25 million for pixel DPPO. These measurements are CALVIN simulator interactions despite the table’s “physical” label; DiWA’s zero excludes shared offline collection and evaluation. Pixel DPPO is faster than latent DPPO on the LED-off row, so the modality advantage is not uniform.

Hybrid models with privileged scene-state supervision adapt faster/more stably, and decoding state for reward can further improve results. These are diagnostic simulator variants; the main physical-compatible variant relies on vision. The comparisons support the importance of representation and reward quality without proving every gain comes from one component.

Table 1: DiWA successfully fine-tunes diffusion policies entirely offline using imagined rollouts in a learned world model. In contrast, DPPO requires hundreds of thousands of online interactions to achieve comparable performance. The DPPO (Vision) variant, operating directly on raw RGB observations without world‑model latents, requires far more interactions to reach similar performance. Results are averaged over three random seeds.
Task Base DiWA (Ours)
DPPO
(Vision WM Encoder)
DPPO
(Vision)
Diffusion Policy Offline Fine-Tuning Online Fine-Tuning
Success Rate Success Rate Env Steps to Match DiWA
open-drawer 57.78±3.8557.78\pm 3.85 74.44±1.92\textbf{74.44}\pm\textbf{1.92} 117,600±23,758117{,}600\pm 23{,}758 134,400±26,508134{,}400\pm 26{,}508
close-drawer 59.14±5.0859.14\pm 5.08 91.95±1.99\textbf{91.95}\pm\textbf{1.99} 345,600±27,651345{,}600\pm 27{,}651 1,545,600±261,3461{,}545{,}600\pm 261{,}346
move-slider-left 62.15±0.6062.15\pm 0.60 83.33±1.80\textbf{83.33}\pm\textbf{1.80} 270,933±28,780270{,}933\pm 28{,}780 1,377,600±251,4391{,}377{,}600\pm 251{,}439
move-slider-right 62.55±3.5562.55\pm 3.55 82.76±3.45\textbf{82.76}\pm\textbf{3.45} 249,600±09,050249{,}600\pm 09{,}050 537,600±23,758537{,}600\pm 23{,}758
turn-on-lightbulb 60.61±3.0360.61\pm 3.03 91.92±1.75\textbf{91.92}\pm\textbf{1.75} 302,933±15,964302{,}933\pm 15{,}964 588,000±62,859588{,}000\pm 62{,}859
turn-off-lightbulb 35.63±1.9935.63\pm 1.99 77.01±1.99\textbf{77.01}\pm\textbf{1.99} 327,066±13,546327{,}066\pm 13{,}546 1,260,000±142,5521{,}260{,}000\pm 142{,}552
turn-on-LED 48.43±3.6748.43\pm 3.67 86.21±3.45\textbf{86.21}\pm\textbf{3.45} 494,933±45,655494{,}933\pm 45{,}655 2,251,200±33,9402{,}251{,}200\pm 33{,}940
turn-off-LED 55.25±4.7955.25\pm 4.79 82.33±6.53\textbf{82.33}\pm\textbf{6.53} 277,333±31,928277{,}333\pm 31{,}928 184,800±23,758184{,}800\pm 23{,}758
Total Physical Interactions: 0 ∼\sim2.5M ∼\sim8M
Refer to caption
Figure 3: Comparison of three DiWA variants on simulated fine-tuning tasks. Blue uses only visual inputs, while green and red both incorporate scene state supervision. Red further decodes rewards from latents instead of relying on a learned classifier. Results demonstrate that more expressive world models and more accurate reward signals lead to improved offline fine-tuning performance.

5.2 Real-World Results

Four hours of physical Panda teleoperation, reported as about 450,000 recorded transitions, train the model; fifty extra expert demonstrations per skill initialize drawer opening, drawer closing, and slider-right pushing. Roughly two million imagined steps improve all three learning curves. Evaluation uses twenty rollouts per skill and three seeds with fixed initial configurations and robot poses. “No interaction” applies to gradient-update data collection, not those demonstrations or physical checkpoint evaluations. The study tests these three local skills rather than broad environment or object generalization.

Refer to caption
(a) Real-world Manipulation Skills
Refer to caption
(b) Real-World Fine-Tuning Results
Figure 4: (a) The three real-world manipulation tasks used for evaluation. (b) Success rates before and after offline fine-tuning with DiWA, averaged over 20 rollouts and three seeds. Values correspond to checkpoints saved during fine-tuning. While pre-trained diffusion policies show limited initial performance, DiWA enables significant improvement through imagination-based reinforcement learning without physical interaction.

6 Conclusion

A reusable offline world model can support diffusion-policy improvement with no new environment rollouts used for the adaptation updates. The physical results demonstrate imagination-to-robot transfer in a trained tabletop setting. Savings are best understood as avoiding additional task-specific online training interaction after collecting play and expert data.

7 Limitations

Frozen dynamics errors persist and may be exploited by the policy; high imagined reward can fail to transfer. The authors explicitly say intermediate checkpoints must be evaluated on the real system to assess true performance. This limits a strict interpretation of evaluation-free offline adaptation. Limited online model correction is proposed as future work, while adequate data coverage, reliable rewards, and checkpoint/regularization choices remain practical dependencies.

Acknowledgments

The authors acknowledge support from the University of Freiburg’s BrainWorlds initiative at the BrainLinks-BrainTools center.

Appendix S.1 Hyperparameters and Training Details

The supplementary settings specify the latent model, diffusion-policy training, reward classifier, and fine-tuning process. The complete original algorithm and every hyperparameter table are retained.

S.1.1 World Model

Two visual encoders feed a DreamerV2-style recurrent model. Thirty-two categorical variables with thirty-two classes form a 1,024-dimensional one-hot stochastic state; another 1,024 deterministic features give a 2,048-dimensional policy input. Training uses fifty-step sequences, batch fifty, recurrent reset probability 0.01, learning rate 3×10⁻⁴, and KL balancing 0.8. KL scale is 0.3 in simulation and 1.0 physically. Random subsequences stay within original episode boundaries.

Table S.1: Hyperparameters used for training the world model. All values are shared across simulation and real-world experiments, except KL loss scale β\beta, which is 0.3 for simulation and 1.0 for real-world settings.
Name Symbol Value
Batch size BB 50
Sequence length LL 50
Deterministic latent state dimensions — 1024
Discrete latent state dimensions — 32
Discrete latent state classes — 32
Latent dimensions kk 2048
KL loss scale β\beta 0.3
KL balancing coefficient δ\delta 0.8
RSSM reset probability ζ\zeta 0.01
World model learning rate — 3×10−43\times 10^{-4}
Gradient clipping — 100
Adam epsilon ϵ\epsilon 10−510^{-5}
Weight decay (decoupled) — 5×10−25\times 10^{-2}
Equations 12–13
minϕ𝔼qϕ[∑t=1T−logpϕ(xt∣st)+βKL(qϕ(zt∣ht,xt)∥pϕ(zt∣ht))],\min_{\phi}\;\mathbb{E}_{q_{\phi}}\left[\sum_{t=1}^{T}-\log p_{\phi}(x_{t}\mid s_{t})+\beta\,\text{KL}\left(q_{\phi}(z_{t}\mid h_{t},x_{t})\,\|\,p_{\phi}(z_{t}\mid h_{t})\right)\right], (12)
KL(q∥p)=δKL(q∥sg(p))⏟posterior regularizer+(1−δ)KL(sg(q)∥p)⏟prior regularizer,\text{KL}(q\parallel p)=\delta\underbrace{\text{KL}(q\parallel\text{sg}(p))}_{\text{posterior regularizer}}+(1-\delta)\underbrace{\text{KL}(\text{sg}(q)\parallel p)}_{\text{prior regularizer}}, (13)

S.1.2 Diffusion Policy

The DDPM policy uses twenty denoising steps, one observation, and four predicted/executed actions per chunk. A three-layer 512-unit MLP trains for five thousand epochs on fifty demonstrations per skill, batch 256, with learning rate decaying from 10⁻⁴ to 10⁻⁵. EMA starts at epoch twenty. Baseline interfaces include 51-dimensional simulator state, two 64×64 RGB images stacked as six channels, or the same 2,048-dimensional world-model representation.

Equation 14
ℒBC(θ)=𝔼𝒟exp[∑t=1T∑k=1K−logπθ(atk−1∣zt,atk)],\mathcal{L}_{\text{BC}}(\theta)=\mathbb{E}_{\mathcal{D}_{\text{exp}}}\left[\sum_{t=1}^{T}\sum_{k=1}^{K}-\log\pi_{\theta}(a_{t}^{k-1}\mid z_{t},a_{t}^{k})\right], (14)
Table S.2: Training and model hyperparameters for diffusion policy across all CALVIN and real-world tasks.
Parameter Symbol Value
Common Training Parameters (All Skills)
Observation Horizon — 1
Number of Demonstrations NN 50
Planning Horizon TpT_{p} 4
Action Horizon TaT_{a} 4
Training Epochs — 5000
Diffusion Denoising Steps KK 20
Initial Learning Rate — 1×10−41\times 10^{-4}
Final Learning Rate — 1×10−51\times 10^{-5}
Weight Decay — 1×10−61\times 10^{-6}
MLP Dimensions — [512, 512, 512]
EMA Decay — 0.995
EMA Start Epoch — 20
EMA Update Frequency — 10
Batch Size — 256
Observation Dimensions
DiWA — 2048
DPPO (Vision WM Encoder) — 2048
DPPO (Vision) — 64×64×664\times 64\times 6
DPPO (State) — 51

S.1.3 Latent Reward Estimator

Two 512/512 MLPs produce a contrastive embedding and success logits, trained with NT-Xent plus cross-entropy for one hundred epochs. Positives are annotated success frames; negatives sample fifteen percent of remaining frames. The retained validation table gives mean latent-classifier precision/recall 0.89/0.98 versus image ResNet-18’s 0.41/0.98. High recall alone does not prevent false-positive reward exploitation, and validation accuracy does not guarantee calibration on policy-generated imagined states.

Equations 15–16
ℒNT-Xent=−log⁡exp⁡(sim​(fψ​(zi),fψ​(zj))/τ)∑k=12​N𝟙[k≠i]exp(sim(fψ(zi),fψ(zk))/τ),\mathcal{L}_{\text{NT-Xent}}=-\log\frac{\exp(\text{sim}(f_{\psi}(z_{i}),f_{\psi}(z_{j}))/\tau)}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq i]}\exp(\text{sim}(f_{\psi}(z_{i}),f_{\psi}(z_{k}))/\tau)}, (15)
ℒreward=ℒNT-Xent+ℒCE.\mathcal{L}_{\text{reward}}=\mathcal{L}_{\text{NT-Xent}}+\mathcal{L}_{\text{CE}}. (16)
Table S.3: Hyperparameters used for training the latent reward classifier.
Parameter Value
Embedding MLP [512, 512]
Classification MLP [512, 512]
Activation ReLU
Output Softmax
Epochs 100
Batch Size 32
Learning Rate 1×10−61\times 10^{-6}
Temperature 0.5
Loss Contrastive+CE
Positives Success frames
Negatives 15% other frames
Table S.4: Validation precision and recall for our latent-based and vision-based reward classifiers.
Task Latents Vision
Prec. Rec. Prec. Rec.
open-drawer 0.92 0.99 0.41 0.99
close-drawer 0.89 0.99 0.52 0.99
move-slider-left 0.87 0.96 0.33 0.97
move-slider-right 0.83 0.98 0.41 0.99
turn-on-lightbulb 0.89 0.96 0.45 0.99
turn-off-lightbulb 0.88 0.99 0.36 1.00
turn-on-LED 0.94 1.00 0.36 0.92
turn-off-LED 0.88 0.99 0.41 0.99
Avg. 0.89 0.98 0.41 0.98

S.1.4 Fine-tuning with DiWA

Algorithm S.1 preserves all thirty-eight lines, including nested imagination/update loops and their indentation. GAE is computed at action-completion steps, then denoising-discounted for policy updates. Only the last ten of twenty denoising steps are finetuned. Actor/critic learning rates are 10⁻⁵/10⁻³, world-model discount 0.999, diffusion discount 0.99, and GAE smoothing 0.95. Default cloning coefficient is 0.05, with task-specific 0.10/0.025/0.025 for opening/closing the drawer and turning on the LED. These selected coefficients are part of the experimental recipe, not a universally fixed setting.

Algorithm S.1 DiWA: Diffusion Policy Adaptation with World Models
Original algorithm s1 with complete labels and original mathematical layout.
Equations 17–19
A^t¯​(t,1)λ=∑l=0∞(γWM​λ)l​δ¯t¯​(t+l,1),where ​δ¯t¯​(t,1)=R¯t¯​(t,1)+γWM​Vν​(s¯t¯​(t+1,1))−Vν​(s¯t¯​(t,1)).\hat{A}_{\bar{t}(t,1)}^{\lambda}=\sum_{l=0}^{\infty}(\gamma_{\text{WM}}\lambda)^{l}\bar{\delta}_{\bar{t}(t+l,1)},\quad\text{where }\bar{\delta}_{\bar{t}(t,1)}=\bar{R}_{\bar{t}(t,1)}+\gamma_{\text{WM}}V_{\nu}(\bar{s}_{\bar{t}(t+1,1)})-V_{\nu}(\bar{s}_{\bar{t}(t,1)}). (17)
ℒθ=ℒPPO−αBC​𝔼π¯θold​[∑k=1Klog⁡πθpre​(a¯tk−1∣zt,a¯tk)],\mathcal{L}_{\theta}=\mathcal{L}_{\text{PPO}}-\alpha_{\text{BC}}\ \mathbb{E}^{\bar{\pi}_{\theta_{\text{old}}}}\left[\sum_{k=1}^{K}\log\pi_{\theta_{\text{pre}}}(\bar{a}_{t}^{k-1}\mid z_{t},\bar{a}_{t}^{k})\right], (18)
ℒν=𝔼𝒟itr​[(∑l=0T−tγWMl​R¯t¯​(t+l,1)−Vν​(zt))2],\mathcal{L}_{\nu}=\mathbb{E}_{\mathcal{D}_{\text{itr}}}\left[\left(\sum_{l=0}^{T-t}\gamma_{\text{WM}}^{l}\bar{R}_{\bar{t}(t+l,1)}-V_{\nu}(z_{t})\right)^{2}\right], (19)
Table S.5: Fine-tuning hyperparameters shared across all skills for DiWA and baseline methods.
Parameter Symbol Value
Planning Horizon (Environment) TpT_{p} 4
Planning Horizon (Actor) TaT_{a} 4
Denoising Steps KK 20
Fine-tuned Denoising Steps K′K^{\prime} 10
Actor Learning Rate — 1×10−51\times 10^{-5}
Critic Learning Rate — 1×10−31\times 10^{-3}
Actor MLP Dimensions — [512, 512, 512]
Critic MLP Dimensions — [256, 256, 256]
Discount Factor (Env /World Model) γENV\gamma_{\text{ENV}} / γWM\gamma_{\text{WM}} 0.999
Discount Factor (Diffusion Policy) γDP\gamma_{\text{DP}} 0.99
GAE Smoothing Parameter λ\lambda 0.95
Behavior Cloning Coefficient (default) αBC\alpha_{\text{BC}} 0.05
Batch Size — 7500

Appendix S.2 Experimental Setup Details

Physical and simulation experiments share a seven-dimensional end-effector action interface. The physical observation/data pipeline is documented separately from the simulator’s privileged-state diagnostic variants.

S.2.1 7-DoF Action Framework

Actions contain three translation deltas, three Euler-angle deltas, and a gripper command, each normalized within ±1. The text says the environment uses +1 to close and −1 to open, but then says nonnegative policy outputs trigger opening and negative outputs closing. This apparent sign-convention mismatch is retained and flagged; deployment must verify the actual threshold mapping rather than copy a silently reconciled version.

Equation
[δ​x,δ​y,δ​z,δ​ϕ,δ​θ,δ​ψ,gripperAction][\delta x,\delta y,\delta z,\delta\phi,\delta\theta,\delta\psi,\textit{gripperAction}]

S.2.2 Real-World Data Collection

Panda teleoperation uses HTC VIVE with a tabletop cabinet/drawer and red cube. An Azure Kinect records static 200×200 views and a wrist RealSense D415 records close-ups, alongside robot state and absolute commands. Relative actions are consecutive command differences. Recording at thirty hertz is reduced to 7.5 hertz for training, so the approximately 450,000 transition headline should not be interpreted as a specified post-downsampling training count.

Refer to caption
(a) Real-World Setup
Refer to caption
(b) Real-World Observations
Figure S.1: (a) Real-world setup showing the Franka Panda robot, VR teleoperation interface (HTC VIVE controller and tracking system), and camera placements (static Kinect and wrist-mounted Realsense). (b) Example observations from the static and gripper-mounted RGB cameras used during data collection.

Appendix S.3 Data Preprocessing

Both cameras are resized to 64×64, converted to floating-point intensity, and normalized. Robot orientation preprocessing converts Euler angles to a continuous six-dimensional representation. These preprocessing details coexist with the paper’s vision-only main-model claim and privileged-state ablations; they do not imply every model receives the same state information.

Appendix S.4 Additional Experiments

Additional experiments examine online-input modalities, regularization, a Gaussian-policy variant, four LIBERO tasks, a failed offline-RL baseline, and qualitative physical-world predictions.

Refer to caption
Figure S.2: Comparison of DiWA with three DPPO variants using different input modalities. DiWA (blue) fine tunes policies entirely offline using a learned world model, requiring no physical interaction during adaptation. In contrast, DPPO (gray, red, green) performs online reinforcement learning with access to environment rewards and dynamics. The DPPO variant using latents from the world model encoder (green) achieves the highest performance among the three, but all require hundreds of thousands of real world interactions per skill.

S.4.1 Comparing DPPO Input Modalities

The frozen world-model representation often improves online DPPO over raw pixels or simulator-state inputs. Online variants can eventually match or exceed DiWA with additional ground-truth dynamics/reward interaction. DiWA’s horizontal band compares its offline-adapted result against that interaction axis; it does not imply no original data or no imagination computation. These are simulated baseline comparisons, not millions of measured hardware training steps.

Refer to caption
Figure S.3: Ablation of behavior cloning regularization strength (αBC\alpha_{\text{BC}}) during fine-tuning. Without regularization (αBC=0.0\alpha_{\text{BC}}=0.0), the agent performs well in imagination but fails in the real environment, indicating exploitation of world model inaccuracies. Excessively high values (e.g., 0.50.5) prevent meaningful adaptation. Intermediate values strike a balance, yielding robust transfer.

S.4.2 Impact of Behavior Cloning Regularization

Without cloning regularization, imagined success rises while true-environment success deteriorates, consistent with model exploitation. A large coefficient such as 0.5 prevents useful adaptation; intermediate values perform better. Figure S.3 therefore demonstrates a substantive dependency on regularization and evaluation outside the model, not an automatic transfer guarantee. Future online model correction could reduce persistent errors.

S.4.3 Fine-tuning a Unimodal Gaussian Policy

Replacing diffusion with a diagonal-covariance Gaussian policy still improves all eight CALVIN tasks. Drawer closing rises from 55.17% to 98.28%, and slider-right from 55.52% to 87.93%. The benefit is therefore not exclusive to diffusion; imagination and reward learning can adapt a simpler policy too. The table is a separate policy-class experiment, not evidence that diffusion uniformly outperforms Gaussian policies.

Table S.6: Offline fine-tuning improves a unimodal Gaussian policy across all tasks. Success rates increase substantially without any additional real-world interaction.
Task Gaussian Policy
Pre-Trained Offline Fine-Tuned
open-drawer 50.00±0.0950.00\pm 0.09 71.67±2.36\textbf{71.67}\pm\textbf{2.36}
close-drawer 55.17±0.1855.17\pm 0.18 98.28±2.44\textbf{98.28}\pm\textbf{2.44}
move-slider-left 54.86±4.7054.86\pm 4.70 82.64±1.59\textbf{82.64}\pm\textbf{1.59}
move-slider-right 55.52±0.7855.52\pm 0.78 87.93±7.31\textbf{87.93}\pm\textbf{7.31}
turn-on-lightbulb 54.55±3.0354.55\pm 3.03 95.96±1.75\textbf{95.96}\pm\textbf{1.75}
turn-off-lightbulb 62.07±4.8862.07\pm 4.88 77.59±2.44\textbf{77.59}\pm\textbf{2.44}
turn-on-LED 44.83±0.5044.83\pm 0.50 77.59±7.31\textbf{77.59}\pm\textbf{7.31}
turn-off-LED 40.94±3.9840.94\pm 3.98 79.69±2.21\textbf{79.69}\pm\textbf{2.21}
Total Physical Interactions: 0

S.4.4 Results on LIBERO-90

A model trained on LIBERO-90 demonstrations is evaluated on four kitchen skills across four scenes. Three-seed mean success improves from 40.67% to 77.33% for opening a top drawer, 54% to 91.33% for turning on a stove, 27.33% to 78% for closing a bottom drawer, and 75.33% to 100% for closing a top drawer. Imagined horizons are task-selected at three, two, one, and three million steps to avoid exploitation. This supports transfer across selected scene/task cases, not evaluation of all ninety tasks.

Table S.7: DiWA improves performance on four LIBERO-90 kitchen tasks, with results averaged over three random seeds.
Task Base DiWA (Ours)
Diffusion Policy Offline Fine-Tuning
Success Rate
open-top-drawer 40.67±3.0640.67\pm 3.06 77.33±3.06\textbf{77.33}\pm\textbf{3.06}
turn-on-stove 54.00±7.2154.00\pm 7.21 91.33±3.08\textbf{91.33}\pm\textbf{3.08}
close-bottom-drawer 27.33±3.1227.33\pm 3.12 78.00±8.72\textbf{78.00}\pm\textbf{8.72}
close-top-drawer 75.33±2.3175.33\pm 2.31 100.00±0.00\textbf{100.00}\pm\textbf{0.00}
Total Physical Interactions: 0

S.4.5 Offline RL Limitations in Our Setting

The attempted CQL baseline used play relabeled by the lower-precision image reward classifier and heuristic pseudo-episodes, then diverged. Reward noise, coverage, and value extrapolation are offered as explanations. This does not establish that all critic-based offline RL is inherently incompatible: the reward representation differs from DiWA’s stronger latent classifier, and no matched successful baseline is reported. DiWA itself also trains a value critic and remains vulnerable to model/reward error.

S.4.6 World Model Rollouts in the Real World

Two held-out frames initialize the model, which predicts eighty latent steps despite training on fifty-step sequences. Decoded static/wrist views qualitatively preserve robot and object structure. These examples illustrate temporal coherence; no quantitative eighty-step dynamics-error bound or guarantee of contact/control fidelity is supplied.

Refer to caption
Figure S.4: Real-world rollout predictions from the learned world model. Each block shows a segment of a held-out trajectory for a specific skill, with static and gripper camera views decoded from imagined latent states. The model produces accurate long-horizon predictions in real-world settings.
Concept demonstration

Open image ↗