Literature
Display
← All papers

Flexible Multitask Learning with Factorized Diffusion Policy

Chaoqi Liu · Haonan Chen · Sigmund H. Høeg · Shaoxiong Yao · Yunzhu Li · Kris Hauser · Yilun Du

· IEEE Robotics and Automation Letters ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

FDP learns several diffusion-policy components jointly and combines their denoising predictions using observation-dependent weights. The aim is to model varied robot behaviors and adapt to new tasks by tuning a small subset of parameters or adding a component.

Simulation and four physical manipulation tasks support improved average performance and efficient adaptation. Freezing old components alone does not prevent forgetting: the encoder and router can still change, and a small replay buffer substantially improves retention.

Introduction

A single policy can struggle to represent varied manipulation behaviors. FDP distributes modeling across diffusion components and continuously combines their outputs, instead of selecting a few experts through discrete routing. New tasks can reuse old components and add a copied, trainable component.

The experiments study multitask success, adaptation, scaling, and component behavior. The evidence supports a useful modular architecture; it does not establish fully independent, uniquely identifiable skills.

Refer to caption
Fig. 1: Overview of FDP. (a) Given an observation 𝐨t\mathbf{o}_{t}, multiple diffusion experts predict score estimates 𝜺i​(𝐚tK,𝐨t)\bm{\varepsilon}_{i}(\mathbf{a}_{t}^{K},\mathbf{o}_{t}) at each denoising step. A lightweight router network computes observation-dependent weights {wi}\{w_{i}\}, which are used to compose the final score as a weighted sum (see (c)). The composed score guides the iterative denoising process over KK steps to generate an action 𝐚t\mathbf{a}_{t}. (b) This compositional structure enables FDP to model complex multimodal distributions and supports modular adaptation via selective tuning or addition of diffusion components.

Related Works

The design connects diffusion-based action generation with modular multitask policies.

II- 1 Diffusion Models for Robotics

Diffusion models generate actions or trajectories by repeatedly denoising a sample. Prior work applies them to visuomotor control, planning, grasping, and bimanual manipulation. FDP builds on Diffusion Policy and compositional diffusion modeling. [10]

II- 2 Multitask Imitation Learning and Adaptation

SDP routes among experts using observations, while MoDE routes according to diffusion noise level. FDP combines complete diffusion components continuously and trains every component on each update. This avoids hard selection in its training design; qualitative analyses examine whether meaningful behavior specialization emerges.

FDP : Factorized Diffusion Policy

The policy factorizes an observation-conditioned action distribution into weighted component distributions. A learned router determines how strongly each component contributes at each observation.

Probabilistic Policy Modeling

Multiplying powered component densities favors actions compatible with their combined preferences. Each component is represented through a diffusion denoiser. The product-of-distributions view motivates interpreting components as soft behavioral constraints, rather than as an exclusive choice between skills.

Equations 1
p⁡(𝐚t|𝐨t)∝∏ipi​(𝐚t|𝐨t)wt,i,p(\mathbf{a}_{t}|\mathbf{o}_{t})\propto\prod_{i}p_{i}(\mathbf{a}_{t}|\mathbf{o}_{t})^{w_{t,i}},
𝐚t,ik−1=αk​(𝐚t,ik−γk​𝜺θi​(𝐚t,ik,𝐨t,k)+𝒩⁡(𝟎,σk2​𝐈)),\mathbf{a}^{k-1}_{t,i}=\alpha_{k}\left(\mathbf{a}^{k}_{t,i}-\gamma_{k}\,\bm{\varepsilon}_{\theta_{i}}(\mathbf{a}^{k}_{t,i},\mathbf{o}_{t},k)+\mathcal{N}(\mathbf{0},\sigma_{k}^{2}\mathbf{I})\right),
ℒMSE=‖ϵk−𝜺θ​(𝐚t,i0+ϵk,𝐨t,k)‖22,\mathcal{L}_{\text{MSE}}=\|\bm{\epsilon}^{k}-\bm{\varepsilon}_{\theta}(\mathbf{a}^{0}_{t,i}+\bm{\epsilon}^{k},\mathbf{o}_{t},k)\|^{2}_{2}, (1)

Compositional Sampling and Routing

The sampler sums component noise predictions with router weights at each denoising step. A single reconstruction loss trains the components and an MLP router jointly. Every component receives gradients; the model does not require separately labeled demonstrations for each component. The original training and inference algorithms are retained.

Equations 2
𝐚tk−1=𝐚tk−γk​∑iwt,i​∇𝐚tkEi​(𝐚tk,𝐨t)+ξk,\mathbf{a}^{k-1}_{t}=\mathbf{a}^{k}_{t}-\gamma_{k}\sum_{i}w_{t,i}\,\nabla_{\mathbf{a}^{k}_{t}}E_{i}(\mathbf{a}^{k}_{t},\mathbf{o}_{t})+\xi_{k}, (2)
𝐚tk−1=𝐚tk−γk​∑iwt,i​𝜺θi​(𝐚tk,𝐨t,k)+ξk.\mathbf{a}^{k-1}_{t}=\mathbf{a}^{k}_{t}-\gamma_{k}\sum_{i}w_{t,i}\,\bm{\varepsilon}_{\theta_{i}}(\mathbf{a}^{k}_{t},\mathbf{o}_{t},k)+\xi_{k}.
ℒMSE=‖ϵk−∑iwt,i​𝜺θi​(𝐚t0+ϵk,𝐨t,k)‖22,\mathcal{L}_{\text{MSE}}=\|\bm{\epsilon}^{k}-\sum_{i}w_{t,i}\,\bm{\varepsilon}_{\theta_{i}}(\mathbf{a}^{0}_{t}+\bm{\epsilon}^{k},\mathbf{o}_{t},k)\|^{2}_{2},
Algorithm 1 FDP Training
0:  Dataset 𝒟\mathcal{D}, Denoisers {𝜺θi}\{\bm{\varepsilon}_{\theta_{i}}\}, Routerψ
1:  while not converged do
2:    Sample (𝐚,𝐨)∼𝒟(\mathbf{a},\mathbf{o})\sim\mathcal{D} and noise ϵk\bm{\epsilon}^{k}
3:    {wi}←\{w_{i}\}\leftarrow Router(𝐨)ψ{}_{\psi}(\mathbf{o})
4:    ℒ←‖ϵk−∑iwi​𝜺θi​(𝐚+ϵk,𝐨,k)‖22\mathcal{L}\leftarrow\left\|\bm{\epsilon}^{k}-\sum_{i}w_{i}\,\bm{\varepsilon}_{\theta_{i}}(\mathbf{a}+\bm{\epsilon}^{k},\mathbf{o},k)\right\|^{2}_{2}
5:    ∀i,θi←θi+∇θiℒ\forall i,\theta_{i}\leftarrow\theta_{i}+\nabla_{\theta_{i}}\mathcal{L}
6:    ψ←ψ+∇ψℒ\psi\leftarrow\psi+\nabla_{\psi}\mathcal{L}
7:  end while
8:  return {𝜺θi}\{\bm{\varepsilon}_{\theta_{i}}\}
Algorithm 2 FDP Inference
0:  Denoisers {𝜺i}\{\bm{\varepsilon}_{i}\}, Router, Observation 𝐨t\mathbf{o}_{t}
1:  {wt,i}←\{w_{t,i}\}\leftarrow Router(𝐨t)(\mathbf{o}_{t})
2:  𝐚tK←𝒩⁡(𝟎,𝐈)\mathbf{a}_{t}^{K}\leftarrow\mathcal{N}(\mathbf{0},\mathbf{I})
3:  for k←K,K−1,…,1k\leftarrow K,K-1,...,1 do
4:    ∇𝐚k←∑iwt,i​𝜺i​(𝐚tk,𝐨t,k)\nabla\mathbf{a}^{k}\leftarrow\sum_{i}w_{t,i}\,\bm{\varepsilon}_{i}(\mathbf{a}_{t}^{k},\mathbf{o}_{t},k)
5:    𝐚tk−1←𝐚tk−γk∇𝐚k+𝒩(𝟎,σk𝐈)\mathbf{a}_{t}^{k-1}\leftarrow\mathbf{a}_{t}^{k}-\gamma_{k}\nabla\mathbf{a}^{k}+\mathcal{N}(\mathbf{0},\sigma_{k}\mathbf{I})
6:  end for
7:  𝐚t←𝐚t0\mathbf{a}_{t}\leftarrow\mathbf{a}_{t}^{0}
8:  return 𝐚t\mathbf{a}_{t}

Multitask Learning and Adaptation

The same composition supports initial multitask training and several forms of adaptation.

1 Multitask Learning

Joint training lets components share the burden of fitting varied observation–action patterns. Their roles are learned from the common demonstration loss, rather than prescribed as distinct named skills.

2 Adapting to New Tasks

A new component is initialized by copying an existing one, then trained with an updated router while old components remain frozen. The experiments also vary encoder and router tuning. Preserving component weights does not guarantee preservation of the complete policy: changes in routing or perception can still degrade old tasks.

Equation
𝜺adapt​(𝐚tk,𝐨t,k)=∑iwi​𝜺θi​(𝐚tk,𝐨t,k)+wnew​𝜺θnew​(𝐚tk,𝐨t,k),\bm{\varepsilon}_{\text{adapt}}(\mathbf{a}_{t}^{k},\mathbf{o}_{t},k)=\sum_{i}w_{i}\,\bm{\varepsilon}_{\theta_{i}}(\mathbf{a}_{t}^{k},\mathbf{o}_{t},k)+w_{\text{new}}\,\bm{\varepsilon}_{\theta_{\text{new}}}(\mathbf{a}_{t}^{k},\mathbf{o}_{t},k),

Experiments

Experiments compare multitask performance and new-task adaptation against DP, SDP, and MoDE, using simulations and physical robot demonstrations.

Experiments Setup

Simulation tasks come from MetaWorld, RLBench, and LIBERO. Physical tests use a UR5e arm, Robotiq gripper, and RealSense D415 camera for placing colored cubes into a bowl and hanging a mug at different heights. MetaWorld and RLBench goals are implicit in scenes; LIBERO and physical tasks use discrete task indices.

Refer to caption
((a)) Real-world workspace.
Refer to caption
((b)) hang-X
Refer to caption
((c)) cube-X
Fig. 2: Real-world setup and task illustrations. (a) Workspace setup with a UR5e arm, Robotiq gripper, and RealSense D415 camera. (b) High-level task illustrations.
Refer to caption
Refer to caption
Fig. 3: Real-world rollouts. Top: cube-X. Bottom: hang-X. Top and bottom rows show success cases and baseline failure modes.
MetaWorld
Door Drawer Window Peg
Policy Open Open Assembly Close Insert Hammer Avg.
DP 87.0 ± 1.92 100.0 ± 0.00 100.0 ± 0.00 94.0 ± 0.89 20.0 ± 1.22 24.0 ± 0.89 70.8 ± 0.24
SDP 80.0 ± 1.87 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 20.5 ± 0.84 18.0 ± 1.64 69.8 ± 0.51
MoDE 100.0 ± 0.00 100.0 ± 0.00 94.0 ± 2.51 100.0 ± 0.00 19.5 ± 2.77 23.5 ± 0.89 72.8 ± 0.76
FDP 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 100.0 ± 0.00 26.5 ± 2.19 22.0 ± 2.39 74.8 ± 0.67
RLBench
Close Close Close Close Toilet Seat Take Reach Pick &
Policy Box Drawer Fridge Microwave Down Umbrella Target Lift Avg.
DP 33.5 ± 1.52 100.0 ± 0.00 77.0 ± 1.64 65.5 ± 1.48 64.0 ± 2.19 19.0 ± 1.95 3.5 ± 0.89 2.0 ± 0.84 45.6 ± 0.43
SDP 24.0 ± 1.34 100.0 ± 0.00 68.0 ± 3.11 42.5 ± 2.00 58.5 ± 2.88 6.5 ± 1.14 36.0 ± 2.61 1.0 ± 0.55 42.1 ± 0.70
MoDE 31.0 ± 2.88 97.0 ± 2.17 70.5 ± 1.64 50.5 ± 1.10 60.0 ± 1.73 10.5 ± 1.92 37.0 ± 1.10 3.5 ± 1.95 45.0 ± 0.87
FDP 72.0 ± 1.30 100.0 ± 0.00 75.5 ± 1.10 93.0 ± 1.92 27.5 ± 2.12 78.0 ± 2.17 61.0 ± 1.52 4.5 ± 2.39 63.9 ± 0.23

TABLE I: Multitask learning evaluation on MetaWorld and RLBench. Results report the mean and standard error over 5 seeds, with 40 samples evaluated per seed.
MetaWorld RLBench
Door Drawer Window Open Open Open Open
Method Policy Close Close Disassemble Open Avg. Box Drawer Fridge Microwave Avg.
Full Parameter DP 100.0 ± 0.00 100.0 ± 0.00 60.5 ± 1.10 100.0 ± 0.00 90.1 ± 0.27 75.5 ± 1.30 81.0 ± 0.55 64.0 ± 2.07 63.5 ± 1.52 71.0 ± 0.76
SDP 89.5 ± 2.05 100.0 ± 0.00 58.0 ± 2.17 100.0 ± 0.00 86.9 ± 0.31 67.0 ± 0.84 88.5 ± 2.88 58.5 ± 2.61 66.0 ± 1.52 70.0 ± 0.90
MoDE 100.0 ± 0.00 100.0 ± 0.00 66.5 ± 2.97 100.0 ± 0.00 91.6 ± 0.74 60.5 ± 0.45 88.0 ± 1.92 71.0 ± 0.89 75.0 ± 1.22 73.6 ± 0.48
FDP 100.0 ± 0.00 100.0 ± 0.00 62.0 ± 2.17 100.0 ± 0.00 90.5 ± 0.54 87.0 ± 1.92 72.5 ± 3.39 65.0 ± 1.87 79.0 ± 1.67 75.9 ± 0.91
Router SDP 72.0 ± 1.30 0.0 ± 0.00 1.0 ± 0.55 4.0 ± 1.52 19.3 ± 0.67 4.5 ± 1.92 23.5 ± 1.95 11.5 ± 1.14 15.5 ± 2.49 13.8 ± 0.40
MoDE 4.0 ± 2.30 1.0 ± 0.89 1.5 ± 1.34 24.0 ± 1.67 7.6 ± 1.08 2.5 ± 1.41 4.0 ± 1.95 19.5 ± 0.84 1.5 ± 1.34 6.9 ± 0.71
FDP 85.5 ± 1.48 64.5 ± 2.28 4.0 ± 2.19 5.5 ± 0.84 39.9 ± 0.67 17.5 ± 1.41 1.5 ± 1.34 17.5 ± 1.73 0.0 ± 0.00 9.1 ± 0.82
+ Observation Encoder SDP 100.0 ± 0.00 100.0 ± 0.00 31.0 ± 0.89 100.0 ± 0.00 82.8 ± 0.22 25.5 ± 0.84 48.0 ± 2.17 41.0 ± 1.14 22.5 ± 1.87 34.3 ± 0.45
MoDE 93.0 ± 0.45 100.0 ± 0.00 40.0 ± 1.58 100.0 ± 0.00 83.2 ± 0.37 19.0 ± 1.34 65.5 ± 1.10 29.5 ± 1.30 25.0 ± 1.00 34.8 ± 0.91
FDP 100.0 ± 0.00 93.0 ± 2.68 52.5 ± 1.87 100.0 ± 0.00 86.4 ± 0.99 30.0 ± 3.08 57.0 ± 1.48 47.5 ± 2.55 18.5 ± 1.82 38.3 ± 1.19
+ New Module SDP 100.0 ± 0.00 100.0 ± 0.00 60.5 ± 1.79 100.0 ± 0.00 90.1 ± 0.45 69.0 ± 1.14 23.5 ± 2.88 40.5 ± 1.92 22.5 ± 1.00 38.9 ± 0.89
MoDE 100.0 ± 0.00 100.0 ± 0.00 61.5 ± 2.30 100.0 ± 0.00 90.4 ± 0.58 58.0 ± 1.92 54.0 ± 1.14 58.0 ± 1.79 53.0 ± 1.92 55.8 ± 0.78
FDP 100.0 ± 0.00 100.0 ± 0.00 68.5 ± 1.52 100.0 ± 0.00 92.1 ± 0.38 86.5 ± 1.34 77.0 ± 1.10 78.0 ± 2.28 77.5 ± 3.08 79.8 ± 0.91
TABLE II: Adaptation evaluation on MetaWorld and RLBench. We pretrain policies on tasks shown in Table I, then adapt them to these successive tasks. We report mean and standard error over 5 seeds, with 60 MetaWorld samples and 40 RLBench samples per seed.

Implementations

Policies receive RGB images and joint angles, use two observations of history, predict 16 joint-angle steps, and execute eight before replanning. FDP uses four U-Net diffusion modules. Baselines follow their original designs, with MoDE reduced in size for comparison; new components copy existing weights.

Multitask Learning

The first comparison learns several tasks together from demonstrations.

1 Simulation

Training uses 25 demonstrations per task for six MetaWorld tasks and 50 for eight RLBench tasks. Evaluation reports 40 rollouts per task per seed across five seeds. Table I gives average success of 74.8% versus 70.8% for DP on MetaWorld and 63.9% versus 45.6% on RLBench. Improvements vary by task, with some baselines matching or exceeding FDP on individual tasks.

Policy Cube Red Hang Low
DP 14 / 20 16 / 20
SDP 15 / 20 13 / 20
MoDE 14 / 20 16 / 20
FDP 15 / 20 17 / 20

TABLE III: Real-world multitask success rates. Average over 20 trials. Tasks: cube red and hang low.

2 Real-world

Physical multitask training uses 300 red-cube and 200 low-mug demonstrations. Across 20 trials per task, FDP succeeds 15 and 17 times, compared with DP’s 14 and 16. These small trial counts show a modest advantage in this setup; selected rollouts illustrate baseline placement errors.

Task Transfer and Adaptation

Adaptation compares full fine-tuning with router-only tuning, encoder-plus-router tuning, and adding a new component. These partial FDP settings update approximately 0.5%, 11%, and 27% of parameters respectively.

Method Policy Cube Blue Hang High
Full Param. DP 15 / 20 17 / 20
SDP 14 / 20 16 / 20
MoDE 15 / 20 16 / 20
FDP 17 / 20 15 / 20
Router + Obs. Enc. SDP 10 / 20 9 / 20
MoDE 10 / 20 11 / 20
FDP 11 / 20 11 / 20
+ New Module SDP 13 / 20 11 / 20
MoDE 14 / 20 13 / 20
FDP 17 / 20 17 / 20

TABLE IV: Adaptation in real-world. Evaluated on cube blue and hang high. Pretrained on cube red and hang low.

1 Simulation

Four new tasks per simulation benchmark use ten demonstrations each in MetaWorld and 25 in RLBench. Evaluation uses 60 and 40 rollouts per seed respectively. Adding a component yields FDP averages of 92.1% and 79.8%, versus 90.5% and 75.9% with full fine-tuning. Router-only adaptation is often insufficient.

2 Real-world

Blue-cube and high-mug adaptation each use 100 demonstrations and 20 evaluation trials. Adding a component reaches 17/20 on both tasks. Full FDP fine-tuning reaches 17/20 and 15/20; full DP reaches 15/20 and 17/20. Modular adaptation therefore works well here without dominating every full-tuning result.

Original figure
Fig. 4: Relative success rate improvement of FDP over DP. FDP ’s advantage increases as the number of tasks grows. Selected from RLBench. We report mean and standard error over 5 seeds.
# Comp MetaWorld RLBench
2 86.7 ± 0.89 54.4 ± 1.22
3 90.0 ± 0.71 58.8 ± 0.84
4 91.3 ± 0.45 63.9 ± 0.23
5 91.3 ± 0.89 64.4 ± 0.71
6 91.7 ± 1.14 65.0 ± 0.45
7 91.9 ± 0.55 65.6 ± 0.84
TABLE V: Multitask performance of FDP with different numbers of components. Performance improves up to 4 components and plateaus thereafter. We report mean and standard error over 5 seeds.

Analysis

Further experiments examine capacity, data, inference cost, continual adaptation, retention, and the interpretation of components.

Scaling of Number of Diffusion Components

Increasing the component count from two to four improves the selected task sets, with diminishing gains beyond four. The authors find four to six components useful for these experiments. Computation grows approximately linearly with component count; pruning offers an explicit speed–performance trade-off.

Policy Close Box Close Drawer Close Fridge Close Microwave Toilet Seat Down Take Umbrella Reach Target Pick & Lift Avg.
DP 33.5 ± 1.52 100.0 ± 0.00 77.0 ± 1.64 65.5 ± 1.48 64.0 ± 2.19 19.0 ± 1.95 3.5 ± 0.89 2.0 ± 0.84 45.6 ± 0.43
FDP 72.0 ± 1.30 100.0 ± 0.00 75.5 ± 1.10 93.0 ± 1.92 27.5 ± 2.12 78.0 ± 2.17 61.0 ± 1.52 4.5 ± 2.39 63.9 ± 0.23
FDPtop2{}_{\text{top2}} 63.5 ± 1.14 95.5 ± 1.64 69.0 ± 1.00 90.5 ± 0.71 22.5 ± 2.70 30.5 ± 2.70 41.5 ± 1.30 1.0 ± 0.55 51.8 ± 0.27
TABLE VI: Partial reconstruction on RLBench. FDPtop2{}_{\text{top2}} uses only top-2 components, achieving a 2×\times speedup in inference time with only 19% relative performance drop. Results report the mean and standard error over 5 seeds, with 40 samples evaluated per seed.
Multitask 4 Experts + Expert Modules (5–12)
Policy
L1 PnP
soup
L2 PnP
cheese
K2 open
top drawer
K3 PnP
pot
L2 PnP
soup
L5 PnP
mug
L6 PnP
mug
K3 turn
on stove
K4 PnP
bowl
K6 PnP
mug
K8 PnP
pot
S1 PnP
book
Avg.
SDP 57.2 ± 1.56 22.4 ± 1.04 100.0 ± 0.00 50.8 ± 1.34 30.4 ± 0.67 56.0 ± 1.50 46.4 ± 0.88 94.8 ± 2.63 56.0 ± 1.13 22.4 ± 0.36 25.6 ± 1.73 35.6 ± 1.43 49.8 ± 0.44
MoDE 65.2 ± 1.84 33.6 ± 0.36 94.8 ± 0.72 45.6 ± 1.91 30.8 ± 0.91 63.6 ± 0.67 41.6 ± 1.91 100.0 ± 0.00 58.8 ± 1.21 35.6 ± 1.54 42.4 ± 1.54 50.8 ± 1.21 55.2 ± 0.30
FDP 83.2 ± 0.91 34.4 ± 1.54 100.0 ± 0.00 60.8 ± 2.30 40.0 ± 0.57 89.6 ± 0.88 56.8 ± 1.75 100.0 ± 0.00 82.0 ± 1.26 40.8 ± 2.50 64.8 ± 1.21 55.6 ± 1.54 67.3 ± 0.39
TABLE VII: Continual adaptation on LIBERO. We pretrain 4-expert FDP on first 4 tasks, and progressively add new experts for each additional adaptation task, ultimately reaching 12 experts. Results report the mean and standard error over 5 seeds, with 50 samples evaluated per seed. PnP stands for pick & place.

Scaling of Number of Tasks

On the selected RLBench task combinations, FDP’s relative advantage over DP grows as more tasks are included. This supports its intended use for varied task distributions within the tested range.

Scaling of Number of Demonstrations

The data-scaling plots distinguish joint learning from adaptation.

((a)) Multitask Learning
Original figure
((b)) Task Adaptation
Original figure
Fig. 5: Performance scaling with number of demonstrations. (a) Metaworld tasks door open, drawer open, assembly, window close, peg insert, hammer; RLBench tasks door open, drawer open, assembly, window close, peg insert, hammer. (b) Metaworld tasks door close, drawer close, disassemble, window open; RLBench tasks toilet seat down, close box

1 Multitask Learning

More demonstrations generally improve multitask success. The reported comparisons favor FDP, particularly on the tested RLBench tasks; this is an empirical scaling observation rather than a universal sample-complexity guarantee.

2 Task Adaptation

Both full fine-tuning and adding a component improve with more adaptation data. On the selected RLBench tasks, component expansion can match or improve full tuning with fewer demonstrations.

Partial Reconstruction of Action Distribution

Keeping only the two highest-weight components at inference reduces mean RLBench success from 63.9% to 51.8%, a roughly 19% relative decrease. Table VI reports a twofold inference speedup without retraining. The table values and caption support 19%; the accompanying source prose says 17%.

Continual Adaptation

A LIBERO sequence starts with four components and four tasks, then adds one component for each of eight new tasks, ending with twelve. Across five seeds and 50 evaluations per seed, the reported overall average is 67.3% for FDP, 55.2% for MoDE, and 49.8% for SDP.

Knowledge Retention

The separate retention experiment reports 57.6% on four base tasks before adaptation, 29.3% after adaptation without replay, and 53.5% with five replay demonstrations per base task. These results show that frozen experts alone do not prevent forgetting. Frozen visual features and cached routers are suggested remedies; replay is the tested intervention.

Policy
L1 PnP
soup
L2 PnP
cheese
K2 open
top drawer
K3 PnP
pot
Avg.
FDPpretrain{}_{\text{pretrain}} 57.2 ± 1.56 22.4 ± 1.04 100.0 ± 0.00 50.8 ± 1.34 57.6 ± 0.61
FDPw/o buffer{}_{\text{w/o buffer}} 32.5 ± 1.00 5.0 ± 1.22 50.5 ± 1.10 29.0 ± 1.14 29.3 ± 0.45
FDPw/ buffer{}_{\text{w/ buffer}} 53.0 ± 1.64 17.5 ± 1.00 96.0 ± 0.89 47.5 ± 1.22 53.5 ± 0.63
TABLE VIII: Knowledge Retention Analysis on LIBERO. We report success rates on four pretraining tasks after adapting policies to two new tasks (L2 PnP soup and L5 PnP mug). We compare: (i) FDPpretrain{}_{\text{pretrain}}: the original model before adaptation; (ii) FDPw/o buffer{}_{\text{w/o buffer}}: the model after adaptation using only new task data; and (iii) FDPw/ buffer{}_{\text{w/ buffer}}: the model after adaptation using a small replay buffer (5 demos/task) to mitigate forgetting. Results show mean and standard error over 5 seeds, 50 evaluations per seed. PnP stands for pick & place.

Diffusion Components Analysis

Assembly and hammer rollouts associate components with approach, alignment, and grasping phases. Pairwise score similarities show distinct but overlapping behavior, rather than orthogonal components. These observations suggest specialization but do not isolate causal roles through component removal. The source uses “Fig. 5” for both the demonstration-scaling and rollout panels; both panels are preserved.

((a)) Assembly
Refer to caption
Original figure
((b)) Hammer
Refer to caption
Original figure
Fig. 5: Rollout trajectories of individual diffusion components in FDP. (a) In assembly, components 0 and 1 align the robot with the stand, component 2 aligns with the ring, and component 3 executes the grasp. (b) In hammer, components 0 and 1 align and approach the pin, component 2 approaches the hammer, and component 3 performs the grasp.
Refer to caption
Fig. 6: Cosine similarity between diffusion component scores. Each heatmap visualizes average pairwise similarity for independent FDP instances configured with 2-5 components, computed over four RLBench tasks. Lower similarity indicates more distinct behavioral specialization. Note that subplots represent separate training runs rather than a single evolving model.
Original figure
Fig. 7: Training convergence curves. Mean squared error (MSE) loss over training epochs for RLBench and MetaWorld tasks. FDP consistently converges faster and more stably than MoDE and SDP, indicating improved training efficiency and optimization stability.

Training Convergence of Policies

Validation denoising losses decrease faster and more steadily for FDP than the tested SDP and MoDE implementations. This measures optimization behavior on the reported datasets, separately from deployment success or wall-clock efficiency.

Conclusion And Limitations

Continuous composition of diffusion components improves several multitask and adaptation comparisons while supporting selective expansion. Current modules share architecture and size, specialization is primarily analyzed through visualizations, and retention still benefits from replay. Heterogeneous modules, component-removal studies, and longer continual-learning sequences remain future work.

Concept demonstration

Open image ↗