Literature
Display
← All papers

Residual Denoising Enables Sample-Efficient Multi-Agent Coordination on Demand

Dayi Dong · Maulik Bhatt · Aayushi Shrivastava · Lasse Peters · Negar Mehr

Preprint posted · arXiv (preprint) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

ALTER adapts a frozen single-arm diffusion policy for two-robot collaboration while trying to preserve solo behavior. This edition retains all four original figures, four tables and nine numbered equations from arXiv v2. The opening framework was restored from the PDF. Results should be read jointly: coordination success measures the new behavior, while source-task success tests whether the same active adapter still permits the old skills.

Original ALTER framework from PDF Figure 1
Fig. 1: Multi-agent adaptation with ALTER. ALTER adapts a frozen single-arm visual diffusion policy to coordinated multi-agent tasks without access to the original expert demonstrations. Blue denotes the frozen source policy and its policy-distilled rollouts, while green denotes the multi-agent demonstrations and trainable coordination head. Each robot runs the adapted policy using only its own visual observations, enabling decentralized coordination without explicit inter-agent communication while retaining the original single-arm skills.

At a glance

A residual denoising head learns partner-dependent coordination from a small collaborative dataset. Single-agent rollouts from the frozen base policy supply replay without access to its original expert demonstrations. All robots share weights but act from their own camera views, without explicit communication or a solo/team label. On the tested simulation task, ALTER improves collaboration and retains roughly 97% combined source success. In the hardware task it ties mixed-data fine-tuning at 18/20 collaborative successes while preserving more source skills.

I Introduction

Pretrained manipulation policies provide useful component skills, but collaborative demonstrations are costly and adaptation can erase solo competence. ALTER first rolls out the base policy in its source environment, then combines those recorded trajectories with a limited set of team demonstrations to train a correction head. At deployment, the head remains active both alone and in a team. The no-source-data premise concerns the original expert dataset: the method still needs source-environment access to generate replay and visual cues that distinguish solo from collaborative situations.

II Related Work

Diffusion Policy supplies the action-chunk formulation. Multi-agent imitation methods learn collaboration directly, while CoDi guides pretrained models with a specified coordination cost. ALTER instead learns a denoiser correction from demonstrations and rehearses base behavior through generated rollouts. Its additive frozen-backbone design is related to side-tuning and ControlNet. Unlike a runtime expert selector, it receives no external instruction to switch between solo and team policies.

III Preliminaries: Diffusion Models

The method uses the EDM continuous-noise formulation. A denoiser predicts clean action data from Gaussian-corrupted samples, with noise-dependent weighting. Tweedie’s relation converts its output into an estimated score, and the probability-flow differential equation transports a high-noise sample toward an action chunk. The paper keeps the base noise schedule and numerical solver during adaptation; the learned correction changes the vector field evaluated by that solver.

Equations 1–3
d​xσd​σ=−σ​s​(xσ,σ,c)\frac{\mathrm{d}x_{\sigma}}{\mathrm{d}\sigma}=-\sigma\,s(x_{\sigma},\sigma,c) (1)
ℒden​(θ,𝒟)=𝔼𝒟,σ,ϵ​[w⁡(σ)​‖Dθ​(x+σ​ϵ,σ,c)−x‖22],\mathcal{L}_{\mathrm{den}}(\theta;\mathcal{D})=\mathbb{E}_{\mathcal{D},\,\sigma,\,\epsilon}\left[w(\sigma)\left\|D_{\theta}(x+\sigma\epsilon,\sigma,c)-x\right\|_{2}^{2}\right], (2)
D⋆​(xσ,σ,c)=xσ+σ2​s​(xσ,σ,c).D^{\star}(x_{\sigma},\sigma,c)=x_{\sigma}+\sigma^{2}s(x_{\sigma},\sigma,c). (3)

IV Problem Formulation

The formal task is a decentralized partially observable process with homogeneous agents, local observation kernels and coupled physical dynamics. Shared policy weights do not provide shared observations: the joint action distribution factors into each robot’s local policy. Each predicts an action chunk, executes a prefix and replans. Collaborative demonstrations record joint trajectories, but training pools separate per-agent observation/action samples. A usable adapter must coordinate when needed while retaining source tasks without a solo/team mode label or the base policy’s original demonstrations. Limited team data and missing expert replay motivate frozen skills plus policy-generated rehearsal.

Equations 4–6
π1:N(a1:N∣o1:N):=∏i=1Nπi(ai∣oi)=∏i=1Nπ(ai∣oi).\pi^{1:N}(a^{1:N}\mid o^{1:N}):=\prod_{i=1}^{N}\pi^{i}(a^{i}\mid o^{i})=\prod_{i=1}^{N}\pi(a^{i}\mid o^{i}). (4)
oti∼Zi(⋅∣st),ati∼π(⋅∣oti),st+1∼P(⋅∣st,at1:N).o^{i}_{t}\sim Z^{i}(\cdot\mid s_{t}),\;a^{i}_{t}\sim\pi(\cdot\mid o^{i}_{t}),\;s_{t+1}\sim P(\cdot\mid s_{t},a^{1:N}_{t}). (5)
𝒟C={τmC}m=1MC,τmC={(ot1:N,at1:N)}t=0T−1,\mathcal{D}_{C}=\left\{\tau^{C}_{m}\right\}_{m=1}^{M_{C}},\qquad\tau^{C}_{m}=\left\{\left(o^{1:N}_{t},a^{1:N}_{t}\right)\right\}_{t=0}^{T-1}, (6)

V ALTER

Three choices define ALTER: freeze the pretrained denoiser, add a trainable residual head, and train on both collaborative demonstrations and base-policy rollouts. The same parameters serve solo and collaborative deployment.

Original figure
Fig. 2: The residual as a correction to the base vector field. Left: the frozen base field of DbaseD_{\mathrm{base}} (blue) alone yields the action distribution of πbase\pi_{\mathrm{base}}. Right: adding the residual field Δϕ\Delta_{\phi} (red) at every noise level bends the same trajectories toward the coordinated action distribution.

V-A Residual Coordination Adapter

The adapter receives the noisy action chunk, noise level, local observation and base denoiser prediction. Adding its output to the base prediction adds the residual divided by noise variance to the score, and adds minus residual divided by noise level to the probability-flow drift. This changes the sampled distribution throughout denoising rather than adding an offset to an already sampled action. A zero residual exactly recovers the base sampler; zero-initializing the output path therefore starts adaptation from the base policy. That identity is conditional on a zero residual, not a guarantee that finite training preserves it everywhere.

Equations 7–8
Dϕ​(xσ,σ,o)=Dbase​(xσ,σ,o)+Δϕ​(xσ,σ,o,Dbase​(xσ,σ,o)).\begin{split}D_{\phi}(x_{\sigma},\sigma,o)=&D_{\mathrm{base}}(x_{\sigma},\sigma,o)\\ &\quad+\Delta_{\phi}\left(x_{\sigma},\sigma,o,D_{\mathrm{base}}(x_{\sigma},\sigma,o)\right).\end{split} (7)
sDϕ​(xσ,σ,o)=sDbase​(xσ,σ,o)+σ−2​Δϕ​(xσ,σ,o),s_{D_{\phi}}(x_{\sigma},\sigma,o)=s_{D_{\mathrm{base}}}(x_{\sigma},\sigma,o)+\sigma^{-2}\,\Delta_{\phi}(x_{\sigma},\sigma,o), (8)

V-B Mixed-Domain Training

Policy-distilled replay records observations and executed actions from base-policy rollouts in the source environment. On collaborative samples, the complete denoiser minimizes the usual reconstruction loss, with gradients entering only the adapter. On replay samples, the objective instead penalizes the weighted squared residual itself; it does not regress the complete denoiser to the recorded action. This directly encourages the condition that recovers the base distribution. A retention weight balances coordination and rehearsal. Routing is learned from observations rather than an explicit team indicator, and preservation is measured on evaluated source episodes rather than established globally.

Equation 9
ℒreplay​(ϕ)=𝔼𝒟S,σ,ϵ​[w⁡(σ)​‖Δϕ​(x+σ​ϵ,σ,o)‖22],\mathcal{L}_{\mathrm{replay}}(\phi)=\mathbb{E}_{\mathcal{D}_{S},\,\sigma,\,\epsilon}\left[w(\sigma)\left\|\Delta_{\phi}(x+\sigma\epsilon,\sigma,o)\right\|_{2}^{2}\right], (9)

V-C Deployment and Implementation Details

Every robot runs the same frozen transformer denoiser plus residual transformer decoder using only its own RGB view. The decoder uses projected base-encoder tokens and a separate lightweight convolutional path over raw images, allowing it to recover partner cues the solo-trained encoder may discard. The original sampler and receding-horizon execution remain in use. The paper describes modest extra inference cost but does not provide a latency benchmark; the largest tested adapter accounts for 29.8% of the combined parameter count.

VI Simulation Experiments

Simulation asks whether limited collaborative data suffice, whether the adapted policy retains solo skills and how adapter capacity changes that tradeoff. Evidence comes from one two-arm collaborative task with randomized layouts, evaluated separately from checkpoint selection.

VI-A Experiment Settings

In TwoArmPlaceWipe, one arm moves a tray aside while the other wipes beneath it, after which both return their objects. Objects, pads and dirt locations vary across demonstrations and tests. Success requires tray x/y error below 1.5 cm, sponge offsets within 8 cm and 6 cm, and at least 60% wipe coverage. Training uses 20, 40 or 60 collaborative demonstrations with equal numbers of distilled solo episodes. Mixed batches contain 256 team and 256 solo samples; team-only training uses 256 samples. AdamW runs 900,000 updates at learning rate 0.0002 and weight decay 0.0001, saving averaged checkpoints every 100,000 steps. Forty episodes select the best coordination checkpoint, which is then evaluated on 200 fresh team episodes and 100 per original solo task. Actions are replanned every 15 environment steps with an 1,800-step cap. The source is inconsistent about pretraining data: this section says 600 expert solo demonstrations, while Table II lists 400 for the base policy; both originals are retained.

Refer to caption
Fig. 3: In the TwoArmPlaceWipe simulated task, one robot places and returns a tray while another wipes up dirt.

VI-B Baselines

All methods use local shoulder-camera images. From Scratch trains on the same mixed demonstrations with parameter count matched to the base-plus-head total. FT-mixed fine-tunes all 17,446,215 base parameters on team and distilled solo data. FT-multi fine-tunes those same parameters using only team demonstrations. Comparing ALTER with FT-mixed tests freezing plus residual adaptation; comparing fine-tuning variants tests mixed replay. These comparisons also differ in architecture and trainable parameter fraction, so no single component alone explains the overall gap.

VI-C Two-Arm Low-Data Coordination

At 20, 40 and 60 collaborative demonstrations, ALTER reaches 35.0%, 71.5% and 87.0% success. From Scratch reaches 13.5%, 28.0% and 59.0%, giving ALTER gains of 21.5, 43.5 and 28.0 percentage points. FT-mixed reaches 33.0%, 59.5% and 63.5%; FT-multi reaches 29.5%, 50.5% and 58.5%. The lowest-data advantage over mixed fine-tuning is only two points, while gains are larger at higher budgets. Table I reports selected-checkpoint episode outcomes rather than confidence intervals or variation across independent training seeds.

TABLE I: TwoArmPlaceWipe coordination success (%, ↑\uparrow) across demonstration budgets. Bold denotes the best result within each data regime.
Multi-agent demos Single-arm demos Method Checkpoint Coordination success ↑\uparrow
20 20 ALTER 500500k 35.0
20 20 FS 200200k 13.5
20 20 FT-mixed 200200k 33.0
20 0 FT-multi 200200k 29.5
40 40 ALTER 300300k 71.5
40 40 FS 900900k 28.0
40 40 FT-mixed 400400k 59.5
40 0 FT-multi 300300k 50.5
60 60 ALTER 500500k 87.0
60 60 FS 400400k 59.0
60 60 FT-mixed 500500k 63.5
60 0 FT-multi 200200k 58.5

VI-D Source-Task Retention

With the coordination head still enabled, ALTER retains 97.0%, 97.5% and 97.0% combined source success across budgets, close to the base policy’s 97.0% average of 100% place-return and 94% wipe. From Scratch averages 32.5–61.0%, mixed fine-tuning 46.0–82.5%, and team-only fine-tuning 20.0–34.0%. The worst forgetting occurs on place-return for team-only fine-tuning and wiping for the scratch model. These are 100-episode evaluations per task; they establish retention on this tested pair, not every behavior of the pretrained policy.

TABLE II: Source-task success (%, ↑\uparrow) after two-arm adaptation.
Single-arm demos Source success ↑\uparrow
Multi-arm demos Expert Distilled rollouts Method Place- return Wipe Combined
0 400 0 Base policy 100.0 94.0 97.0
20 0 20 ALTER 99.0 95.0 97.0
20 0 20 FS 56.0 9.0 32.5
20 0 20 FT-mixed 55.0 37.0 46.0
20 0 0 FT-multi 0.0 40.0 20.0
40 0 40 ALTER 100.0 95.0 97.5
40 0 40 FS 75.0 14.0 44.5
40 0 40 FT-mixed 84.0 81.0 82.5
40 0 0 FT-multi 0.0 68.0 34.0
60 0 60 ALTER 99.0 95.0 97.0
60 0 60 FS 76.0 46.0 61.0
60 0 60 FT-mixed 83.0 82.0 82.5
60 0 0 FT-multi 0.0 50.0 25.0

VI-E Ablation Studies

With 20 collaborative and 20 distilled solo demonstrations, adapter sizes XS/S/M/L yield 25.0/27.0/33.5/35.0% coordination, while each retains 97.0% combined solo success. Total sizes range from 17.881 to 24.846 million parameters and trainable fractions from 2.4% to 29.8%. Scratch comparators change transformer/feed-forward widths to match total counts within the reported 0.0013%, but perform at 10.0–13.5% coordination. This motivates the largest head for the main experiments, while leaving larger-team scaling and sensitivity to the retention weight untested.

TABLE III: Effect of coordination-head size on coordination and source-task success (%, ↑\uparrow) in TwoArmPlaceWipe.
Size Method Total params. (M) Trainable fraction (%) Coordination success ↑\uparrow Source success ↑\uparrow
XS ALTER 17.881 2.4 25.0 97.0
XS FS 17.881 100.0 12.0 34.5
S ALTER 19.077 8.5 27.0 97.0
S FS 19.076 100.0 10.0 37.5
M ALTER 20.815 16.2 33.5 97.0
M FS 20.815 100.0 10.5 39.5
L ALTER 24.846 29.8 35.0 97.0
L FS 24.846 100.0 13.5 32.5

VII Hardware Experiments

A separate hardware study tests the adaptation strategy on a collaborative box task. The base and adapted models are trained on physical demonstrations; this is not direct transfer of the simulation-trained tray/wiping policy.

VII-A Setup

Two xArm7 arms remove a box lid, place a plush bird inside and replace the lid. Success requires the bird inside and a stable unsupported lid covering more than half the box. Base pretraining uses 332 solo demonstrations across lid removal, lid replacement and bird pick-place. Each adaptation method receives 69 collaborative and 68 distilled solo episodes, trains for 25,000 updates and uses only each robot’s shoulder camera. Planning and execution horizons are both 20 steps, with a 900-step trial cutoff. Hardware comparisons include the strongest simulated fine-tuning variant and matched training from scratch.

Refer to caption
Fig. 4: Hardware setup with two xArm7 robots, a lidded box, and a plush bird. One robot removes the lid, the other places the bird in the box, and the first robot replaces the lid.

VII-B Results

ALTER and FT-mixed each succeed in 18/20 collaborative trials, versus 13/20 for From Scratch. ALTER’s source-task results are 19/20 lid removal, 20/20 lid replacement and 20/20 bird pick-place. In Table IV’s column order, From Scratch gives 9/20, 19/20 and 12/20; FT-mixed gives 11/20, 19/20 and 17/20. The narrative paragraph assigns those FT-mixed values to a different task order; this summary follows the labeled table and preserves that discrepancy. The unadapted base is shown at 40/40 on each solo task. With only 20 adapted-policy trials per task, the observed collaboration tie and retention advantage do not establish population-level statistical equality or significance.

TABLE IV: Hardware coordination and source success.
Source success ↑\uparrow
Method Coordination success ↑\uparrow Lid removal Lid replacement Bird pick-place
Base policy – 40/40 40/40 40/40
ALTER 18/20 19/20 20/20 20/20
FS 13/20 9/20 19/20 12/20
FT-mixed 18/20 11/20 19/20 17/20

VIII Conclusion

Residual denoising plus policy-distilled rehearsal supports both coordination and solo-skill retention in the reported two-arm tasks. The largest simulation coordination gains are 43.5 points over scratch and 23.5 over mixed fine-tuning; the hardware advantage over fine-tuning appears in retained source skills rather than collaborative success. Scope is limited to one simulation task, one physical task, homogeneous two-arm teams and a replay-capable source environment. Broader embodiments, larger teams and diverse task transfer remain future work.

Concept demonstration

Open image ↗