Literature
Display
← All papers

FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning

Zhilin He · Gauri Joshi

Preprint posted · CoRL 2026 (accepted) ↗

Robotics17 min text · CondensedOriginal paper ↗
Contents

Overview

Federated reinforcement learning through personalized behavior priors: local policies and critics learn with shared diffusion-prior experts, with all original results, proofs and rollout grids retained.

At a glance

FedGuide exchanges behavior-prior heads among clients with different environments, routes them into server experts using optimal transport, and returns a personalized mixture to each client. Local PPO combines prior regularization with a frozen DICE value baseline. Simulated benchmarks show higher returns in many settings, with important exceptions and weak-overlap limits. Policies and raw trajectories remain local in standard FedGuide, but this is not a formal privacy guarantee.

1 Introduction

Heterogeneous clients induce different trajectory distributions, so averaging their policy parameters can erase useful local behavior. FedGuide instead communicates diffusion behavior priors, preserves local policies, and uses a local offline value estimate to stabilize advantage estimation. The novelty is this federated combination of existing diffusion, transport and DICE components. The analysis gives conditional bias/variance and approximate-stationarity statements; experiments test simulation return, not physical multi-robot deployment.

2 Related Work

Prior work addresses federated heterogeneity through parameter sharing, policy-divergence control, update correction and personalized experts. FedGuide changes the collaboration interface to behavior-prior models while keeping policy optimization local.

Heterogeneous Federated Reinforcement Learning.

Federated RL adds policy-induced data shift to ordinary non-IID learning. Methods such as FedKL, variance-reduced gradient sharing and federated Q-learning stabilize shared policies or updates. Distribution-aware supervised MoE/transport methods motivate preserving multiple modes, but their static-data setting differs from evolving client trajectories.

Diffusion Priors and Value-Guided Policy Learning.

Diffusion priors model data-supported actions without inherently preferring high return. DICE supplies offline value/distribution-correction information. Here its value stays local and enters only as a state baseline: it neither scores sampled actions directly nor guides the diffusion sampler. This distinction prevents reading “value guidance” as classifier-style diffusion guidance.

3 Preliminaries

Notation separates diffusion step, environment time, communication round, local update, client and server expert. Policy parameters, diffusion-prior parameters and DICE value parameters are distinct objects.

3.1 Problem Formulation

Clients share state/action interfaces but may differ in transition dynamics, rewards and initial-state distributions. The objective is the average discounted return of personalized local policies. Shared interfaces and meaningful behavior-support overlap are substantive assumptions; arbitrary incompatible robot embodiments are not covered.

Equations 1–2
pi​(τi∣πi)=ρi​(si0)​∏t=0T−1πi​(ait∣sit)​Pi​(sit+1∣sit,ait).p_{i}(\tau_{i}\mid\pi_{i})\;=\;\rho_{i}(s_{i}^{0})\prod_{t=0}^{T-1}\pi_{i}(a_{i}^{t}\mid s_{i}^{t})\,P_{i}(s_{i}^{t+1}\mid s_{i}^{t},a_{i}^{t}). (1)
max{πi}i=1N⁡J⁡({πi})=1N​∑i=1NJi​(πi),Ji​(πi)=𝔼τ∼pi​(τi∣πi)​[∑t=0T−1γ​ri​(sit,ait)].\max_{\{\pi_{i}\}_{i=1}^{N}}\;J(\{\pi_{i}\})\;=\;\frac{1}{N}\sum_{i=1}^{N}J_{i}(\pi_{i}),\quad J_{i}(\pi_{i})\;=\;\mathbb{E}_{\tau\sim p_{i}(\tau_{i}\mid\pi_{i})}\!\!\left[\sum_{t=0}^{T-1}\gamma r_{i}(s_{i}^{t},a_{i}^{t})\right]. (2)

3.2 Diffusion Priors and DICE Value Functions

The two offline components serve different roles: a conditional behavior distribution constrains plausible actions, while a state-only value function contributes return information to advantage centering.

Diffusion prior as behavior model.

A state-conditioned diffusion model learns noisy-action scores through denoising score matching, then defines a clean-action behavior distribution. It captures the offline buffer’s action support irrespective of reward. The full forward process and score definitions remain available. Behavior modeling alone is not policy improvement.

Equation 3
qk(atk∣at0,st)=𝒩(atk;α¯kat0,(1−α¯k)𝐈),k=0,…,K,q_{k}({}^{k}a^{t}\mid{}^{0}a^{t},s^{t})\;=\;\mathcal{N}\!\bigl({}^{k}a^{t};\,\sqrt{\bar{\alpha}_{k}}\,{}^{0}a^{t},\,(1-\bar{\alpha}_{k})\mathbf{I}\bigr),\quad k=0,\ldots,K, (3)

DICE value function.

DICE estimates discounted occupancy corrections using offline value/Q functions and a divergence-regularized objective, drawing on DualDICE and OptiDICE. After pretraining, only the frozen state value is used online. This avoids directly evaluating unseen actions with an offline Q function, but a stale or inaccurate state value can still bias bootstrapped GAE and fail to reduce variance.

Equations 4–6
wi​(st,at)=diπi⋆​(st,at)di𝒟i​(st,at)w_{i}(s^{t},a^{t})\;=\;\frac{d_{i}^{\pi_{i}^{\star}}(s^{t},a^{t})}{d_{i}^{\mathcal{D}_{i}}(s^{t},a^{t})} (4)
wi​(st,at)≈exp⁡(1ω​(Qϕ,i​(st,at)−Vϕ,i​(st))),w_{i}(s^{t},a^{t})\;\approx\;\exp\!\Big(\tfrac{1}{\omega}\bigl(Q_{\phi,i}(s^{t},a^{t})-V_{\phi,i}(s^{t})\bigr)\Big), (5)
minVϕ,i⁡𝔼(st,at)∼di𝒟i​[(1−γ)​Vϕ,i​(st)+ω​f∗​(1ω​(Qϕ,i​(st,at)−Vϕ,i​(st)))],\displaystyle\min_{V_{\phi,i}}\;\mathbb{E}_{(s^{t},a^{t})\sim d_{i}^{\mathcal{D}_{i}}}\!\Big[(1-\gamma)\,V_{\phi,i}(s^{t})+\omega\,f^{*}\!\!\Big(\tfrac{1}{\omega}\bigl(Q_{\phi,i}(s^{t},a^{t})-V_{\phi,i}(s^{t})\bigr)\Big)\Big], (6)
minQϕ,i⁡𝔼(st,at,st+1)∼di𝒟i​[(ri​(st,at)+γ​Vϕ,i​(st+1)−Qϕ,i​(st,at))2],\displaystyle\min_{Q_{\phi,i}}\;\mathbb{E}_{(s^{t},a^{t},s^{t+1})\sim d_{i}^{\mathcal{D}_{i}}}\!\Big[(r_{i}(s^{t},a^{t})+\gamma V_{\phi,i}(s^{t+1})-Q_{\phi,i}(s^{t},a^{t}))^{2}\Big],
Avg Error𝐱(t,0)\mathbf{x}^{(t,0)}𝐱(t,H)\mathbf{x}^{(t,H)}𝐱1∗\mathbf{x}_{1}^{*}𝐱2∗\mathbf{x}_{2}^{*}𝐱∗\mathbf{x}^{*}
(a) Policy-based Aggregation
π¯D\bar{\pi}_{D}π¯D,1\bar{\pi}_{D,1}π¯D,2\bar{\pi}_{D,2}𝐱(t,0)\mathbf{x}^{(t,0)}𝐱1(t,H)\mathbf{x}_{1}^{(t,H)}𝐱2(t,H)\mathbf{x}_{2}^{(t,H)}𝐱1∗\mathbf{x}_{1}^{*}𝐱2∗\mathbf{x}_{2}^{*}𝐱∗\mathbf{x}^{*}
(b) Priors Only (FedGuide-P)
Vϕ,iV_{\phi,i}Vϕ,iV_{\phi,i}π¯D\bar{\pi}_{D}π¯D,1\bar{\pi}_{D,1}π¯D,2\bar{\pi}_{D,2}Var↓\mathrm{Var}\downarrowVar↓\mathrm{Var}\downarrow𝐱(t,0)\mathbf{x}^{(t,0)}𝐱1(t,H)\mathbf{x}_{1}^{(t,H)}𝐱2(t,H)\mathbf{x}_{2}^{(t,H)}𝐱1∗\mathbf{x}_{1}^{*}𝐱2∗\mathbf{x}_{2}^{*}𝐱∗\mathbf{x}^{*}
(c) Priors and Value Baseline (FedGuide)
Figure 1: Method Visualization with 2 Clients Heterogeneous Settings. (a) Policy aggregation averages local optima 𝐱1∗,𝐱2∗\mathbf{x}_{1}^{*},\mathbf{x}_{2}^{*} into global policy 𝐱(t,H)\mathbf{x}^{(t,H)}, away from the global optimum 𝐱∗\mathbf{x}^{*}. (b) OT-MoE aggregates diffusion priors in distribution space, forming shared prior support π¯D\bar{\pi}_{D} and personalized priors π¯D,1,π¯D,2\bar{\pi}_{D,1},\bar{\pi}_{D,2} (pink). The prior regularizer ℛ^\widehat{\mathcal{R}} (yellow) pulls local policies toward personalized prior supports. (c) FedGuide further uses the DICE value baseline (green) for policy improvement, shown as straighter policies updates and smaller endpoint dispersion (Variance↓\mathrm{Variance}\downarrow).

4 Methodology

Server-side transport routes prior heads into multiple experts; clients receive personalized mixture distributions. Local policy updates then combine prior regularization, a trust-region term and a blended online/offline value baseline.

4.1 OT-MoE Aggregation of the Diffusion Priors

The ideal transport cost compares client/expert action scores; implementation uses a parameter-space proxy. Entropic Sinkhorn transport uses uniform client/expert marginals. Each expert head is refreshed by a column-normalized weighted average of client head parameters, then each client receives a row-weighted mixture of expert distributions. Thus the method still averages prior-head parameters; its distinction is preserving multiple expert distributions and local policies. The prior cross-entropy is estimated with self-normalized importance weights over rollout samples, introducing approximation error. Proxy-cost quality and within-expert averaging are not automatic guarantees of mode preservation.

Equations 7–11
Ci,m(h)=𝔼(st,at)∼di𝒟i​[‖∇at​log​πD,ψi(h)​(at∣st)−∇at​log​πD,ψ(m,h)​(at∣st)‖22],C^{(h)}_{i,m}\;=\;\mathbb{E}_{(s^{t},a^{t})\sim d_{i}^{\mathcal{D}_{i}}}\!\Bigl[\bigl\|\nabla_{a^{t}}\log\pi_{D,\psi_{i}^{(h)}}(a^{t}\mid s^{t})-\nabla_{a^{t}}\log\pi_{D,\psi^{(m,h)}}(a^{t}\mid s^{t})\bigr\|_{2}^{2}\Bigr], (7)
Y⋆=arg⁡minY∈Π⁡(μ,ν)​∑i=1N∑m=1MYi,m​Ci,m(h)+η​∑i=1N∑m=1MYi,m​log⁡Yi,m,Y^{\star}\;=\;\arg\min_{Y\in\Pi(\mu,\nu)}\;\sum_{i=1}^{N}\sum_{m=1}^{M}Y_{i,m}C^{(h)}_{i,m}\;+\;\eta\sum_{i=1}^{N}\sum_{m=1}^{M}Y_{i,m}\log Y_{i,m}, (8)
ψ(m,h+1)←∑i=1NYi,m⋆∑j=1NYj,m⋆ψi(h),m=1,…,M.\psi^{(m,h+1)}\;\leftarrow\;\sum_{i=1}^{N}\frac{Y_{i,m}^{\star}}{\sum_{j=1}^{N}Y_{j,m}^{\star}}\,\psi_{i}^{(h)},\quad m=1,\dots,M. (9)
π¯D,i(h+1)(⋅∣st)=∑m=1MYi,m⋆∑j=1MYi,j⋆πD,ψ(m,h+1)(⋅∣st),\bar{\pi}_{D,i}^{(h+1)}(\cdot\mid s^{t})\;=\;\sum_{m=1}^{M}\frac{Y_{i,m}^{\star}}{\sum_{j=1}^{M}Y_{i,j}^{\star}}\,\pi_{D,\psi^{(m,h+1)}}(\cdot\mid s^{t}), (10)
minθi𝒥i(θi)=−Ji(πθ,i)+λ1D^KL(πθ,iold∥πθ,i)+λ2ℛ^prior(θi;π¯D,i).\min_{\theta_{i}}\;\mathcal{J}_{i}(\theta_{i})\;=\;-J_{i}(\pi_{\theta,i})\;+\;\lambda_{1}\,\widehat{D}_{\mathrm{KL}}\!\bigl(\pi_{\theta,i}^{\,\text{old}}\,\big\|\,\pi_{\theta,i}\bigr)\;+\;\lambda_{2}\,\widehat{\mathcal{R}}_{\mathrm{prior}}(\theta_{i};\bar{\pi}_{D,i}). (11)

4.2 DICE Value Baseline Advantage Blending

Blend the online critic with the frozen DICE value using beta, then form GAE and optimize a clipped PPO surrogate with prior regularization. Default beta=0.5 gives equal value weights; beta=1 removes DICE. A state-only baseline has zero direct score-function expectation, but bootstrapping, clipping and approximate prior weights still introduce errors outside that identity.

Equations 12–13
𝐕i​(s)=β​Vθ,i​(s)+(1−β)​Vϕ,i​(s),β∈[0,1],\mathbf{V}_{i}(s)\;=\;\beta\,V_{\theta,i}(s)\;+\;(1-\beta)\,V_{\phi,i}(s),\quad\beta\in[0,1], (12)
A^t=∑ℓ=0T−t−1(γ​λGAE)ℓ​(rt+ℓ+γ​𝐕i​(st+ℓ+1)−𝐕i​(st+ℓ)),\hat{A}^{t}\;=\;\sum_{\ell=0}^{T-t-1}(\gamma\lambda_{\mathrm{GAE}})^{\ell}\,\Bigl(r^{t+\ell}+\gamma\mathbf{V}_{i}(s^{t+\ell+1})-\mathbf{V}_{i}(s^{t+\ell})\Bigr), (13)
Algorithm 1 FedGuide (Server)
1: Experts { ψ ( m ) } m = 1 M \{\psi^{(m)}\}_{m=1}^{M} ; Sinkhorn temperature η \eta ; marginals μ i = 1 / N \mu_{i}\!=\!1/N , ν m = 1 / M \nu_{m}\!=\!1/M . 2: for round h = 1 , … , H h=1,\dots,H do 3: If h = 1 h=1 , initialize π ¯ D , i \bar{\pi}_{D,i} from seeded experts; 4: otherwise form π ¯ D , i \bar{\pi}_{D,i} from previous Y ⋆ Y^{\star} ( 10 ). 5: Broadcast π ¯ D , i \bar{\pi}_{D,i} (and θ ¯ \bar{\theta} ) to client i i . 6: // Server side update via Alg. 2 . 7: Receive { ψ i } i = 1 N \{\psi_{i}\}_{i=1}^{N} (and { θ i } i = 1 N \{\theta_{i}\}_{i=1}^{N} ) from clients. 8: Build cost matrix C i , m C_{i,m} via score-MSE ( 7 ). 9: Solve Sinkhorn OT-based plan Y ⋆ Y^{\star} ( 8 ). 10: Refresh expert heads ψ ( m ) \psi^{(m)} via ( 9 ). 11: Store Y ⋆ Y^{\star} for next-round personalized routing. 12: Averaging received policies into θ ¯ \bar{\theta} . 13: end for
Algorithm 2 FedGuide (Client ii)
1: Personalized prior π ¯ D , i \bar{\pi}_{D,i} from server; local params θ i \theta_{i} (or received θ ¯ \bar{\theta} ) , online critic V θ , i V_{\theta,i} , DICE value baseline V ϕ , i V_{\phi,i} , prior head ψ i \psi_{i} . 2: Collect rollout { ( s t , a t , r t , s t + 1 ) } t = 0 T − 1 \{(s^{t},a^{t},r^{t},s^{t+1})\}_{t=0}^{T-1} with π θ , i \pi_{\theta,i} . 3: Form blended 𝐕 i \mathbf{V}_{i} with β = 0.5 \beta=0.5 (or β = 1 \beta=1 ) ( 12 ). 4: Compute advantage A ^ t \hat{A}^{t} from 𝐕 i \mathbf{V}_{i} ( 13 ). 5: for each epoch do 6: Update θ i \theta_{i} on PPO surrogate ℒ i \mathcal{L}_{i} ( 14 ). 7: Update online critic V θ , i V_{\theta,i} on ( s t , R ^ t ) (s^{t},\hat{R}^{t}) . 8: end for 9: Optionally adapt prior head ψ i \psi_{i} locally, or keep the pretrained head. 10: Upload ψ i \psi_{i} (and θ i \theta_{i} ) to server. 11: // Uncolored lines are standard FedGuide; colored parts apply only to FedGuide-A and FedGuide-P .
Equation 14
ℒi​(θi)=𝔼⁡[−min⁡(πθiπθiold​A^t,clip⁡(πθiπθiold,1−ε,1+ε)​A^t)]+λ2​∑t=0T−1ℛ^prior​(θi,π¯D,i)​(st).\mathcal{L}_{i}(\theta_{i})=\mathbb{E}\!\Biggl[-\min\!\Biggl(\frac{\pi_{\theta_{i}}}{\pi_{\theta_{i}}^{\mathrm{old}}}\hat{A}^{t},\mathrm{clip}\!\Biggl(\frac{\pi_{\theta_{i}}}{\pi_{\theta_{i}}^{\mathrm{old}}},1{-}\varepsilon,1{+}\varepsilon\Biggr)\hat{A}^{t}\Biggr)\Biggr]+\lambda_{2}\!\sum_{t=0}^{T-1}\widehat{\mathcal{R}}_{\mathrm{prior}}(\theta_{i};\bar{\pi}_{D,i})(s^{t}). (14)

4.3 Putting Everything Together: FedGuide

Each round broadcasts personalized priors, collects local prior heads and updates transport/expert parameters. Policy and both value models remain local in standard FedGuide. Local head adaptation is optional in Algorithm 2; the pretrained head may be retained. FedGuide-P removes DICE; FedGuide-A additionally averages policies every five rounds. The native algorithms preserve the source’s colored variant-only lines and exact indentation. A possible beta schedule is discussed, while the reported default table uses fixed beta=0.5.

5 Theoretical Analysis

The convergence analysis assumes a smooth surrogate, bounded gradient noise, bounded local KL drift, informative bounded values, and bounded transport/proxy/prior/value approximation errors. It describes an idealized stochastic descent direction under these conditions, not a proof that every neural PPO run satisfies them.

Lemma 1 (State-only baseline does not add score function bias).

Subtracting a state-only value does not add an action-dependent score-function term: its expected contribution is zero under the current policy. This identity does not establish that finite-horizon GAE with an imperfect baseline, PPO clipping or importance approximations is completely unbiased.

Equation 15
𝔼at∼πθ,i(⋅∣st)[∇θilogπθ,i(at∣st)𝐕i(st)]=0.\mathbb{E}_{a^{t}\sim\pi_{\theta,i}(\cdot\mid s^{t})}\!\left[\nabla_{\theta_{i}}\log\pi_{\theta,i}(a^{t}\mid s^{t})\mathbf{V}_{i}(s^{t})\right]=0. (15)

Lemma 2 (Variance reduction).

Variance can fall when the offline-minus-online value correction correlates positively with the online-baseline return residual and the blending coefficient lies in the derived interval. Positive correlation alone does not guarantee improvement for the fixed beta=0.5. The proof analyzes the return/advantage control variate; effective gradient-noise reduction enters the convergence assumptions separately.

Theorem 1 (Convergence of FedGuide).

With a sufficiently small policy step, the average squared surrogate-gradient norm is bounded by initial suboptimality over rounds, a stochastic term scaling as effective variance/(clients×local updates), and a residual approximation/drift floor. Local policies avoid an explicit policy-averaging heterogeneity term, but heterogeneity still affects prior costs, overlap and approximation assumptions. The result concerns stationarity of the surrogate, not global optimal return.

Equation 16
1H​∑h=0H−1𝔼​‖∇F~​(ψ(h),θ(h))‖2≤𝒪⁡(F~(0)−F~⋆ηpg​H)+𝒪⁡(ηpg​L​ςeff2N​U)+𝒪⁡(Δ).\frac{1}{H}\sum_{h=0}^{H-1}\mathbb{E}\bigl\|\nabla\tilde{F}(\psi^{(h)},\theta^{(h)})\bigr\|^{2}\;\leq\;\mathcal{O}\!\left(\frac{\tilde{F}^{(0)}-\tilde{F}^{\star}}{\eta_{\mathrm{pg}}H}\right)+\mathcal{O}\!\left(\eta_{\mathrm{pg}}L\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}\right)+\mathcal{O}\!\left(\Delta\right). (16)

6 Experimental Analysis

Experiments ask whether distribution sharing improves average client return, survives harder heterogeneity and benefits from both the prior and value baseline. All evaluations are simulated, with common client splits and five random seeds.

Variants Diffusion Prior DICE Value Client Policy
FG-A OT-MoE Local Avg.
FG-P OT-MoE N/A Local
FG OT-MoE Local Local
Table 1: Comparison of FedGuide (FG) variants: FedGuide keeps the DICE value baseline 𝐕i\mathbf{V}_{i} local, while FedGuide-P does not use Vϕ,iV_{\phi,i} (β=1\beta=1). FedGuide-A uses Vϕ,iV_{\phi,i} but periodically averages client policies at the server. All variants use OT-MoE aggregation for diffusion prior.
CVσT\mathrm{CV}_{\sigma_{T}} ↓\downarrow
Env FG-A FG-P FG
Reacher 0.380 0.068 0.053
Hopper 0.402 0.164 0.155
Walker2D 0.208 0.179 0.127
HalfCheetah 0.628 0.235 0.241
MetaWorld10 0.364 0.272 0.260
Reacher-Hard 2.097 0.229 0.342
Hopper-Hard 0.377 0.185 0.133
Walker2D-Hard 0.178 0.155 0.088
HalfCheetah-Hard 0.591 0.231 0.201
Table 2: Normalized temporal volatility CVσT\mathrm{CV}_{\sigma_{T}} for FedGuide (FG) and variants, computed over the last 20 rounds as the across-seed mean of return standard deviation normalized by the absolute mean return. Smaller values indicate smoother late-stage returns and are consistent with, but do not directly measure, the variance-reduction effect in Lemma 2.

Experimental Setup.

Benchmarks include a four-mode 2-D bandit, Reacher, three MuJoCo locomotion tasks and ten MetaWorld manipulation tasks. Reacher/locomotion use eight clients; MetaWorld uses ten clients and eight experts. Baselines share interaction/evaluation settings and behavior-cloning warm-up, but use different optimizers and local epoch counts; this is not a matched total-compute comparison. FedRL is a DDPGAvg-style actor-critic and FedSVRPG-M a practical PPO/SVRPG implementation.

Figure 2: Left: Demonstrations of different heterogeneous FRL environments: Reacher, Walker2D, Hopper, HalfCheetah, and MetaWorld10 with 10 heterogeneous manipulation tasks. Right: Visualization of toy-case Bandit2D with 4 isotropic Gaussian reward peaks (σ=0.2\sigma{=}0.2) evenly spaced on the unit circle, each client sampled from an overlapping 120∘120^{\circ} sector of the annulus r∈[0.7,1.3]r\in[0.7,1.3] centered on one peak. All clients in FedAvg collapse to one peak, while FedGuide (including its two variants) captures each peak; its global diffusion prior recovers the ring-shaped union of client supports through OT-MoE aggregation.
Reacher Hopper Walker2D HalfCheetah MetaWorld10 Bandit2D Ground Truth OTMoE Global Prior FedAvg FedGuide Client 1 Client 2 Client 3 Client 4 Global
Figure 3: Client average return for 100 rounds at different heterogeneous FRL environments. We set client numbers to 10 in MetaWorld10 and keep 8 clients in other environments, evaluating on 5 random seeds with Behavior Cloning warm-up. Color regions indicate one standard deviation.
Refer to caption
Environment Metric FedAvg FedKL FedRL FedSVRPG-M† FedGuide-A FedGuide-P FedGuide
Reacher 𝐅↑\mathbf{F}\uparrow -19.4±\pm1.82 -20.9±\pm1.63 -13.4±\pm1.16 -24.9±\pm3.46 -16.1±\pm0.48 -12.2±\pm0.35 -12.3±\pm0.67
𝐖↑\mathbf{W}\uparrow -21.4 -22.3 -14.4 -29.8 -16.8 -12.6 -13.0
Hopper 𝐅↑\mathbf{F}\uparrow 227±\pm42.5 253±\pm48.1 527±\pm120 525±\pm360 481±\pm26.5 507±\pm19.3 632±\pm19.0
𝐖↑\mathbf{W}\uparrow 180 207 380 166 451 481 614
Walker2D 𝐅↑\mathbf{F}\uparrow 283±\pm27.4 265±\pm21.4 360±\pm113 156±\pm113 422±\pm31.5 383±\pm18.7 480±\pm18.6
𝐖↑\mathbf{W}\uparrow 252 250 219 21.3 377 353 461
HalfCheetah 𝐅↑\mathbf{F}\uparrow 600±\pm92.1 950±\pm90.7 579±\pm145 -42.2±\pm393 1439±\pm176 1768±\pm500 1994±\pm304
𝐖↑\mathbf{W}\uparrow 486 821 421 -409 1246 978 1681
MetaWorld10 𝐅↑\mathbf{F}\uparrow 631±\pm238 445±\pm213 767±\pm53.6 668±\pm102 829±\pm82.5 914±\pm58.2 939±\pm141
𝐖↑\mathbf{W}\uparrow 248 313 724 547 744 816 716
Reacher-Hard 𝐅↑\mathbf{F}\uparrow -2531±\pm1823 -4274±\pm2070 -90.2±\pm4.97 -8339±\pm6580 -1995±\pm128 90.7±\pm15.7 86.5±\pm37.6
𝐖↑\mathbf{W}\uparrow -5210 -7118 -94.8 -16456 -2148 67.4 37.6
Hopper-Hard 𝐅↑\mathbf{F}\uparrow 216±\pm20.1 242±\pm41.0 699±\pm256 387±\pm285 398±\pm10.9 437±\pm51.6 510±\pm44.0
𝐖↑\mathbf{W}\uparrow 187 197 404 169 384 383 445
Walker2D-Hard 𝐅↑\mathbf{F}\uparrow 256±\pm13.2 286±\pm24.3 321±\pm57.8 57.8±\pm24.5 380±\pm10.9 361±\pm25.3 415±\pm14.5
𝐖↑\mathbf{W}\uparrow 241 258 258 27.3 368 327 396
HalfCheetah-Hard 𝐅↑\mathbf{F}\uparrow 231±\pm451 373±\pm110 832±\pm349 96.1±\pm251 1272±\pm203 1519±\pm327 2146±\pm304
𝐖↑\mathbf{W}\uparrow -544 217 328 -282 928 1168 1893

Table 3: Quantitative results of 𝐅\mathbf{F}inal-round and 𝐖\mathbf{W}orst-round client average return performance across different heterogeneous FRL environments and seeds, along with results for stronger heterogeneity robustness (“Hard” ) experiments. We use M=8M=8 for all environments. † denotes the method is implemented by us.

6.1 Experimental Results

The result tables distinguish final-round return, worst-round client-average return and late-training temporal volatility. None is a worst-client guarantee or a measured task-success rate.

Performance for Heterogeneous FRL.

FedGuide’s final returns are 632 on Hopper, 480 on Walker2D, 1,994 on HalfCheetah and 939 on MetaWorld10, exceeding the compared methods’ means there. FedGuide-P is marginally better on ordinary Reacher (−12.2 versus −12.3). MetaWorld gains are modest and variable; FedGuide’s worst-round value 716 is below FedGuide-P’s 816 and FedRL’s 724 despite its best final mean. Table 3 retains all means/spreads and exceptions; return should not be interpreted as manipulation success rate.

Refer to caption
Figure 4: Client average return for FRL environments with stronger heterogeneity (“Hard”). We fixed client numbers at 8 and the same experimental settings with Fig. 3.

Discussion of Robustness.

Under harder heterogeneity, FedGuide leads final Walker2D/ HalfCheetah returns, while FedRL leads Hopper-Hard (699 versus 510). Reacher-Hard exposes a sharp variant difference: FedGuide-P and FedGuide reach about 90.7/86.5, but policy-averaging FedGuide-A collapses to −1,995. Thus the whole family is not uniformly robust. These benchmark comparisons vary client counts across tasks, rather than constituting a controlled scaling study in client number.

Why Diffusion Prior and DICE Value Baseline?

Bandit plots show that policy averaging can lose reward modes while shared priors preserve the union of represented supports. The ring is prescribed by overlapping annular training sectors; it is not evidence of discovering unsupported behavior. Table 2 measures temporal variation of return over the last twenty rounds, not policy-gradient variance or the residual correlations assumed by Lemma 2. DICE usually helps this metric, but FedGuide-P has lower volatility on Reacher-Hard.

7 Limitations

No physical robots, perception noise, actuation latency or uncertain real contacts are evaluated. Keeping trajectories and policies local does not establish privacy against model-update leakage. Shared state/action spaces, useful prior overlap and manageable client/expert counts are required. Severe task-semantic or reward shifts can make guidance uninformative, and adaptive expert-count selection remains open.

8 Conclusions

Sharing multiple behavior-prior experts can improve heterogeneous local RL without synchronizing every policy. The experiments support benefits on the tested simulations and expose limits when useful support is absent. Extension to task-semantic representations and physical multi-robot systems remains future work.

Acknowledgments

The work acknowledges NSF/ONR funding and Bridges-2 computing through the ACCESS program. These resources are separate from the algorithmic communication and interaction budgets.

Appendix Contents

The appendices contain all theoretical assumptions/proofs, benchmark metadata, baseline mechanisms, hyperparameters, sensitivity tests and complete qualitative rollout grids.

Appendix A Theoretical Proofs

The theory justifies a state-only baseline, controlled-error transport aggregation and keeping policies local. It explicitly targets approximate stationarity and conditional error bounds, not global optimality of heterogeneous RL.

A.1 Assumptions

Assume smoothness/Lipschitz prior scores, bounded stochastic gradients and self-normalized-prior bias, bounded KL drift, bounded informative DICE values, and bounded Sinkhorn/cost-proxy errors. The descent-direction decomposition assumes zero-mean noise and a bias bounded by the sum of approximation terms. Its 1/(NU) variance scaling is assumed after client/update averaging, rather than derived from arbitrary correlated local trajectories.

Equations 17–21
g(h)=∇F~​(x(h))+ξ(h)+b(h),𝔼⁡[ξ(h)∣ℱ​h]=0,g^{(h)}=\nabla\tilde{F}(x^{(h)})+\xi^{(h)}+b^{(h)},\quad\mathbb{E}[\xi^{(h)}\mid\mathcal{F}h]=0, (17)
𝔼⁡[|ξ(h)|2∣ℱ​h]≤ς​eff2N​U.\mathbb{E}\left[|\xi^{(h)}|^{2}\mid\mathcal{F}h\right]\leq\frac{\varsigma{\mathrm{eff}}^{2}}{NU}. (18)
DKL​(πθ,i(h+1)∥πθ,i(h))≤δ.D_{\mathrm{KL}}\!\left(\pi_{\theta,i}^{(h+1)}\;\middle\|\;\pi_{\theta,i}^{(h)}\right)\leq\delta. (19)
κi=Corr⁡(Git−Vθ,i​(st),Vϕ,i​(st)−Vθ,i​(st)).\kappa_{i}=\mathrm{Corr}\!\left(G_{i}^{t}-V_{\theta,i}(s^{t}),\;V_{\phi,i}(s^{t})-V_{\theta,i}(s^{t})\right). (20)
‖b(h)‖2≤(ϵOT+ϵC+ϵSNIS+ϵV+δ).\|b^{(h)}\|^{2}\leq\left(\epsilon_{\mathrm{OT}}+\epsilon_{C}+\epsilon_{\mathrm{SNIS}}+\epsilon_{V}+\delta\right). (21)

On the role of assumptions.

These are regularity and approximation conditions. The implementation’s trust-region control motivates a drift bound, but the theorem is conditional on that bound and informative values actually holding. Recording an error symbol does not empirically establish a small error.

A.2 Variance Reduction from the Value Baseline

The baseline proof first establishes a score-function identity, then analyzes when an offline value correction reduces residual variance. These are separate claims from full algorithmic convergence.

Lemma 1 (State-only baseline does not add score-function bias).

For a baseline depending only on state, its score-function expectation vanishes under the current action distribution. Blending online and offline state values preserves that property, with GAE approximation handled separately.

Equations 22–23
𝔼at∼πθ,i(⋅∣st)[∇θilogπθ,i(at∣st)𝐕i(st)]=0.\mathbb{E}_{a^{t}\sim\pi_{\theta,i}(\cdot\mid s^{t})}\!\left[\nabla_{\theta_{i}}\log\pi_{\theta,i}(a^{t}\mid s^{t})\mathbf{V}_{i}(s^{t})\right]=0. (22)
bi​(s)=β​Vθ,i​(s)+(1−β)​Vϕ,i​(s)b_{i}(s)=\beta V_{\theta,i}(s)+(1-\beta)V_{\phi,i}(s) (23)

Proof.

Pull the state baseline outside the action integral and differentiate the policy’s unit normalization. The remaining derivative is zero. This argument requires the baseline contribution to be treated as a baseline, not an action-dependent objective term.

Equations 24–26
𝔼at∼πθ,i​[∇θi​log​πθ,i​(at∣st)​𝐕i​(st)]\displaystyle\mathbb{E}_{a^{t}\sim\pi_{\theta,i}}\!\left[\nabla_{\theta_{i}}\log\pi_{\theta,i}(a^{t}\mid s^{t})\mathbf{V}_{i}(s^{t})\right] =𝐕i​(st)​∫πθ,i​(at∣st)​∇θi​log⁡πθ,i​(at∣st)​d​at\displaystyle=\mathbf{V}_{i}(s^{t})\int\pi_{\theta,i}(a^{t}\mid s^{t})\nabla_{\theta_{i}}\log\pi_{\theta,i}(a^{t}\mid s^{t})\,da^{t} (24)
=𝐕i​(st)​∫∇θiπθ,i​(at∣st)​d​at\displaystyle=\mathbf{V}_{i}(s^{t})\int\nabla_{\theta_{i}}\pi_{\theta,i}(a^{t}\mid s^{t})\,da^{t} (25)
=𝐕i​(st)​∇θi​∫πθ,i​(at∣st)​d​at=𝐕i​(st)​∇θi1=0.\displaystyle=\mathbf{V}_{i}(s^{t})\nabla_{\theta_{i}}\int\pi_{\theta,i}(a^{t}\mid s^{t})\,da^{t}=\mathbf{V}_{i}(s^{t})\nabla_{\theta_{i}}1=0. (26)

Lemma 2 (Variance reduction).

Write the blended residual as X−alpha Z, where X is return minus online value and Z is offline minus online value, with alpha=1−beta. Positive covariance permits some alpha between zero and one that reduces variance. It does not identify the default coefficient as optimal.

Equations 27–29
Xit=Git−Vθ,i​(st),Zit=Vϕ,i​(st)−Vθ,i​(st).X_{i}^{t}=G_{i}^{t}-V_{\theta,i}(s^{t}),\quad Z_{i}^{t}=V_{\phi,i}(s^{t})-V_{\theta,i}(s^{t}). (27)
A^β,it=Git−β​Vθ,i​(st)−(1−β)​Vϕ,i​(st)=Xit−(1−β)​Zit.\hat{A}_{\beta,i}^{t}=G_{i}^{t}-\beta V_{\theta,i}(s^{t})-(1-\beta)V_{\phi,i}(s^{t})=X_{i}^{t}-(1-\beta)Z_{i}^{t}. (28)
Var⁡[A^β⋆,it]<Var⁡[A^θ,it],\mathrm{Var}\!\left[\hat{A}_{\beta^{\star},i}^{t}\right]<\mathrm{Var}\!\left[\hat{A}_{\theta,i}^{t}\right], (29)

Proof.

The variance change is alpha squared times Var(Z) minus twice alpha Cov(X,Z). It is negative for alpha below twice covariance/variance, also constrained below one. The unconstrained optimum is covariance/variance when admissible; correlation then yields the familiar multiplicative reduction. A fixed common beta must lie in each relevant client’s reducing interval to obtain a positive uniform coefficient. Zero-variance corrections cannot satisfy the assumed positive covariance.

Equations 30–39
Var⁡[A^β,it]\displaystyle\mathrm{Var}\!\left[\hat{A}_{\beta,i}^{t}\right] =Var⁡[Xit−α​Zit]\displaystyle=\mathrm{Var}\!\left[X_{i}^{t}-\alpha Z_{i}^{t}\right] (30)
=Var⁡[Xit]−2​α​Cov​(Xit,Zit)+α2​Var​[Zit].\displaystyle=\mathrm{Var}[X_{i}^{t}]-2\alpha\,\mathrm{Cov}(X_{i}^{t},Z_{i}^{t})+\alpha^{2}\mathrm{Var}[Z_{i}^{t}]. (31)
Var⁡[A^β,it]−Var⁡[Xit]=−2​α​Cov​(Xit,Zit)+α2​Var​[Zit].\mathrm{Var}\!\left[\hat{A}_{\beta,i}^{t}\right]-\mathrm{Var}[X_{i}^{t}]=-2\alpha\,\mathrm{Cov}(X_{i}^{t},Z_{i}^{t})+\alpha^{2}\mathrm{Var}[Z_{i}^{t}]. (32)
0<α<min⁡{1,2​C​o​v​(Xit,Zit)Var⁡(Zit)},0<\alpha<\min\left\{1,\,\frac{2\mathrm{Cov}(X_{i}^{t},Z_{i}^{t})}{\mathrm{Var}(Z_{i}^{t})}\right\}, (33)
α⋆=Cov⁡(Xit,Zit)Var⁡(Zit).\alpha^{\star}=\frac{\mathrm{Cov}(X_{i}^{t},Z_{i}^{t})}{\mathrm{Var}(Z_{i}^{t})}. (34)
Var⁡[A^β⋆,it]=Var⁡[Xit]−Cov​(Xit,Zit)2Var⁡(Zit)<Var⁡[Xit].\mathrm{Var}\!\left[\hat{A}_{\beta^{\star},i}^{t}\right]=\mathrm{Var}[X_{i}^{t}]-\frac{\mathrm{Cov}(X_{i}^{t},Z_{i}^{t})^{2}}{\mathrm{Var}(Z_{i}^{t})}<\mathrm{Var}[X_{i}^{t}]. (35)
Cov​(Xit,Zit)2=κi2​Var​(Xit)​Var​(Zit),\mathrm{Cov}(X_{i}^{t},Z_{i}^{t})^{2}=\kappa_{i}^{2}\,\mathrm{Var}(X_{i}^{t})\,\mathrm{Var}(Z_{i}^{t}), (36)
Var⁡[A^β⋆,it]=Var⁡(Xit)​(1−κi2).\mathrm{Var}\!\left[\hat{A}_{\beta^{\star},i}^{t}\right]=\mathrm{Var}(X_{i}^{t})(1-\kappa_{i}^{2}). (37)
Var⁡[A^β,it]≤Var⁡(Xit)​(1−cβ,i​κi2).\mathrm{Var}\!\left[\hat{A}_{\beta,i}^{t}\right]\leq\mathrm{Var}(X_{i}^{t})(1-c_{\beta,i}\kappa_{i}^{2}). (38)
ςeff2≤ς2​(1−cβ​κ¯2),κ¯2=∑i=1Nζi​κi2,\varsigma_{\mathrm{eff}}^{2}\leq\varsigma^{2}\left(1-c_{\beta}\bar{\kappa}^{2}\right),\quad\bar{\kappa}^{2}=\sum_{i=1}^{N}\zeta_{i}\kappa_{i}^{2}, (39)

A.3 Federated Convergence

The convergence argument treats the combined prior/policy surrogate as stochastic smooth optimization with bounded bias. A separate PL result adds a stronger geometry assumption.

Theorem 1 (Convergence of FedGuide).

Average squared gradient norm decays with the initial surrogate gap divided by step size×rounds, plus a stochastic term and approximation floor. A step size proportional to inverse square root of rounds gives the usual nonconvex stationarity rate under the stated assumptions. Persistent proxy/value/drift error prevents convergence to an exact stationary point.

Equation 40
1H​∑h=0H−1𝔼​‖∇F~​(ψ(h),θ(h))‖2≤𝒪⁡(F~(0)−F~⋆ηpg​H)+𝒪⁡(ηpg​L​ςeff2N​U)+𝒪⁡(Δ).\frac{1}{H}\sum_{h=0}^{H-1}\mathbb{E}\bigl\|\nabla\tilde{F}(\psi^{(h)},\theta^{(h)})\bigr\|^{2}\leq\mathcal{O}\!\left(\frac{\tilde{F}^{(0)}-\tilde{F}^{\star}}{\eta_{\mathrm{pg}}H}\right)+\mathcal{O}\!\left(\eta_{\mathrm{pg}}L\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}\right)+\mathcal{O}\!\left(\Delta\right). (40)

Proof.

Apply smooth descent, split direction into true gradient, zero-mean noise and bounded bias, bound cross terms with Young’s inequality, then telescope across rounds. The proof depends on the assumed direction/noise model and sufficiently small step; it is not an independent derivation that the implemented transport/PPO loop realizes exact gradient descent.

Equations 41–51
x(h+1)=x(h)−ηpg​g(h).x^{(h+1)}=x^{(h)}-\eta_{\mathrm{pg}}g^{(h)}. (41)
F~​(x(h+1))\displaystyle\tilde{F}(x^{(h+1)}) ≤F~​(x(h))−ηpg​⟨∇F~​(x(h)),g(h)⟩+L​ηpg22​‖g(h)‖2.\displaystyle\leq\tilde{F}(x^{(h)})-\eta_{\mathrm{pg}}\left\langle\nabla\tilde{F}(x^{(h)}),g^{(h)}\right\rangle+\frac{L\eta_{\mathrm{pg}}^{2}}{2}\|g^{(h)}\|^{2}. (42)
𝔼⁡[F~​(x(h+1))∣ℱh]\displaystyle\mathbb{E}\!\left[\tilde{F}(x^{(h+1)})\mid\mathcal{F}_{h}\right] ≤F~​(x(h))−ηpg​‖∇F~​(x(h))‖2−ηpg​⟨∇F~​(x(h)),b(h)⟩\displaystyle\leq\tilde{F}(x^{(h)})-\eta_{\mathrm{pg}}\|\nabla\tilde{F}(x^{(h)})\|^{2}-\eta_{\mathrm{pg}}\left\langle\nabla\tilde{F}(x^{(h)}),b^{(h)}\right\rangle (43)
+L​ηpg22​𝔼​[‖∇F~​(x(h))+ξ(h)+b(h)‖2∣ℱh].\displaystyle\quad+\frac{L\eta_{\mathrm{pg}}^{2}}{2}\mathbb{E}\!\left[\|\nabla\tilde{F}(x^{(h)})+\xi^{(h)}+b^{(h)}\|^{2}\mid\mathcal{F}_{h}\right]. (44)
−⟨∇F~​(x(h)),b(h)⟩≤14​‖∇F~​(x(h))‖2+‖b(h)‖2,-\left\langle\nabla\tilde{F}(x^{(h)}),b^{(h)}\right\rangle\leq\frac{1}{4}\|\nabla\tilde{F}(x^{(h)})\|^{2}+\|b^{(h)}\|^{2}, (45)
‖u+v+w‖2≤3​‖u‖2+3​‖v‖2+3​‖w‖2.\|u+v+w\|^{2}\leq 3\|u\|^{2}+3\|v\|^{2}+3\|w\|^{2}. (46)
𝔼⁡[F~​(x(h+1))∣ℱh]\displaystyle\mathbb{E}\!\left[\tilde{F}(x^{(h+1)})\mid\mathcal{F}_{h}\right] ≤F~​(x(h))−(3​ηpg4−3​L​ηpg22)​‖∇F~​(x(h))‖2\displaystyle\leq\tilde{F}(x^{(h)})-\left(\frac{3\eta_{\mathrm{pg}}}{4}-\frac{3L\eta_{\mathrm{pg}}^{2}}{2}\right)\|\nabla\tilde{F}(x^{(h)})\|^{2} (47)
+C1​L​ηpg2​ςeff2N​U+C2​ηpg​Δ,\displaystyle\quad+C_{1}L\eta_{\mathrm{pg}}^{2}\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}+C_{2}\eta_{\mathrm{pg}}\Delta, (48)
ηpg4​𝔼​‖∇F~​(x(h))‖2≤𝔼​F~​(x(h))−𝔼​F~​(x(h+1))+C1​L​ηpg2​ςeff2N​U+C2​ηpg​Δ.\frac{\eta_{\mathrm{pg}}}{4}\mathbb{E}\|\nabla\tilde{F}(x^{(h)})\|^{2}\leq\mathbb{E}\tilde{F}(x^{(h)})-\mathbb{E}\tilde{F}(x^{(h+1)})+C_{1}L\eta_{\mathrm{pg}}^{2}\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}+C_{2}\eta_{\mathrm{pg}}\Delta. (49)
ηpg4​∑h=0H−1𝔼​‖∇F~​(x(h))‖2\displaystyle\frac{\eta_{\mathrm{pg}}}{4}\sum_{h=0}^{H-1}\mathbb{E}\|\nabla\tilde{F}(x^{(h)})\|^{2} ≤F~(0)−F~⋆+H​C1​L​ηpg2​ςeff2N​U+H​C2​ηpg​Δ.\displaystyle\leq\tilde{F}^{(0)}-\tilde{F}^{\star}+HC_{1}L\eta_{\mathrm{pg}}^{2}\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}+HC_{2}\eta_{\mathrm{pg}}\Delta. (50)
1H​∑h=0H−1𝔼​‖∇F~​(x(h))‖2≤𝒪⁡(F~(0)−F~⋆ηpg​H)+𝒪⁡(ηpg​L​ςeff2N​U)+𝒪⁡(Δ),\frac{1}{H}\sum_{h=0}^{H-1}\mathbb{E}\|\nabla\tilde{F}(x^{(h)})\|^{2}\leq\mathcal{O}\!\left(\frac{\tilde{F}^{(0)}-\tilde{F}^{\star}}{\eta_{\mathrm{pg}}H}\right)+\mathcal{O}\!\left(\eta_{\mathrm{pg}}L\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}\right)+\mathcal{O}(\Delta), (51)

Theorem 2 (Linear convergence under PL).

If the surrogate additionally satisfies a Polyak–Łojasiewicz inequality, expected objective error contracts geometrically toward a neighborhood. The neighborhood depends on stochastic variance and approximation/drift terms. This stronger condition is not established for the neural RL benchmark objectives.

Equations 52–53
F~​(x)−F~⋆≤cPL​‖∇F~​(x)‖2.\tilde{F}(x)-\tilde{F}^{\star}\leq c_{\mathrm{PL}}\|\nabla\tilde{F}(x)\|^{2}. (52)
𝔼⁡[F~​(x(h))−F~⋆]≤(1−μ)h​(F~(0)−F~⋆)+𝒪⁡(ηpg​L​ςeff2N​U+Δ),\mathbb{E}\!\left[\tilde{F}(x^{(h)})-\tilde{F}^{\star}\right]\leq(1-\mu)^{h}\left(\tilde{F}^{(0)}-\tilde{F}^{\star}\right)+\mathcal{O}\!\left(\frac{\eta_{\mathrm{pg}}L\varsigma_{\mathrm{eff}}^{2}}{NU}+\Delta\right), (53)

Proof.

Substitute the PL inequality into the one-step descent bound and unroll a linear recursion. The initial error shrinks geometrically, while accumulated noise/bias forms a nonzero floor. Reducing effective variance improves that floor only when the baseline assumptions hold.

Equations 54–61
Rh+1≤Rh−ηpg4​𝔼​‖∇F~​(x(h))‖2+C1​L​ηpg2​ςeff2N​U+C2​ηpg​Δ.R_{h+1}\leq R_{h}-\frac{\eta_{\mathrm{pg}}}{4}\mathbb{E}\|\nabla\tilde{F}(x^{(h)})\|^{2}+C_{1}L\eta_{\mathrm{pg}}^{2}\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}+C_{2}\eta_{\mathrm{pg}}\Delta. (54)
𝔼​‖∇F~​(x(h))‖2≥1cPL​𝔼​[F~​(x(h))−F~⋆]=1cPL​Rh.\mathbb{E}\|\nabla\tilde{F}(x^{(h)})\|^{2}\geq\frac{1}{c_{\mathrm{PL}}}\mathbb{E}\left[\tilde{F}(x^{(h)})-\tilde{F}^{\star}\right]=\frac{1}{c_{\mathrm{PL}}}R_{h}. (55)
Rh+1≤(1−ηpg4​cPL)​Rh+C1​L​ηpg2​ςeff2N​U+C2​ηpg​Δ.R_{h+1}\leq\left(1-\frac{\eta_{\mathrm{pg}}}{4c_{\mathrm{PL}}}\right)R_{h}+C_{1}L\eta_{\mathrm{pg}}^{2}\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}+C_{2}\eta_{\mathrm{pg}}\Delta. (56)
A=C1​L​ηpg2​ςeff2N​U+C2​ηpg​Δ.A=C_{1}L\eta_{\mathrm{pg}}^{2}\frac{\varsigma_{\mathrm{eff}}^{2}}{NU}+C_{2}\eta_{\mathrm{pg}}\Delta. (57)
Rh+1≤(1−μ)​Rh+A.R_{h+1}\leq(1-\mu)R_{h}+A. (58)
Rh\displaystyle R_{h} ≤(1−μ)h​R0+A​∑r=0h−1(1−μ)r\displaystyle\leq(1-\mu)^{h}R_{0}+A\sum_{r=0}^{h-1}(1-\mu)^{r} (59)
≤(1−μ)h​R0+Aμ.\displaystyle\leq(1-\mu)^{h}R_{0}+\frac{A}{\mu}. (60)
Aμ=𝒪⁡(ηpg​L​ςeff2N​U+Δ),\frac{A}{\mu}=\mathcal{O}\!\left(\frac{\eta_{\mathrm{pg}}L\varsigma_{\mathrm{eff}}^{2}}{NU}+\Delta\right), (61)

A.4 Policy Averaging under Prior-Regularized Alignment (FedGuide-A)

A separate theorem bounds the return effect of policy averaging in FedGuide-A when actual local-versus-averaged policy distributions remain close.

Theorem 3 (Controlled policy-averaging error).

Assume return is Lipschitz in expected action-distribution total variation. Define dispersion by client-weighted KL to the averaged policy. The average return gap is bounded by a constant times the square root of that dispersion. The theorem does not guarantee that parameter averaging produces small dispersion.

Equations 62–64
Γπ(h)=∑i=1Nζi𝔼s∼di(h)[DKL(πθi(h)(⋅∣s)∥π¯θ(h)(⋅∣s))],\Gamma_{\pi}^{(h)}=\sum_{i=1}^{N}\zeta_{i}\,\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{KL}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s)\;\|\;\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\right], (62)
|Ji(π)−Ji(π′)|≤LJ𝔼s∼di(h)[DTV(π(⋅∣s),π′(⋅∣s))].\left|J_{i}(\pi)-J_{i}(\pi^{\prime})\right|\leq L_{J}\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{TV}}\bigl(\pi(\cdot\mid s),\pi^{\prime}(\cdot\mid s)\bigr)\right]. (63)
|∑i=1Nζi​Ji​(πθi(h))−∑i=1Nζi​Ji​(π¯θ(h))|≤Cπ​Γπ(h),\left|\sum_{i=1}^{N}\zeta_{i}J_{i}(\pi_{\theta_{i}}^{(h)})-\sum_{i=1}^{N}\zeta_{i}J_{i}(\bar{\pi}_{\theta}^{(h)})\right|\leq C_{\pi}\sqrt{\Gamma_{\pi}^{(h)}}, (64)

Proof.

Use the return-Lipschitz assumption clientwise, convert total variation to square-root KL with Pinsker, and apply Jensen over states and client weights. Every original inequality and definition remains available.

Equations 65–73
|Ji(πθi(h))−Ji(π¯θ(h))|≤LJ𝔼s∼di(h)[DTV(πθi(h)(⋅∣s),π¯θ(h)(⋅∣s))].\left|J_{i}(\pi_{\theta_{i}}^{(h)})-J_{i}(\bar{\pi}_{\theta}^{(h)})\right|\leq L_{J}\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{TV}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s),\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\right]. (65)
DTV(πθi(h)(⋅∣s),π¯θ(h)(⋅∣s))≤12DKL(πθi(h)(⋅∣s)∥π¯θ(h)(⋅∣s)).D_{\mathrm{TV}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s),\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\leq\sqrt{\frac{1}{2}D_{\mathrm{KL}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s)\;\|\;\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)}. (66)
𝔼s∼di(h)[DTV(πθi(h)(⋅∣s),π¯θ(h)(⋅∣s))]\displaystyle\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{TV}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s),\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\right] ≤12𝔼s∼di(h)[DKL(πθi(h)(⋅∣s)∥π¯θ(h)(⋅∣s))].\displaystyle\leq\sqrt{\frac{1}{2}\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{KL}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s)\;\|\;\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\right]}. (67)
|∑i=1Nζi​Ji​(πθi(h))−∑i=1Nζi​Ji​(π¯θ(h))|\displaystyle\left|\sum_{i=1}^{N}\zeta_{i}J_{i}(\pi_{\theta_{i}}^{(h)})-\sum_{i=1}^{N}\zeta_{i}J_{i}(\bar{\pi}_{\theta}^{(h)})\right| (68)
≤∑i=1Nζi​|Ji​(πθi(h))−Ji​(π¯θ(h))|\displaystyle\quad\leq\sum_{i=1}^{N}\zeta_{i}\left|J_{i}(\pi_{\theta_{i}}^{(h)})-J_{i}(\bar{\pi}_{\theta}^{(h)})\right| (69)
≤LJ2​∑i=1Nζi​𝔼s∼di(h)[DKL(πθi(h)(⋅∣s)∥π¯θ(h)(⋅∣s))].\displaystyle\quad\leq\frac{L_{J}}{\sqrt{2}}\sum_{i=1}^{N}\zeta_{i}\sqrt{\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{KL}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s)\;\|\;\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\right]}. (70)
∑i=1Nζi​Ai\displaystyle\sum_{i=1}^{N}\zeta_{i}\sqrt{A_{i}} ≤∑i=1Nζi​Ai=Γπ(h),\displaystyle\leq\sqrt{\sum_{i=1}^{N}\zeta_{i}A_{i}}=\sqrt{\Gamma_{\pi}^{(h)}}, (71)
Ai=𝔼s∼di(h)[DKL(πθi(h)(⋅∣s)∥π¯θ(h)(⋅∣s))].A_{i}=\mathbb{E}_{s\sim d_{i}^{(h)}}\!\left[D_{\mathrm{KL}}\bigl(\pi_{\theta_{i}}^{(h)}(\cdot\mid s)\;\|\;\bar{\pi}_{\theta}^{(h)}(\cdot\mid s)\bigr)\right]. (72)
|∑i=1Nζi​Ji​(πθi(h))−∑i=1Nζi​Ji​(π¯θ(h))|≤LJ2​Γπ(h).\left|\sum_{i=1}^{N}\zeta_{i}J_{i}(\pi_{\theta_{i}}^{(h)})-\sum_{i=1}^{N}\zeta_{i}J_{i}(\bar{\pi}_{\theta}^{(h)})\right|\leq\frac{L_{J}}{\sqrt{2}}\sqrt{\Gamma_{\pi}^{(h)}}. (73)

Implication for FedGuide.

Prior alignment may make policy averaging less destructive when it actually reduces dispersion, explaining why FedGuide-A sometimes performs reasonably. This is conditional; its severe Reacher-Hard failure demonstrates that the required compatibility is not assured by using OT-MoE.

Appendix B Environmental Setup

Environment descriptions and fixed client metadata specify exactly what differs across clients, with harder versions increasing distribution mismatch. Shared observation/action interfaces are maintained within each benchmark family.

B.1 Environment Description

The benchmark progression moves from controlled action modes to target differences, contact/dynamics changes and distinct manipulation tasks. It tests transfer across predefined client distributions rather than open-ended differences in sensors or action semantics.

Bandit2D.

Bandit2D has four Gaussian reward peaks on the unit circle. Clients sample overlapping 120° annular sectors with radii 0.7–1.3, making their union a ring. The global behavior prior should recover that support, while locally rewarded policies favor peaks. Table 6 uses closed-form Gaussian priors for this toy case, so it is not a full learned-diffusion ablation.

Equation 74
r⁡(𝐱)=maxm∈{1,…,4}⁡exp⁡(−‖𝐱−𝐜m‖222​σ2),σ=0.2,r(\mathbf{x})=\max_{m\in\{1,\ldots,4\}}\exp\left(-\frac{\|\mathbf{x}-\mathbf{c}_{m}\|_{2}^{2}}{2\sigma^{2}}\right),\quad\sigma=0.2, (74)

Reacher.

Reacher controls a two-link planar arm with torque actions and distance/control rewards. Different goal regions and environment parameters change local occupancy while retaining the same interface. It isolates target/reward variation before adding more complex contact dynamics.

MuJoCo Locomotion.

Hopper, Walker2D and HalfCheetah vary mass, damping, friction, actuation and reward preferences within fixed nominal spaces. Contact-sensitive hopping, biped balance and planar running produce different gait distributions. These parameter variations do not establish transfer between arbitrary morphological topologies.

MetaWorld10.

MetaWorld assigns a distinct manipulation task to each client, changing objects, contact sequences, reward geometry and success conditions. Shared dimensions permit aggregation, but useful behavior overlap is weaker than in dynamics-only perturbations. This setting reveals the method’s task-semantic limitation.

Client and Expert Configuration.

Use four clients/experts in Bandit2D, eight/eight for Reacher and locomotion, and ten clients/eight experts for MetaWorld. Each client owns its policy, online critic and frozen DICE value. Standard communication carries prior heads only; FedGuide-A adds policy synchronization as an explicit ablation.

B.2 Heterogeneity Setup

Fixed metadata gives every method the same client assignments and local MDP variants. Trajectories and buffers are not exchanged. The following ranges describe the evaluated split, not universal distributions over all possible robots.

Bandit2D.

Four clients receive 1,000 offline samples each around unit-circle centers, with Gaussian reward width 0.2 and area-uniform annular radii 0.7–1.3. The authors describe adjacent support overlap as 30%. Ring-shaped global support is built into this sampling construction.

Reacher.

Eight Reacher goal sectors are spaced every 45°, around radius 0.18 with radial width 0.06 and angular width 30°. Fixed action offsets, reward scales and initial-angle variation add heterogeneity: offset norms range 0.265–1.173 and reward scales 0.621–1.332.

Hopper and Walker2D.

Hopper/Walker2D cycle nominal, heavy, weak-actuator and high-damping/friction presets with jitter, plus speed/efficiency/stability rewards. Evaluated mass scales are 0.972–1.462, damping 0.779–1.367, friction 0.887–1.267 and action gain 0.818–1.002. Reward and instability settings vary independently as listed in the source.

HalfCheetah.

HalfCheetah uses similar preset categories with narrower main mass variation (0.959–1.097), action gains 0.850–1.023 and task-specific control costs 0.0899–0.1232. Reset noise also varies. The main split has no instability penalty, unlike its Hard version.

MetaWorld10.

The ten clients are reach, push, pick-place, door-open, drawer-close, button-press-topdown, peg-insert-side, window-open, sweep and basketball v3 tasks. Heatmaps for other environments normalize numerical metadata within each environment; MetaWorld heatmaps encode semantic task attributes. Their colors therefore cannot be compared as common physical units.

Refer to caption
Figure 5: Client-level heterogeneity heatmaps for the main benchmark settings. Hopper, Walker2D, and HalfCheetah show normalized dynamics and reward metadata; Reacher shows normalized goal, action-noise, reward, and angle-shift metadata; MetaWorld10 shows task-semantic attributes.

B.3 Heterogeneity Setup for Stronger Environments

Hard settings widen or separate dynamics/reward regimes to increase disagreement among clients. They are heterogeneous-client stress tests, not a uniform increase in every individual task’s difficulty.

Reacher-Hard.

Reacher-Hard retains goal sectors while adding default/speed/stability/efficiency preferences and strong structural scaling. Mass spans 0.598–1.955, damping 0.568–1.923 and actuator gear 0.587–1.954. This produces the notable failure of the policy-averaging variant despite useful local-prior variants.

Hopper-Hard and Walker2D-Hard.

Hopper/Walker2D-Hard use eight regimes, including very-heavy, weak-actuator, extreme damping/friction and low-friction combinations. Sprint, miser, tightrope and explorer rewards separate speed, effort and stability preferences. Mass spans 0.862–1.698, friction 0.417–1.476 and action gain 0.620–0.902; the full metadata plots preserve other ranges.

HalfCheetah-Hard.

HalfCheetah-Hard broadens actuation/friction/reward/reset variation and adds an instability penalty of 0.0121–0.0352. Action gains span 0.670–1.066 and friction 0.851–1.376. Main/Hard violin plots normalize each metadata dimension using its pooled range, so they compare spread within dimensions rather than absolute units across dimensions.

Refer to caption
Figure 6: Main-to-Hard metadata range expansion. For each metadata dimension, client values are normalized by the pooled Main and Hard range and plotted as side-by-side Main and Hard distributions.

Appendix C Baseline Details

Baseline descriptions specify what is communicated and what remains local. Interaction budgets, client splits, warm-up and evaluation are shared, while Table 9 documents optimizer and epoch differences. The tables’ guarantee labels refer to distinct methods/assumptions and should not be read as one directly comparable theorem.

FedAvg.

FedAvg broadcasts a common policy, runs local PPO and averages updated policy parameters with sample-count weights; equal rollout budgets give uniform weights here. It has no behavior prior, expert routing or DICE baseline. Its failure modes reflect this entire learning setup, not parameter averaging in isolation.

Equation 75
θ¯(h+1)=∑i=1Nζi​θi(h+1),ζi=ni∑j=1Nnj,\bar{\theta}^{(h+1)}=\sum_{i=1}^{N}\zeta_{i}\theta_{i}^{(h+1)},\quad\zeta_{i}=\frac{n_{i}}{\sum_{j=1}^{N}n_{j}}, (75)

FedKL.

FedKL uses a shared global policy plus penalties for divergence from the global policy and from the previous local update. The experiment instantiates it with PPO. Output-space KL limits drift, but maintaining one aggregated policy can still conflict with distinct client behaviors.

Equation 76
ℒiFedKL(θi)=ℒiPPO(θi)+λgD^KL(πθi(⋅|s)∥πθ¯(h)(⋅|s))+λlD^KL(πθi(⋅|s)∥πθiold(⋅|s)).\mathcal{L}_{i}^{\mathrm{FedKL}}(\theta_{i})=\mathcal{L}_{i}^{\mathrm{PPO}}(\theta_{i})+\lambda_{\mathrm{g}}\widehat{D}_{\mathrm{KL}}\left(\pi_{\theta_{i}}(\cdot|s)\,\|\,\pi_{\bar{\theta}^{(h)}}(\cdot|s)\right)+\lambda_{\mathrm{l}}\widehat{D}_{\mathrm{KL}}\left(\pi_{\theta_{i}}(\cdot|s)\,\|\,\pi_{\theta_{i}^{\mathrm{old}}}(\cdot|s)\right). (76)
Method Heterogeneity    Type Aggregation    Target Alignment    Space Multi-Modal    Support Local    Policy Return-Aware
Baseline
Baselines FedAvg [23] Data Policy Params. Parameter Space ✗ ✗ ✗
FedKL [44] Data Policy Params. Policy Output ✗ ✗ P
FedRL [13] MDP / Dynamics Actor + Critic Parameter Space ✗ ✗ P
FedSVRPG-M [37] MDP / Dynamics Policy Grad. Gradient Space ✗ ✗ P
Variants FedGuide-P Data / MDP / Task Diffusion Prior Distribution Space ✓ ✓ ✗
FedGuide-A Data / MDP / Task Prior + Policy Distribution Space ✓ ✗ ✓
FedGuide Data / MDP / Task Diffusion Prior Distribution Space ✓ ✓ ✓
Table 4: Comparison of the baselines and FedGuide variants used in our experiments. FedAvg aggregates policy parameters, FedKL regularizes local policies through KL penalties in the policy-output space, FedRL aggregates actor-critic models, and FedSVRPG-M aggregates variance-reduced policy-gradient updates. FedGuide instead aggregates diffusion behavior priors in distribution space while keeping each client’s policy local. FedGuide-P removes the DICE value baseline, and FedGuide-A additionally averages client policies. Here, ✓ denotes explicit support, ✗ denotes no explicit support, and P denotes partial or indirect support.
Method Convergence Communication    Speedup Local Updates    Allowed Distribution-Level    Analysis
Baselines FedAvg [23] FL convergence Empirical ✓ ✗
FedKL [44] Asymptotic Not claimed P Policy-output KL
FedRL [13] Finite to subopt. Not claimed ✓ ✗
FedSVRPG-M [37] Variance-reduced PG Claimed ✓ Policy-gradient correction
Variants FedGuide-P Prior-only ablation Not isolated ✓ OT-MoE diffusion prior
FedGuide-A Controlled avg. error Not isolated ✓ OT-MoE prior + policy avg.
FedGuide Finite to controlled error Variance-controlled ✓ OT-MoE diffusion prior
Table 5: Theoretical and mechanism-level comparison of the baselines and FedGuide variants. FedAvg provides the standard federated policy averaging baseline, FedKL provides convergence analysis for KL-regularized policy-output alignment, FedRL analyzes federated RL under environment heterogeneity with convergence to a heterogeneity dependent suboptimal solution, and FedSVRPG-M uses momentum-based variance-reduced policy-gradient correction for efficient heterogeneous FRL. FedGuide keeps policies local and aggregates only diffusion-prior heads through OT-MoE; its analysis separates the effective stochastic variance, OT errors, prior or value surrogate errors, and local KL drift, while the DICE value baseline reduces the variance of local policy updates.

FedRL.

FedRL is the continuous-control DDPGAvg-style extension: clients retain local replay buffers, train actor/critic networks and average both at round boundaries. Its off-policy backbone differs from the PPO methods, so early learning and replay coverage confound a pure communication-rule comparison.

Equation 77
θ¯μ,(h+1)=∑i=1Nζi​θiμ,(h+1),θ¯Q,(h+1)=∑i=1Nζi​θiQ,(h+1).\bar{\theta}^{\mu,(h+1)}=\sum_{i=1}^{N}\zeta_{i}\theta_{i}^{\mu,(h+1)},\quad\bar{\theta}^{Q,(h+1)}=\sum_{i=1}^{N}\zeta_{i}\theta_{i}^{Q,(h+1)}. (77)

FedSVRPG-M.

FedSVRPG-M shares momentum/variance-reduced update directions around a global policy. The tested implementation uses the paper’s practical PPO/SVRPG gradient path, rather than claiming an unchanged reference implementation. Table 3 marks it as implemented by the authors. It shares the warm-up and rollout protocol but no multimodal behavior priors.

Parameter Symbol Bandit2D Reacher MuJoCo locomotion MetaWorld10
Offline buffer size |𝒟i||\mathcal{D}_{i}| 10310^{3} 2×1042{\times}10^{4} 2×1052{\times}10^{5} cap 5×1035{\times}10^{3}
Offline source – local modes client rollouts D4RL scripted policy
Diffusion steps KK Gaussian 10001000 10001000 10001000
Prior width / horizon dD,LDd_{D},L_{D} Gaussian 64,6464,64 64,6464,64 64,6464,64
Prior epochs / batch ED,BDE_{D},B_{D} closed form 40,51240,512 40,51240,512 40,51240,512
Prior learning rate ηD\eta_{D} – 10−410^{-4} 10−410^{-4} 10−410^{-4}
DICE width dϕd_{\phi} 256256 256256 256256 256256
DICE learning rates ηϕ,ηw\eta_{\phi},\eta_{w} 10−4,10−410^{-4},10^{-4} 10−4,10−410^{-4},10^{-4} 10−4,10−410^{-4},10^{-4} 10−4,10−410^{-4},10^{-4}
DICE temperature ω\omega 0.50.5 0.50.5 0.50.5 0.50.5
DICE warm-up epochs EϕE_{\phi} 100100 8080 8080 8080
BC epochs / batch EBC,BBCE_{\mathrm{BC}},B_{\mathrm{BC}} prior init 40,51240,512 40,51240,512 40,51240,512
BC width / learning rate dBC,ηBCd_{\mathrm{BC}},\eta_{\mathrm{BC}} prior init 256,3×10−4256,3{\times}10^{-4} 256,3×10−4256,3{\times}10^{-4} 256,3×10−4256,3{\times}10^{-4}
Table 6: Offline pretraining hyperparameters.

FedGuide-P.

FedGuide-P keeps transport/expert prior sharing and fully local policies while setting beta=1. It isolates removal of the DICE baseline within the FedGuide implementation and sometimes outperforms the full method, notably on Reacher-Hard.

Equation 78
λ2​ℛ^prior​(θi,π¯D,i).\lambda_{2}\widehat{\mathcal{R}}_{\mathrm{prior}}(\theta_{i};\bar{\pi}_{D,i}). (78)

FedGuide-A.

FedGuide-A adds server-side policy averaging to otherwise prior-regularized, DICE-blended local learning. The hyperparameter table synchronizes policies every five rounds. It tests whether aligned behavior support makes averaging tolerable; its failures show that personalized adaptation can still be lost.

Equation 79
θ¯(h+1)=∑i=1Nζi​θi(h+1).\bar{\theta}^{(h+1)}=\sum_{i=1}^{N}\zeta_{i}\theta_{i}^{(h+1)}. (79)

FedGuide.

Standard FedGuide communicates only prior heads, receives a personalized mixture and learns policy/online value locally with a frozen DICE baseline. Distribution sharing and local value blending are distinct components; neither involves sharing raw replay data or evaluating sampled actions with DICE Q values.

Equation 80
𝐕i​(s)=β​Vθ,i​(s)+(1−β)​Vϕ,i​(s).\mathbf{V}_{i}(s)=\beta V_{\theta,i}(s)+(1-\beta)V_{\phi,i}(s). (80)

Appendix D Hyperparameters

All offline, online, variant and baseline hyperparameters are retained. These tables also expose special cases and resource differences that qualify the main comparisons.

D.1 Offline Pretraining

Offline buffers contain 1,000 local-mode samples for Bandit2D, 20,000 Reacher client transitions, up to 200,000 D4RL locomotion samples and 5,000 MetaWorld scripted-policy samples. Bandit priors are closed-form Gaussian; other priors use 1,000 diffusion steps, 64-width/horizon settings and 40 epochs. DICE has width 256, temperature 0.5 and 100/80 warm-up epochs, then stays frozen. Behavior-cloning warm-up is shared by all methods, while prior/DICE pretraining adds method-specific work.

D.2 Online FedGuide Settings

Online FedGuide uses 60 bandit or 100 other rounds, with 200/2,048/4,096 samples per client round by domain and four local PPO epochs. Defaults are trust-region weight 0.05, prior weight 0.5 and Sinkhorn regularization 0.05. Prior guidance anneals over 60 bandit or 50 other rounds. Evaluation uses ten episodes outside the toy bandit. The tables preserve action ranges and exploration initialization.

Parameter Symbol Bandit2D Reacher MuJoCo locomotion MetaWorld10
Clients NN 44 88 88 1010
Prior experts MM 44 88 88 88
Federated rounds HH 6060 100100 100100 100100
Rollout size / round TrollT_{\mathrm{roll}} 200200 20482048 40964096 40964096
Local PPO epochs UU 44 44 44 44
PPO mini-batch BB 6464 6464 6464 6464
Trust-region weight λ1\lambda_{1} 0.050.05 0.050.05 0.050.05 0.050.05
Prior regularizer weight λ2\lambda_{2} 0.50.5 0.50.5 0.50.5 0.50.5
Prior annealing horizon Hλ2H_{\lambda_{2}} 6060 5050 5050 5050
Sinkhorn regularization η\eta 0.050.05 0.050.05 0.050.05 0.050.05
Initial log std. log⁡σ0\log\sigma_{0} −1-1 00 00 00
Evaluation episodes nevaln_{\mathrm{eval}} grid / return 1010 1010 1010
Action range 𝒜\mathcal{A} [−1.5,1.5][-1.5,1.5] [−1,1][-1,1] [−1,1][-1,1] [−1,1][-1,1]
Table 7: Shared online hyperparameters for FedGuide variants.

D.3 FedGuide Variants

All three variants share priors, data and local PPO. FedGuide and FedGuide-A use beta=0.5; FedGuide-P uses beta=1. Only FedGuide-A averages policy parameters, every five rounds. These exact settings take precedence over optional schedules discussed in prose.

Method Prior aggregation Value baseline Policy aggregation Key coefficients
FedGuide OT-MoE on {ψi}i=1N\{\psi_{i}\}_{i=1}^{N} β=0.5\beta=0.5 none λ1=0.05\lambda_{1}=0.05, λ2=0.5\lambda_{2}=0.5, η=0.05\eta=0.05
FedGuide-P OT-MoE on {ψi}i=1N\{\psi_{i}\}_{i=1}^{N} β=1\beta=1 none λ1=0.05\lambda_{1}=0.05, λ2=0.5\lambda_{2}=0.5, η=0.05\eta=0.05
FedGuide-A OT-MoE on {ψi}i=1N\{\psi_{i}\}_{i=1}^{N} β=0.5\beta=0.5 every 5 rounds λ1=0.05\lambda_{1}=0.05, λ2=0.5\lambda_{2}=0.5, η=0.05\eta=0.05
Table 8: FedGuide variant hyperparameters.

D.4 Baseline Hyperparameters

FedAvg/FedKL use ten local PPO epochs versus four for FedGuide and the PPO/SVRPG baseline; FedRL uses DDPG with replay capacity 100,000 and warm-up 1,000. Shared interaction budgets therefore do not imply equal gradient work or wall time. Table 9 preserves KL, clipping, learning-rate, momentum and exploration settings; end-to-end pretraining/communication cost is not quantified by return curves.

Parameter Symbol FedAvg FedKL FedRL FedSVRPG-M
Local optimizer – PPO PPO DDPG PPO/SVRPG
Aggregated parameters – θi\theta_{i} θi\theta_{i} θiμ,θiQ\theta_{i}^{\mu},\theta_{i}^{Q} Δ​θi\Delta\theta_{i}
Global KL weight λg\lambda_{\mathrm{g}} 00 0.10.1 – –
Local KL weight λl\lambda_{\mathrm{l}} 00 0.050.05 – –
Local PPO epochs UU 1010 1010 – 44
Mini-batch size BB 6464 6464 6464 6464
Discount factor γ\gamma 0.990.99 0.990.99 0.990.99 0.990.99
GAE coefficient λGAE\lambda_{\mathrm{GAE}} 0.950.95 0.950.95 – 0.950.95
PPO clipping radius ϵ\epsilon {0.1,0.2}\{0.1,0.2\} {0.1,0.2}\{0.1,0.2\} – {0.1,0.2}\{0.1,0.2\}
Value loss weight cVc_{V} 0.50.5 0.50.5 – 0.50.5
Gradient clipping cgradc_{\mathrm{grad}} 0.50.5 0.50.5 – 0.50.5
Learning rate ηopt\eta_{\mathrm{opt}} 10−410^{-4} 10−410^{-4} 10−410^{-4} 10−410^{-4}
Reference update freq. KrefK_{\mathrm{ref}} – – – 55
Momentum weight βmom\beta_{\mathrm{mom}} – – – {0.2,0.5}\{0.2,0.5\}
BC initialization weight αBC\alpha_{\mathrm{BC}} 0.30.3 0.30.3 0.30.3 0.30.3
Replay capacity RR – – 10510^{5} –
Replay warm-up R0R_{0} – – 10310^{3} –
Target update coefficient τDDPG\tau_{\mathrm{DDPG}} – – 0.0010.001 –
Exploration noise (μOU,θOU,σOU)(\mu_{\mathrm{OU}},\theta_{\mathrm{OU}},\sigma_{\mathrm{OU}}) – – (0,0.15,0.2)(0,0.15,0.2) –
Table 9: Baseline hyperparameters.

Appendix E Additional Analysis

Additional tests isolate expert count, transport softness, prior weight and missing expert support using the controlled bandit. They clarify operating conditions rather than proving robustness across arbitrary client populations.

E.1 Sensitivity to M , η , and λ 2

Five-seed, four-client Bandit2D sweeps vary expert count, Sinkhorn temperature and prior weight while the true mode count is known. This is a diagnostic setting with the toy prior construction described in Table 6.

Original figure

(a) Expert count MM

Original figure

(b) Sinkhorn η\eta

Original figure

(c) Prior weight λ2\lambda_{2}

Figure 7: Bandit2D sensitivity to the expert count MM, the Sinkhorn temperature η\eta, and the prior weight λ2\lambda_{2}.

Expert count M .

One or two experts underrepresent four modes and learn more slowly; four is most sample-efficient, while larger banks are stable but redundant in this test. The result motivates matching effective behavior modes, not necessarily client count. Online estimation and adaptive resizing remain unsolved.

Sinkhorn temperature η .

Temperature changes assignment softness between near-hard routing and broader mixtures. The tested range does not change the reported qualitative ordering, but this limited sweep is not a guarantee that arbitrary temperatures preserve useful client specialization.

Prior weight λ 2 .

Zero prior weight removes shared-prior guidance; positive weights become less different late in training because the implementation anneals guidance away. The local DICE baseline can still remain, so zero prior weight is not necessarily ordinary PPO. Shared behavior-cloning warm-up cannot by itself explain relative method differences.

E.2 Behavior under Weak Overlap

A fifth client targets the origin while only the four ring clients update server experts; the new client participates in routing only. Equidistance produces a near-uniform mixture on the ring, without creating the absent center mode. This deliberately excludes learning the fifth mode into the bank. The result demonstrates recombination of represented supports, and why missing support can make transferred guidance uninformative or mismatched.

Figure 8: Weak-overlap stress test. Client 5 has a true target mode at the origin, while the four experts contain only ring-shaped behavior support. OT-MoE routes Client 5 to a mixture of the existing experts, keeping its personalized prior on the ring rather than creating the unseen center mode.
Bandit2D Ground Truth OT-MoE Global Prior Client 1 Client 2 Client 3 Client 4 Client 5 Overall

Appendix F Visualizations

All five complete rollout grids retain every client/method panel. Rows compare FedGuide-A, FedGuide-P and FedGuide; each cell overlays frames from a single final-round episode. Visualizations select the best of five seeds by final return, whereas numerical results average seeds. These selected trajectories illustrate motion and contact behavior and are not independent success-rate evidence.

Refer to caption
Figure 9: FedGuide-family qualitative rollouts on Reacher, visualized with motion from a single final-round evaluation episode.
Refer to caption
Figure 10: FedGuide-family qualitative rollouts on Hopper. Each cell overlays sampled frames from the final-round rollout to show the learned motion trace.
Refer to caption
Figure 11: FedGuide-family qualitative rollouts on Walker2D, visualized with motion across heterogeneous clients.
Refer to caption
Figure 12: FedGuide-family qualitative rollouts on HalfCheetah, visualized with motion overlays across the first eight clients.
Refer to caption
Figure 13: FedGuide-family qualitative rollouts on MetaWorld10, visualized with across the ten task clients.
Concept demonstration

Open image ↗