Overview
Federated reinforcement learning through personalized behavior priors: local policies and critics learn with shared diffusion-prior experts, with all original results, proofs and rollout grids retained.
At a glance
FedGuide exchanges behavior-prior heads among clients with different environments, routes them into server experts using optimal transport, and returns a personalized mixture to each client. Local PPO combines prior regularization with a frozen DICE value baseline. Simulated benchmarks show higher returns in many settings, with important exceptions and weak-overlap limits. Policies and raw trajectories remain local in standard FedGuide, but this is not a formal privacy guarantee.
1 Introduction
Heterogeneous clients induce different trajectory distributions, so averaging their policy parameters can erase useful local behavior. FedGuide instead communicates diffusion behavior priors, preserves local policies, and uses a local offline value estimate to stabilize advantage estimation. The novelty is this federated combination of existing diffusion, transport and DICE components. The analysis gives conditional bias/variance and approximate-stationarity statements; experiments test simulation return, not physical multi-robot deployment.
2 Related Work
Prior work addresses federated heterogeneity through parameter sharing, policy-divergence control, update correction and personalized experts. FedGuide changes the collaboration interface to behavior-prior models while keeping policy optimization local.
Heterogeneous Federated Reinforcement Learning.
Federated RL adds policy-induced data shift to ordinary non-IID learning. Methods such as FedKL, variance-reduced gradient sharing and federated Q-learning stabilize shared policies or updates. Distribution-aware supervised MoE/transport methods motivate preserving multiple modes, but their static-data setting differs from evolving client trajectories.
Diffusion Priors and Value-Guided Policy Learning.
Diffusion priors model data-supported actions without inherently preferring high return. DICE supplies offline value/distribution-correction information. Here its value stays local and enters only as a state baseline: it neither scores sampled actions directly nor guides the diffusion sampler. This distinction prevents reading “value guidance” as classifier-style diffusion guidance.
3 Preliminaries
Notation separates diffusion step, environment time, communication round, local update, client and server expert. Policy parameters, diffusion-prior parameters and DICE value parameters are distinct objects.
3.1 Problem Formulation
Clients share state/action interfaces but may differ in transition dynamics, rewards and initial-state distributions. The objective is the average discounted return of personalized local policies. Shared interfaces and meaningful behavior-support overlap are substantive assumptions; arbitrary incompatible robot embodiments are not covered.
Equations 1–2
| (1) |
| (2) |
3.2 Diffusion Priors and DICE Value Functions
The two offline components serve different roles: a conditional behavior distribution constrains plausible actions, while a state-only value function contributes return information to advantage centering.
Diffusion prior as behavior model.
A state-conditioned diffusion model learns noisy-action scores through denoising score matching, then defines a clean-action behavior distribution. It captures the offline buffer’s action support irrespective of reward. The full forward process and score definitions remain available. Behavior modeling alone is not policy improvement.
Equation 3
| (3) |
DICE value function.
DICE estimates discounted occupancy corrections using offline value/Q functions and a divergence-regularized objective, drawing on DualDICE and OptiDICE. After pretraining, only the frozen state value is used online. This avoids directly evaluating unseen actions with an offline Q function, but a stale or inaccurate state value can still bias bootstrapped GAE and fail to reduce variance.
Equations 4–6
| (4) |
| (5) |
| (6) | |||
4 Methodology
Server-side transport routes prior heads into multiple experts; clients receive personalized mixture distributions. Local policy updates then combine prior regularization, a trust-region term and a blended online/offline value baseline.
4.1 OT-MoE Aggregation of the Diffusion Priors
The ideal transport cost compares client/expert action scores; implementation uses a parameter-space proxy. Entropic Sinkhorn transport uses uniform client/expert marginals. Each expert head is refreshed by a column-normalized weighted average of client head parameters, then each client receives a row-weighted mixture of expert distributions. Thus the method still averages prior-head parameters; its distinction is preserving multiple expert distributions and local policies. The prior cross-entropy is estimated with self-normalized importance weights over rollout samples, introducing approximation error. Proxy-cost quality and within-expert averaging are not automatic guarantees of mode preservation.
Equations 7–11
| (7) |
| (8) |
| (9) |
| (10) |
| (11) |
4.2 DICE Value Baseline Advantage Blending
Blend the online critic with the frozen DICE value using beta, then form GAE and optimize a clipped PPO surrogate with prior regularization. Default beta=0.5 gives equal value weights; beta=1 removes DICE. A state-only baseline has zero direct score-function expectation, but bootstrapping, clipping and approximate prior weights still introduce errors outside that identity.
Equations 12–13
| (12) |
| (13) |
Equation 14
| (14) |
4.3 Putting Everything Together: FedGuide
Each round broadcasts personalized priors, collects local prior heads and updates transport/expert parameters. Policy and both value models remain local in standard FedGuide. Local head adaptation is optional in Algorithm 2; the pretrained head may be retained. FedGuide-P removes DICE; FedGuide-A additionally averages policies every five rounds. The native algorithms preserve the source’s colored variant-only lines and exact indentation. A possible beta schedule is discussed, while the reported default table uses fixed beta=0.5.
5 Theoretical Analysis
The convergence analysis assumes a smooth surrogate, bounded gradient noise, bounded local KL drift, informative bounded values, and bounded transport/proxy/prior/value approximation errors. It describes an idealized stochastic descent direction under these conditions, not a proof that every neural PPO run satisfies them.
Lemma 1 (State-only baseline does not add score function bias).
Subtracting a state-only value does not add an action-dependent score-function term: its expected contribution is zero under the current policy. This identity does not establish that finite-horizon GAE with an imperfect baseline, PPO clipping or importance approximations is completely unbiased.
Equation 15
| (15) |
Lemma 2 (Variance reduction).
Variance can fall when the offline-minus-online value correction correlates positively with the online-baseline return residual and the blending coefficient lies in the derived interval. Positive correlation alone does not guarantee improvement for the fixed beta=0.5. The proof analyzes the return/advantage control variate; effective gradient-noise reduction enters the convergence assumptions separately.
Theorem 1 (Convergence of FedGuide).
With a sufficiently small policy step, the average squared surrogate-gradient norm is bounded by initial suboptimality over rounds, a stochastic term scaling as effective variance/(clients×local updates), and a residual approximation/drift floor. Local policies avoid an explicit policy-averaging heterogeneity term, but heterogeneity still affects prior costs, overlap and approximation assumptions. The result concerns stationarity of the surrogate, not global optimal return.
Equation 16
| (16) |
6 Experimental Analysis
Experiments ask whether distribution sharing improves average client return, survives harder heterogeneity and benefits from both the prior and value baseline. All evaluations are simulated, with common client splits and five random seeds.
| Variants | Diffusion Prior | DICE Value | Client Policy |
|---|---|---|---|
| FG-A | OT-MoE | Local | Avg. |
| FG-P | OT-MoE | N/A | Local |
| FG | OT-MoE | Local | Local |
| Env | FG-A | FG-P | FG |
|---|---|---|---|
| Reacher | 0.380 | 0.068 | 0.053 |
| Hopper | 0.402 | 0.164 | 0.155 |
| Walker2D | 0.208 | 0.179 | 0.127 |
| HalfCheetah | 0.628 | 0.235 | 0.241 |
| MetaWorld10 | 0.364 | 0.272 | 0.260 |
| Reacher-Hard | 2.097 | 0.229 | 0.342 |
| Hopper-Hard | 0.377 | 0.185 | 0.133 |
| Walker2D-Hard | 0.178 | 0.155 | 0.088 |
| HalfCheetah-Hard | 0.591 | 0.231 | 0.201 |
Experimental Setup.
Benchmarks include a four-mode 2-D bandit, Reacher, three MuJoCo locomotion tasks and ten MetaWorld manipulation tasks. Reacher/locomotion use eight clients; MetaWorld uses ten clients and eight experts. Baselines share interaction/evaluation settings and behavior-cloning warm-up, but use different optimizers and local epoch counts; this is not a matched total-compute comparison. FedRL is a DDPGAvg-style actor-critic and FedSVRPG-M a practical PPO/SVRPG implementation.

| Environment | Metric | FedAvg | FedKL | FedRL | FedSVRPG-M† | FedGuide-A | FedGuide-P | FedGuide |
| Reacher | -19.41.82 | -20.91.63 | -13.41.16 | -24.93.46 | -16.10.48 | -12.20.35 | -12.30.67 | |
| -21.4 | -22.3 | -14.4 | -29.8 | -16.8 | -12.6 | -13.0 | ||
| Hopper | 22742.5 | 25348.1 | 527120 | 525360 | 48126.5 | 50719.3 | 63219.0 | |
| 180 | 207 | 380 | 166 | 451 | 481 | 614 | ||
| Walker2D | 28327.4 | 26521.4 | 360113 | 156113 | 42231.5 | 38318.7 | 48018.6 | |
| 252 | 250 | 219 | 21.3 | 377 | 353 | 461 | ||
| HalfCheetah | 60092.1 | 95090.7 | 579145 | -42.2393 | 1439176 | 1768500 | 1994304 | |
| 486 | 821 | 421 | -409 | 1246 | 978 | 1681 | ||
| MetaWorld10 | 631238 | 445213 | 76753.6 | 668102 | 82982.5 | 91458.2 | 939141 | |
| 248 | 313 | 724 | 547 | 744 | 816 | 716 | ||
| Reacher-Hard | -25311823 | -42742070 | -90.24.97 | -83396580 | -1995128 | 90.715.7 | 86.537.6 | |
| -5210 | -7118 | -94.8 | -16456 | -2148 | 67.4 | 37.6 | ||
| Hopper-Hard | 21620.1 | 24241.0 | 699256 | 387285 | 39810.9 | 43751.6 | 51044.0 | |
| 187 | 197 | 404 | 169 | 384 | 383 | 445 | ||
| Walker2D-Hard | 25613.2 | 28624.3 | 32157.8 | 57.824.5 | 38010.9 | 36125.3 | 41514.5 | |
| 241 | 258 | 258 | 27.3 | 368 | 327 | 396 | ||
| HalfCheetah-Hard | 231451 | 373110 | 832349 | 96.1251 | 1272203 | 1519327 | 2146304 | |
| -544 | 217 | 328 | -282 | 928 | 1168 | 1893 |
6.1 Experimental Results
The result tables distinguish final-round return, worst-round client-average return and late-training temporal volatility. None is a worst-client guarantee or a measured task-success rate.
Performance for Heterogeneous FRL.
FedGuide’s final returns are 632 on Hopper, 480 on Walker2D, 1,994 on HalfCheetah and 939 on MetaWorld10, exceeding the compared methods’ means there. FedGuide-P is marginally better on ordinary Reacher (−12.2 versus −12.3). MetaWorld gains are modest and variable; FedGuide’s worst-round value 716 is below FedGuide-P’s 816 and FedRL’s 724 despite its best final mean. Table 3 retains all means/spreads and exceptions; return should not be interpreted as manipulation success rate.
Discussion of Robustness.
Under harder heterogeneity, FedGuide leads final Walker2D/ HalfCheetah returns, while FedRL leads Hopper-Hard (699 versus 510). Reacher-Hard exposes a sharp variant difference: FedGuide-P and FedGuide reach about 90.7/86.5, but policy-averaging FedGuide-A collapses to −1,995. Thus the whole family is not uniformly robust. These benchmark comparisons vary client counts across tasks, rather than constituting a controlled scaling study in client number.
Why Diffusion Prior and DICE Value Baseline?
Bandit plots show that policy averaging can lose reward modes while shared priors preserve the union of represented supports. The ring is prescribed by overlapping annular training sectors; it is not evidence of discovering unsupported behavior. Table 2 measures temporal variation of return over the last twenty rounds, not policy-gradient variance or the residual correlations assumed by Lemma 2. DICE usually helps this metric, but FedGuide-P has lower volatility on Reacher-Hard.
7 Limitations
No physical robots, perception noise, actuation latency or uncertain real contacts are evaluated. Keeping trajectories and policies local does not establish privacy against model-update leakage. Shared state/action spaces, useful prior overlap and manageable client/expert counts are required. Severe task-semantic or reward shifts can make guidance uninformative, and adaptive expert-count selection remains open.
8 Conclusions
Sharing multiple behavior-prior experts can improve heterogeneous local RL without synchronizing every policy. The experiments support benefits on the tested simulations and expose limits when useful support is absent. Extension to task-semantic representations and physical multi-robot systems remains future work.
Acknowledgments
The work acknowledges NSF/ONR funding and Bridges-2 computing through the ACCESS program. These resources are separate from the algorithmic communication and interaction budgets.
Appendix Contents
The appendices contain all theoretical assumptions/proofs, benchmark metadata, baseline mechanisms, hyperparameters, sensitivity tests and complete qualitative rollout grids.
Appendix A Theoretical Proofs
The theory justifies a state-only baseline, controlled-error transport aggregation and keeping policies local. It explicitly targets approximate stationarity and conditional error bounds, not global optimality of heterogeneous RL.
A.1 Assumptions
Assume smoothness/Lipschitz prior scores, bounded stochastic gradients and self-normalized-prior bias, bounded KL drift, bounded informative DICE values, and bounded Sinkhorn/cost-proxy errors. The descent-direction decomposition assumes zero-mean noise and a bias bounded by the sum of approximation terms. Its 1/(NU) variance scaling is assumed after client/update averaging, rather than derived from arbitrary correlated local trajectories.
Equations 17–21
| (17) |
| (18) |
| (19) |
| (20) |
| (21) |
On the role of assumptions.
These are regularity and approximation conditions. The implementation’s trust-region control motivates a drift bound, but the theorem is conditional on that bound and informative values actually holding. Recording an error symbol does not empirically establish a small error.
A.2 Variance Reduction from the Value Baseline
The baseline proof first establishes a score-function identity, then analyzes when an offline value correction reduces residual variance. These are separate claims from full algorithmic convergence.
Lemma 1 (State-only baseline does not add score-function bias).
For a baseline depending only on state, its score-function expectation vanishes under the current action distribution. Blending online and offline state values preserves that property, with GAE approximation handled separately.
Equations 22–23
| (22) |
| (23) |
Proof.
Pull the state baseline outside the action integral and differentiate the policy’s unit normalization. The remaining derivative is zero. This argument requires the baseline contribution to be treated as a baseline, not an action-dependent objective term.
Equations 24–26
| (24) | ||||
| (25) | ||||
| (26) |
Lemma 2 (Variance reduction).
Write the blended residual as X−alpha Z, where X is return minus online value and Z is offline minus online value, with alpha=1−beta. Positive covariance permits some alpha between zero and one that reduces variance. It does not identify the default coefficient as optimal.
Equations 27–29
| (27) |
| (28) |
| (29) |
Proof.
The variance change is alpha squared times Var(Z) minus twice alpha Cov(X,Z). It is negative for alpha below twice covariance/variance, also constrained below one. The unconstrained optimum is covariance/variance when admissible; correlation then yields the familiar multiplicative reduction. A fixed common beta must lie in each relevant client’s reducing interval to obtain a positive uniform coefficient. Zero-variance corrections cannot satisfy the assumed positive covariance.
Equations 30–39
| (30) | ||||
| (31) |
| (32) |
| (33) |
| (34) |
| (35) |
| (36) |
| (37) |
| (38) |
| (39) |
A.3 Federated Convergence
The convergence argument treats the combined prior/policy surrogate as stochastic smooth optimization with bounded bias. A separate PL result adds a stronger geometry assumption.
Theorem 1 (Convergence of FedGuide).
Average squared gradient norm decays with the initial surrogate gap divided by step size×rounds, plus a stochastic term and approximation floor. A step size proportional to inverse square root of rounds gives the usual nonconvex stationarity rate under the stated assumptions. Persistent proxy/value/drift error prevents convergence to an exact stationary point.
Equation 40
| (40) |
Proof.
Apply smooth descent, split direction into true gradient, zero-mean noise and bounded bias, bound cross terms with Young’s inequality, then telescope across rounds. The proof depends on the assumed direction/noise model and sufficiently small step; it is not an independent derivation that the implemented transport/PPO loop realizes exact gradient descent.
Equations 41–51
| (41) |
| (42) |
| (43) | ||||
| (44) |
| (45) |
| (46) |
| (47) | ||||
| (48) |
| (49) |
| (50) |
| (51) |
Theorem 2 (Linear convergence under PL).
If the surrogate additionally satisfies a Polyak–Łojasiewicz inequality, expected objective error contracts geometrically toward a neighborhood. The neighborhood depends on stochastic variance and approximation/drift terms. This stronger condition is not established for the neural RL benchmark objectives.
Equations 52–53
| (52) |
| (53) |
Proof.
Substitute the PL inequality into the one-step descent bound and unroll a linear recursion. The initial error shrinks geometrically, while accumulated noise/bias forms a nonzero floor. Reducing effective variance improves that floor only when the baseline assumptions hold.
Equations 54–61
| (54) |
| (55) |
| (56) |
| (57) |
| (58) |
| (59) | ||||
| (60) |
| (61) |
A.4 Policy Averaging under Prior-Regularized Alignment (FedGuide-A)
A separate theorem bounds the return effect of policy averaging in FedGuide-A when actual local-versus-averaged policy distributions remain close.
Theorem 3 (Controlled policy-averaging error).
Assume return is Lipschitz in expected action-distribution total variation. Define dispersion by client-weighted KL to the averaged policy. The average return gap is bounded by a constant times the square root of that dispersion. The theorem does not guarantee that parameter averaging produces small dispersion.
Equations 62–64
| (62) |
| (63) |
| (64) |
Proof.
Use the return-Lipschitz assumption clientwise, convert total variation to square-root KL with Pinsker, and apply Jensen over states and client weights. Every original inequality and definition remains available.
Equations 65–73
| (65) |
| (66) |
| (67) |
| (68) | ||||
| (69) | ||||
| (70) |
| (71) |
| (72) |
| (73) |
Implication for FedGuide.
Prior alignment may make policy averaging less destructive when it actually reduces dispersion, explaining why FedGuide-A sometimes performs reasonably. This is conditional; its severe Reacher-Hard failure demonstrates that the required compatibility is not assured by using OT-MoE.
Appendix B Environmental Setup
Environment descriptions and fixed client metadata specify exactly what differs across clients, with harder versions increasing distribution mismatch. Shared observation/action interfaces are maintained within each benchmark family.
B.1 Environment Description
The benchmark progression moves from controlled action modes to target differences, contact/dynamics changes and distinct manipulation tasks. It tests transfer across predefined client distributions rather than open-ended differences in sensors or action semantics.
Bandit2D.
Bandit2D has four Gaussian reward peaks on the unit circle. Clients sample overlapping 120° annular sectors with radii 0.7–1.3, making their union a ring. The global behavior prior should recover that support, while locally rewarded policies favor peaks. Table 6 uses closed-form Gaussian priors for this toy case, so it is not a full learned-diffusion ablation.
Equation 74
| (74) |
Reacher.
Reacher controls a two-link planar arm with torque actions and distance/control rewards. Different goal regions and environment parameters change local occupancy while retaining the same interface. It isolates target/reward variation before adding more complex contact dynamics.
MuJoCo Locomotion.
Hopper, Walker2D and HalfCheetah vary mass, damping, friction, actuation and reward preferences within fixed nominal spaces. Contact-sensitive hopping, biped balance and planar running produce different gait distributions. These parameter variations do not establish transfer between arbitrary morphological topologies.
MetaWorld10.
MetaWorld assigns a distinct manipulation task to each client, changing objects, contact sequences, reward geometry and success conditions. Shared dimensions permit aggregation, but useful behavior overlap is weaker than in dynamics-only perturbations. This setting reveals the method’s task-semantic limitation.
Client and Expert Configuration.
Use four clients/experts in Bandit2D, eight/eight for Reacher and locomotion, and ten clients/eight experts for MetaWorld. Each client owns its policy, online critic and frozen DICE value. Standard communication carries prior heads only; FedGuide-A adds policy synchronization as an explicit ablation.
B.2 Heterogeneity Setup
Fixed metadata gives every method the same client assignments and local MDP variants. Trajectories and buffers are not exchanged. The following ranges describe the evaluated split, not universal distributions over all possible robots.
Bandit2D.
Four clients receive 1,000 offline samples each around unit-circle centers, with Gaussian reward width 0.2 and area-uniform annular radii 0.7–1.3. The authors describe adjacent support overlap as 30%. Ring-shaped global support is built into this sampling construction.
Reacher.
Eight Reacher goal sectors are spaced every 45°, around radius 0.18 with radial width 0.06 and angular width 30°. Fixed action offsets, reward scales and initial-angle variation add heterogeneity: offset norms range 0.265–1.173 and reward scales 0.621–1.332.
Hopper and Walker2D.
Hopper/Walker2D cycle nominal, heavy, weak-actuator and high-damping/friction presets with jitter, plus speed/efficiency/stability rewards. Evaluated mass scales are 0.972–1.462, damping 0.779–1.367, friction 0.887–1.267 and action gain 0.818–1.002. Reward and instability settings vary independently as listed in the source.
HalfCheetah.
HalfCheetah uses similar preset categories with narrower main mass variation (0.959–1.097), action gains 0.850–1.023 and task-specific control costs 0.0899–0.1232. Reset noise also varies. The main split has no instability penalty, unlike its Hard version.
MetaWorld10.
The ten clients are reach, push, pick-place, door-open, drawer-close, button-press-topdown, peg-insert-side, window-open, sweep and basketball v3 tasks. Heatmaps for other environments normalize numerical metadata within each environment; MetaWorld heatmaps encode semantic task attributes. Their colors therefore cannot be compared as common physical units.
B.3 Heterogeneity Setup for Stronger Environments
Hard settings widen or separate dynamics/reward regimes to increase disagreement among clients. They are heterogeneous-client stress tests, not a uniform increase in every individual task’s difficulty.
Reacher-Hard.
Reacher-Hard retains goal sectors while adding default/speed/stability/efficiency preferences and strong structural scaling. Mass spans 0.598–1.955, damping 0.568–1.923 and actuator gear 0.587–1.954. This produces the notable failure of the policy-averaging variant despite useful local-prior variants.
Hopper-Hard and Walker2D-Hard.
Hopper/Walker2D-Hard use eight regimes, including very-heavy, weak-actuator, extreme damping/friction and low-friction combinations. Sprint, miser, tightrope and explorer rewards separate speed, effort and stability preferences. Mass spans 0.862–1.698, friction 0.417–1.476 and action gain 0.620–0.902; the full metadata plots preserve other ranges.
HalfCheetah-Hard.
HalfCheetah-Hard broadens actuation/friction/reward/reset variation and adds an instability penalty of 0.0121–0.0352. Action gains span 0.670–1.066 and friction 0.851–1.376. Main/Hard violin plots normalize each metadata dimension using its pooled range, so they compare spread within dimensions rather than absolute units across dimensions.
Appendix C Baseline Details
Baseline descriptions specify what is communicated and what remains local. Interaction budgets, client splits, warm-up and evaluation are shared, while Table 9 documents optimizer and epoch differences. The tables’ guarantee labels refer to distinct methods/assumptions and should not be read as one directly comparable theorem.
FedAvg.
FedAvg broadcasts a common policy, runs local PPO and averages updated policy parameters with sample-count weights; equal rollout budgets give uniform weights here. It has no behavior prior, expert routing or DICE baseline. Its failure modes reflect this entire learning setup, not parameter averaging in isolation.
Equation 75
| (75) |
FedKL.
FedKL uses a shared global policy plus penalties for divergence from the global policy and from the previous local update. The experiment instantiates it with PPO. Output-space KL limits drift, but maintaining one aggregated policy can still conflict with distinct client behaviors.
Equation 76
| (76) |
| Method | Heterogeneity Type | Aggregation Target | Alignment Space | Multi-Modal Support | Local Policy |
Return-Aware
Baseline | |
|---|---|---|---|---|---|---|---|
| Baselines | FedAvg [23] | Data | Policy Params. | Parameter Space | ✗ | ✗ | ✗ |
| FedKL [44] | Data | Policy Params. | Policy Output | ✗ | ✗ | P | |
| FedRL [13] | MDP / Dynamics | Actor + Critic | Parameter Space | ✗ | ✗ | P | |
| FedSVRPG-M [37] | MDP / Dynamics | Policy Grad. | Gradient Space | ✗ | ✗ | P | |
| Variants | FedGuide-P | Data / MDP / Task | Diffusion Prior | Distribution Space | ✓ | ✓ | ✗ |
| FedGuide-A | Data / MDP / Task | Prior + Policy | Distribution Space | ✓ | ✗ | ✓ | |
| FedGuide | Data / MDP / Task | Diffusion Prior | Distribution Space | ✓ | ✓ | ✓ | |
| Method | Convergence | Communication Speedup | Local Updates Allowed | Distribution-Level Analysis | |
|---|---|---|---|---|---|
| Baselines | FedAvg [23] | FL convergence | Empirical | ✓ | ✗ |
| FedKL [44] | Asymptotic | Not claimed | P | Policy-output KL | |
| FedRL [13] | Finite to subopt. | Not claimed | ✓ | ✗ | |
| FedSVRPG-M [37] | Variance-reduced PG | Claimed | ✓ | Policy-gradient correction | |
| Variants | FedGuide-P | Prior-only ablation | Not isolated | ✓ | OT-MoE diffusion prior |
| FedGuide-A | Controlled avg. error | Not isolated | ✓ | OT-MoE prior + policy avg. | |
| FedGuide | Finite to controlled error | Variance-controlled | ✓ | OT-MoE diffusion prior | |
FedRL.
FedRL is the continuous-control DDPGAvg-style extension: clients retain local replay buffers, train actor/critic networks and average both at round boundaries. Its off-policy backbone differs from the PPO methods, so early learning and replay coverage confound a pure communication-rule comparison.
Equation 77
| (77) |
FedSVRPG-M.
FedSVRPG-M shares momentum/variance-reduced update directions around a global policy. The tested implementation uses the paper’s practical PPO/SVRPG gradient path, rather than claiming an unchanged reference implementation. Table 3 marks it as implemented by the authors. It shares the warm-up and rollout protocol but no multimodal behavior priors.
| Parameter | Symbol | Bandit2D | Reacher | MuJoCo locomotion | MetaWorld10 |
|---|---|---|---|---|---|
| Offline buffer size | cap | ||||
| Offline source | – | local modes | client rollouts | D4RL | scripted policy |
| Diffusion steps | Gaussian | ||||
| Prior width / horizon | Gaussian | ||||
| Prior epochs / batch | closed form | ||||
| Prior learning rate | – | ||||
| DICE width | |||||
| DICE learning rates | |||||
| DICE temperature | |||||
| DICE warm-up epochs | |||||
| BC epochs / batch | prior init | ||||
| BC width / learning rate | prior init |
FedGuide-P.
FedGuide-P keeps transport/expert prior sharing and fully local policies while setting beta=1. It isolates removal of the DICE baseline within the FedGuide implementation and sometimes outperforms the full method, notably on Reacher-Hard.
Equation 78
| (78) |
FedGuide-A.
FedGuide-A adds server-side policy averaging to otherwise prior-regularized, DICE-blended local learning. The hyperparameter table synchronizes policies every five rounds. It tests whether aligned behavior support makes averaging tolerable; its failures show that personalized adaptation can still be lost.
Equation 79
| (79) |
FedGuide.
Standard FedGuide communicates only prior heads, receives a personalized mixture and learns policy/online value locally with a frozen DICE baseline. Distribution sharing and local value blending are distinct components; neither involves sharing raw replay data or evaluating sampled actions with DICE Q values.
Equation 80
| (80) |
Appendix D Hyperparameters
All offline, online, variant and baseline hyperparameters are retained. These tables also expose special cases and resource differences that qualify the main comparisons.
D.1 Offline Pretraining
Offline buffers contain 1,000 local-mode samples for Bandit2D, 20,000 Reacher client transitions, up to 200,000 D4RL locomotion samples and 5,000 MetaWorld scripted-policy samples. Bandit priors are closed-form Gaussian; other priors use 1,000 diffusion steps, 64-width/horizon settings and 40 epochs. DICE has width 256, temperature 0.5 and 100/80 warm-up epochs, then stays frozen. Behavior-cloning warm-up is shared by all methods, while prior/DICE pretraining adds method-specific work.
D.2 Online FedGuide Settings
Online FedGuide uses 60 bandit or 100 other rounds, with 200/2,048/4,096 samples per client round by domain and four local PPO epochs. Defaults are trust-region weight 0.05, prior weight 0.5 and Sinkhorn regularization 0.05. Prior guidance anneals over 60 bandit or 50 other rounds. Evaluation uses ten episodes outside the toy bandit. The tables preserve action ranges and exploration initialization.
| Parameter | Symbol | Bandit2D | Reacher | MuJoCo locomotion | MetaWorld10 |
|---|---|---|---|---|---|
| Clients | |||||
| Prior experts | |||||
| Federated rounds | |||||
| Rollout size / round | |||||
| Local PPO epochs | |||||
| PPO mini-batch | |||||
| Trust-region weight | |||||
| Prior regularizer weight | |||||
| Prior annealing horizon | |||||
| Sinkhorn regularization | |||||
| Initial log std. | |||||
| Evaluation episodes | grid / return | ||||
| Action range |
D.3 FedGuide Variants
All three variants share priors, data and local PPO. FedGuide and FedGuide-A use beta=0.5; FedGuide-P uses beta=1. Only FedGuide-A averages policy parameters, every five rounds. These exact settings take precedence over optional schedules discussed in prose.
D.4 Baseline Hyperparameters
FedAvg/FedKL use ten local PPO epochs versus four for FedGuide and the PPO/SVRPG baseline; FedRL uses DDPG with replay capacity 100,000 and warm-up 1,000. Shared interaction budgets therefore do not imply equal gradient work or wall time. Table 9 preserves KL, clipping, learning-rate, momentum and exploration settings; end-to-end pretraining/communication cost is not quantified by return curves.
| Parameter | Symbol | FedAvg | FedKL | FedRL | FedSVRPG-M |
|---|---|---|---|---|---|
| Local optimizer | – | PPO | PPO | DDPG | PPO/SVRPG |
| Aggregated parameters | – | ||||
| Global KL weight | – | – | |||
| Local KL weight | – | – | |||
| Local PPO epochs | – | ||||
| Mini-batch size | |||||
| Discount factor | |||||
| GAE coefficient | – | ||||
| PPO clipping radius | – | ||||
| Value loss weight | – | ||||
| Gradient clipping | – | ||||
| Learning rate | |||||
| Reference update freq. | – | – | – | ||
| Momentum weight | – | – | – | ||
| BC initialization weight | |||||
| Replay capacity | – | – | – | ||
| Replay warm-up | – | – | – | ||
| Target update coefficient | – | – | – | ||
| Exploration noise | – | – | – |
Appendix E Additional Analysis
Additional tests isolate expert count, transport softness, prior weight and missing expert support using the controlled bandit. They clarify operating conditions rather than proving robustness across arbitrary client populations.
E.1 Sensitivity to M , η , and λ 2
Five-seed, four-client Bandit2D sweeps vary expert count, Sinkhorn temperature and prior weight while the true mode count is known. This is a diagnostic setting with the toy prior construction described in Table 6.
(a) Expert count
(b) Sinkhorn
(c) Prior weight
Expert count M .
One or two experts underrepresent four modes and learn more slowly; four is most sample-efficient, while larger banks are stable but redundant in this test. The result motivates matching effective behavior modes, not necessarily client count. Online estimation and adaptive resizing remain unsolved.
Sinkhorn temperature η .
Temperature changes assignment softness between near-hard routing and broader mixtures. The tested range does not change the reported qualitative ordering, but this limited sweep is not a guarantee that arbitrary temperatures preserve useful client specialization.
Prior weight λ 2 .
Zero prior weight removes shared-prior guidance; positive weights become less different late in training because the implementation anneals guidance away. The local DICE baseline can still remain, so zero prior weight is not necessarily ordinary PPO. Shared behavior-cloning warm-up cannot by itself explain relative method differences.
E.2 Behavior under Weak Overlap
A fifth client targets the origin while only the four ring clients update server experts; the new client participates in routing only. Equidistance produces a near-uniform mixture on the ring, without creating the absent center mode. This deliberately excludes learning the fifth mode into the bank. The result demonstrates recombination of represented supports, and why missing support can make transferred guidance uninformative or mismatched.
Appendix F Visualizations
All five complete rollout grids retain every client/method panel. Rows compare FedGuide-A, FedGuide-P and FedGuide; each cell overlays frames from a single final-round episode. Visualizations select the best of five seeds by final return, whereas numerical results average seeds. These selected trajectories illustrate motion and contact behavior and are not independent success-rate evidence.