Overview
MoE-Loco trains one proprioceptive quadruped policy to switch between four-legged and rear-leg walking across several terrains. Its learned experts can also be selectively recombined for new motions.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/b8/b83133b79c67dd0682146caa162a7812ac11bbcfcc34f8b8d32e0925a94ee173.webp)
At a glance
A six-expert actor and critic share a learned gating network, allowing locomotion tasks to use different mixtures of parameters. Compared with a parameter-matched ordinary network, this improves most tested tasks and reduces several measured conflicts between their training gradients. A privileged-state teacher is adapted to proprioceptive deployment on a Unitree Go2. Manually selected experts demonstrate dribbling, while an additional adaptation experiment produces three-legged walking; these examples do not establish automatic composition of arbitrary skills.
I INTRODUCTION
Sharing one network across terrain traversal and fundamentally different gaits can cause competing parameter updates. The paper introduces expert routing into both policy and value prediction, trains quadrupedal and bipedal behaviors jointly, and examines whether the resulting modules expose reusable motion components. Terrain sensing is absent at deployment: the robot receives body observations and a commanded gait rather than a visual terrain label.
II RELATED WORKS
The work combines locomotion reinforcement learning, multitask optimization, and modular mixture-of-experts networks.
II-A Reinforcement Learning for Robot Locomotion
Prior reinforcement-learning controllers demonstrate agile walking, parkour, and specialized bipedal motions. The challenge here is retaining several terrain and gait skills in one policy rather than optimizing a separate controller for each. The training framework builds on Robust Robot Walker [20].
II-B MultiTask Learning
Multitask learning can share useful knowledge but also introduce conflicting gradients. Previous locomotion approaches combine pretrained experts or hierarchical terrain controllers. MoE-Loco instead learns experts jointly across both gait modes. The related-work discussion cites MELA [39] and MTAC [43] as nearby modular and hierarchical approaches.
II-C Mixture of Experts (MoE)
Mixture-of-experts models route inputs among specialized subnetworks. Here, their purpose is to separate competing control updates while retaining shared behavior. The design uses a weighted combination of all experts rather than claiming the sparse inference savings of large language-model routing.
III METHOD
The method specifies gait-dependent objectives, a privileged-to-proprioceptive training procedure, and optional manual changes to expert weights.
III-A Task Definition
Nine locomotion tasks cover quadrupedal bars, pits, baffles, stairs, and slopes, plus bipedal standing, flat walking, slopes, and stair descent. A joystick supplies velocity and a gait indicator. The unified objective sums discounted rewards across tasks, whose state distributions, rewards, and termination rules differ. Only proprioception and commands are available to the deployed policy; visual locomotion is suggested as future use rather than evaluated here.
Equation 1
| (1) |
III-B MoE Based Multitask Locomotion Learning
The actor outputs twelve desired joint positions. Inputs include gravity direction, angular velocity, joint states, previous action, and commands. During teacher training, base linear velocity and friction are explicit privileged quantities; link contact forces are encoded into a latent state. Recurrent processing feeds actor and critic expert modules with a shared gate. Their weighted expert outputs support task-dependent behavior without a separately provided terrain identity.
Training combines PPO policy/value losses with privileged-state reconstruction. A recurrent estimator learns to infer the missing state from proprioception. Probability Annealing Selection, following SARO [54], gradually replaces true privileged information with estimates during adaptation. Gait-specific tracking, standing, and regularization rewards remain hand designed; the expert architecture does not remove reward engineering.
Equations 2–4
| (2) |
| (3) |
| (4) |
III-C Skill Decomposition and Composition
Expert gates can be multiplied by manually chosen or learned weights. Masking experts and amplifying selected contributions produces alternative combinations of learned behavior. This is a controllable architectural interface, but identifying a useful combination still requires inspection or additional optimization. The displayed recomposition rule is retained with the original weighting convention.
Equation 5
| (5) |
IV EXPERIMENTS
Experiments compare task performance, gradient interactions, training curves, expert usage, and two examples of recomposed behavior.
IV-A Experiment Setup
IsaacGym runs 4,096 robots in parallel on an RTX 3090. Training uses 40,000 iterations of flat walking, 80,000 of harder terrain, and 10,000 of proprioceptive adaptation. Six experts are used. Simulation and hardware run the policy at 50 Hz; the real Go2 uses a Jetson Orin and proportional–derivative joint control with gains 40 and 0.5. Training iteration counts are reported, but wall-clock training cost is not.
IV-B Multitask Performance
The main comparison replaces the expert module with an ordinary network of similar total parameter count. A second baseline uses the original Rapid Motor Adaptation [3] architecture, so that comparison also changes the temporal model and adaptation design.
| Method | Success Rate | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Mix | Bar (q) | Baffle (q) | Stair (q) | Pit (q) | Slope (q) | Walk (b) | Slope (b) | Stair (b) | |
| Ours | 0.879 | 0.886 | 0.924 | 0.684 | 0.902 | 0.956 | 0.932 | 0.961 | 0.964 |
| Ours w/o MoE | 0.571 | 0.848 | 0.264 | 0.568 | 0.698 | 0.988 | 0.826 | 0.504 | 0.453 |
| RMA | 0.000 | 0.871 | 0.058 | 0.017 | 0.017 | 0.437 | 0.000 | 0.000 | 0.000 |
| Average Pass Time (s) | |||||||||
| Ours | 230.98 | 102.42 | 87.84 | 179.14 | 91.86 | 76.75 | 92.37 | 86.14 | 86.44 |
| Ours w/o MoE | 315.47 | 125.46 | 318.68 | 214.52 | 161.38 | 65.28 | 156.76 | 236.67 | 253.62 |
| RMA | 400.00 | 107.84 | 385.25 | 395.34 | 394.49 | 272.06 | 400.00 | 400.00 | 400.00 |
| Average Travel Distance (m) | |||||||||
| Ours | 89.41 | 28.05 | 28.02 | 20.42 | 27.82 | 27.62 | 27.20 | 27.99 | 28.04 |
| Ours w/o MoE | 57.12 | 27.59 | 17.41 | 22.66 | 25.59 | 28.49 | 22.73 | 26.21 | 14.23 |
| RMA | 13.40 | 27.39 | 11.31 | 3.92 | 12.48 | 21.33 | 2.00 | 2.00 | 2.00 |
IV-B 1 Simulation Experiment
A mixed-terrain runway is 100 m long, with separate 30 m single-task tracks. Success requires reaching within 1 m of the target in 400 seconds; failed trials receive the full 400-second time penalty. Thus average pass time includes failures and is not merely successful-run speed.
Mixed-task success is 87.9% versus 57.1% without experts and 0% for the tested RMA implementation. Corresponding travel distances are 89.41, 57.12, and 13.40 m. Most single-task success rates improve, but quadrupedal slopes favor the ordinary network (98.8% versus 95.6%); quadrupedal stair travel distance also favors that baseline despite its lower success. The evidence supports broader competence under this benchmark, not dominance on every metric.
| Obstacle Type | Specification | Gait Mode |
|---|---|---|
| Bars | 5 bars, height: 0.05m – 0.2m | Quadrupedal |
| Pits | 5 pits, width: 0.05m – 0.2m | Quadrupedal |
| Baffles | 5 baffles, height: 0.3m – 0.22m | Quadrupedal |
| Up Stairs | 3 sets, step height: 5cm – 15cm | Quadrupedal |
| Down Stairs | 3 sets, step height: 5cm – 15cm | Quadrupedal |
| Up Slopes | 3 sets, incline: – | Quadrupedal |
| Down Slopes | 3 sets, incline: – | Quadrupedal |
| Plane | 10m flat surface | Bipedal |
| Up Slopes | 3 sets, incline: – | Bipedal |
| Down Slopes | 3 sets, incline: – | Bipedal |
| Down Stairs | 3 sets, step height: 5cm – 15cm | Bipedal |
IV-B 2 Real World Experiments
The same trained policy is deployed without hardware fine-tuning. The mixed course combines a 20 cm bar, 22 cm baffle, 15 cm stairs, 20 cm pits, and a 30-degree slope before a commanded switch to bipedal motion. Each test receives twenty trials. Figure 4 reports higher success for the expert policy across the tested categories, with the largest separation on the combined course. Outdoor examples show additional terrain encounters but are qualitative rather than a separate statistical benchmark.
IV-C Gradient Conflict Alleviation
Gradients are measured over 500 multitask training epochs after a selected 15,000-epoch checkpoint. The cosine-similarity and negative-entry analyses show reduced conflict for several cross-gait pairs: for example, bar crossing versus bipedal uphill changes from −0.132 to 0.278 cosine similarity. Improvements are not uniform: baffle versus quadrupedal stairs has slightly worse similarity and negative-entry ratio. These diagnostics support a mechanism at the sampled checkpoint, not a proof that all training interference disappears.
| MoE/Standard | Gradient Cosine Similarity | ||||
|---|---|---|---|---|---|
| Bar (q) | Baffle (q) | Stair (q) | Slope Up (b) | Slope down (b) | |
| Bar (q) | - | 0.519/0.474 | 0.606/0.592 | 0.278/-0.132 | 0.091/-0.128 |
| Baffle (q) | - | - | 0.369/0.384 | 0.062/-0.091 | 0.061/-0.101 |
| Stair (q) | - | - | - | 0.046/-0.023 | 0.052/0.015 |
| Slope up (b) | - | - | - | - | 0.806/0.709 |
| Slope down (b) | - | - | - | - | - |
| MoE/Standard | Gradient Negative Entries (%) | ||||
|---|---|---|---|---|---|
| Bar (q) | Baffle (q) | Stair (q) | Slope Up (b) | Slope down (b) | |
| Bar (q) | - | 35.72/37.33 | 32.67/32.62 | 45.50/ 55.12 | 49.83/50.80 |
| Baffle (q) | - | - | 39.90/38.52 | 49.86/55.91 | 49.91/51.68 |
| Stair (q) | - | - | - | 49.52/50.15 | 50.04/50.34 |
| Slope up (b) | - | - | - | - | 23.17/30.91 |
| Slope down (b) | - | - | - | - | - |
IV-D Training Performance
During initial flat-ground training, the expert policy attains higher mean reward and episode length than the parameter-matched standard network. These curves show faster acquisition in the plotted training stage; they do not provide an end-to-end wall-clock comparison across every stage.
IV-E Expert Specialization Analysis
Average gating weights differ across walking, crawling, and crossing phases. A t-SNE view separates bipedal and quadrupedal regimes, while tasks with similar gaits cluster more closely. These are observations of routed representations: cluster separation alone does not establish that each expert implements an independently complete skill.
IV-F Skill Composition
After inspecting experts, the authors retain one associated with balance and one with front-leg crossing, double the latter’s gating weight, and mask the rest. The resulting motion periodically kicks a ball while walking. Figure 9 demonstrates this manually constructed dribbling pattern without retraining; no aggregate dribbling success rate or automated expert-selection procedure is reported.
IV-G Additional Experiment
A separate experiment freezes the original experts and introduces a new expert to obtain three-footed walking on flat ground and slopes. The source describes updating only the gate while also saying that the new expert learns to lift a leg; these descriptions leave its precise update schedule ambiguous. The retained example supports a demonstrated adapted gait, without resolving that implementation detail or quantifying adaptation efficiency.
V CONCLUSION
Expert routing offers a useful way to combine substantially different gaits in one blind locomotion policy, with improved mixed-terrain results and interpretable intervention points. Evidence is limited to the tested robot, terrains, baselines, and manually examined compositions. Cameras and lidar are proposed extensions; neither predictive visual planning nor a general safety guarantee is established.
APPENDIX
The appendix retains the complete reward definitions, both training algorithms, module dimensions, randomization ranges, observation noise, and PPO settings.
V-A Reward Functions
Quadrupedal rewards emphasize velocity tracking, body stability, contact balance, and actuator limits. Bipedal rewards add upright orientation and height, condition tracking on standing, and regularize front/rear legs differently. Table V preserves all formulas and weights, which are essential task-specific components of the result.
| Type | Item | Formula | Weight |
|---|---|---|---|
| Quadrupedal Tracking | Tracking lin vel | ||
| Tracking ang vel | |||
| Termination | |||
| Alive | |||
| Quadrupedal Regularization | Joint pos | ||
| Joint vel | |||
| Joint acc | |||
| Ang vel stability | |||
| Feet in air | |||
| Front hip pos | |||
| Rear hip pos | |||
| Base height | |||
| Balance | |||
| Joint limit | |||
| Torque exceed limit | |||
| Bipedal Stand | Orientation | ||
| Base height linear | |||
| Bipedal Tracking | Tracking lin vel | ||
| Tracking ang vel | |||
| Termination | |||
| Alive | |||
| Bipedal Regularization | Rear air | ||
| Front hip pos | |||
| Rear hip pos | |||
| Rear pos balance | |||
| Front joint pos | |||
| Front joint vel | |||
| Front joint acc | |||
| Legs energy substeps | |||
| Torque exceed limits | |||
| Joint limits | |||
| Collision | |||
| Action rate | |||
| Joint vel | |||
| Joint acc |
V-B Algorithm Pseudocode
Algorithm 1 trains an oracle from privileged state while learning its reconstruction. Algorithm 2 initializes from the oracle and anneals the probability of substituting estimated state during continued PPO training. Both algorithms retain the rollout, reset, reconstruction, and policy/value update details.
V-C Network Architecture Details
Actor, critic, and estimator use 256-dimensional recurrent modules. The table specifies privileged encoders, expert heads, the larger ordinary-network comparison head, and the gating network. These architectural differences explain how parameter counts are matched while changing the routing structure.
V-D Domain Randomization
Simulation randomizes mass, center of mass, friction, initial joint positions, motor strength, and proprioception latency; latency spans 5–45 ms. Gaussian noise perturbs velocity, gravity, and joint observations. The complete distributions are retained. Randomization supports transfer under these disturbances but does not by itself certify safe behavior outside them.
| Parameters | Range | Unit |
|---|---|---|
| Base mass | [1, 3] | |
| Mass position of X axis | [-0.2, 0.2] | |
| Mass position of Y axis | [-0.1, 0.1] | |
| Mass position of Z axis | [-0.05, 0.05] | |
| Friction | [0, 2] | - |
| Initial joint positions | [0.5, 1.5]nominal value | |
| Motor strength | [0.9, 1.1]nominal value | - |
| Proprioception latency | [0.005, 0.045] |
| Observation | Gaussian Noise Amplitude | Unit |
|---|---|---|
| Linear velocity | 0.05 | |
| Angular velocity | 0.2 | |
| Gravity | 0.05 | |
| Joint position | 0.01 | |
| Joint velocity | 1.5 |
V-E Training Details
Fractal ground noise with maximum scale 0.1 discourages dragging feet. PPO uses a 0.2 clipping parameter, 0.99 discount, 0.95 advantage parameter, 0.001 learning rate, four minibatches, and 24 rollout steps per environment, with the remaining settings preserved in Table IX.
| Hyperparameter | Value |
|---|---|
| clip min std | 0.05 |
| clip param | 0.2 |
| gamma | 0.99 |
| lam | 0.95 |
| desired kl | 0.01 |
| entropy coef | 0.01 |
| learning rate | 0.001 |
| max grad norm | 1 |
| num mini batch | 4 |
| num steps per env | 24 |