Literature
Display
← All papers

MoE-Loco: Mixture of Experts for Multitask Locomotion

Runhan Huang · Shaoting Zhu · Yilun Du · Hang Zhao

· IROS 2025 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

MoE-Loco trains one proprioceptive quadruped policy to switch between four-legged and rear-leg walking across several terrains. Its learned experts can also be selectively recombined for new motions.

[Uncaptioned image]
Fig. 1: We introduce MoE-Loco. With a single MoE policy, quadruped robot can traverse a variety of challenging terrains and perform different locomotion modes, including bipedal and quadrupedal gaits.

At a glance

A six-expert actor and critic share a learned gating network, allowing locomotion tasks to use different mixtures of parameters. Compared with a parameter-matched ordinary network, this improves most tested tasks and reduces several measured conflicts between their training gradients. A privileged-state teacher is adapted to proprioceptive deployment on a Unitree Go2. Manually selected experts demonstrate dribbling, while an additional adaptation experiment produces three-legged walking; these examples do not establish automatic composition of arbitrary skills.

I INTRODUCTION

Sharing one network across terrain traversal and fundamentally different gaits can cause competing parameter updates. The paper introduces expert routing into both policy and value prediction, trains quadrupedal and bipedal behaviors jointly, and examines whether the resulting modules expose reusable motion components. Terrain sensing is absent at deployment: the robot receives body observations and a commanded gait rather than a visual terrain label.

II RELATED WORKS

The work combines locomotion reinforcement learning, multitask optimization, and modular mixture-of-experts networks.

II-A Reinforcement Learning for Robot Locomotion

Prior reinforcement-learning controllers demonstrate agile walking, parkour, and specialized bipedal motions. The challenge here is retaining several terrain and gait skills in one policy rather than optimizing a separate controller for each. The training framework builds on Robust Robot Walker [20].

II-B MultiTask Learning

Multitask learning can share useful knowledge but also introduce conflicting gradients. Previous locomotion approaches combine pretrained experts or hierarchical terrain controllers. MoE-Loco instead learns experts jointly across both gait modes. The related-work discussion cites MELA [39] and MTAC [43] as nearby modular and hierarchical approaches.

II-C Mixture of Experts (MoE)

Mixture-of-experts models route inputs among specialized subnetworks. Here, their purpose is to separate competing control updates while retaining shared behavior. The design uses a weighted combination of all experts rather than claiming the sparse inference savings of large language-model routing.

III METHOD

The method specifies gait-dependent objectives, a privileged-to-proprioceptive training procedure, and optional manual changes to expert weights.

III-A Task Definition

Nine locomotion tasks cover quadrupedal bars, pits, baffles, stairs, and slopes, plus bipedal standing, flat walking, slopes, and stair descent. A joystick supplies velocity and a gait indicator. The unified objective sums discounted rewards across tasks, whose state distributions, rewards, and termination rules differ. Only proprioception and commands are available to the deployed policy; visual locomotion is suggested as future use rather than evaluated here.

Refer to caption
Fig. 2: A snapshot of the terrain settings. From left to right: bar, pit, baffle, slope, and stairs.
Equation 1
J⁡(π)=𝔼⁡[∑τ∑t=0∞γt​Rτ​(st,at,st+1)]J(\pi)=\mathbb{E}\left[\sum_{\tau}\sum_{t=0}^{\infty}\gamma^{t}R_{\tau}(s_{t},a_{t},s_{t+1})\right] (1)
Refer to caption
Fig. 3: Overview of our MoELoco pipeline. With the design of MoE architecture, our policy achieves robust multitask locomotion ability on various challenging terrains with multiple gaits.

III-B MoE Based Multitask Locomotion Learning

The actor outputs twelve desired joint positions. Inputs include gravity direction, angular velocity, joint states, previous action, and commands. During teacher training, base linear velocity and friction are explicit privileged quantities; link contact forces are encoded into a latent state. Recurrent processing feeds actor and critic expert modules with a shared gate. Their weighted expert outputs support task-dependent behavior without a separately provided terrain identity.

Training combines PPO policy/value losses with privileged-state reconstruction. A recurrent estimator learns to infer the missing state from proprioception. Probability Annealing Selection, following SARO [54], gradually replaces true privileged information with estimates during adaptation. Gait-specific tracking, standing, and regularization rewards remain hand designed; the expert architecture does not remove reward engineering.

Equations 2–4
𝒈i^=softmax​(g⁡(𝒉t))​[i],\hat{\boldsymbol{g}_{i}}=\text{softmax}(g(\boldsymbol{h}_{t}))[i],\vskip-2.84526pt (2)
𝒂t=∑i=1N𝒈i^⋅fi​(𝒉t),\boldsymbol{a}_{t}=\sum_{i=1}^{N}\hat{\boldsymbol{g}_{i}}\cdot f_{i}(\boldsymbol{h}_{t}),\vskip-2.84526pt (3)
Lsurro+Lvalue+Lrecon,L_{\text{surro}}+L_{\text{value}}+L_{\text{recon}},\vskip-2.84526pt (4)

III-C Skill Decomposition and Composition

Expert gates can be multiplied by manually chosen or learned weights. Masking experts and amplifying selected contributions produces alternative combinations of learned behavior. This is a controllable architectural interface, but identifying a useful combination still requires inspection or additional optimization. The displayed recomposition rule is retained with the original weighting convention.

Equation 5
𝒈i^=w⁡[i]⋅softmax​(g⁡(𝒉t))​[i],\hat{\boldsymbol{g}_{i}}=w[i]\cdot\text{softmax}(g(\boldsymbol{h}_{t}))[i],\vskip-2.84526pt (5)

IV EXPERIMENTS

Experiments compare task performance, gradient interactions, training curves, expert usage, and two examples of recomposed behavior.

IV-A Experiment Setup

IsaacGym runs 4,096 robots in parallel on an RTX 3090. Training uses 40,000 iterations of flat walking, 80,000 of harder terrain, and 10,000 of proprioceptive adaptation. Six experts are used. Simulation and hardware run the policy at 50 Hz; the real Go2 uses a Jetson Orin and proportional–derivative joint control with gains 40 and 0.5. Training iteration counts are reported, but wall-clock training cost is not.

IV-B Multitask Performance

The main comparison replaces the expert module with an ordinary network of similar total parameter count. A second baseline uses the original Rapid Motor Adaptation [3] architecture, so that comparison also changes the temporal model and adaptation design.

TABLE I: Quantitative Comparison in Simulation. Metrics include success rate, average travel distance, and average passing time.
Method Success Rate ↑\uparrow
Mix Bar (q) Baffle (q) Stair (q) Pit (q) Slope (q) Walk (b) Slope (b) Stair (b)
Ours 0.879 0.886 0.924 0.684 0.902 0.956 0.932 0.961 0.964
Ours w/o MoE 0.571 0.848 0.264 0.568 0.698 0.988 0.826 0.504 0.453
RMA 0.000 0.871 0.058 0.017 0.017 0.437 0.000 0.000 0.000
Average Pass Time (s) ↓\downarrow
Ours 230.98 102.42 87.84 179.14 91.86 76.75 92.37 86.14 86.44
Ours w/o MoE 315.47 125.46 318.68 214.52 161.38 65.28 156.76 236.67 253.62
RMA 400.00 107.84 385.25 395.34 394.49 272.06 400.00 400.00 400.00
Average Travel Distance (m) ↑\uparrow
Ours 89.41 28.05 28.02 20.42 27.82 27.62 27.20 27.99 28.04
Ours w/o MoE 57.12 27.59 17.41 22.66 25.59 28.49 22.73 26.21 14.23
RMA 13.40 27.39 11.31 3.92 12.48 21.33 2.00 2.00 2.00

IV-B 1 Simulation Experiment

A mixed-terrain runway is 100 m long, with separate 30 m single-task tracks. Success requires reaching within 1 m of the target in 400 seconds; failed trials receive the full 400-second time penalty. Thus average pass time includes failures and is not merely successful-run speed.

Mixed-task success is 87.9% versus 57.1% without experts and 0% for the tested RMA implementation. Corresponding travel distances are 89.41, 57.12, and 13.40 m. Most single-task success rates improve, but quadrupedal slopes favor the ordinary network (98.8% versus 95.6%); quadrupedal stair travel distance also favors that baseline despite its lower success. The evidence supports broader competence under this benchmark, not dominance on every metric.

TABLE II: Benchmark tasks for simulation experiments
Obstacle Type Specification Gait Mode
Bars 5 bars, height: 0.05m – 0.2m Quadrupedal
Pits 5 pits, width: 0.05m – 0.2m Quadrupedal
Baffles 5 baffles, height: 0.3m – 0.22m Quadrupedal
Up Stairs 3 sets, step height: 5cm – 15cm Quadrupedal
Down Stairs 3 sets, step height: 5cm – 15cm Quadrupedal
Up Slopes 3 sets, incline: 10∘10^{\circ} – 35∘35^{\circ} Quadrupedal
Down Slopes 3 sets, incline: 10∘10^{\circ} – 35∘35^{\circ} Quadrupedal
Plane 10m flat surface Bipedal
Up Slopes 3 sets, incline: 10∘10^{\circ} – 35∘35^{\circ} Bipedal
Down Slopes 3 sets, incline: 10∘10^{\circ} – 35∘35^{\circ} Bipedal
Down Stairs 3 sets, step height: 5cm – 15cm Bipedal

IV-B 2 Real World Experiments

The same trained policy is deployed without hardware fine-tuning. The mixed course combines a 20 cm bar, 22 cm baffle, 15 cm stairs, 20 cm pits, and a 30-degree slope before a commanded switch to bipedal motion. Each test receives twenty trials. Figure 4 reports higher success for the expert policy across the tested categories, with the largest separation on the combined course. Outdoor examples show additional terrain encounters but are qualitative rather than a separate statistical benchmark.

Refer to caption
Fig. 4: Real world success rate over multiple terrains and gaits.
Refer to caption
Fig. 5: Real-world experiments over multiple terrains and gaits: 1. Bar (Quad), 2. Pit (Quad), 3. Baffle (Quad), 4. Stair (Quad), 5. Slope (Quad), 6. Stand up (Bip), 7. Walk (Bip), 8. Slope (Bip), 9. Stair (Bip).

IV-C Gradient Conflict Alleviation

Gradients are measured over 500 multitask training epochs after a selected 15,000-epoch checkpoint. The cosine-similarity and negative-entry analyses show reduced conflict for several cross-gait pairs: for example, bar crossing versus bipedal uphill changes from −0.132 to 0.278 cosine similarity. Improvements are not uniform: baffle versus quadrupedal stairs has slightly worse similarity and negative-entry ratio. These diagnostics support a mechanism at the sampled checkpoint, not a proof that all training interference disappears.

TABLE III: Cosine similarity of gradients of different tasks. Left represents MoE policy and right represents standard policy.
MoE/Standard Gradient Cosine Similarity ↑\uparrow
Bar (q) Baffle (q) Stair (q) Slope Up (b) Slope down (b)
Bar (q) - 0.519/0.474 0.606/0.592 0.278/-0.132 0.091/-0.128
Baffle (q) - - 0.369/0.384 0.062/-0.091 0.061/-0.101
Stair (q) - - - 0.046/-0.023 0.052/0.015
Slope up (b) - - - - 0.806/0.709
Slope down (b) - - - - -
TABLE IV: Negative entry ratio of MoE and Standard policy on different tasks. Left represents MoE policy and right represents standard policy.
MoE/Standard Gradient Negative Entries (%) ↓\downarrow
Bar (q) Baffle (q) Stair (q) Slope Up (b) Slope down (b)
Bar (q) - 35.72/37.33 32.67/32.62 45.50/ 55.12 49.83/50.80
Baffle (q) - - 39.90/38.52 49.86/55.91 49.91/51.68
Stair (q) - - - 49.52/50.15 50.04/50.34
Slope up (b) - - - - 23.17/30.91
Slope down (b) - - - - -

IV-D Training Performance

During initial flat-ground training, the expert policy attains higher mean reward and episode length than the parameter-matched standard network. These curves show faster acquisition in the plotted training stage; they do not provide an end-to-end wall-clock comparison across every stage.

Refer to caption
Fig. 6: Training curve of our multitask policy in the pretraining stage.

IV-E Expert Specialization Analysis

Average gating weights differ across walking, crawling, and crossing phases. A t-SNE view separates bipedal and quadrupedal regimes, while tasks with similar gaits cluster more closely. These are observations of routed representations: cluster separation alone does not establish that each expert implements an independently complete skill.

Refer to caption
Fig. 7: Expert usage in different tasks. From left to right is: Bipedal walking, Quadrupedal Baffle crawling, Front Leg Crossing Bars and Rear Leg Crossing Bars.
Refer to caption
Fig. 8: t-SNE result of gating network output on different terrains and gaits.

IV-F Skill Composition

After inspecting experts, the authors retain one associated with balance and one with front-leg crossing, double the latter’s gating weight, and mask the rest. The resulting motion periodically kicks a ball while walking. Figure 9 demonstrates this manually constructed dribbling pattern without retraining; no aggregate dribbling success rate or automated expert-selection procedure is reported.

Refer to caption
Fig. 9: Manually designed new dribbling gait by selecting two experts.

IV-G Additional Experiment

A separate experiment freezes the original experts and introduces a new expert to obtain three-footed walking on flat ground and slopes. The source describes updating only the gate while also saying that the new expert learns to lift a leg; these descriptions leave its precise update schedule ambiguous. The retained example supports a demonstrated adapted gait, without resolving that implementation detail or quantifying adaptation efficiency.

Refer to caption
Fig. 10: MoE-Loco can quickly adapt to a three-footed gait by training a new expert. 1) ground plane, 2) slope up, and 3) slope down.

V CONCLUSION

Expert routing offers a useful way to combine substantially different gaits in one blind locomotion policy, with improved mixed-terrain results and interpretable intervention points. Evidence is limited to the tested robot, terrains, baselines, and manually examined compositions. Cameras and lidar are proposed extensions; neither predictive visual planning nor a general safety guarantee is established.

APPENDIX

The appendix retains the complete reward definitions, both training algorithms, module dimensions, randomization ranges, observation noise, and PPO settings.

V-A Reward Functions

Quadrupedal rewards emphasize velocity tracking, body stability, contact balance, and actuator limits. Bipedal rewards add upright orientation and height, condition tracking on standing, and regularize front/rear legs differently. Table V preserves all formulas and weights, which are essential task-specific components of the result.

TABLE V: Reward functions
Type Item Formula Weight
Quadrupedal Tracking Tracking lin vel exp⁡(−‖𝐯c,x​y−𝐯b,x​y‖2σ)\exp\left(-\frac{\left\|\mathbf{v}_{c,xy}-\mathbf{v}_{b,xy}\right\|^{2}}{\mathbf{\sigma}}\right) 7.07.0
Tracking ang vel exp⁡(−(𝐜yaw−ωyaw)2σ)\exp\left(-\frac{\left(\mathbf{c}_{\text{yaw}}-\mathbf{\omega}_{\text{yaw}}\right)^{2}}{\mathbf{\sigma}}\right) 2.52.5
Termination −1-1 −1.0-1.0
Alive 11 1.01.0
Quadrupedal Regularization Joint pos (𝐪−𝐪default)2\left(\mathbf{q}-\mathbf{q}_{\text{default}}\right)^{2} −0.05-0.05
Joint vel ‖𝐪˙‖2\left\|\dot{\mathbf{q}}\right\|_{2} −0.002-0.002
Joint acc ‖𝐪¨‖2\left\|\ddot{\mathbf{q}}\right\|_{2} −2×10−6-2\times 10^{-6}
Ang vel stability (‖ωt,x‖2+‖ωt,y‖2)\left(\left\|\mathbf{\omega}_{t,x}\right\|_{2}+\left\|\mathbf{\omega}_{t,y}\right\|_{2}\right) −0.2-0.2
Feet in air ∏i=1𝐍𝕀{𝐅i,zfoot<1}+∑i=1𝐍𝕀{𝐅i,zfoot<1}𝕀{𝐅i,zcalf≥1}\prod_{i=1}^{\mathbf{N}}\mathbb{I}\Bigl\{\mathbf{F}^{\text{foot}}_{i,z}<1\Bigr\}\;+\;\sum_{i=1}^{\mathbf{N}}\mathbb{I}\Bigl\{\mathbf{F}^{\text{foot}}_{i,z}<1\Bigr\}\,\mathbb{I}\Bigl\{\mathbf{F}^{\text{calf}}_{i,z}\geq 1\Bigr\} −0.05-0.05
Front hip pos ∑i∈ℐfront hip(𝐪i−𝐪idefault)2\sum_{i\in\mathcal{I}_{\text{front hip}}}\Bigl(\mathbf{q}_{i}-\mathbf{q}_{i}^{\text{default}}\Bigr)^{2} −0.2-0.2
Rear hip pos ∑i∈ℐrear hip(𝐪i−𝐪idefault)2\sum_{i\in\mathcal{I}_{\text{rear hip}}}\Bigl(\mathbf{q}_{i}-\mathbf{q}_{i}^{\text{default}}\Bigr)^{2} −0.5-0.5
Base height (𝐳base−1𝐍​∑i=1𝐍𝐡i−𝐡target)2\Bigl(\mathbf{z}_{\text{base}}-\frac{1}{\mathbf{N}}\sum_{i=1}^{\mathbf{N}}\mathbf{h}_{i}-\mathbf{h}_{\text{target}}\Bigr)^{2} −0.1-0.1
Balance ‖𝐅feet,0+𝐅feet,2−𝐅feet,1−𝐅feet,3‖2\left\|\mathbf{F}_{\text{feet},0}+\mathbf{F}_{\text{feet},2}-\mathbf{F}_{\text{feet},1}-\mathbf{F}_{\text{feet},3}\right\|_{2} −2×10−5-2\times 10^{-5}
Joint limit ‖(𝐪<𝐪min)∨(𝐪>𝐪max)‖1\left\|\Bigl(\mathbf{q}<\mathbf{q}_{\min}\Bigr)\lor\Bigl(\mathbf{q}>\mathbf{q}_{\max}\Bigr)\right\|_{1} −0.01-0.01
Torque exceed limit ∑i=1𝐍sub∑j=112max⁡{|τi​j|−τjlimit, 0}\sum_{i=1}^{\mathbf{N}_{\text{sub}}}\sum_{j=1}^{12}\max\Bigl\{\,|\mathbf{\tau}_{ij}|-\mathbf{\tau}^{\text{limit}}_{j},\;0\Bigr\} −2.0-2.0
Bipedal Stand Orientation (0.5​cos⁡θ+0.5)2,θ=arccos⁡(𝐠⋅𝐭‖𝐠‖​‖𝐭‖)\Bigl(0.5\,\cos\theta+0.5\Bigr)^{2},\quad\theta=\arccos\Bigl(\frac{\mathbf{g}\cdot\mathbf{t}}{\|\mathbf{g}\|\|\mathbf{t}\|}\Bigr) 1.01.0
Base height linear min⁡(max⁡(𝐳root−TminTmax−Tmin, 0), 1)\min\Bigl(\max\Bigl(\frac{\mathbf{z}_{\text{root}}-T_{\min}}{T_{\max}-T_{\min}},\,0\Bigr),\,1\Bigr) 0.80.8
Bipedal Tracking Tracking lin vel exp(−‖𝐯c−𝐯a‖2σ)𝕀{cosθ>0.95}𝐳base−𝐬low𝐬high−𝐬low\exp\Bigl(-\frac{\left\|\mathbf{v}_{c}-\mathbf{v}_{a}\right\|^{2}}{\mathbf{\sigma}}\Bigr)\,\mathbb{I}\Bigl\{\cos\mathbf{\theta}>0.95\Bigr\}\,\frac{\mathbf{z}_{\mathrm{base}}-\mathbf{s}_{\mathrm{low}}}{\mathbf{s}_{\mathrm{high}}-\mathbf{s}_{\mathrm{low}}} 3.03.0
Tracking ang vel exp(−(𝐜roll−ωroll)2σa​n​g)𝕀{cosθ>0.95}𝐳base−𝐬low𝐬high−𝐬low\exp\Bigl(-\frac{\left(\mathbf{c}_{\mathrm{roll}}-\mathbf{\omega}_{\mathrm{roll}}\right)^{2}}{\mathbf{\sigma}_{ang}}\Bigr)\,\mathbb{I}\Bigl\{\cos\mathbf{\theta}>0.95\Bigr\}\,\frac{\mathbf{z}_{\mathrm{base}}-\mathbf{s}_{\mathrm{low}}}{\mathbf{s}_{\mathrm{high}}-\mathbf{s}_{\mathrm{low}}} 2.52.5
Termination −1-1 −1.0-1.0
Alive 11 1.01.0
Bipedal Regularization Rear air ∏i∈𝐑𝕀{𝐟i,zrear<1}+∑i∈𝐑𝕀{𝐟i,zrear<1}𝕀{𝐟i,zrear−calf≥1}\prod_{i\in\mathbf{R}}\mathbb{I}\Bigl\{\mathbf{f}^{\mathrm{rear}}_{i,z}<1\Bigr\}\;+\;\sum_{i\in\mathbf{R}}\mathbb{I}\Bigl\{\mathbf{f}^{\mathrm{rear}}_{i,z}<1\Bigr\}\,\mathbb{I}\Bigl\{\mathbf{f}^{\mathrm{rear-calf}}_{i,z}\geq 1\Bigr\} −0.5-0.5
Front hip pos ∑i∈𝐅H(𝐪i−𝐪idefault)2\sum_{i\in\mathbf{F}_{H}}\Bigl(\mathbf{q}_{i}-\mathbf{q}_{i}^{\mathrm{default}}\Bigr)^{2} −0.1-0.1
Rear hip pos ∑i∈𝐑H(𝐪i−𝐪idefault)2\sum_{i\in\mathbf{R}_{H}}\Bigl(\mathbf{q}_{i}-\mathbf{q}_{i}^{\mathrm{default}}\Bigr)^{2} −0.18-0.18
Rear pos balance ‖𝐪irear​left−𝐪irear​right‖2\left\|\mathbf{q}^{\mathrm{rear\,left}}_{i}-\mathbf{q}^{\mathrm{rear\,right}}_{i}\right\|_{2} −0.05-0.05
Front joint pos 𝟏{𝐭>𝐓allow}∑i∈𝐅(𝐪i−𝐪idefault)2\mathbf{1}\Bigl\{\mathbf{t}>\mathbf{T}_{\mathrm{allow}}\Bigr\}\sum_{i\in\mathbf{F}}\Bigl(\mathbf{q}_{i}-\mathbf{q}_{i}^{\mathrm{default}}\Bigr)^{2} −0.2-0.2
Front joint vel 𝟏{𝐭>𝐓allow}∑i∈𝐅𝐪˙i2\mathbf{1}\Bigl\{\mathbf{t}>\mathbf{T}_{\mathrm{allow}}\Bigr\}\sum_{i\in\mathbf{F}}\dot{\mathbf{q}}_{i}^{2} −1×10−3-1\times 10^{-3}
Front joint acc 𝟏{𝐭>𝐓allow}∑i∈𝐅(Δ​𝐪˙iΔ​𝐭)2\mathbf{1}\Bigl\{\mathbf{t}>\mathbf{T}_{\mathrm{allow}}\Bigr\}\sum_{i\in\mathbf{F}}\Bigl(\frac{\Delta\dot{\mathbf{q}}_{i}}{\Delta\mathbf{t}}\Bigr)^{2} −2×10−6-2\times 10^{-6}
Legs energy substeps 1𝐍s​∑j=1𝐍s∑i=1𝐍dof(τi​j​𝐪˙i​j)2\frac{1}{\mathbf{N}_{s}}\sum_{j=1}^{\mathbf{N}_{s}}\sum_{i=1}^{\mathbf{N}_{\mathrm{dof}}}\Bigl(\mathbf{\tau}_{ij}\,\dot{\mathbf{q}}_{ij}\Bigr)^{2} −1×10−6-1\times 10^{-6}
Torque exceed limits ∑j∑imax⁡{|τi​j|−τilimit, 0}\sum_{j}\sum_{i}\max\Bigl\{|\mathbf{\tau}_{ij}|-\mathbf{\tau}^{\mathrm{limit}}_{i},\;0\Bigr\} −2.0-2.0
Joint limits 1𝐍s∑j∑i𝕀{𝐪i​j∉[𝐪imin,𝐪imax]}\frac{1}{\mathbf{N}_{s}}\sum_{j}\sum_{i}\mathbb{I}\Bigl\{\mathbf{q}_{ij}\notin[\mathbf{q}_{i}^{\min},\mathbf{q}_{i}^{\max}]\Bigr\} −0.06-0.06
Collision ∑k𝕀{∥𝐟k∥>1}\sum_{k}\mathbb{I}\Bigl\{\|\mathbf{f}_{k}\|>1\Bigr\} −2.0-2.0
Action rate ‖𝐚ilast−𝐚i‖2\left\|\mathbf{a}^{\mathrm{last}}_{i}-\mathbf{a}_{i}\right\|_{2} −0.03-0.03
Joint vel ‖𝐪˙‖2\left\|\dot{\mathbf{q}}\right\|_{2} −2×10−3-2\times 10^{-3}
Joint acc ‖𝐪¨‖2\left\|\ddot{\mathbf{q}}\right\|_{2} −3×10−6-3\times 10^{-6}

V-B Algorithm Pseudocode

Algorithm 1 trains an oracle from privileged state while learning its reconstruction. Algorithm 2 initializes from the oracle and anneals the probability of substituting estimated state during continued PPO training. Both algorithms retain the rollout, reset, reconstruction, and policy/value update details.

Algorithm 1 Training Stage 1
1: for total iteration do
2:   Initialize rollout buffer 𝒟←∅\mathcal{D}\leftarrow\emptyset
3:   for num steps do
4:    𝐳t←Enc⁡(𝐢t)\mathbf{z}_{t}\leftarrow\mathrm{Enc}(\mathbf{i}_{t})
5:    𝐥^t←[Estimator⁡(𝐩t,𝐜t),𝐩t]\hat{\mathbf{l}}_{t}\leftarrow[\mathrm{Estimator}(\mathbf{p}_{t},\mathbf{c}_{t}),\mathbf{p}_{t}]
6:    𝐥t←[𝐳t,𝐞t,𝐩t]\mathbf{l}_{t}\leftarrow[\mathbf{z}_{t},\mathbf{e}_{t},\mathbf{p}_{t}]
7:    𝐡t←LSTM⁡([𝐥t,𝐜t])\mathbf{h}_{t}\leftarrow\mathrm{LSTM}([\mathbf{l}_{t},\mathbf{c}_{t}])
8:    𝐠^←softmax⁡(g⁡(𝐡t))\hat{\mathbf{g}}\leftarrow\mathrm{softmax}(g(\mathbf{h}_{t}))
9:    𝐚t←∑i=1N𝐠^i⋅fi​(𝐡t)\mathbf{a}_{t}\leftarrow\sum_{i=1}^{N}\hat{\mathbf{g}}_{i}\cdot f_{i}(\mathbf{h}_{t})
10:    Execute 𝐚t\mathbf{a}_{t}, observe reward rtr_{t} and next state
11:    Reset if terminated
12:    t←t+1t\leftarrow t+1
13:    Store rollout in 𝒟\mathcal{D}
14:   end for
15:   Compute PPO losses: Lsurro,LvalueL_{\mathrm{surro}},\,L_{\mathrm{value}}
16:   Compute reconstruction loss:
17:    Lrecon=∑𝐥^i,𝐥i∈𝒟‖𝐥^i−𝐥i‖2L_{\mathrm{recon}}=\sum_{\begin{subarray}{c}\hat{\mathbf{l}}_{i},\,\mathbf{l}_{i}\in\mathcal{D}\end{subarray}}\left\|\hat{\mathbf{l}}_{i}-\mathbf{l}_{i}\right\|^{2}
18:   L=Lsurro+Lvalue+Lr​e​c​o​nL=L_{\mathrm{surro}}+L_{\mathrm{value}}+L_{recon}
19:   Update policy and value network
20: end for
21: return Oracle Policy
Algorithm 2 Training Stage 2
1: Copy parameters from Oracle Policy
2: for total iteration do
3:   Initialize rollout buffer 𝒟←∅\mathcal{D}\leftarrow\emptyset
4:   for num steps do
5:    𝐥^t←[Estimator⁡(𝐩t,𝐜t),𝐩t]\hat{\mathbf{l}}_{t}\leftarrow[\mathrm{Estimator}(\mathbf{p}_{t},\mathbf{c}_{t}),\,\mathbf{p}_{t}]
6:    𝐥t←[𝐳t,𝐞t,𝐩t]\mathbf{l}_{t}\leftarrow[\mathbf{z}_{t},\mathbf{e}_{t},\mathbf{p}_{t}]
7:    𝐏t←αt\mathbf{P}_{t}\leftarrow\alpha^{\text{t}}
8:    𝐥¯t←Probability​Selection​(𝐏t,𝐥^t,𝐥t)\bar{\mathbf{l}}_{t}\leftarrow\mathrm{Probability\ Selection}(\mathbf{P}_{t},\hat{\mathbf{l}}_{t},\mathbf{l}_{t})
9:    𝐡t←LSTM⁡([𝐥¯t,𝐜t])\mathbf{h}_{t}\leftarrow\mathrm{LSTM}([\bar{\mathbf{l}}_{t},\,\mathbf{c}_{t}])
10:    𝐠^←softmax⁡(g⁡(𝐡t))\hat{\mathbf{g}}\leftarrow\mathrm{softmax}(g(\mathbf{h}_{t}))
11:    𝐚t←∑i=1N𝐠^i⋅fi​(𝐡t)\mathbf{a}_{t}\leftarrow\sum_{i=1}^{N}\hat{\mathbf{g}}_{i}\cdot f_{i}(\mathbf{h}_{t})
12:    Execute 𝐚t\mathbf{a}_{t}, observe reward rtr_{t} and next state
13:    Reset if terminated
14:    t←t+1t\leftarrow t+1
15:    Store rollout in 𝒟\mathcal{D}
16:   end for
17:   Compute PPO losses: Lsurro,LvalueL_{\mathrm{surro}},\,L_{\mathrm{value}}
18:   Compute reconstruction loss:
19:    Lrecon=∑𝐥^i,𝐥i∈𝒟‖𝐥^i−𝐥i‖2L_{\mathrm{recon}}=\sum_{\begin{subarray}{c}\hat{\mathbf{l}}_{i},\,\mathbf{l}_{i}\in\mathcal{D}\end{subarray}}\left\|\hat{\mathbf{l}}_{i}-\mathbf{l}_{i}\right\|^{2}
20:   L=Lsurro+Lvalue+Lr​e​c​o​nL=L_{\mathrm{surro}}+L_{\mathrm{value}}+L_{recon}
21:   Update policy and value network
22: end for
23: return Final Policy

V-C Network Architecture Details

Actor, critic, and estimator use 256-dimensional recurrent modules. The table specifies privileged encoders, expert heads, the larger ordinary-network comparison head, and the gating network. These architectural differences explain how parameter counts are matched while changing the routing structure.

TABLE VI: Network architecture details
Network Type Dims
Actor RNN LSTM [256]
Critic RNN LSTM [256]
Estimator Module LSTM [256]
Estimator Latent Encoder MLP [256, 128]
Implicit Encoder MLP [32, 16]
Expert Head MLP [256, 128, 128]
Standard Head MLP [640, 640, 128]
Gating Network MLP [128]

V-D Domain Randomization

Simulation randomizes mass, center of mass, friction, initial joint positions, motor strength, and proprioception latency; latency spans 5–45 ms. Gaussian noise perturbs velocity, gravity, and joint observations. The complete distributions are retained. Randomization supports transfer under these disturbances but does not by itself certify safe behavior outside them.

TABLE VII: Domain randomization
Parameters Range Unit
Base mass [1, 3] k​gkg
Mass position of X axis [-0.2, 0.2] mm
Mass position of Y axis [-0.1, 0.1] mm
Mass position of Z axis [-0.05, 0.05] mm
Friction [0, 2] -
Initial joint positions [0.5, 1.5]×~\times~nominal value r​a​drad
Motor strength [0.9, 1.1]×~\times~nominal value -
Proprioception latency [0.005, 0.045] ss
TABLE VIII: Gaussian noise
Observation Gaussian Noise Amplitude Unit
Linear velocity 0.05 m/sm/s
Angular velocity 0.2 r​a​d/srad/s
Gravity 0.05 m/s2m/s^{2}
Joint position 0.01 r​a​drad
Joint velocity 1.5 r​a​d/srad/s

V-E Training Details

Fractal ground noise with maximum scale 0.1 discourages dragging feet. PPO uses a 0.2 clipping parameter, 0.99 discount, 0.95 advantage parameter, 0.001 learning rate, four minibatches, and 24 rollout steps per environment, with the remaining settings preserved in Table IX.

TABLE IX: PPO hyperparameters
   Hyperparameter       Value   
   clip min std       0.05   
   clip param       0.2   
   gamma       0.99   
   lam       0.95   
   desired kl       0.01   
   entropy coef       0.01   
   learning rate       0.001   
   max grad norm       1   
   num mini batch       4   
   num steps per env       24   
Concept demonstration

Open image ↗