At a glance
MCP represents a policy as a weighted product of learned Gaussian motor primitives. Several primitives influence each action simultaneously, allowing new combinations after pretraining. A new gating policy learns how to combine frozen primitives for a transfer task.
Experiments use simulated ants, bipeds, humanoids, and a T-Rex. The largest gains appear on difficult locomotion-and-manipulation tasks; the results depend on useful motion-imitation pretraining and do not show physical robot deployment.
1 Introduction
Relearning each task from scratch wastes useful motor experience. Temporal skill selection activates one behavior at a time; MCP instead combines primitives within the current action distribution. This creates a continuous control space that a transfer policy can explore while retaining the pretrained motor representation.
2 Preliminaries
Pretraining and transfer tasks share states, actions, and dynamics, while goals and rewards may differ. Transfer assumes some relevant shared skills. An additive mixture samples one primitive before sampling an action; MCP’s product combines contributions before drawing the action.
Equation 1
| (1) |
3 Multiplicative Compositional Policies
Each primitive supplies an action density, raised to a nonnegative gating weight. Multiplication and normalization produce the composite density. Weights need not sum to one; experiments bound them between zero and one.
Equation 2
| (2) |
3.1 Gaussian Primitives
A product of diagonal Gaussian primitives is another diagonal Gaussian. Its mean uses precision-weighted primitive means, and its precision sums the weighted primitive precisions. Consequently, both the gate and each primitive’s uncertainty determine influence on each action coordinate. This closed form supports ordinary end-to-end policy optimization.
Equation 3
| (3) |
3.2 Pre-Training and Transfer
All primitives train jointly across pretraining tasks. Only the gate sees the goal; primitives see state, encouraging reusable behavior rather than a goal-specific controller inside every primitive. At transfer, primitive parameters are frozen and a new gate learns composition weights. The original algorithm gives both stages.
Equation 4
| (4) |
4 Related Work
The approach relates to transfer learning, hierarchical policies, and continuous latent skill spaces. Freezing primitives preserves their learned parameters while allowing the new gate to repurpose them.
Hierarchical Policies:
Hierarchical methods typically learn or pretrain low-level skills and select among them over time. MCP’s comparison asks whether simultaneous composition provides a richer range of actions than selecting one skill at a time.
Latent Space Models:
Continuous latent models map a code into motor actions. [14] MCP gives that code explicit structure as primitive-composition weights. The proposed benefit is a reusable action representation whose combinations remain flexible on new goals.
5 Experiments
The benchmark uses a 14-degree-of-freedom ant, 23-degree-of-freedom biped, 34-degree-of-freedom humanoid, and 55-degree-of-freedom T-Rex. Tests combine locomotion, balance, object transport, and ball manipulation in simulation.
5.1 Experimental Setup
The protocol first learns imitation skills, then trains transfer controllers on new rewards.
Pre-Training Tasks:
Pretraining tracks walking and turning motion clips. The gate receives the next two target states. Random clip changes during an episode encourage transitions between skills; the primitives remain state-conditioned.
Transfer Tasks:
Heading follows a changing direction. Carry picks up a box, transports it, and places it at a target; the simulator attaches the box through a virtual joint after designated contact. Dribble moves a ball to a target through contact while maintaining balance. Goal inputs provide relevant object and target state.
| Environment | Scratch | Finetune | Hierarchical | Option-Critic | MOE | Latent Space | MCP (Ours) |
|---|---|---|---|---|---|---|---|
| Heading: Biped | |||||||
| Carry: Biped | |||||||
| Dribble: Biped | |||||||
| Dribble: Humanoid | |||||||
| Dribble: T-Rex | |||||||
| Holdout: Ant |
Model Representation:
The main model has eight primitives and separate gate and primitive networks. State includes body-relative link poses and velocities; actions specify joint PD targets at 30 Hz. PPO trains the policies, with the full network structure retained in Figure 3. [34]
5.2 Comparisons
Three seeds per method and environment are evaluated over about 100 episodes each. Scores are normalized returns, not success percentages. MCP reaches 0.575 on biped carrying versus 0.456 for the latent model, 0.805 versus 0.751 on humanoid dribbling, and 0.781 versus 0.115 on T-Rex dribbling. It is the only tested model reported to solve the latter task.
Benefits are not universal: heading is easier for several baselines, and scratch training scores 0.951 on held-out ant directions versus MCP’s 0.812. The authors attribute MCP’s harder-task gains partly to flexible combinations of primitives, while latent decoders can overfit pretraining behaviors.
5.3 Exploration Behaviors
Randomly combining pretrained primitives produces coherent motion in different directions. Direct independent action noise instead makes the humanoid fall quickly. Latent and hierarchical pretraining also improve exploration, so structured exploration is not unique to MCP.
5.4 Primitive Specializations
Primitive activation varies with gait phase: some components peak during left stance and others during right stance. PCA plots cluster their proposed actions. These observations support specialization, although the primitives are learned rather than assigned explicit semantic labels.
6 Conclusion
Products of motor distributions enable reusable spatial composition and strong transfer on selected difficult simulated tasks. Choosing an expressive motion corpus remains important. Combining temporal abstraction with MCP and learning comparable primitives without motion supervision are future directions.
Acknowledgements
The authors acknowledge computational support and funding from university, government, and industry partners.
Supplementary Material
The supplement retains additional comparisons, reference-motion details, task rewards, and complete model hyperparameters.
7 Additional Experiments
Supplementary curves count environment interactions, including the lower-frequency hierarchical controller fairly in physical timesteps. Four and eight MCP primitives perform similarly; increasing to sixteen or thirty-two slows learning in these tests. Thirty-two composition weights exceed the humanoid’s 28-dimensional action space, weakening the intended reduction in control dimensionality.
| Environment | Scratch | Finetune | Hierarchical | Option-Critic | MOE | Latent Space | MCP (Ours) |
|---|---|---|---|---|---|---|---|
| Heading: Biped | |||||||
| Heading: Humanoid | |||||||
| Heading: T-Rex | |||||||
| Carry: Biped | |||||||
| Dribble: Biped | |||||||
| Dribble: Humanoid | |||||||
| Dribble: T-Rex | |||||||
| Holdout: Ant |
| Property | Biped | Humanoid | T-Rex |
|---|---|---|---|
| Links | 12 | 13 | 20 |
| Total Mass (kg) | 42 | 45 | 54.5 |
| Height (m) | 1.34 | 1.62 | 1.66 |
| Degrees-of-Freedom | 23 | 34 | 55 |
| State Features | 105 | 196 | 261 |
| Action Parameters | 17 | 28 | 49 |
8 Reference Motions
Biped and humanoid pretraining uses 230 seconds of motion-capture walking and turning. Retargeting to the biped removes unused upper-body joints. The T-Rex uses only eleven seconds of hand-authored walking and turning animation, showing reuse from a small but deliberately relevant repertoire.
9 Transfer Tasks
These sections specify goal state, randomization, and reward shaping for each transfer task.
Heading:
Heading changes by a uniform perturbation between −0.15 and 0.15 radians each step. Reward favors center-of-mass velocity of one meter per second along the requested direction.
Equation
Carry:
The five-kilogram box and destination are randomized up to ten meters apart. A virtual joint attaches the box after contact with a designated body link and detaches it at the destination. Reward combines approaching the box, staying near it, moving it toward the target, and reducing target distance, with weights 0.1/0.2/0.3/0.4. This is simplified carrying rather than learned physical grasping.
Equations
Dribble:
Ball and goal locations are randomized up to ten meters apart. Reward balances approaching and remaining near the ball with advancing it toward the target, using weights 0.1/0.1/0.3/0.5. Unlike carrying, motion of the ball depends on contact dynamics without an attachment joint.
Equations
Holdout:
The ant is pretrained on heading angles from zero to 3π/2, then transferred to the remaining quarter-circle. Reward adapts the usual locomotion objective to the requested direction while retaining health, control, and contact terms. This measures adaptation beyond the pretrained direction range.
Equations
10 Model Setup
Most models use PPO; option-critic follows its original updates. Discount factors are 0.95 during pretraining and 0.99 during transfer. Value fitting uses multi-step TD returns and policy updates use GAE.
10.1 mcp
MCP uses eight primitives and a value network with 1,024 and 512 hidden units. Table 5 specifies character-specific learning rates, batch size 4,096, minibatches 256, momentum 0.9, and PPO clipping 0.02.
10.2 Scratch
The scratch baseline learns a Gaussian action policy directly on each transfer task using 1,024- and 512-unit hidden layers, fixed diagonal covariance, and a matching value architecture. Table 6 preserves its tuned learning rates.
10.3 Finetuning
The fine-tuning baseline first imitates a reference motion, then updates the same full policy on transfer tasks. Additional goal inputs are inserted without changing its initial behavior; its architecture otherwise matches scratch training.
10.4 Hierarchical
The hierarchical baseline has seven separately trained walking or turning primitives. Frozen skills are selected once per approximately one-second gait cycle by a learned gate. This temporal selection frequency differs from the other controllers and is accounted for using environment-step comparisons.
10.5 Option-Critic
Option-critic learns eight options and termination functions during imitation. Both see state only, while the option-value function also sees goals. Transfer freezes options and terminations and learns new option values; joint fine-tuning was tried but produced degradation without clear gains.
| Parameter | Biped | Humanoid | T-Rex |
|---|---|---|---|
| Options | |||
| Stepsize | |||
| Stepsize | |||
| Stepsize | |||
| Batch Size | |||
| SGD Momentum | |||
| Termination Cost |
10.6 Mixture-of-Experts
The additive MOE comparison also trains eight state-conditioned primitives and a goal-aware gate. It freezes primitives for transfer but samples only one primitive at each step. This isolates an important difference from simultaneous product composition.
10.7 Latent Space
The latent baseline uses an eight-dimensional Gaussian code with KL regularization toward a standard Gaussian. A pretrained decoder maps state and code into actions; transfer freezes that decoder and learns a new state-and-goal encoder. Its complete architecture and hyperparameters remain in Table 11.
Equation 5
| (5) |