At a glance
A policy can be split into a task module that interprets the goal and a robot module that produces robot-specific controls. Joint training shares each module across several combinations, then recombines familiar modules for a previously unseen robot–task pairing. Simulated experiments demonstrate zero-shot transfer in two settings and faster learning in a harder contact task; they do not establish transfer to entirely unseen robots or tasks.
I Introduction
The method separates reusable task knowledge, such as visual recognition and manipulation goals, from robot kinematics and dynamics. Instead of learning every combination independently, it trains a grid of shared modules and composes them at deployment. A small interface and dropout discourage memorizing individual training combinations.
II Related Work
Earlier transfer approaches reuse reward shaping, symbolic knowledge, policy parameters, or learned mappings. This work focuses on the representation: interchangeable modules for two independent factors of variation. Guided policy search supplies training signals, but the modular architecture is intended to support other policy-learning algorithms. [21], [22], [4].


III Modular Policy Networks
A world is one combination of robot identity and task identity. Robots may have different numbers of links, joints, observations, and actions, but each task must be achievable by every robot considered. Training covers only part of the robot–task grid; testing holds out combinations rather than entire factors. [4].
III-A Preliminaries
Observations and states are divided into intrinsic robot information and extrinsic task information. Costs similarly separate robot-dependent action penalties from task-dependent goals. This prescribed decomposition is essential: the method does not discover the factorization from arbitrary observations.
III-B Modularity
The task module processes extrinsic observations into an unsupervised latent interface. The robot module combines that interface with intrinsic observations to output the policy mean. Modules with the same robot or task identity share parameters. A held-out pair can be assembled only when its robot and its task have each appeared elsewhere during training.
Equations 1
| (1) |
III-C Architecture and Training
Neural modules parameterize Gaussian policies with learned, observation-independent covariance. Training minimizes losses summed across worlds while backpropagating through their shared modules. Experiments use synchronized data collection and guided policy search to regress onto locally optimized actions; the displayed derivatives give the shared-parameter gradient structure.
Equations
III-D Regularization
A narrow task-to-robot bottleneck limits the information available for memorizing combinations. Dropout further discourages specialized interface channels. Both regularizers aim to make independently trained module pairings compatible, especially when the robot–task grid is small.
IV Experiments
All experiments use simulated MuJoCo environments with contact, manipulation, and in some cases raw camera observations. The comparisons test zero-shot recombination or learning speed on a held-out pairing, using separate policies, incorrect task modules, and random initialization as baselines.
IV-A Reinforcement Learning Algorithm
Guided policy search alternates local trajectory optimization with supervised learning of a global neural policy. The BADMM variant penalizes disagreement between local controllers and the neural policy. This relatively sample-efficient training procedure is distinct from the modular representation itself.
IV-B Network Architecture
For visual tasks, convolution and spatial softmax form the task module, followed by fully connected robot-specific layers. Nonvisual tasks replace convolution with fully connected layers. End-effector position enters the task module because all robot morphologies expose it.
IV-C Reaching Colored Blocks in Simulation
Three arms and four target colors form twelve combinations; the four-link arm reaching the black block is withheld. Recombining its robot module with the black-target module achieves final target-distance errors of 0.08–0.28 across four test positions, versus 1.16–1.35 for random policies and 1.65–2.41 for the wrong task module. These are distances, not success percentages.
| Test Position | Random network | Wrong task module | Ours |
|---|---|---|---|
| 1 | 1.16 | 2.34 | 0.12 |
| 2 | 1.29 | 1.75 | 0.28 |
| 3 | 1.35 | 1.65 | 0.21 |
| 4 | 1.29 | 2.41 | 0.08 |
IV-D Object Manipulation
Three arms perform horizontal drawer pulling, vertical drawer pushing, and block pushing. The withheld three-link horizontal-drawer task does not succeed zero shot. Its composed policy nevertheless provides a strong initialization: it learns without reward shaping faster than training from scratch with shaping, while unshaped learning from scratch makes no progress.
IV-E Visually Distinct Manipulation Tasks
A second visual experiment combines reaching, block pushing, and peg insertion with three arm variants. The withheld four-link block-pushing pairing transfers without additional training. Its four reported final-distance values are 0.48, 0.19, 0.25, and 0.23, below both comparison policies; the retained table and images show the individual outcomes.
| Test Position | Random network | Wrong task module | Ours |
|---|---|---|---|
| 1 | 0.95 | 0.95 | 0.48 |
| 2 | 1.79 | 1.14 | 0.19 |
| 3 | 1.54 | 1.27 | 0.25 |
| 4 | 0.94 | 1.32 | 0.23 |
V Discussion and Future Work
Shared modules support immediate transfer when their interface generalizes and useful initialization when it does not. The evidence is limited to simulation and small grids of familiar robots and tasks. Simultaneous training is an operational limitation; sequential or lifelong module learning and larger robot populations remain future work.