Literature
Display
← All papers

Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer

Coline Devin · Abhishek Gupta · Trevor Darrell · Pieter Abbeel · Sergey Levine

· ICRA 2017 ↗

Robotics4 min text · CondensedOriginal paper ↗
Contents

At a glance

A policy can be split into a task module that interprets the goal and a robot module that produces robot-specific controls. Joint training shares each module across several combinations, then recombines familiar modules for a previously unseen robot–task pairing. Simulated experiments demonstrate zero-shot transfer in two settings and faster learning in a harder contact task; they do not establish transfer to entirely unseen robots or tasks.

I Introduction

The method separates reusable task knowledge, such as visual recognition and manipulation goals, from robot kinematics and dynamics. Instead of learning every combination independently, it trains a grid of shared modules and composes them at deployment. A small interface and dropout discourage memorizing individual training combinations.

II Related Work

Earlier transfer approaches reuse reward shaping, symbolic knowledge, policy parameters, or learned mappings. This work focuses on the representation: interchangeable modules for two independent factors of variation. Guided policy search supplies training signals, but the modular architecture is intended to support other policy-learning algorithms. [21], [22], [4].

Refer to caption
Refer to caption
Fig. 1: The 3DoF and a 4DoF robot which specify one degree of variation (robots) in the universe described in Section III as well as the tasks of opening a drawer and pushing a block which specify the other degree of variation (tasks) in the universe.
Refer to caption
Fig. 2: The possible worlds enumerated for all combinations of tasks and robots for the universe described in Section III

III Modular Policy Networks

A world is one combination of robot identity and task identity. Robots may have different numbers of links, joints, observations, and actions, but each task must be achievable by every robot considered. Training covers only part of the robot–task grid; testing holds out combinations rather than entire factors. [4].

III-A Preliminaries

Observations and states are divided into intrinsic robot information and extrinsic task information. Costs similarly separate robot-dependent action penalties from task-dependent goals. This prescribed decomposition is essential: the method does not discover the factorization from arbitrary observations.

III-B Modularity

The task module processes extrinsic observations into an unsupervised latent interface. The robot module combines that interface with intrinsic observations to output the policy mean. Modules with the same robot or task identity share parameters. A held-out pair can be assembled only when its robot and its task have each appeared elsewhere during training.

Equations 1
ϕwr​k​(ow)=ϕwr​k​(ow,𝒯,ow,ℛ)=fr​(gk​(ow,𝒯),ow,ℛ)\phi_{w_{rk}}(o_{w})=\phi_{w_{rk}}(o_{w,\mathcal{T}},o_{w,\mathcal{R}})=f_{r}(g_{k}(o_{w,\mathcal{T}}),o_{w,\mathcal{R}}) (1)
ϕwtest​(otest)\displaystyle\phi_{w_{\text{test}}}(o_{\text{test}}) =ϕwtest​(owtest,𝒯,owtest,ℛ)\displaystyle=\phi_{w_{\text{test}}}(o_{w_{\text{test}},\mathcal{T}},o_{w_{\text{test}},\mathcal{R}})
=frtest​(gktest​(owtest,𝒯),owtest,ℛ)\displaystyle=f_{r_{\text{test}}}(g_{k_{\text{test}}}(o_{w_{\text{test}},\mathcal{T}}),o_{w_{\text{test}},\mathcal{R}})
Original figure
Fig. 3: Modular policy composition for a universe with 2 tasks and 2 robots. There are 4 available modules - 2 task modules and 2 robot modules, and each module is a nerual network. For the training worlds, these modules are composed together to form the individual policy networks. Modules of the same color share their weights. Policy networks for the same task share task modules and those for the same robot share robot modules. The training worlds are composed and then trained end-to-end. On encountering an unseen world, the appropriate previously trained modules are composed to give a policy capable of good zero-shot performance

III-C Architecture and Training

Neural modules parameterize Gaussian policies with learned, observation-independent covariance. Training minimizes losses summed across worlds while backpropagating through their shared modules. Experiments use synchronized data collection and guided policy search to regress onto locally optimized actions; the displayed derivatives give the shared-parameter gradient structure.

Equations
∂πwr​k∂θr=∂πwr​k∂ϕwr​k​∂ϕwr​kθr\frac{\partial\pi_{w_{rk}}}{\partial\theta_{r}}=\frac{\partial\pi_{w_{rk}}}{\partial\phi_{w_{rk}}}\frac{\partial\phi_{w_{rk}}}{\theta_{r}}
∂πwr​k∂θk=∂πwr​k∂ϕwr​k​∂ϕwr​kθk\frac{\partial\pi_{w_{rk}}}{\partial\theta_{k}}=\frac{\partial\pi_{w_{rk}}}{\partial\phi_{w_{rk}}}\frac{\partial\phi_{w_{rk}}}{\theta_{k}}
∂πwr​k∂θr=∂πwr​k∂fr​∂fr∂θr\frac{\partial\pi_{w_{rk}}}{\partial\theta_{r}}=\frac{\partial\pi{{}_{w_{rk}}}}{\partial f_{r}}\frac{\partial f_{r}}{\partial\theta_{r}}
∂πwr​k∂θk=∂πwr​k∂fr​∂fr∂gk​∂gk∂θk\frac{\partial\pi_{w_{rk}}}{\partial\theta_{k}}=\frac{\partial\pi{{}_{w_{rk}}}}{\partial f_{r}}\frac{\partial f_{r}}{\partial g_{k}}\frac{\partial g_{k}}{\partial\theta_{k}}

III-D Regularization

A narrow task-to-robot bottleneck limits the information available for memorizing combinations. Dropout further discourages specialized interface channels. Both regularizers aim to make independently trained module pairings compatible, especially when the robot–task grid is small.

IV Experiments

All experiments use simulated MuJoCo environments with contact, manipulation, and in some cases raw camera observations. The comparisons test zero-shot recombination or learning speed on a held-out pairing, using separate policies, incorrect task modules, and random initialization as baselines.

IV-A Reinforcement Learning Algorithm

Guided policy search alternates local trajectory optimization with supervised learning of a global neural policy. The BADMM variant penalizes disagreement between local controllers and the neural policy. This relatively sample-efficient training procedure is distinct from the modular representation itself.

Refer to caption
Fig. 4: Basic visuomotor policy network for a single robot. The two convolutional layers and spatial softmax form the task module, while the last few fully connected layers form the robot module

IV-B Network Architecture

For visual tasks, convolution and spatial softmax form the task module, followed by fully connected robot-specific layers. Nonvisual tasks replace convolution with fully connected layers. End-effector position enters the task module because all robot morphologies expose it.

Refer to caption
Fig. 5: Grid of tasks vs robots for the reaching colored blocks task in simulation described in  IV-C. We train on all the worlds besides the 4link robot reaching to black, and test on this held out world.

IV-C Reaching Colored Blocks in Simulation

Three arms and four target colors form twelve combinations; the four-link arm reaching the black block is withheld. Recombining its robot module with the black-target module achieves final target-distance errors of 0.08–0.28 across four test positions, versus 1.16–1.35 for random policies and 1.65–2.41 for the wrong task module. These are distances, not success percentages.

Test Position Random network Wrong task module Ours
1 1.16 2.34 0.12
2 1.29 1.75 0.28
3 1.35 1.65 0.21
4 1.29 2.41 0.08
TABLE I: We evaluate the zero shot performance of the 4-link arm reach to the black block. The numbers shown in the table are average distances from the end-effector to the black block over the last 5 timesteps of a sample from the policy; a perfect policy would get 0. We see that composing the 4-link robot module with the reach to black-block task module generates very good performance (under the column Ours), while composing a different task module with the correct robot module, or running a random policy does quite poorly.
Refer to caption
Fig. 6: Grid of tasks vs robots for the object manipulation tasks described in  IV-D. The horizontal drawer tasks involve sliding the grey drawer horizontally to the target in the direction indicated by the arrow on the image. The vertical drawer tasks involve sliding the grey drawer vertically up in the direction indicated by the arrow on the image. The block push tasks involve pushing the white block to the red goal. All positions shown in this image are the final successful positions for each world.

IV-D Object Manipulation

Three arms perform horizontal drawer pulling, vertical drawer pushing, and block pushing. The withheld three-link horizontal-drawer task does not succeed zero shot. Its composed policy nevertheless provides a strong initialization: it learns without reward shaping faster than training from scratch with shaping, while unshaped learning from scratch makes no progress.

Refer to caption
Fig. 7: Results on the 3-link robot performing horizontal drawer pulling. The initialization from composing the 3-link robot module with the horizontal pull task module provides the fastest learning. Although the vertical drawer module was trained with the 3-link robot, the task is too different to directly transfer. Random initialization performs well with reward shaping, but without it is unable to learn the task.
Refer to caption
Fig. 8: R-obots for visually distinct tasks mentioned in  IV-E. The reach task involves reaching the white target. The push task involves pushing the white block to the red goal. The peg insertion task involves inserting the peg at the tip of the robot into the hole specified by the black square. These tasks involve manipulation and are also visually distinct in that they do not use the colors in the same way (the target and block colors are not consistent). We train on all combinations besides the 4link robot pushing the block.

IV-E Visually Distinct Manipulation Tasks

A second visual experiment combines reaching, block pushing, and peg insertion with three arm variants. The withheld four-link block-pushing pairing transfers without additional training. Its four reported final-distance values are 0.48, 0.19, 0.25, and 0.23, below both comparison policies; the retained table and images show the individual outcomes.

Test Position Random network Wrong task module Ours
1 0.95 0.95 0.48
2 1.79 1.14 0.19
3 1.54 1.27 0.25
4 0.94 1.32 0.23
TABLE II: Zero-shot results on the 4-link performing the block-pushing task from section IV-E. The values are the distance between the drawer and its target position averaged over tha last five time steps of each sample. Forming the policy by composing the 4-link module with the block pushing module performs best even those modules were not trained together. Choosing the reach module instead performs on par with a random network. We show that the task and robot modules are able to generalize to unseen robot-task combinations without additional training.
Refer to caption
Fig. 9: The final positions of the zero-shot performance of our method on the blockpushing task. Our method performs the task very well on zero shot and gets the white block to the red goal.

V Discussion and Future Work

Shared modules support immediate transfer when their interface generalizes and useful initialization when it does not. The evidence is limited to simulation and small grids of familiar robots and tasks. Simultaneous training is an operational limitation; sequential or lifelong module learning and larger robot populations remain future work.

Concept demonstration

Open image ↗