Literature
Display
← All papers

Policy Architectures for Compositional Generalization in Control

Allan Zhou · Vikash Kumar · Chelsea Finn · Aravind Rajeswaran

· Reinforcement Learning Journal ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

Entity-centered policy architectures can learn and generalize better than a flat MLP when goals are compositions of object-level subgoals. Deep Sets pool independently encoded entities; self attention additionally models their relationships. Both handle varying entity counts without predefined action primitives.

Simulated Fetch manipulation demonstrates faster learning, transfer to more or fewer entities, and some novel combinations of pushing, switching, and stacking. Inputs are supplied compact object states, and most non-stacking tests disable cube–cube collisions.

1 Introduction

A scene-level goal can often be split into entity–subgoal pairs: each object has a desired state, but interactions may determine the order of operations. The paper formalizes this structure and builds invariant actor and critic networks to exploit it during ordinary reinforcement or imitation learning.

Refer to caption
Figure 1: A family of tasks where the agent is trained to re-arrange three cubes (top-left), but tested zero-shot to re-arrange anywhere from three to six cubes (bottom-left). Reinforcement learning (RL) with standard MLPs fails to even learn the 3-cube task. Our self-attention based policy can learn the three cube task, and also solve tasks with more cubes using no additional data.

2 Related Work

Unlike hierarchical methods that explicitly learn macro-actions, this approach changes policy architecture while retaining end-to-end low-level learning. Its attention operates over objects and goals, rather than trajectory history. It connects entity-centric control with permutation-invariant neural networks.

3 Problem Formulation and Architectures

The framework specifies entity-based tasks and compares three ways to represent their policies.

3.1 Problem Setup

An EFMDP contains an agent state, a set of entity states, and matching subgoals. Rewards aggregate entity-level outcomes; transitions and rewards are unchanged by consistently reordering entity–goal pairs. Entity identities can remain encoded in their features.

The symmetry makes optimal action values permutation-invariant and permits an invariant optimal policy. The source states the stronger claim that every optimal policy is invariant; when optimal actions tie, this requires a consistent invariant tie-breaking rule. The architecture motivation does not depend on that stronger wording.

Refer to caption
Figure 2: In an EFMDP, an agent interacts with entities that have corresponding subgoals. This framework can model rearrangement, strategic game playing, and multi-agent systems. In this “pushing and stacking” example, the agent must move the green cube to its subgoal (indicated by the green sphere), then stack the yellow cube on top of the green cube.
Equations 1–5
ℛ⁡(𝐬,𝐠)\displaystyle\mathcal{R}\left(\mathbf{s},\mathbf{g}\right) :=ℛ⁡({r~​(ei,gi,u)}i=1N)\displaystyle:=\mathcal{R}\left(\{\tilde{r}(e_{i},g_{i},u)\}_{i=1}^{N}\right) (1)
ℙ⁡(𝐬′|𝐬,𝐚)\displaystyle\mathbb{P}(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a}) :=ℙ((u′,{ei′}i=1N)|(u,{ei}i=1N),𝐚)\displaystyle\vcentcolon=\mathbb{P}\left(\left(u^{\prime},\{e_{i}^{\prime}\}_{i=1}^{N}\right)|\left(u,\{e_{i}\}_{i=1}^{N}\right),\mathbf{a}\right) (2)
π⋆:=argmaxπ{J(π):=𝔼π[∑t=0∞γtℛ(𝐬t,𝐠)]},\pi^{\star}\vcentcolon=\arg\max_{\pi}\ \ \left\{J(\pi):=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(\mathbf{s}_{t},\mathbf{g})\right]\right\}, (3)
σ​𝐬\displaystyle\sigma\mathbf{s} :=(u,eσ⁡(1),⋯,eσ⁡(N))\displaystyle\vcentcolon=(u,e_{\sigma(1)},\cdots,e_{\sigma(N)}) (4)
σ​𝐠\displaystyle\sigma\mathbf{g} :=(gσ⁡(1),⋯,gσ⁡(N))\displaystyle\vcentcolon=(g_{\sigma(1)},\cdots,g_{\sigma(N)}) (5)
π⋆​(σ​𝐬,σ​𝐠)\displaystyle\pi^{\star}(\sigma\mathbf{s},\sigma\mathbf{g}) =π⋆​(𝐬,𝐠)\displaystyle=\pi^{\star}(\mathbf{s},\mathbf{g})
Q⋆​(σ​𝐬,𝐚,σ​𝐠)\displaystyle Q^{\star}(\sigma\mathbf{s},\mathbf{a},\sigma\mathbf{g}) =Q⋆​(𝐬,𝐚,𝐠)\displaystyle=Q^{\star}(\mathbf{s},\mathbf{a},\mathbf{g})
Original figure
Figure 3: Visualizations of implementing an entity-based goal conditioned policy using either Deep Sets (left) or Self Attention (right). The policy π:(𝐬,𝐠)↦𝐚\pi:(\mathbf{s},\mathbf{g})\mapsto\mathbf{a} receives state 𝐬=(u,e1,⋯,eN)\mathbf{s}=(u,e_{1},\cdots,e_{N}) containing agent state uu and entity states eie_{i}. The goal (g1,⋯,gN)(g_{1},\cdots,g_{N}) contains subgoals for each entity. Both policies arrange the input into NN vectors yi=(u,ei,gi)y_{i}=(u,e_{i},g_{i}), one per entity. The Deep Set policy processes each yiy_{i} independently with MLP ϕ⁡(⋅)\phi(\cdot), aggregates the outputs, and maps the result to an action using MLP ρ⁡(⋅)\rho(\cdot). The self attention encoder SA​(⋅)\text{SA}(\cdot) produces output z1,⋯,zNz_{1},\cdots,z_{N} and uses self-attention (Eq. 11) to model interactions between the entities/subgoals. The ziz_{i} are mapped to an action by summation and an MLP ρ⁡(⋅)\rho(\cdot).

3.2 Multilayer Perceptrons (MLPs)

An MLP concatenates all states and goals into a fixed-size vector. Testing with more entities requires padding during training to reserve enough input slots; permutation invariance must be learned from data.

Equations 6–7
vec​(𝐬,𝐠):=Concatenate​(u,e1,⋯,eN⏟=𝐬,g1,⋯,gN⏟=𝐠)\text{vec}(\mathbf{s},\mathbf{g})\vcentcolon=\text{Concatenate}(\underbrace{u,e_{1},\cdots,e_{N}}_{=\mathbf{s}},\underbrace{g_{1},\cdots,g_{N}}_{=\mathbf{g}}) (6)
π⁡(𝐬,𝐠):=MLP​(vec​(𝐬,𝐠))\pi(\mathbf{s},\mathbf{g})\vcentcolon=\text{MLP}(\text{vec}(\mathbf{s},\mathbf{g})) (7)

3.3 Deep Sets

Deep Sets applies a shared encoder to each entity, its subgoal, and the shared agent state, sums those features, then predicts an action. Pooling removes dependence on entity order and accepts variable set sizes. General approximation properties require sufficiently expressive component networks. [38]

Equations 8–10
DS​(x):=ρ⁡(∑iϕ⁡(xi)),\text{DS}(x)\vcentcolon=\rho\left(\sum_{i}\phi(x_{i})\right), (8)
π⁡(𝐬,𝐠)\displaystyle\pi(\mathbf{s},\mathbf{g}) :=DS({yi}i=1N)\displaystyle\vcentcolon=\text{DS}\left(\{y_{i}\}_{i=1}^{N}\right) (9)
yi\displaystyle y_{i} :=Concatenate(u,ei⏟∈𝐬,gi⏟∈𝐠)\displaystyle\vcentcolon=\text{Concatenate}(\underbrace{u,e_{i}}_{\in\mathbf{s}},\underbrace{g_{i}}_{\in\mathbf{g}}) (10)
Refer to caption
Figure 4: Illustrations of the robot manipulation environments we study. They consist of subtasks such as pushing a cube to its (spherical) target, flipping a switch to a specified position, and stacking one cube on top of another. The overall task can involve multiple entities and subtasks as well as their combinations (pushing cubes and flipping switches).

3.4 Self Attention

Self attention exchanges information among entity–goal representations before summing them into an action. The encoder omits positional encodings and causal masks so arbitrary input order has no effect. Compared with independent entity encoding, it provides a stronger bias for interactions among objects. [36]

Equation 11
hi=∑j=1Nsoftmax​{(qi⋅kl)l=1N}j​vjh_{i}=\sum_{j=1}^{N}\text{softmax}\left\{(q_{i}\cdot k_{l})_{l=1}^{N}\right\}_{j}v_{j} (11)

4 Experiments and Evaluation

Evaluation separates training efficiency, zero-shot transfer to different entity counts, and combining previously learned subtasks without new training.

Environment Description

A simulated Fetch robot pushes cubes, flips switches, and stacks blocks. Task families vary object counts and combinations. A hand-designed oracle sequentially invokes single-subtask policies; it can fail because it does not account for interactions, so it is not a guaranteed-success upper bound.

4.1 Efficiency of Learning

Success requires every subgoal. Reinforcement learning uses DDPG with HER, applying the chosen architecture to both actor and critic. Imitation learning fits actions from successful trajectories generated by trained agents. Curves report five seeds with 95% confidence intervals.

Original figure
Figure 5: Training on environments of varying complexity using either reinforcement or imitation learning. Each row corresponds to a single environment family (N-Push, N-Switch, and N-Switch + N-Push), where environments with larger NN contain more entities and are more complex. For RL (left), each plot is a training curve of success rate vs the number of steps taken in the environment. RL with standard MLPs can solve the simpler tasks such as 1-Push, but Deep Set and Self Attention policies are superior on the more complex environments. For IL (right), we show success rates of behavior cloning against number of expert demonstrations in the dataset. Both Deep Set and Self Attention policies far outperform the MLP even when given less data. Shaded regions indicate 95%95\% CIs over 55 seeds.

RL results

All architectures learn simple single-entity tasks. Deep Sets and self attention learn substantially better as entity counts or mixed subtasks grow; the MLP often fails on harder tasks. Deep Sets commonly learns faster than attention, although their eventual performance is often similar.

IL Results

The entity-based models also use demonstrations more effectively. In the reported three-cube pushing comparison with 5,000 demonstrations, self attention approaches full success while the MLP remains near zero.

Conclusions

The results favor explicit entity structure over a flat representation. Attention is especially useful on some interaction-heavy tasks, while the simpler Deep Sets design often has better sample efficiency. Neither architecture dominates every comparison.

Refer to caption
Figure 6: Extrapolation capabilities of RL-trained policies with different architectures. Each row is an environment family that contains environments with a varying number of entities. Policies are trained on a single environment from each family before being tested on all the others, with no additional training. Bar charts show success rates in each environment, with the hatched bars corresponding to training environments. The Self Attention and Deep Set policies extrapolate beyond the training environment to solve tasks with more or fewer entities than seen in training, while MLP policies struggle on more complex testing environments. Error bars are 95%95\% CIs on 5 seeds.

4.2 Zero-Shot Extrapolation Capabilities

Policies trained on three cubes or switches are tested with one through six; mixed policies train with two of each and test with one through three. Checkpoints are selected by performance on training tasks. Structured policies often extrapolate well, but attention can fail on single-entity cases after multi-entity training. Deep Sets generalizes better in many of these tests.

4.3 Zero-Shot Stitching to solve novel tasks

A single policy trains on separate task families and is then tested on their combination. Attention achieves about 60% success on pushing followed by stacking, versus roughly 5% for an MLP, despite never training on that combination. Mixed switch-and-push performance remains limited partly because joint training itself is difficult; learned composition is useful but incomplete.

Refer to caption
Figure 7: Left: each row corresponds to a setting where policies are evaluated on test tasks that require stitching together skills learned in training, with no additional data. In the top setting, a single agent is trained on 2-Switch and 2-Push, and must solve 2-Switch+2-Push at test time. In the bottom setting, a single agent is trained on 2-Push and Stack, and must solve Push + Stack at test time. Right: average success rates of policies with different architectures in each setting. Deep Set and Self Attention policies are moderate successful at solving the test tasks, and are comparable to the Oracle in Push + Stack. The MLP fails to achieve nontrivial success rates on both test environments. Error bars indicate 95%95\% CIs over 55 seeds.

5 Conclusion

Architectural symmetry improves sample efficiency and selected forms of compositional generalization without handcrafted action primitives. The experiments assume structured state and goal vectors; discovering equivalent entities directly from images is left for future work.

Acknowledgements

The authors acknowledge colleagues who provided feedback, mentoring, and code review.

Appendix A Permutation invariance

The proof initializes value iteration with an invariant value function and shows each Bellman update preserves the symmetry by reindexing successor states. It assumes discrete state components for simplicity. Invariant optimal action values imply invariant sets of maximizing actions; a consistently chosen maximizer gives an invariant optimal policy.

Equations 12–17
π⋆​(σ​𝐬,σ​𝐠)=π⋆​(𝐬,𝐠)\displaystyle\pi^{\star}(\sigma\mathbf{s},\sigma\mathbf{g})=\pi^{\star}(\mathbf{s},\mathbf{g}) (12)
Q⋆​(σ​𝐬,𝐚,σ​𝐠)=Q⋆​(𝐬,𝐚,𝐠)\displaystyle Q^{\star}(\sigma\mathbf{s},\mathbf{a},\sigma\mathbf{g})=Q^{\star}(\mathbf{s},\mathbf{a},\mathbf{g}) (13)
ℛ⁡(𝐬,𝐚,𝐠)=ℛ⁡(σ​𝐬,𝐚,σ​𝐠)\displaystyle\mathcal{R}(\mathbf{s},\mathbf{a},\mathbf{g})=\mathcal{R}(\sigma\mathbf{s},\mathbf{a},\sigma\mathbf{g})
ℙ⁡(𝐬′|𝐬,𝐚)=ℙ⁡(σ​𝐬′|σ​𝐬,𝐚)\displaystyle\mathbb{P}(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})=\mathbb{P}(\sigma\mathbf{s}^{\prime}|\sigma\mathbf{s},\mathbf{a})
Qk+1⋆​(σ​𝐬,𝐚,σ​𝐠)\displaystyle Q^{\star}_{k+1}(\sigma\mathbf{s},\mathbf{a},\sigma\mathbf{g}) =ℛ⁡(σ​𝐬,𝐚,σ​𝐠)+γ​max⁡∑𝐬′∈𝒮𝐚′⁡ℙ⁡(𝐬′|σ​𝐬,𝐚)​Qk⋆​(𝐬′,𝐚′)\displaystyle=\mathcal{R}(\sigma\mathbf{s},\mathbf{a},\sigma\mathbf{g})+\gamma\max_{\mathbf{a}^{\prime}}\sum_{\mathbf{s}^{\prime}\in\mathcal{S}}\mathbb{P}(\mathbf{s}^{\prime}|\sigma\mathbf{s},\mathbf{a})Q^{\star}_{k}(\mathbf{s}^{\prime},\mathbf{a}^{\prime}) (14)
=ℛ⁡(𝐬,𝐚,𝐠)+γ​max⁡∑𝐬′∈𝒮𝐚′⁡ℙ⁡(σ−1​𝐬′|𝐬,𝐚)​Qk⋆​(σ−1​𝐬′,𝐚′)\displaystyle=\mathcal{R}(\mathbf{s},\mathbf{a},\mathbf{g})+\gamma\max_{\mathbf{a}^{\prime}}\sum_{\mathbf{s}^{\prime}\in\mathcal{S}}\mathbb{P}(\sigma^{-1}\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})Q^{\star}_{k}(\sigma^{-1}\mathbf{s}^{\prime},\mathbf{a}^{\prime}) (15)
=ℛ⁡(𝐬,𝐚,𝐠)+γ​max⁡∑𝐬′∈𝒮𝐚′⁡ℙ⁡(𝐬′|𝐬,𝐚)​Qk⋆​(𝐬′,𝐚′)\displaystyle=\mathcal{R}(\mathbf{s},\mathbf{a},\mathbf{g})+\gamma\max_{\mathbf{a}^{\prime}}\sum_{\mathbf{s}^{\prime}\in\mathcal{S}}\mathbb{P}(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a})Q^{\star}_{k}(\mathbf{s}^{\prime},\mathbf{a}^{\prime}) (16)
=Qk+1⋆​(𝐬,𝐚,𝐠)\displaystyle=Q^{\star}_{k+1}(\mathbf{s},\mathbf{a},\mathbf{g}) (17)
∑𝐬∈𝒮f(σ−1𝐬)=∑u∈𝒰∑e1∈ℰ⋯∑eN∈ℰf(u,eσ−1​(1),⋯,eσ−1​(N))=∑𝐬∈𝒮f(𝐬)\sum_{\mathbf{s}\in\mathcal{S}}f(\sigma^{-1}\mathbf{s})=\sum_{u\in\mathcal{U}}\sum_{e_{1}\in\mathcal{E}}\cdots\sum_{e_{N}\in\mathcal{E}}f(u,e_{\sigma^{-1}(1)},\cdots,e_{\sigma^{-1}(N)})=\sum_{\mathbf{s}\in\mathcal{S}}f(\mathbf{s})
π⋆​(σ​𝐬,σ​𝐠)=arg⁡max𝐚​Q⋆​(σ​𝐬,𝐚,σ​𝐠)=arg⁡max𝐚​Q⋆​(𝐬,𝐚,𝐠)=π⋆​(𝐬,𝐠)\pi^{\star}(\sigma\mathbf{s},\sigma\mathbf{g})=\arg\max_{\mathbf{a}}Q^{\star}(\sigma\mathbf{s},\mathbf{a},\sigma\mathbf{g})=\arg\max_{\mathbf{a}}Q^{\star}(\mathbf{s},\mathbf{a},\mathbf{g})=\pi^{\star}(\mathbf{s},\mathbf{g})

Appendix B Environments

Actions command three end-effector displacements and one gripper coordinate; the latter is disabled when stacking is unnecessary. Episode length adds twenty steps per switch and fifty per cube subtask. States provide object pose, velocity, type, and robot information.

Most non-stacking experiments disable cube–cube collisions, although the gripper can still disturb other objects. Dense rewards reflect average goal distance; sparse rewards require every subgoal within tolerance. A supplementary experiment restores collisions.

Appendix C Training details

Training details specify algorithms, network sizes, optimization, and computational costs.

C.1 Reinforcement learning

DDPG gathers experience from sixteen parallel environments into one replay buffer, with two episodes per five gradient updates and HER on 80% of sampled minibatches. Shared discount is 0.98. Learning rates are 0.001 for MLP/Deep Sets and 0.0001 for attention.

Exploration and training duration vary by environment; fast and slow decay schedules are compared where indicated. Attention uses two four-head encoder blocks without dropout. Complete actor/critic specifications and schedules remain in the original tables.

Equation
a~∼{a+𝒩⁡(0,η),with prob ​1−ϵUniform​(−1,1),with prob ​ϵ\tilde{a}\sim\begin{cases}a+\mathcal{N}(0,\eta),&\text{with prob }1-\epsilon\\ \text{Uniform}(-1,1),&\text{with prob }\epsilon\end{cases}
Table 1: General shared RL hyperparameters
Hyperparameter Value
Discount γ\gamma 0.98
Parallel envs 16
Replay buffer size 10610^{6}
Relabel prob 0.8
Ratio of episodes : updates 2:52:5
Optimizer Adam
Learning rate MLP, Deep Set: 0.001
Self Attention: 0.0001
Reward Scale Sparse: 1; Dense: 5
Action noise η0\eta_{0} (initial) 0.2
Random action prob ϵ0\epsilon_{0} (initial) 0.3
Table 2: Environment specific RL hyperparameters
Environment Reward Epochs Exploration decay Target update speed τ\tau
1-Push Sparse 50 Constant(1) 0.95
2-Push Dense 150 Lin(.01, 75, 125) 0.99
3-Push Dense 250 Fast: Lin(.01, 30, 80) 0.99
Slow: Lin(.01, 100, 175)
{1,2,3}-Switch Sparse {10, 50, 100} Constant(1) 0.95
1-Switch + 1-Push Dense 150 Lin(.01, 60, 100) 0.99
2-Switch + 2-Push Dense 250 Fast: Lin(.01, 75, 150) 0.99
Slow: Lin(.01, 100, 150)
Table 3: RL architectures
Family Actor Critic
MLP Linear(256)×3\times 3, Linear(A) Linear(256) ×3\times 3, Linear(1)
Deep Set ϕ\phi: Linear(256) ×3\times 3 ϕ\phi: Linear(256) ×2\times 2
ρ\rho: Linear(A) ρ\rho: Linear(256), Linear(1)
Self Attention SA: Linear(256), Block(256, 256, 4)×2\times 2 SA: Linear(256), Block(256, 256, 4)×2\times 2
ρ\rho: Linear(A) ρ\rho: Linear(A)

C.2 Imitation Learning

Behavior-cloning datasets contain 1,000–5,000 generated trajectories, filtered to successful episodes. Policies minimize action squared error for 60,000 Adam updates with batch size 128 and the same architecture-specific learning rates as reinforcement learning.

Equation
arg⁡minπ⁡J⁡(π):=1|𝒟|​∑(𝐬,𝐚)∼𝒟‖π⁡(𝐬)−𝐚‖2\arg\min_{\pi}J(\pi)\vcentcolon=\frac{1}{|\mathcal{D}|}\sum_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}}||\pi(\mathbf{s})-\mathbf{a}||^{2}

C.3 Training and inference speed

Deep Sets has parameter count independent of entity count and linear per-entity computation. Attention’s pairwise interactions require quadratic work and memory. Its GPU forward pass is slower, but total training times are similar in these small environments because simulation dominates; tests contain fewer than ten entities.

Original figure
Figure 8: Left: the time (in milliseconds) it takes for each policy architecture to execute a single forward pass on a single input observation from the N-Push environments, where N∈{1,2,3}N\in\{1,2,3\}. The self attention policy is significantly slower, while the Deep Set and MLP policies are comparable. Right: Real world reinforcement learning times (in hours) training each policy/critic architecture on the N-Push environments. Although the Self Attention policy is slightly slower, all policies train at comparable speeds in the same environment. This suggests that environment simulation, not policy execution, is the dominant time consuming element.

Appendix D Further comparisons

Additional comparisons vary the Deep Sets output network and restore cube collisions.

D.1 Deep set architecture size

A two-layer output network improves some training and push-then-stack results, but extrapolates worse to five or six cubes than the one-layer version. Model capacity therefore trades off different forms of generalization rather than uniformly helping.

Original figure
Figure 9: Comparison of N-Push extrapolation and Push + Stack stitching performance when using small and large variants of the Deep Set policy architecture. The small version implements ρ\rho with a 1-layer linear map, while the large version implements ρ\rho with a 2-layer MLP. For N-Push, the larger network achieves greater success rates in the training environment (3 cubes) but is actually worse when extrapolating to 5 or 6 cubes. On the other hand, the larger Deep Set displays superior stitching capability and achieves a higher average success rate when generalizing to Push + Stack from 2-Push and Stack.

D.2 N-Push with cube-cube collisions

With cube–cube collisions enabled, success drops across methods by roughly the reported 15%, and attention often overtakes Deep Sets. These results support relational modeling when entities physically interact, while also showing the main collision-disabled setting is easier.

Original figure
Figure 10: N-Push extrapolation with cube-cube collisions enabled. All methods observe some drop in performance relative to Figure 6, where N-Push has cube-cube collisions disabled. Self Attention tends to outperform Deep Sets when collisions enabled, likely because its relational inductive biases are better suited to handling interactions between entities that arise from collisions.
Concept demonstration

Open image ↗