Literature
Display
← All papers

CompILE: Compositional Imitation Learning and Execution

Kipf, Thomas · Li, Yujia · Dai, Hanjun · Zambaldi, Vinicius · Sanchez-Gonzalez, Alvaro · Grefenstette, Edward · Kohli, Pushmeet · Battaglia, Peter

· ICML 2019 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

CompILE learns reusable, variable-length behavior segments from state–action demonstrations. A differentiable segmentation model infers boundaries and latent codes, while subpolicies reconstruct the demonstrated actions. Grid-world and simulated reacher experiments test segmentation and longer task sequences; discovered grid-world skills also help hierarchical reinforcement learning under sparse rewards.

1 Introduction

The model jointly discovers where one behavior ends and what that behavior represents. After explaining one segment, it masks that portion of the trajectory and processes the remainder with shared parameters. This repeated structure supports new combinations and longer sequences. Segment labels are not required, although experiments provide the number of segments as weak supervision.

Original figure
Figure 1: Joint unsupervised learning of task segmentation and encoding in CompILE. CompILE auto-encodes sequential demonstration data by 1) softly breaking an input sequence into segments of variable length, and 2) mapping each such segment into a latent code, which can be executed to reconstruct the input sequence. At test time, the latent code can be re-composed to produce novel behavior.

2 Model overview

A conditional variational autoencoder (VAE) represents each demonstrated state–action trajectory as a sequence of latent behavior codes and boundaries. The state sequence conditions reconstruction; the model learns which action to take at each state.

2.1 Behavioral cloning

Behavioral cloning (BC) maximizes the likelihood of expert actions given observed states. CompILE retains this imitation objective while dividing responsibility among latent subtask policies.

Equation 1
θ∗=arg​maxθ𝔼ρ∈𝒟[pθ(a1:T|s1:T)].\theta^{*}=\argmax_{\theta}\mathbb{E}_{\rho\in\mathcal{D}}\left[p_{\theta}(a_{1:T}|s_{1:T})\right]. (1)

2.2 Sub-task identification and imitation

Boundaries partition a trajectory into at most M ordered segments, each controlled by a policy conditioned on its latent code. The model permits empty segments and categorical or continuous codes. A truncated Poisson-inspired boundary prior discourages both single-step fragmentation and one segment covering everything. The demonstrated environments are fully observable, and the decoder policy has no recurrent memory.

Equations 2–4
ci=((sbi′,abi′),(sbi′+1,abi′+1),…,(sbi−1,abi−1)),c_{i}=((s_{b_{i^{\prime}}},a_{b_{i^{\prime}}}),(s_{b_{i^{\prime}}+1},a_{b_{i^{\prime}}+1}),...,(s_{b_{i}-1},a_{b_{i}-1})), (2)
pθ(a1:T|s1:T)=\displaystyle p_{\theta}(a_{1:T}|s_{1:T})= (3)
∑b1:M∑z1:Mpθ(a1:T|s1:T,b1:M,z1:M)p(b1:M,z1:M)=\displaystyle\sum_{b_{1:M}}\sum_{z_{1:M}}p_{\theta}(a_{1:T}|s_{1:T},b_{1:M},z_{1:M})p(b_{1:M},z_{1:M})=
∑b1:Mz1:M∏i=1:Mpθ(abi′:bi−1|sbi′:bi−1,zi)p(bi|bi′)p(zi)=\displaystyle\sum_{\begin{subarray}{c}b_{1:M}\\ z_{1:M}\end{subarray}}\prod_{i=1:M}p_{\theta}(a_{b_{i^{\prime}}:b_{i}-1}|s_{b_{i^{\prime}}:b_{i}-1},z_{i})p(b_{i}|b_{i^{\prime}})p(z_{i})=
∑b1:Mz1:M∏i=1:M[∏j=bi′:bi−1πθ(aj|sj,zi)]p(bi|bi′)p(zi),\displaystyle\sum_{\begin{subarray}{c}b_{1:M}\\ z_{1:M}\end{subarray}}\prod_{i=1:M}\left[\quad\,\,\prod_{\mathclap{j=b_{i^{\prime}}:b_{i}-1}}\,\,\pi_{\theta}(a_{j}|s_{j},z_{i})\right]p(b_{i}|b_{i^{\prime}})p(z_{i}),
p⁡(bi|bi′)∝Poisson⁡(bi−bi′,λ)=e−λ​λbi−bi′(bi−bi′)!,p(b_{i}|b_{i^{\prime}})\propto\mathrm{Poisson}(b_{i}-b_{i^{\prime}},\lambda)=e^{-\lambda}\frac{\lambda^{b_{i}-b_{i^{\prime}}}}{(b_{i}-b_{i^{\prime}})!}, (4)

2.2.1 Recognition model

A shared long short-term memory (LSTM) encoder makes repeated passes over the trajectory. Separate output heads predict boundary positions and segment codes. Previously explained time steps are masked out, allowing the same inference process to operate on longer unseen sequences.

Equations 5–7
qϕ(b1:M,\displaystyle q_{\phi}(b_{1:M}, z1:M|x1:T)=\displaystyle z_{1:M}|x_{1:T})=
∏i=1:Mqϕz(zi|xbi′:bi−1)qϕb(bi|xbi′:T),\displaystyle\prod_{i=1:M}q_{\phi_{z}}(z_{i}|x_{b_{i^{\prime}}:b_{i}-1})q_{\phi_{b}}(b_{i}|x_{b_{i^{\prime}}:T}), (5)
hzi\displaystyle h_{z_{i}} =MLPz(LSTMbi−1(x~bi′:bi−1)),\displaystyle=\mathrm{MLP}_{z}(\mathrm{LSTM}_{b_{i}-1}(\tilde{x}_{b_{i^{\prime}}:b_{i}-1})), (6)
hbit\displaystyle\ h_{b_{i}}^{t} =MLPb(LSTMt(x~bi′:T)),\displaystyle=\mathrm{MLP}_{b}(\mathrm{LSTM}_{t}(\tilde{x}_{b_{i^{\prime}}:T})), (7)

2.2.2 Continuous relaxation

During training, Gumbel-softmax relaxations replace discrete boundary and code samples, enabling reparameterized gradients. Testing returns to discrete boundaries and codes.

Refer to caption

Figure 2: Differentiable segmentation of an input trajectory xx composed of a sequence of sub-tasks. The recognition model (encoder, marked as inference) predicts relaxed categorical (Gumbel softmax) boundary distributions q⁡(bi|x)q(b_{i}|x) from which we can obtain soft segment masks P⁡(t∈Ci)P(t\in C_{i}). Each segment CiC_{i} is encoded via q⁡(zi|x)q(z_{i}|x). The generative model p⁡(x|zi)p(x|z_{i}) is executed once for every latent variable ziz_{i}. The reconstruction loss is masked with P⁡(t∈Ci)P(t\in C_{i}), so that only the reconstructed part corresponding to the ii-th segment receives a training signal. For imitation learning, the generative model (decoder, marked as generation) takes the form of a policy πθ​(at|st,zi)\pi_{\theta}(a_{t}|s_{t},z_{i}).

Soft segment masks

Soft segment membership is computed from cumulative boundary probabilities. The lower boundary is the maximum of all earlier boundaries, so inconsistent relaxed predictions can yield empty segments. The resulting masks partition responsibility over time.

Equation 8
P⁡(t∈Ci)=P⁡(max0≤j≤i−1⁡bj≤t<bi)=\displaystyle P(t\in C_{i})=P\left(\max_{0\leq j\leq i-1}b_{j}\leq t<b_{i}\right)= (8)
[1−cumsum(qϕb(bi|x),t)]∏j=0:i−1cumsum(qϕb(bj|x),t),\displaystyle\left[1-\mathrm{cumsum}(q_{\phi_{b}}(b_{i}|x),t)\right]\,\,\prod_{\mathclap{j=0:i-1}}\,\,\mathrm{cumsum}(q_{\phi_{b}}(b_{j}|x),t),

RNN state masking

Each pass multiplies recurrent hidden and cell states by the probability that the current time step remains unexplained. This suppresses earlier segments while preserving gradients through boundary decisions.

Equation 9
maski​(t)=P⁡(t≥max0≤j≤i−1⁡bj)=\displaystyle\mathrm{mask}_{i}(t)=P\left(t\geq\max_{0\leq j\leq i-1}b_{j}\right)= (9)
∏j=0:i−1P(t≥bj)=∏j=0:i−1cumsum(qϕb(bj|x),t),\displaystyle\prod_{j=0:i-1}P(t\geq b_{j})=\prod_{j=0:i-1}\mathrm{cumsum}(q_{\phi_{b}}(b_{j}|x),t),

Soft RNN readout

Invalid boundary locations and padding are masked. Code logits are read as a boundary-weighted average of recurrent outputs, shifted by one time step because a boundary identifies the next segment’s start.

Equation 10
qϕz(zi|x)=concreteτ(∑t=1:Tqϕb(bi=t+1|x)hzit),\displaystyle q_{\phi_{z}}(z_{i}|x)=\mathrm{concrete}_{\tau}\left(\sum_{t=1:T}q_{\phi_{b}}(b_{i}=t+1|x)\,h_{z_{i}}^{t}\right), (10)

Loss masking

The action-reconstruction loss is weighted by each segment’s soft membership mask, so every subpolicy explains only its assigned portion of the demonstration. Training estimates the expected loss with one reparameterized posterior sample.

Equation 11
ℒi=𝔼qϕ(b,z|a,s)[segi⋅logpθ(a|s,zi)],\displaystyle\mathcal{L}_{i}=\mathbb{E}_{q_{\phi}(b,z|a,s)}[\mathrm{seg}_{i}\cdot\log p_{\theta}(a|s,z_{i})], (11)

Number of segments

M is supplied during training and fixed across a minibatch. Giving the true number of subtasks is a form of weak supervision; the method does not automatically select an unrestricted number of segments.

Complexity

A forward pass costs O(TM) for trajectory length T and M segments, because inference and decoding revisit the sequence for every segment.

3 Related work

CompILE relates to option discovery, hierarchical imitation, sequence segmentation, and latent-variable models. It learns from offline demonstrations rather than requiring environment interaction during skill discovery. Unlike task-sketch approaches such as TACO, the main model need not receive subtask identities or their order. Its latent codes can be discrete or continuous.

4 Experiments

Experiments ask whether the model discovers useful boundaries and codes, generalizes from three to five subtasks, and provides useful actions for a hierarchical reinforcement learning agent.

4.1 Multi-task environments

Both environments are fully observable simulations with generated demonstrations and known evaluation boundaries.

Refer to caption
Refer to caption
Figure 3: Example instances of multi-task, instruction-following environments used in our experiments. Left: Grid world with walls. An agent has to pick up or visit certain objects. Right: Continuous control reacher task with multiple targets. The tip of the reacher arm has to touch multiple colored spheres in a pre-specified order.

Grid world

A 10×10 grid contains walls and objects. Tasks visit or pick up specified object types in sequence; demonstrations follow shortest paths. Training uses three-task sequences, while testing includes five-task sequences and new layouts.

Continuous control

A two-link reacher touches a sequence of colored targets. Observations give target locations and arm state, and a hand-written controller generates demonstrations. Tasks contain three or five target reaches.

4.2 Imitation learning

The model trains on three-task demonstrations with Adam, batch size 256, and learning rate 0.0001. Grid-world training uses 50,000 iterations and reacher training 500,000. A separate causal termination policy learns the inferred boundaries without future observations. Evaluation uses 1,024 new instances and supplies M=3 or M=5 according to the test sequence.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 4: Imitation learning results in grid world domain. We report accuracy of segmentation boundary recovery, reconstruction accuracy (average over sequence vs. percentage of exact full-sequence matches) and online evaluation: average reward obtained when deploying the generative model (with termination policy) using the inferred latent code from the demonstration sequence in the environment, without re-training. See main text for additional details.

Baselines

Baselines include an LSTM that detects surprising actions and a single-segment VAE behavioral-cloning model. Partially supervised variants provide code identities (z-CompILE) or boundaries (b-CompILE) during training; these labels are not supplied at test time.

Grid world results

Pickup boundaries and reconstructions are accurate, including on longer sequences; the single-segment baseline generalizes poorly. Visit tasks are more ambiguous because an agent can accidentally pass over an object, which can confuse termination. Reconstruction given expert states and successful online execution are reported separately.

Continuous control results

Unsupervised CompILE improves reacher boundary detection over the surprisal baseline but often misses the exact generating segmentation. Allowing a one-step boundary tolerance improves some scores, while partial supervision yields much stronger results. The paper does not report continuous-control reconstruction or execution performance because behavioral-cloning covariate shift remains unresolved.

Model Accuracy F1 (tol=0) F1 (tol=1)
3 tasks
LSTM surprisal 24.8±0.624.8\pm 0.6 39.0±0.339.0\pm 0.3 47.1±0.447.1\pm 0.4
CompILE 62.0±4.562.0\pm 4.5 74.3±3.374.3\pm 3.3 78.9±2.578.9\pm 2.5
z-CompILE 99.5±0.299.5\pm 0.2 99.7±0.299.7\pm 0.2 99.8±0.199.8\pm 0.1
b-CompILE 99.8±0.199.8\pm 0.1 99.9±0.199.9\pm 0.1 100±0.0100\pm 0.0
5 tasks – generalization
LSTM surprisal 21.6±0.521.6\pm 0.5 44.9±0.544.9\pm 0.5 54.4±0.554.4\pm 0.5
CompILE 41.7±8.041.7\pm 8.0 69.3±4.769.3\pm 4.7 74.0±4.674.0\pm 4.6
z-CompILE 98.4±0.598.4\pm 0.5 99.3±0.299.3\pm 0.2 99.8±0.199.8\pm 0.1
b-CompILE 98.8±0.398.8\pm 0.3 99.5±0.199.5\pm 0.1 99.8±0.199.8\pm 0.1
Table 1: Segmentation results in continuous control domain. We report accuracy (mean and standard deviation over 5 runs) of exact segmentation boundary recovery and two F1 scores (in %\%), which measure the harmonic mean between precision and recall for boundary prediction, with (tol=1) and without (tol=0) tolerance for boundaries that are misplaced by 1 time step in either direction.

4.3 Hierarchical reinforcement learning

A hierarchical IMPALA agent chooses either primitive actions or pretrained CompILE skills with learned termination. It solves sparse-reward pickup tasks that the tested flat and single-segment baselines struggle with, including five-task compositions after three-task pretraining. With dense subtask rewards, the flat baseline can learn with fewer episodes. These results concern grid-world control, not the reacher or a physical robot.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 5: Learning curves for agents trained in the multi-task grid world environment for a single representative seed. We found that the qualitative behavior was consistent across seeds. Original learning curve (reward at every episode) plotted as shaded line; overlaid with solid line using exponential smoothing for easier visibility. BC denotes a VAE-based behavioral cloning baseline that was exposed to the same number of task demonstrations as our CompILE model. The low-level baseline is an agent without internal hierarchy. The CompILE-based hierarchical agent benefits from significantly improved exploration and is the only agent that succeeds at all sparse reward tasks. Best viewed in color.

4.4 Limitations and future work

Discovered codes need not correspond to human task labels: the model often learns location-based rather than object-based skills. The number of segments supplies weak supervision, and partially observable environments would require decoder memory and mechanisms preventing it from ignoring latent codes.

5 Conclusions

Joint segmentation and imitation can yield reusable skills for longer compositions and sparse-reward learning. Applications to other sequential data, episodic memory, partial observability, and higher-level planning remain proposed extensions.

Acknowledgements

The authors acknowledge colleagues for discussions and feedback.

Appendix A CompILE model details

The appendix specifies architectures, termination, variational training, alternative latent codes, and sensitivity to segmentation settings.

A.1 Encoder architecture

The encoder converts observations and actions into recurrent inputs; its state is reset between trajectories and segment passes.

Grid world

Grid observations pass through two 3×3 convolutional layers with 64 feature maps, followed by a linear projection and layer normalization. The recognition network additionally receives an action embedding.

Continuous control environment

For continuous control, a two-layer perceptron with 256 hidden units per layer replaces the convolutional encoder.

A.2 Sub-task policies

Discrete codes select separate policy heads sharing the visual encoder. A soft mixture supplies gradients during relaxed training. This design generalizes better in initial experiments than feeding an embedded categorical code into one shared policy.

A.3 Termination policy

A separate recurrent termination network learns to mark the boundary predicted by the inference model. During execution it terminates a skill when its sigmoid output exceeds 0.5.

b1b_{1}b2b_{2}......bM−1b_{M-1}bMb_{M}b0b_{0}z1z_{1}z2z_{2}zMz_{M}x1x_{1}x2x_{2}...xb1−1x_{b_{1}-1}xb1x_{b_{1}}xb1+1x_{b_{1}+1}...xb2−1x_{b_{2}-1}xbM−1x_{b_{M-1}}...xTx_{T}
Figure 6: Dependencies between observed and latent variables in our generative model pθ(x1:T|b1:M,z1:M)p_{\theta}(x_{1:T}|b_{1:M},z_{1:M}). The state-action pair (at,st)(a_{t},s_{t}) is summarized into a single observed variable xtx_{t}. The latent variables bib_{i} determine the location of the boundaries between segments, whereas ziz_{i} summarize the content of each segment.

A.4 ELBO objective for learning

The evidence lower bound (ELBO) combines action reconstruction with a Kullback–Leibler (KL) penalty between posterior and prior. The retained derivation follows from Jensen’s inequality, and Figure 6 shows the graphical model.

Equations 12–13
ELBO\displaystyle\mathrm{ELBO} =𝔼qϕ(b,z|a,s)[logpθ(a|s,b,z)\displaystyle=\mathbb{E}_{q_{\phi}(b,z|a,s)}[\log p_{\theta}(a|s,b,z)
+logp(b,z)−logqϕ(b,z|a,s)],\displaystyle+\log p(b,z)-\log q_{\phi}(b,z|a,s)], (12)
log⁡pθ​(a|s)\displaystyle\log p_{\theta}(a|s) =log∑b,zpθ(a|s,b,z)p(b,z)\displaystyle=\log\sum_{b,z}p_{\theta}(a|s,b,z)p(b,z)
=log𝔼qϕ(b,z|a,s)[pθ​(a|s,b,z)​p​(b,z)qϕ(b,z|a,s)]\displaystyle=\log\,\mathbb{E}_{q_{\phi}(b,z|a,s)}\left[\frac{p_{\theta}(a|s,b,z)p(b,z)}{q_{\phi}(b,z|a,s)}\right]
≥𝔼qϕ(b,z|a,s)[logpθ​(a|s,b,z)​p​(b,z)qϕ(b,z|a,s)]\displaystyle\geq\mathbb{E}_{q_{\phi}(b,z|a,s)}\left[\log\frac{p_{\theta}(a|s,b,z)p(b,z)}{q_{\phi}(b,z|a,s)}\right]
=ELBO\displaystyle=\mathrm{ELBO} (13)

A.5 KL term

A coefficient β controls prior regularization. Training uses categorical KL terms even for relaxed samples. Because exact previous boundaries are unavailable, the boundary-prior term is evaluated for the first boundary and multiplied by M, relying on shared segment-distribution assumptions.

A.6 Gaussian latent variables

A variant replaces categorical codes with 32-dimensional Gaussian variables and uses one code-conditioned decoder. The appendix reports grid-world and reacher segmentation results for this alternative.

Original figure
Original figure
Original figure
Original figure
Figure 7: Imitation learning results in grid world domain for CompILE model variant with Gaussian latent variables.
Model Accuracy F1 (tol=0) F1 (tol=1)
3 tasks
LSTM surprisal 24.8±0.624.8\pm 0.6 39.0±0.339.0\pm 0.3 47.1±0.447.1\pm 0.4
CompILE 45.2±13.845.2\pm 13.8 59.3±12.159.3\pm 12.1 68.8±8.168.8\pm 8.1
z-CompILE 99.6±0.299.6\pm 0.2 99.6±0.199.6\pm 0.1 99.9±0.199.9\pm 0.1
b-CompILE 99.8±0.199.8\pm 0.1 99.9±0.199.9\pm 0.1 99.9±0.099.9\pm 0.0
5 tasks – generalization
LSTM surprisal 21.6±0.521.6\pm 0.5 44.9±0.544.9\pm 0.5 54.4±0.554.4\pm 0.5
CompILE 28.7±7.028.7\pm 7.0 56.4±9.156.4\pm 9.1 63.8±6.563.8\pm 6.5
z-CompILE 98.3±0.598.3\pm 0.5 99.2±0.399.2\pm 0.3 99.7±0.199.7\pm 0.1
b-CompILE 98.5±0.398.5\pm 0.3 99.3±0.299.3\pm 0.2 99.7±0.199.7\pm 0.1
Table 2: Segmentation results in continuous control domain for CompILE model variant with Gaussian latent variables. Values are in % and we report mean and standard deviation for runs with 5 different random seeds.
Hyperparam. Accuracy F1 (tol=0) F1 (tol=1)
# Segments MM
M=3M=3 62.0±4.562.0\pm 4.5 74.3±3.374.3\pm 3.3 78.9±2.578.9\pm 2.5
M=4M=4 53.4±6.353.4\pm 6.3 66.6±5.566.6\pm 5.5 75.3±3.175.3\pm 3.1
M=5M=5 29.8±6.129.8\pm 6.1 47.3±5.547.3\pm 5.5 65.6±2.965.6\pm 2.9
Softmax temperature τ\tau
τ=1\tau=1 62.0±4.562.0\pm 4.5 74.3±3.374.3\pm 3.3 78.9±2.578.9\pm 2.5
τ=0.1\tau=0.1 40.9±5.340.9\pm 5.3 55.3±5.755.3\pm 5.7 67.0±2.467.0\pm 2.4
τ=0.01\tau=0.01 36.3±6.036.3\pm 6.0 50.3±5.250.3\pm 5.2 66.4±3.666.4\pm 3.6
Table 3: Segmentation results in continuous control domain with 3 tasks for CompILE model trained with (a) different number of segments ranging from M=3M=3 (correct setting) to M=5M=5 (too many boundaries provided to the model), and (b) softmax temperature ranging from τ=1\tau=1 to τ=0.01\tau=0.01. Values are in % and we report mean and standard deviation for runs with 5 different random seeds.

A.7 Attentive readout

An alternative attention head reads code information from any point within the softly masked segment. Results are similar to final-step readout in these tasks, with attention often focusing near the segment end.

Equations 14–15
a~it=ait+log⁡P⁡(t∈Ci),\tilde{a}_{i}^{t}=a_{i}^{t}+\log P(t\in C_{i}), (14)
qϕz(zi|x)=concreteτ(∑t=1:Tsithzit).\displaystyle q_{\phi_{z}}(z_{i}|x)=\mathrm{concrete}_{\tau}(\sum_{t=1:T}s_{i}^{t}h_{z_{i}}^{t}). (15)

A.8 Other hyperparameters

The following settings and ablations explain the model’s sensitivity to capacity, segment count, and relaxation temperature.

Number of hidden units and MLP layers

Most perceptron and LSTM layers have 256 hidden units. Boundary prediction uses a hidden-layer perceptron, code prediction a linear head, and the termination head two hidden layers. Smaller networks mainly reduce reconstruction quality.

Number of segments

Providing the true number of segments generally works best. Extra segments sometimes collapse to the sequence end but can also split one behavior unnecessarily, slightly reducing boundary accuracy.

Poisson prior rate

All experiments use Poisson rate λ=3; the authors report limited sensitivity to its precise value.

Softmax temperature

Reported experiments usually fix Gumbel-softmax temperature at 1. Annealing can help, whereas starting with a lower fixed temperature reduces unsupervised boundary accuracy in the tested reacher settings.

Appendix B Reinforcement learning agent details

The hierarchical reinforcement learning experiments reuse pretrained skills and train a task-conditioned controller over them.

B.1 Architecture and hyperparameters

The controller uses a smaller convolutional network and 128-unit perceptron layers. It can call ten learned skills or primitive actions; the single-segment comparison uses a larger categorical code space. Task embeddings condition both the image encoder and policy head. Adam uses learning rate 0.001 and batch size 256; discounting is 0.99.

B.2 Distributed training

IMPALA distributes environment interaction among CPU actors and learning to a GPU learner. Off-policy correction accounts for actor trajectories generated with older policy parameters.

Appendix C Environment implementation details

The environment specifications describe observations, demonstrations, task construction, and episode limits.

C.1 Grid world

The grid world provides four movement and four directional pickup actions, with a 10×10×12 observation tensor. Random mazes are simplified by wall subsampling and checked for solvable tasks. Breadth-first search generates demonstrations, truncated at 42 steps during training and 200 during testing.

C.2 Continuous control

The simulated reacher has two links of length 0.12 and targets of diameter 0.05. A 32-dimensional observation records ten targets’ visibility and coordinates plus two joint angles. Episodes last at most 100 control steps. The demonstration controller independently adjusts shoulder direction and elbow opening toward the current target.

Appendix D Evaluation details

Evaluation distinguishes segmentation, reconstruction under expert observations, and execution in the environment.

D.1 Metrics

Boundary accuracy checks exact task transitions; F1 also evaluates unordered boundary matches with optional one-step tolerance. Reconstruction measures per-action agreement and exact full-sequence agreement while using expert states. Online evaluation executes inferred skills with learned termination and reports complete-task success, allowing errors to affect subsequent observations.

Equations
1M−1∑i=1M−1𝕀[arg​maxbiqϕb(bi|x)=bi],\frac{1}{M-1}\sum_{i=1}^{M-1}\mathbb{I}[\argmax_{b_{i}}q_{\phi_{b}}(b_{i}|x)=b_{i}],
1T∑i=1:M(∑j=bi′:bi−1𝕀[arg​maxajπθ(aj|sj,zi)=aj]),\frac{1}{T}\sum_{i=1:M}\left(\quad\,\,\sum_{\mathclap{j=b_{i^{\prime}}:b_{i}-1}}\,\,\mathbb{I}[\argmax_{a_{j}}\pi_{\theta}(a_{j}|s_{j},z_{i})=a_{j}]\right),
#predictions that matches the ground truthtotal #predictions,\frac{\text{\#predictions that matches the ground truth}}{\text{total \#predictions}},
#ground truth that has matches in predictionstotal #ground truth.\frac{\text{\#ground truth that has matches in predictions}}{\text{total \#ground truth}}.
F​1=2⋅precision⋅recallprecision+recall.F1=\frac{2\cdot\mathrm{precision}\cdot\mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}.

D.2 Segmentation baseline (LSTM surprisal)

The LSTM-surprisal baseline marks the M−1 least-likely demonstrated actions as boundaries. It shares comparable encoders and is trained only on three-task sequences. Its stronger pickup boundary detection on five-task sequences may reflect surprise at events absent during training rather than learned reusable structure.

Equations 16–17
maxθPθ(a1:T|s1:T)=∏i=1TP(ai|a1:i−1,s1:i)\max_{\theta}P_{\theta}(a_{1:T}|s_{1:T})=\prod_{i=1}^{T}P(a_{i}|a_{1:i-1},s_{1:i}) (16)
arg​min[b1,b2,…,bM−1],bi≤bi+1∑i=1M−1P(abi|a1:bi−1,s1:bi)\argmin_{[b_{1},b_{2},\ldots,b_{M-1}],b_{i}\leq b_{i+1}}\sum_{i=1}^{M-1}P(a_{b_{i}}|a_{1:b_{i}-1},s_{1:b_{i}}) (17)

Appendix E Qualitative results

The retained qualitative figures visualize each discovered policy, its chosen object, and its termination location. They show that codes often partition space rather than encode object identity, although the model can distinguish nearby objects in some scenes.

Refer to caption
Figure 8: Example of sub-task policies discovered by the agent.
Refer to caption
Figure 9: Example of sub-task policies discovered by the agent.
Refer to caption
Figure 10: Heatmap of termination locations for each policy (for 1000 random environment instances).
Concept demonstration

Open image ↗