Literature
Display
← All papers

TACO: Learning Task Decomposition via Temporal Alignment for Control

Shiarlis, Kyriacos · Wulfmeier, Markus · Salter, Sasha · Whiteson, Shimon · Posner, Ingmar

· ICML 2018 ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

At a glance

TACO learns reusable subpolicies from demonstrations paired with task sketches: ordered subtask names without time boundaries. It jointly optimizes sketch alignment and action imitation, including when each subpolicy should stop. Four simulated domains show stronger control than aligning first and cloning afterward, with performance approaching fully segmented supervision in several settings.

1 Introduction

Complex demonstrations often combine simpler skills that can be reused in other orders. TACO receives the ordered skill list during training and execution, learns one policy per named skill, and infers temporal boundaries. A new sketch can therefore specify a previously unseen composition. This is weakly supervised imitation, not unsupervised discovery of the skill vocabulary or autonomous high-level planning.

2 Related Work

Earlier modular imitation approaches often segment trajectories before fitting skills or require particular motion models. TACO allows differentiable policy architectures and couples segmentation to how well those policies explain actions. It extends connectionist temporal classification (CTC), originally used for sequence alignment, with an imitation objective and learned termination. [9]

3 Preliminaries

The derivation combines behavioral cloning, sketch-based modular policies [2], and dynamic-programming sequence alignment.

3.1 Behavioural Cloning

Behavioral cloning (BC) fits demonstrated actions from states, but small execution errors can move the agent outside the training distribution. Most experiments use noisy demonstration collection inspired by DART to teach recovery behavior. [15] Modularity then lets the same skill use data from several tasks.

Equation 1
θ∗=arg⁡maxθ⁡𝔼ρ​[∑t=1Tlog⁡πθ​(at|st)].\displaystyle\theta^{*}=\argmax_{\theta}\mathbb{E}_{\rho}[\sum_{t=1}^{T}\log\pi_{\theta}(a_{t}|s_{t})]. (1)

3.2 Modular Learning from Demonstration

Each subpolicy includes an unobserved STOP action. Fully segmented demonstrations would directly label both skills and terminations; a sketch supplies only the ordered skill names, so their durations and switch points must be inferred. Sketches constrain the allowable policy sequence at both training and test time.

Equation 2
θk=1,…,K∗=arg⁡maxθk⁡𝔼ρk​[∑t=1Tρlog⁡πθk​(at+|st)].\displaystyle\theta_{k=1,...,K}^{*}=\argmax_{\theta_{k}}\mathbb{E}_{\rho_{k}}[\sum_{t=1}^{T_{\rho}}\log\pi_{\theta_{k}}(a^{+}_{t}|s_{t})]. (2)
Original figure
Figure 1: Problem setting: The trajectory ρ\rho (red) is augmented by a task sketch τ\tau (blue). The two sequences operate at different timescales. The whole trajectory is aligned (manually or automatically) and segmented into three parts. From this alignment three separate policies are learned. The unobserved aS​T​O​Pa_{STOP} action for each policy is inferred to occur at the point where the policies switch from one to the other.

3.3 Sequence Alignment

A CTC path assigns one sketch element to each demonstration time step. Summing over all paths consistent with the sketch is tractable using forward dynamic programming. This variant assumes consecutive subtasks without intervening blank segments.

Equations 3–7
ψ∗\displaystyle\psi^{*} =arg⁡maxψ⁡𝔼(ρ,τ)​[pψ​(τ|ρ)]\displaystyle=\argmax_{\psi}\mathbb{E}_{(\rho,\tau)}[p_{\psi}(\tau|\rho)] (3)
=arg⁡maxψ⁡𝔼(ρ,τ)​[∑ζ∈𝒵T,τpψ​(ζ|ρ)]\displaystyle=\argmax_{\psi}\mathbb{E}_{(\rho,\tau)}\Big[\sum_{\zeta\in\mathcal{Z}_{T,\tau}}p_{\psi}(\zeta|\rho)\Big] (4)
=arg⁡maxψ⁡𝔼(ρ,τ)​[∑ζ∈𝒵T,τ∏t=1Tpψ​(ζt|ρ)]\displaystyle=\argmax_{\psi}\mathbb{E}_{(\rho,\tau)}\Big[\sum_{\zeta\in\mathcal{Z}_{T,\tau}}\prod_{t=1}^{T}~p_{\psi}(\zeta_{t}|\rho)\Big] (5)
αt​(l)\displaystyle\alpha_{t}(l) =p(bl|ρt)[αt−1(l−1)+αt−1(l))],\displaystyle=p(b_{l}|\rho_{t})[\alpha_{t-1}(l-1)+\alpha_{t-1}(l))], (6)
α1​(l)\displaystyle\alpha_{1}(l) ={1,if​l=1,0,otherwise.\displaystyle=\begin{cases}1,&\text{if}\ l=1,\\ 0,&\text{otherwise}.\end{cases} (7)

4 Methods

The paper compares separate alignment and imitation with a single objective that lets policy fit influence the alignment.

Original figure
(a) CTC - Computation of forward variables αt​(l)\alpha_{t}(l)
Original figure
(b) TACO - Computation of forward variables αt​(l)\alpha_{t}(l)
Figure 2: Visualisation of the forward recursion for all paths ζ\zeta corresponding to the sketch τ\tau. The horizontal axis denotes time, while the vertical denotes the task sequence. While a node at l,tl,t in CTC is only weighted by the meta-controller probability p⁡(bl|st,at)p(b_{l}|s_{t},a_{t}), nodes and edges are weighted in TACO respectively via the sub-policies πl​(at|st)\pi_{l}(a_{t}|s_{t}) and πl​(aS​T​O​P|st)\pi_{l}(a_{STOP}|s_{t})

4.1 Naively Adapted CTC

CTC-BC first predicts an alignment, then clones actions within each inferred segment. Alignment errors can mix incompatible actions at similar states. A probabilistically weighted version is also derived, but the reported main baseline uses a single alignment and does not feed control performance back into segmentation.

4.2 Temporal Alignment for Control (TACO)

TACO maximizes the joint likelihood of the demonstrated actions and sketch, conditioned on states. A forward recursion combines action likelihood with probabilities of continuing or stopping each policy. Automatic differentiation trains the complete model, and normalization prevents numerical underflow. Dropout helps avoid premature collapse onto one alignment path by exposing policies to a wider range of plausible assignments.

Equations 8–12
p(τ,𝐚ρ|𝐬ρ)=∑ζ∈𝒵T,τp(ζ|𝐬ρ)∏t=1Tπθζt(at|st),p(\tau,\mathbf{a}_{\rho}|\mathbf{s}_{\rho})=\sum_{\zeta\in\mathcal{Z}_{T,\tau}}p(\zeta|\mathbf{s}_{\rho})\prod_{t=1}^{T}\pi_{\theta{{}_{\zeta_{t}}}}(a_{t}|s_{t}), (8)
αt(l)≔∑ζ1:t∈𝒵t,τ1:lp(ζ|𝐬ρ)∏t′=1tπθζt′(at′|st′).\alpha_{t}(l)\coloneqq\sum_{\zeta_{1:t}\in\mathcal{Z}_{t,\tau_{1:l}}}p(\zeta|\mathbf{s}_{\rho})\prod_{t^{\prime}=1}^{t}\pi_{\theta_{\zeta_{t^{\prime}}}}(a_{t^{\prime}}|s_{t^{\prime}}). (9)
α1​(l)={πθ​b1​(a1|s1),if​l=1,0,otherwise.\alpha_{1}(l)=\begin{cases}\pi_{\theta{b_{1}}}(a_{1}|s_{1}),&\text{if}\ l=1,\\ 0,&\text{otherwise}.\end{cases} (10)
αt(l)=πθ​bl(at|st)[αt−1(l−1)πθ​bl−1(aS​T​O​P|st)\displaystyle\alpha_{t}(l)=\pi_{\theta{b_{l}}}(a_{t}|s_{t})\big[\alpha_{t-1}(l-1)\pi_{\theta{b_{l-1}}}(a_{STOP}|s_{t}) (11)
+αt−1(l)(1−πθ​bl(aS​T​O​P|st))].\displaystyle+\alpha_{t-1}(l)(1-\pi_{\theta{b_{l}}}(a_{STOP}|s_{t}))\big].
αT​(L)=p⁡(τ,𝐚ρ|𝐬ρ).\alpha_{T}(L)=p(\tau,\mathbf{a}_{\rho}|\mathbf{s}_{\rho}). (12)

5 Experiments

Evaluation measures complete-task success, subtask success, and agreement with known boundaries. Baselines include fully segmented GT-BC and CTC-BC with recurrent or perceptron alignment networks. Tests vary demonstration counts and, where specified, use unseen longer sketches. All robot-arm experiments are simulated.

5.1 Nav-World Domain

Nav-World learns four destination-reaching policies from continuous state–action trajectories. Training sketches contain three destinations and zero-shot tests contain four. Across 50, 400, and 1,000 demonstrations, TACO approaches fully supervised performance as data grows. Even reasonably accurate CTC boundaries can produce poor control because small switch errors corrupt the cloned action targets.

Original figure
Figure 3: The Nav-World. The agent (Blue) receives a route as a task sketch. In this case. τ=(Black, Green, Yellow, Red)\tau=(\text{Black, Green, Yellow, Red})
Refer to caption
Figure 4: Nav-World results: Mean accuracy over 100 agents on 100 test tasks. Task length at test time is Lt​e​s​t=4L_{test}=4 and at training time is L=3L=3. TACO (red) approaches the performance of a fully supervised sequence (grey) given enough data.
Domain
Algorithm Nav-World Craft Dial Dial (Image)
TACO 95.3 95.6 99.8 99.0
CTC-BC (MLP) 89.0 41.4 90.1 84.6
CTC-BC (GRU) 80.0 57.1 93.4 48.8
GT-BC (aligned w. TACO) 94.6 99.4 97.3 98.2
Table 1: Alignment accuracy of each algorithm for all domains. TACO always outperforms CTC emphasising the importance of maximising the joint likelihood of task sequences and actions. The alignment for GT-BC was obtained by computing the arg⁡max\argmax of the TACO forward variables on the policies learned using GT-BC.

5.2 Craft Domain

Craft combines discrete navigation and object-use skills using a 1,076-dimensional binary state. Demonstrations come from a trained reinforcement learning agent, and evaluation samples from the training task distribution. TACO reaches rewards comparable to or above GT-BC with enough data, while separate CTC alignment struggles.

Original figure
Figure 5: Craft results: Mean reward over 2000 tasks. The tasks are the same as during training but the environment instantiation is different. The RL agent used to derive the policies achieves a mean reward of 0.9. Taco (red) approaches the performance of a fully supervised method (grey) given enough data.

5.3 Dial Domain

A simulated six-degree-of-freedom JACO arm presses digit sequences on a keypad. Demonstrations come from a proportional–integral–derivative (PID) controller. Subsampling produces roughly 200 time steps for a four-digit sketch; actions command joint torques.

Refer to caption
Figure 6: Dial Domain: A 6 DoF JACO arm must dial a PIN of arbitrary length.

5.3.1 Joint Space Dial Domain

Joint-state observations include arm angles and distances to digits. Training randomizes digit positions, while testing uses a standard layout absent from training. TACO improves with data and exceeds GT-BC at some larger training sizes. Success declines for longer digit strings, but more slowly than the comparison policy; improved regularization is a proposed explanation, not a separately established mechanism.

Original figure
Figure 7: Joint Space Dial results: Task accuracy over 100 tasks of Lt​e​s​t=5L_{test}=5. During training L=4L=4. Evaluation is performed in an unseen configuration of the dial pad. Bars for CTC based methods do not appear as they did not finish any tasks.
Original figure
Figure 8: Task accuracy for increasing values of Lt​e​s​tL_{test}. The accuracy is measured over 100 runs for TACO  and GT-BC.

5.3.2 Image Space Dial Domain

The image variant uses 112×112 RGB observations and a shared convolutional encoder. It removes joint-angle observations to ensure visual information matters and keeps the digit layout fixed. TACO remains competitive with fully supervised cloning and stronger than the tested CTC-BC baselines.

Original figure
(a) Task Accuracy
Original figure
(b) Sub-Task Accuracy
Figure 9: Image-Space Dial: Accuracy over 25 test tasks. CTC-BC is unable to perform full task sequences and only solves a minor percentage of the sub-tasks, while both GT-BC and TACO  are able to complete most sub-tasks.

6 Discussion

A distribution over alignments can give each subpolicy more useful training evidence than committing early to one segmentation. However, the method still requires highly structured ordered sketches and a predefined skill vocabulary. Natural-language instructions, less rigid ordering, more complex hierarchies, and real robots remain future work.

7 Conclusion

Joint alignment and imitation reduce boundary annotation while learning executable modular policies. The evidence supports generalization to new compositions in the tested simulations; it does not eliminate the need for task sketches.

Acknowledgements

The work acknowledges UK and European research funding and help with early development.

Appendix A Technical Details

The appendix specifies policy architectures, action distributions, termination networks, and demonstration noise.

A.1 Implementation Details and Architecture

Perceptrons model state-based policies; image policies share convolutional features. Continuous actions use Gaussian distributions with fixed unit standard deviation, and discrete actions use categorical distributions. Separate action and termination networks perform better, especially for Dial. The retained architecture and training tables include dropout schedules and optimization settings.

Experiment State Dim Core Stop Policy Action Policy Output Dim
Nav World 8 - FC [100] FC [100] 2
Craft 1076 - FC [400,100] FC [400,100] 7
Dial 39 - FC [300,200,100] FC [300,200,100] 9
Dial (Images) [112,112,3]
conv[10,5,3]
kernel[5,5,3]
stride[2,2,1]
FC [400,300] FC [400,300] 9
Table 2: Specification of the architectures used in each experiment. For experiment Dial (Images) conv[], kernel[] and stride[] represent the number of channels, dimension of square kernels and 2D strides per convolutional layer respectively

A.2 Data Collection

Demonstration collection injects varying action noise to cover states reached after execution errors. In Dial, manually tuned noise scales with each joint’s maximum torque. This robustness support is part of the experimental setup, not a consequence of alignment alone.

Appendix B Detailed Results

Additional plots report ordinary Nav-World testing, Dial subtask success, and held-out alignment accuracy.

B.1 NavWorld Domain

On Nav-World tasks drawn from the training setting, increasing demonstrations brings TACO toward fully segmented GT-BC while CTC-BC remains much weaker.

Original figure
Figure 10: Task accuracy on NavWorld. L=Lt​e​s​t=3L=L_{test}=3.

B.2 Joint-State Dial Domain

The retained Dial plots separate individual digit-press accuracy from complete-sequence success, which compounds errors across subtasks.

Original figure
Figure 11: Sub-task accuracy for the Dial domain. We measure the number of sub-tasks succeeded over the total given, for 5 independent evaluations of 20 tasks of task length 5. Due to the complexity of the task and bad alignment performance, CTC based methods are unable to complete any sub-tasks.

B.3 Sequence Alignment Accuracy

Alignment accuracy is the percentage of time steps assigned the correct subtask on 100 held-out trajectories. TACO and CTC use the most likely forward assignment; GT-BC policies are evaluated through TACO’s alignment calculation without additional fitting. The complete table reports each domain and training-set size.

Domain
Algorithm Nav-World Craft Dial Dial (Image)
TACO 95.3 95.6 98.9 99.0
CTC-BC (MLP) 89.0 41.4 31.4 84.6
CTC-BC (GRU) 80.0 57.1 28.6 48.8
GT-BC (aligned with TACO) 94.6 99.4 98.7 98.2
Table 3: Alignment accuracy of each algorithm for all domains. TACO always outperforms CTC emphasising the importance of maximising the joint likelihood of task sequences and actions.

Appendix C CTC Probabilistic Sub-Policy Training

The alternative CTC-BC formulation weights actions by normalized forward probabilities instead of selecting one alignment. Because demonstrations lack STOP labels, transition edges between successive time steps determine probabilistic continue/stop targets. The retained derivation gives the action objective and recursive termination targets; it remains separate alignment followed by policy fitting.

Equations 13–14
θk=1,..,K∗=arg⁡maxθk⁡𝔼ρk​[∑t=1T∑l=1Llog⁡pt​(l)​πθk​(at+|st)]\displaystyle\theta_{k=1,..,K}^{*}=\argmax_{\theta_{k}}\mathbb{E}_{\rho_{k}}[\sum_{t=1}^{T}\sum_{l=1}^{L}\log p_{t}(l)\pi_{\theta_{k}}(a^{+}_{t}|s_{t})] (13)
where ​pt​(l)=αt​(l)∑li=1Lαt​(li).\displaystyle\text{where ~}p_{t}(l)=\frac{\alpha_{t}(l)}{\sum_{l_{i}=1}^{L}\alpha_{t}(l_{i})}. (14)
Refer to caption
Figure 12: CTC-based computation of stop action targets. The green and blue areas respectively depict the relations for l=1l=1 in Equation 15 and l>1l>1 in Equation 16
Equations 15–18
pt+1(1)=pt(1)⋅π1(a¯s​t​o​p|st+1)]\displaystyle\color[rgb]{0,0.6,0}{p_{t+1}(1)=p_{t}(1)\cdot\pi_{1}(\bar{a}_{stop}|s_{t+1})]} (15)
pt+1​(l+1)=pt​(l+1)⋅πl+1​(a¯s​t​o​p|st+1)+\displaystyle\color[rgb]{0.06,0.46,1}{p_{t+1}(l+1)=p_{t}(l+1)\cdot\pi_{l+1}(\bar{a}_{stop}|s_{t+1})+} (16)
pt​(l)⋅πl​(as​t​o​p|st+1)\displaystyle\color[rgb]{0.06,0.46,1}{p_{t}(l)\cdot\pi_{l}(a_{stop}|s_{t+1})}
πl​(as​t​o​p|st)=1−πl​(a¯s​t​o​p|st)\displaystyle\pi_{l}(a_{stop}|s_{t})=1-\pi_{l}(\bar{a}_{stop}|s_{t}) (17)
πl​(a¯s​t​o​p|st+1)​{pt​(l)pt+1​(l),if​l=1,pt+1​(l+1)pt​(l+1)if​l>1.−pt​(l)⋅(1−πl​(a¯s​t​o​p|st+1))pt​(l+1)\displaystyle\pi_{l}(\bar{a}_{stop}|s_{t+1})\begin{cases}\frac{p_{t}(l)}{p_{t+1}(l)},&\text{if}\ l=1,\\ \\ \frac{p_{t+1}(l+1)}{p_{t}(l+1)}&\text{if}\ l>1.\\ ~~-\frac{p_{t}(l)\cdot(1-\pi_{l}(\bar{a}_{stop}|s_{t+1}))}{p_{t}(l+1)}\end{cases} (18)
Concept demonstration

Open image ↗