At a glance
TACO learns reusable subpolicies from demonstrations paired with task sketches: ordered subtask names without time boundaries. It jointly optimizes sketch alignment and action imitation, including when each subpolicy should stop. Four simulated domains show stronger control than aligning first and cloning afterward, with performance approaching fully segmented supervision in several settings.
1 Introduction
Complex demonstrations often combine simpler skills that can be reused in other orders. TACO receives the ordered skill list during training and execution, learns one policy per named skill, and infers temporal boundaries. A new sketch can therefore specify a previously unseen composition. This is weakly supervised imitation, not unsupervised discovery of the skill vocabulary or autonomous high-level planning.
2 Related Work
Earlier modular imitation approaches often segment trajectories before fitting skills or require particular motion models. TACO allows differentiable policy architectures and couples segmentation to how well those policies explain actions. It extends connectionist temporal classification (CTC), originally used for sequence alignment, with an imitation objective and learned termination. [9]
3 Preliminaries
The derivation combines behavioral cloning, sketch-based modular policies [2], and dynamic-programming sequence alignment.
3.1 Behavioural Cloning
Behavioral cloning (BC) fits demonstrated actions from states, but small execution errors can move the agent outside the training distribution. Most experiments use noisy demonstration collection inspired by DART to teach recovery behavior. [15] Modularity then lets the same skill use data from several tasks.
Equation 1
| (1) |
3.2 Modular Learning from Demonstration
Each subpolicy includes an unobserved STOP action. Fully segmented demonstrations would directly label both skills and terminations; a sketch supplies only the ordered skill names, so their durations and switch points must be inferred. Sketches constrain the allowable policy sequence at both training and test time.
Equation 2
| (2) |
3.3 Sequence Alignment
A CTC path assigns one sketch element to each demonstration time step. Summing over all paths consistent with the sketch is tractable using forward dynamic programming. This variant assumes consecutive subtasks without intervening blank segments.
Equations 3–7
| (3) | ||||
| (4) | ||||
| (5) |
| (6) | ||||
| (7) |
4 Methods
The paper compares separate alignment and imitation with a single objective that lets policy fit influence the alignment.
4.1 Naively Adapted CTC
CTC-BC first predicts an alignment, then clones actions within each inferred segment. Alignment errors can mix incompatible actions at similar states. A probabilistically weighted version is also derived, but the reported main baseline uses a single alignment and does not feed control performance back into segmentation.
4.2 Temporal Alignment for Control (TACO)
TACO maximizes the joint likelihood of the demonstrated actions and sketch, conditioned on states. A forward recursion combines action likelihood with probabilities of continuing or stopping each policy. Automatic differentiation trains the complete model, and normalization prevents numerical underflow. Dropout helps avoid premature collapse onto one alignment path by exposing policies to a wider range of plausible assignments.
Equations 8–12
| (8) |
| (9) |
| (10) |
| (11) | |||
| (12) |
5 Experiments
Evaluation measures complete-task success, subtask success, and agreement with known boundaries. Baselines include fully segmented GT-BC and CTC-BC with recurrent or perceptron alignment networks. Tests vary demonstration counts and, where specified, use unseen longer sketches. All robot-arm experiments are simulated.
5.1 Nav-World Domain
Nav-World learns four destination-reaching policies from continuous state–action trajectories. Training sketches contain three destinations and zero-shot tests contain four. Across 50, 400, and 1,000 demonstrations, TACO approaches fully supervised performance as data grows. Even reasonably accurate CTC boundaries can produce poor control because small switch errors corrupt the cloned action targets.
| Domain | ||||
|---|---|---|---|---|
| Algorithm | Nav-World | Craft | Dial | Dial (Image) |
| TACO | 95.3 | 95.6 | 99.8 | 99.0 |
| CTC-BC (MLP) | 89.0 | 41.4 | 90.1 | 84.6 |
| CTC-BC (GRU) | 80.0 | 57.1 | 93.4 | 48.8 |
| GT-BC (aligned w. TACO) | 94.6 | 99.4 | 97.3 | 98.2 |
5.2 Craft Domain
Craft combines discrete navigation and object-use skills using a 1,076-dimensional binary state. Demonstrations come from a trained reinforcement learning agent, and evaluation samples from the training task distribution. TACO reaches rewards comparable to or above GT-BC with enough data, while separate CTC alignment struggles.
5.3 Dial Domain
A simulated six-degree-of-freedom JACO arm presses digit sequences on a keypad. Demonstrations come from a proportional–integral–derivative (PID) controller. Subsampling produces roughly 200 time steps for a four-digit sketch; actions command joint torques.
5.3.1 Joint Space Dial Domain
Joint-state observations include arm angles and distances to digits. Training randomizes digit positions, while testing uses a standard layout absent from training. TACO improves with data and exceeds GT-BC at some larger training sizes. Success declines for longer digit strings, but more slowly than the comparison policy; improved regularization is a proposed explanation, not a separately established mechanism.
5.3.2 Image Space Dial Domain
The image variant uses 112×112 RGB observations and a shared convolutional encoder. It removes joint-angle observations to ensure visual information matters and keeps the digit layout fixed. TACO remains competitive with fully supervised cloning and stronger than the tested CTC-BC baselines.
6 Discussion
A distribution over alignments can give each subpolicy more useful training evidence than committing early to one segmentation. However, the method still requires highly structured ordered sketches and a predefined skill vocabulary. Natural-language instructions, less rigid ordering, more complex hierarchies, and real robots remain future work.
7 Conclusion
Joint alignment and imitation reduce boundary annotation while learning executable modular policies. The evidence supports generalization to new compositions in the tested simulations; it does not eliminate the need for task sketches.
Acknowledgements
The work acknowledges UK and European research funding and help with early development.
Appendix A Technical Details
The appendix specifies policy architectures, action distributions, termination networks, and demonstration noise.
A.1 Implementation Details and Architecture
Perceptrons model state-based policies; image policies share convolutional features. Continuous actions use Gaussian distributions with fixed unit standard deviation, and discrete actions use categorical distributions. Separate action and termination networks perform better, especially for Dial. The retained architecture and training tables include dropout schedules and optimization settings.
| Experiment | State Dim | Core | Stop Policy | Action Policy | Output Dim | |||
| Nav World | 8 | - | FC [100] | FC [100] | 2 | |||
| Craft | 1076 | - | FC [400,100] | FC [400,100] | 7 | |||
| Dial | 39 | - | FC [300,200,100] | FC [300,200,100] | 9 | |||
| Dial (Images) | [112,112,3] |
|
FC [400,300] | FC [400,300] | 9 |
A.2 Data Collection
Demonstration collection injects varying action noise to cover states reached after execution errors. In Dial, manually tuned noise scales with each joint’s maximum torque. This robustness support is part of the experimental setup, not a consequence of alignment alone.
Appendix B Detailed Results
Additional plots report ordinary Nav-World testing, Dial subtask success, and held-out alignment accuracy.
B.1 NavWorld Domain
On Nav-World tasks drawn from the training setting, increasing demonstrations brings TACO toward fully segmented GT-BC while CTC-BC remains much weaker.
B.2 Joint-State Dial Domain
The retained Dial plots separate individual digit-press accuracy from complete-sequence success, which compounds errors across subtasks.
B.3 Sequence Alignment Accuracy
Alignment accuracy is the percentage of time steps assigned the correct subtask on 100 held-out trajectories. TACO and CTC use the most likely forward assignment; GT-BC policies are evaluated through TACO’s alignment calculation without additional fitting. The complete table reports each domain and training-set size.
| Domain | ||||
|---|---|---|---|---|
| Algorithm | Nav-World | Craft | Dial | Dial (Image) |
| TACO | 95.3 | 95.6 | 98.9 | 99.0 |
| CTC-BC (MLP) | 89.0 | 41.4 | 31.4 | 84.6 |
| CTC-BC (GRU) | 80.0 | 57.1 | 28.6 | 48.8 |
| GT-BC (aligned with TACO) | 94.6 | 99.4 | 98.7 | 98.2 |
Appendix C CTC Probabilistic Sub-Policy Training
The alternative CTC-BC formulation weights actions by normalized forward probabilities instead of selecting one alignment. Because demonstrations lack STOP labels, transition edges between successive time steps determine probabilistic continue/stop targets. The retained derivation gives the action objective and recursive termination targets; it remains separate alignment followed by policy fitting.
Equations 13–14
| (13) | |||
| (14) |
Equations 15–18
| (15) | |||
| (16) | |||
| (17) |
| (18) |