Literature
Display
← All papers

Diagnosing Compositional Generalization in Sequential Robot Tasks

Yixiao Wang · Cheng-En Wu · Lingfeng Sun · Pengcheng Wang · Xiang Ji · Boyuan Liang · Guojian Zhan · Masayoshi Tomizuka

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

The paper studies compositional generalization and data-efficient imitation learning through controlled instruction-coverage experiments.

At a glance

The study asks which instruction combinations a robot policy needs to see during training. A bounded-loss distribution-shift analysis distinguishes factor marginals, instruction recombinations, and conditional context/action changes. Simulated manipulation experiments show that pair-covering task subsets can match full enumeration at a fixed demonstration budget, while sparse pretrained policies can benefit from small amounts of broader supervision. Dependent tasks require preserving cross-stage information. The results concern a small controlled task family and do not provide a universal coverage rule for pretrained robot models.

1 Introduction

A policy may observe every object, container, and button individually yet fail when they appear in a new combination. The paper treats instruction coverage as a dataset-design variable, comparing sparse correlated task sets with structured subsets that cover factor pairs. It also separates recovering already learned behaviors under new instructions from learning entirely new manipulation skills. The proposed principle is to cover action-relevant dependencies, with the required interaction order depending on the task.

2 Related Work

The related work connects instruction-conditioned policies, compositional benchmarks, experimental design, and hierarchical control.

Language-Conditioned Robot Policies.

Generalist robot policies can map language or multimodal prompts to actions, but scaling a policy does not specify which combinations its demonstrations must cover. This paper isolates instruction-tuple coverage in a common flow-matching architecture rather than proposing a new large pretrained robot policy.

Compositional Generalization in Language and Robotics.

Compositional benchmarks hold out recombinations of familiar factors. In robotics, changing those combinations also changes the observations reached and the appropriate actions. The analysis therefore includes conditional context/action shift in addition to changes in the instruction distribution.

Data Coverage and Combinatorial Designs.

Data-generation approaches such as MimicGen [17] increase demonstrations, while orthogonal designs [12] motivate covering interactions without enumerating every tuple. The experiments use this design idea to compare task breadth with the number of demonstrations per selected task.

Modularity and Hierarchy.

Hierarchical policies can expose only the active subtask, reducing dependence on irrelevant instruction components. This helps when stages are independent, but can discard information needed for cross-stage constraints. The idealized modular comparison assumes a correct stage indicator, whose construction is explicitly documented.

3 Problem Statement

An instruction is an ordered tuple of discrete subtask values. A conditional policy maps the current observation/state and instruction tuple to an action distribution. A stage indicator identifies the currently active subtask. The task family is designed so individual factor values are familiar while selected combinations are withheld.

3.1 Compositional Generalization Gap

Training and test joint distributions include instructions, contexts, and actions. Held-out instructions are valid tuples absent from the selected training support. The formal generalization gap is the difference in expected policy loss, instantiated experimentally with flow-matching loss. This loss gap is distinct from the measured difference in rollout success.

Equations 1–4
Etrain:\displaystyle E_{\mathrm{train}}: =supp⁡(p⁡(l))\displaystyle=\operatorname{supp}(p(l)) (1)
={l∈ℒ1×⋯×ℒn|p⁡(l)>0}.\displaystyle=\left\{l\in\mathcal{L}_{1}\times\dots\times\mathcal{L}_{n}\;\big|\;p(l)>0\right\}.
Eood:=Etotal∖Etrain.E_{\mathrm{ood}}:=E_{\mathrm{total}}\setminus E_{\mathrm{train}}. (2)
ℛ𝒟​(θ):=∫𝒵×𝒜×ℒ𝒟⁡(z,a,l)​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l.\mathcal{R}_{\mathcal{D}}(\theta):=\int_{\mathcal{Z}\times\mathcal{A}\times\mathcal{L}}\mathcal{D}(z,a,l)L_{\theta}(z,a,l)\,dz\,da\,dl. (3)
Δq​(θ):=\displaystyle\Delta_{q}(\theta):= ℛq​(θ)−ℛp​(θ)=∫𝒵×𝒜×ℒ(q⁡(z,a,l)−p⁡(z,a,l))​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l\displaystyle\mathcal{R}_{q}(\theta)-\mathcal{R}_{p}(\theta)=\int_{\mathcal{Z}\times\mathcal{A}\times\mathcal{L}}\bigl(q(z,a,l)-p(z,a,l)\bigr)\,L_{\theta}(z,a,l)\,dz\,da\,dl (4)

4 Theoretical Analysis

The proposition bounds the absolute risk gap by a loss bound times three L1 distribution discrepancies: active-factor marginals, the remaining instruction factors conditional on that factor, and context/action distributions conditional on the complete instruction. The proof is a distribution-factorization and triangle-inequality argument.

The result assumes a uniformly bounded loss. It is a diagnostic decomposition, not a finite-sample guarantee that pairwise task coverage suffices. Matching low-order pairs need not match the full instruction-tuple distribution, and merely observing every factor value does not ensure matching marginal frequencies. The experiments, rather than the bound alone, establish the usefulness of the chosen designs in these tasks.

Equations
0≤Lθ​(z,a,l)≤M∀(z,a,l)∈supp⁡(p)∪supp⁡(q).0\leq L_{\theta}(z,a,l)\leq M\qquad\forall(z,a,l)\in\operatorname{supp}(p)\cup\operatorname{supp}(q).
Original inequality S4.E5 reproduced from PDF page 3
‖μ−ν‖1:=∫𝒳|μ⁡(x)−ν⁡(x)|​𝑑x\|\mu-\nu\|_{1}:=\int_{\mathcal{X}}|\mu(x)-\nu(x)|\,dx

4.1 Stage-wise Modular Policy

A stage-wise controller receives the current context and only the active instruction. Under bounded stage losses, integrating out inactive instruction factors removes the explicit full-instruction composition term from the bound. Their influence can still remain through context/action distributions. If the correct current action depends on an omitted future instruction, the modular input is insufficient even with an oracle stage indicator.

Equations
πθ​(a∣z,l)=πθ,σ⁡(z)​(a∣z,lσ⁡(z)).\pi_{\theta}(a\mid z,l)=\pi_{\theta,\sigma(z)}(a\mid z,l_{\sigma(z)}).
Lθ(z,a,l)=∑i=1n𝟏{σ(z)=i}ℓi,θ(z,a,li),L_{\theta}(z,a,l)=\sum_{i=1}^{n}\mathbf{1}\{\sigma(z)=i\}\,\ell_{i,\theta}(z,a,l_{i}),
0≤ℓi,θ​(z,a,li)≤Mi∀(z,a,li)∈supp⁡(pi)∪supp⁡(qi),0\leq\ell_{i,\theta}(z,a,l_{i})\leq M_{i}\quad\forall(z,a,l_{i})\in\operatorname{supp}(p_{i})\cup\operatorname{supp}(q_{i}),
Original inequality S4.E6 reproduced from PDF page 4

5 Experiments

The experiments test whether exhaustive tuple coverage is necessary, what sparse-pretraining failures reflect, and what coverage is required when one instruction factor constrains another stage’s actions.

Refer to caption
Figure 1: Illustration of Pick-Place-Press and Dependent Two-Stage Pick-and-Place tasks.
Original figure
(a) Pick-and-Place (PP).
Original figure
(b) Pick-Place-Press (PPP).
Figure 2: Success rates versus training-set size BB under a fixed total demonstration budget.

5.1 Experimental Setup

All main tasks are simulated sequential manipulation problems with controlled discrete factors, generated demonstrations, a common policy architecture, and held-out combinations.

Task Design.

Pick-and-Place has four objects and four containers, giving sixteen tasks. Pick-Place-Press adds four buttons, giving 64 tuples. Two-Stage Pick-and-Place uses different objects and different containers across the two placements, giving 144 fully specified combinations. Its policy input omits the first container and contains only the two objects and final container: this leaves 48 instruction tuples, each with three valid first-container choices. The first placement must therefore account for a later instruction while remaining multimodal.

Data and Policy.

A customized motion planner supplies demonstrations. Policies use a frozen DINOv2 [21] image encoder and a transformer flow-matching action model trained from scratch. Across different numbers of selected training tasks, total demonstration count is fixed and allocated uniformly among those tasks. Fewer selected tasks consequently receive more examples per task.

Evaluation Protocol.

Success is reported over all tasks, selected training tasks, and held-out instruction tuples. The full-task baseline has no held-out tuples; its overall success is repeated in the plots only as a reference. Most experiments use three training seeds and thirty rollouts per task. Same-container violation rate is conditional on completing the first placement, excluding earlier failures, so it must be read alongside success rate.

5.2 Q1: Is Full Tuple Coverage Necessary?

Small task sets perform poorly on held-out tuples even when every individual instruction value appears. Performance saturates near eight of sixteen Pick-and-Place tasks and sixteen of 64 Pick-Place-Press tasks. For three PPP training sizes, held-out flow loss falls from 0.03421 to 0.01548 to 0.00756 while held-out success rises from 0.83% to 23.93% to 82.29%; training losses remain similar.

Within a fixed budget, tasks whose object–container, object–button, and container–button pairs have been seen tend to perform better. A sixteen-task orthogonal design covers every pair exactly once and achieves 78.2% held-out success, close to the 78.6% overall success of full-task training. This supports pair coverage in this independent three-factor task, without proving that pair coverage is sufficient for longer or more dependent tasks.

Original figure
(a) OOD success in PPP grouped by seen pair count, for each training budget BB. Hatched bars contain fewer than 10 held-out tasks. Within each BB, OOD success increases with the seen pair count, and the effect is largest before saturation (B≤12B\leq 12).
Original figure
(b) Representative PPP training sets (B=10B=10; rows). Left: success rate, where darker means higher success. Right: number of seen instruction pairs for each held-out task. Held-out tasks with more seen pairs tend to achieve higher success.
Figure 3: OOD success in PPP increases with the number of seen instruction pairs within each training budget BB (left), and in two example B=10B{=}10 training sets the held-out success pattern closely matches the pair-overlap pattern (right).
Original figure
Figure 4: Orthogonal PPP training set. The 16 training tasks (blue; listed at right) cover every pairwise combination of instruction values exactly once.
Original figure
Figure 5: Orthogonal-set training outperforms full-set training when demonstrations are limited.

Fixed-budget comparison.

At low and medium total demonstration budgets, the orthogonal subset outperforms full enumeration. Concentrating data on fewer structurally informative tasks provides more examples per selected tuple. The gap narrows with larger budgets. This comparison tests allocation at a fixed total data count, not a claim that reducing both tasks and demonstrations always improves performance.

5.3 Q2: What Is Missing When Sparse Coverage Fails?

A diagonal four-task set ties each object, container, and button together. It yields high training success but only 0.4% success on the originally unseen combinations. Finetuning with one demonstration for each of all 64 tasks raises that group’s success to 54.7%, substantially more than learning from the small broader dataset alone.

Those formerly held-out tuples are now represented in the finetuning data. The 54.7% figure therefore measures adaptation to combinations unseen during pretraining, not final-model zero-shot success on tuples withheld from every training phase. The result supports reusable learned skills but does not uniquely isolate instruction routing as the causal failure mechanism.

Separate matched-budget comparisons favor orthogonal over random finetuning task selection for sparse pretrained checkpoints. Training-task success can decrease, and strong pretrained policies can worsen after tiny finetuning sets, exposing reweighting and overfitting tradeoffs.

Original figure
Figure 6: Finetuning from sparse diagonal pretraining. Small amounts of broad-coverage data substantially improve OOD success, suggesting that sparse pretraining learns reusable skills but not reliable instruction steering, which can be acquired with limited additional data.
Original figure
Figure 7: OOD success with 16 or 32 fine-tuning demonstrations. Stars: orthogonal set (16 tasks, one or two demonstrations per task); diamonds: five random sets (16 or 32 tasks, one demonstration per task); boxes: interquartile ranges.

5.4 Q3: What Coverage Is Needed for Dependent Instructions?

For the dependent task, vary coverage of the first-object/final-container pair while fixing the number of selected training tuples. In the 36-instruction version, twelve training tasks cover twelve, nine, six, or four such pairs. Broader pair coverage improves held-out success and reduces same-container violations; narrower coverage can improve in-distribution performance by concentrating supervision. The relevant pairs are known from the task design rather than automatically discovered by the method.

Refer to caption
Figure 8: Performance and same-container violations in 2S-PP. Broader coverage of dependent pairs improves success and reduces relational errors, such as placing o1o_{1} into c2c_{2}.
Refer to caption
(a) Task with independent instructions (PPP).
Refer to caption
(b) Task with dependent instructions (2S-PP).
Figure 9: GT stage selects the active instruction; Full instruction uses all factors. Cov16 covers all dependent pairs; Full covers all 48 tasks.

Comparison with stage-wise modular policies.

Oracle stage selection helps the independent PPP task under sparse coverage. In the dependent task, the modular controller receives only the first object during the first placement and cannot condition on the forbidden final container. A full-instruction controller retains that information and reduces held-out container violations with broader dependent-pair coverage. This comparison exposes a deliberate information bottleneck in the specified modular input, not a general failure of all hierarchical policies.

6 Conclusion and Limitations

The evidence favors structured coverage of action-relevant instruction interactions over automatic full enumeration in these controlled tasks. Limitations include the small simulated task family, known dependency structure, combinatorial evaluation cost, and policies trained from scratch over frozen vision features. The analysis assumes bounded losses and does not estimate a universally sufficient interaction order. Transfer to large pretrained vision–language–action models remains future work.

Acknowledgments

The study acknowledges support from FANUC.

Appendix A Theoretical Analysis

The general proof factors the joint density into an active instruction marginal, the remaining-instruction conditional, and a context/action conditional. Adding and subtracting two intermediate products separates the density difference into three terms. Integrating against a bounded loss and applying the triangle inequality gives the stated bound.

The modular proof expands risk by stage, uses the stage indicator, and integrates away inactive instructions because each stage loss depends only on the current factor. A second density decomposition yields marginal and conditional-context terms for each stage. This algebra explains the simplified bound but does not remove information required by dependent actions or establish a nontrivial numerical guarantee for the empirical flow loss.

Equations 8–9
0≤Lθ​(z,a,l)≤M∀(z,a,l)∈supp⁡(p)∪supp⁡(q).0\leq L_{\theta}(z,a,l)\leq M\qquad\forall(z,a,l)\in\operatorname{supp}(p)\cup\operatorname{supp}(q).
Original inequality A1.E7 reproduced from PDF page 13
Δq​(θ)=∫𝒵×𝒜×ℒ(q⁡(z,a,l)−p⁡(z,a,l))​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l,\Delta_{q}(\theta)=\int_{\mathcal{Z}\times\mathcal{A}\times\mathcal{L}}\bigl(q(z,a,l)-p(z,a,l)\bigr)\,L_{\theta}(z,a,l)\,dz\,da\,dl,
q⁡(z,a,l)−p⁡(z,a,l)=q⁡(li)​q​(l−i∣li)​q​(z,a∣l)−p⁡(li)​p​(l−i∣li)​p​(z,a∣l).q(z,a,l)-p(z,a,l)=q(l_{i})q(l_{-i}\mid l_{i})q(z,a\mid l)-p(l_{i})p(l_{-i}\mid l_{i})p(z,a\mid l).
p⁡(li)​q​(l−i∣li)​q​(z,a∣l)andp⁡(li)​p​(l−i∣li)​q​(z,a∣l),p(l_{i})q(l_{-i}\mid l_{i})q(z,a\mid l)\quad\text{and}\quad p(l_{i})p(l_{-i}\mid l_{i})q(z,a\mid l),
q⁡(z,a,l)−p⁡(z,a,l)=\displaystyle q(z,a,l)-p(z,a,l)= q⁡(li)​q​(l−i∣li)​q​(z,a∣l)−p⁡(li)​q​(l−i∣li)​q​(z,a∣l)\displaystyle q(l_{i})q(l_{-i}\mid l_{i})q(z,a\mid l)-p(l_{i})q(l_{-i}\mid l_{i})q(z,a\mid l) (8)
+p⁡(li)​q​(l−i∣li)​q​(z,a∣l)−p⁡(li)​p​(l−i∣li)​q​(z,a∣l)\displaystyle+p(l_{i})q(l_{-i}\mid l_{i})q(z,a\mid l)-p(l_{i})p(l_{-i}\mid l_{i})q(z,a\mid l)
+p⁡(li)​p​(l−i∣li)​q​(z,a∣l)−p⁡(li)​p​(l−i∣li)​p​(z,a∣l)\displaystyle+p(l_{i})p(l_{-i}\mid l_{i})q(z,a\mid l)-p(l_{i})p(l_{-i}\mid l_{i})p(z,a\mid l)
Δq​(θ)=Ai+Bi+Ci,\Delta_{q}(\theta)=A_{i}+B_{i}+C_{i}, (9)
Ai=∫(q⁡(li)−p⁡(li))​q​(l−i∣li)​q​(z,a∣l)​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l,A_{i}=\int\bigl(q(l_{i})-p(l_{i})\bigr)\,q(l_{-i}\mid l_{i})\,q(z,a\mid l)\,L_{\theta}(z,a,l)\,dz\,da\,dl,
Bi=∫p⁡(li)​(q⁡(l−i∣li)−p⁡(l−i∣li))​q​(z,a∣l)​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l,B_{i}=\int p(l_{i})\,\bigl(q(l_{-i}\mid l_{i})-p(l_{-i}\mid l_{i})\bigr)\,q(z,a\mid l)\,L_{\theta}(z,a,l)\,dz\,da\,dl,
Ci=∫p⁡(li)​p​(l−i∣li)​(q⁡(z,a∣l)−p⁡(z,a∣l))​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l.C_{i}=\int p(l_{i})\,p(l_{-i}\mid l_{i})\,\bigl(q(z,a\mid l)-p(z,a\mid l)\bigr)\,L_{\theta}(z,a,l)\,dz\,da\,dl.
|Ai|≤M​∫|q⁡(li)−p⁡(li)|​q​(l−i∣li)​q​(z,a∣l)​𝑑z​𝑑a​𝑑l=M​‖q⁡(li)−p⁡(li)‖1,|A_{i}|\leq M\int|q(l_{i})-p(l_{i})|\,q(l_{-i}\mid l_{i})\,q(z,a\mid l)\,dz\,da\,dl=M\|q(l_{i})-p(l_{i})\|_{1},
|Bi|≤M​∫p⁡(li)​|q⁡(l−i∣li)−p⁡(l−i∣li)|​q​(z,a∣l)​𝑑z​𝑑a​𝑑l=M​𝔼li∼p⁡(li)​[‖q⁡(l−i∣li)−p⁡(l−i∣li)‖1],|B_{i}|\leq M\int p(l_{i})\,|q(l_{-i}\mid l_{i})-p(l_{-i}\mid l_{i})|\,q(z,a\mid l)\,dz\,da\,dl=M\,\mathbb{E}_{l_{i}\sim p(l_{i})}\big[\|q(l_{-i}\mid l_{i})-p(l_{-i}\mid l_{i})\|_{1}\big],
|Ci|≤M​∫p⁡(l)|q⁡(z,a∣l)−p⁡(z,a∣l)|d​z​𝑑a​𝑑l=M​𝔼l∼p⁡(l)​[‖q⁡(z,a∣l)−p⁡(z,a∣l)‖1].|C_{i}|\leq M\int p(l)\,|q(z,a\mid l)-p(z,a\mid l)|\,dz\,da\,dl=M\,\mathbb{E}_{l\sim p(l)}\big[\|q(z,a\mid l)-p(z,a\mid l)\|_{1}\big].
𝔼li∼p⁡(li)​[‖q⁡(l−i∣li)−p⁡(l−i∣li)‖1]\mathbb{E}_{l_{i}\sim p(l_{i})}\big[\|q(l_{-i}\mid l_{i})-p(l_{-i}\mid l_{i})\|_{1}\big]
𝔼l∼p⁡(l)​[‖q⁡(z,a∣l)−p⁡(z,a∣l)‖1]\mathbb{E}_{l\sim p(l)}\big[\|q(z,a\mid l)-p(z,a\mid l)\|_{1}\big]
0≤ℓi,θ​(z,a,li)≤Mi∀(z,a,li)∈supp⁡(pi)∪supp⁡(qi).0\leq\ell_{i,\theta}(z,a,l_{i})\leq M_{i}\qquad\forall(z,a,l_{i})\in\operatorname{supp}(p_{i})\cup\operatorname{supp}(q_{i}).
Original inequality A1.E10 reproduced from PDF page 14
Δqm​(θ)=∫(q⁡(z,a,l)−p⁡(z,a,l))​Lθ​(z,a,l)​𝑑z​𝑑a​𝑑l,\Delta_{q}^{\mathrm{m}}(\theta)=\int\bigl(q(z,a,l)-p(z,a,l)\bigr)\,L_{\theta}(z,a,l)\,dz\,da\,dl,
Lθ(z,a,l)=∑i=1n𝟏{σ(z)=i}ℓi,θ(z,a,li),L_{\theta}(z,a,l)=\sum_{i=1}^{n}\mathbf{1}\{\sigma(z)=i\}\,\ell_{i,\theta}(z,a,l_{i}),
Δqm(θ)=∑i=1n∫𝟏{σ(z)=i}ℓi,θ(z,a,li)(q(z,a,l)−p(z,a,l))dzdadl.\Delta_{q}^{\mathrm{m}}(\theta)=\sum_{i=1}^{n}\int\mathbf{1}\{\sigma(z)=i\}\,\ell_{i,\theta}(z,a,l_{i})\,\bigl(q(z,a,l)-p(z,a,l)\bigr)\,dz\,da\,dl.
Δqm​(θ)=∑i=1n∫ℓi,θ​(z,a,li)​(q⁡(z,a,li)−p⁡(z,a,li))​𝑑z​𝑑a​d​li.\Delta_{q}^{\mathrm{m}}(\theta)=\sum_{i=1}^{n}\int\ell_{i,\theta}(z,a,l_{i})\,\bigl(q(z,a,l_{i})-p(z,a,l_{i})\bigr)\,dz\,da\,dl_{i}.
q⁡(z,a,li)−p⁡(z,a,li)=(q⁡(li)−p⁡(li))​q​(z,a∣li)+p⁡(li)​(q⁡(z,a∣li)−p⁡(z,a∣li)).q(z,a,l_{i})-p(z,a,l_{i})=\bigl(q(l_{i})-p(l_{i})\bigr)\,q(z,a\mid l_{i})+p(l_{i})\,\bigl(q(z,a\mid l_{i})-p(z,a\mid l_{i})\bigr).
Δqm​(θ)=∑i=1n(Ui+Vi),\Delta_{q}^{\mathrm{m}}(\theta)=\sum_{i=1}^{n}(U_{i}+V_{i}),
Ui=∫ℓi,θ​(z,a,li)​(q⁡(li)−p⁡(li))​q​(z,a∣li)​𝑑z​𝑑a​d​li,U_{i}=\int\ell_{i,\theta}(z,a,l_{i})\,\bigl(q(l_{i})-p(l_{i})\bigr)\,q(z,a\mid l_{i})\,dz\,da\,dl_{i},
Vi=∫ℓi,θ​(z,a,li)​p​(li)​(q⁡(z,a∣li)−p⁡(z,a∣li))​𝑑z​𝑑a​d​li.V_{i}=\int\ell_{i,\theta}(z,a,l_{i})\,p(l_{i})\,\bigl(q(z,a\mid l_{i})-p(z,a\mid l_{i})\bigr)\,dz\,da\,dl_{i}.
|Ui|≤Mi​∫|q⁡(li)−p⁡(li)|​(∫q⁡(z,a∣li)​𝑑z​𝑑a)​d​li=Mi​‖q⁡(li)−p⁡(li)‖1,|U_{i}|\leq M_{i}\int|q(l_{i})-p(l_{i})|\left(\int q(z,a\mid l_{i})\,dz\,da\right)\,dl_{i}=M_{i}\,\|q(l_{i})-p(l_{i})\|_{1},
|Vi|≤Mi​∫p⁡(li)​(∫|q⁡(z,a∣li)−p⁡(z,a∣li)|​𝑑z​𝑑a)​d​li=Mi​𝔼li∼p​‖q⁡(z,a∣li)−p⁡(z,a∣li)‖1.|V_{i}|\leq M_{i}\int p(l_{i})\left(\int|q(z,a\mid l_{i})-p(z,a\mid l_{i})|\,dz\,da\right)\,dl_{i}=M_{i}\mathbb{E}_{l_{i}\sim p}\|q(z,a\mid l_{i})-p(z,a\mid l_{i})\|_{1}.

Appendix B Experimental Details

The supplementary protocol separates generated demonstration pools from the smaller budgets actually selected for each experiment and documents architecture, evaluation, and exceptions.

B.1 Data Generation and Training Protocol

Generated pools contain 500 demonstrations per PP task, 300 per PPP task, and 100 per fully specified 2S-PP task: 8,000, 19,200, and 14,400 demonstrations. Experiments select subsets under their stated budgets; they do not necessarily use the full pool. PPP trains for 100,000 steps at batch size 64, and dependent 2S-PP for 150,000 steps. Seeds vary demonstration selection and initialization.

B.2 Policy Architecture

Inputs include third-person and wrist images plus robot state, including gripper pose. Instruction tokens query image tokens in a Q-Former-style interaction and condition a flow transformer through cross-attention. Actions specify end-effector position/orientation deltas and gripper actuation. The vision encoder is frozen across these experiments.

Refer to caption
Figure 10: Policy network structure.

B.3 Evaluation Protocol

Metrics are computed per training seed and then averaged. Full-task curves display overall success in both training and held-out panels because no unseen instruction remains. The 48-instruction dependent experiments instead use five seeds and ten rollouts per task. Conditional violation rates omit trajectories that fail before the first placement and can therefore conceal differences in earlier-stage failures.

Table 1: Sensitivity to evaluation trials per task on 2S-PP.
Trials per task ID SR (%) OOD SR (%)
5 56.25 15.00
10 63.75 15.70
15 62.92 16.30
30 61.88 15.89

Sensitivity to evaluation trial count.

Increasing evaluation from fifteen to thirty rollouts per task changes training and held-out success by 1.04 and 0.41 percentage points in the listed sensitivity example. This supports the selected trial count for that setting but is not a general statistical confidence bound for every comparison.

Appendix C Q1: Is Full Tuple Coverage Necessary?

Additional PPP experiments expand the task-subset sweep and include a separate simulation experiment with human demonstrations.

Pick-Place-Press.

Fifty-five PPP training-set configurations reproduce the broad relationship between more informative pair coverage and held-out success. The complete figure retains both the training-size trend and held-out performance grouped by seen-pair count.

Human demonstrations.

A separate Isaac Lab task has three objects, containers, and target stations, giving 27 combinations. Human SpaceMouse demonstrations supply thirty examples per task, and the policy trains on a nine-task orthogonal subset. Across three seeds and thirty trials per task, training success is 11.67% and held-out success 12.96%. The similar rates suggest limited compositional degradation, but the low absolute success prevents interpreting this as strong manipulation performance or a physical-robot demonstration.

Refer to caption
(a) Success rate versus budget BB (full experiments). The orthogonal set achieves the same performance as the full task set using only one-fourth of the tasks.
Refer to caption
(b) OOD tasks with more seen pairs achieve higher success.
Figure 11: Additional experiments on the PPP task.
Original figure
Figure 12: Finetuning from sparse pretraining (𝚍𝚒𝚊𝚐​_​𝟶𝟹,B=6\mathtt{diag\_03},B=6). Small amounts of broad-coverage data substantially improve OOD success, suggesting that sparse pretraining learns reusable skills but not reliable instruction steering, which can be acquired with limited additional data.

Appendix D Q2: What Is Missing When Sparse Coverage Fails?

Finetuning lasts 5,000 steps after 100,000 pretraining steps, both at batch size 64. Six-task pretraining shows similar adaptation benefits, and stronger pretrained checkpoints usually yield better final performance. However, an already strong sixteen-task orthogonal checkpoint can degrade when finetuned on just 64 broader demonstrations, consistent with overfitting and a shift in task weighting. The full finetuning curves preserve both training and formerly held-out performance.

Refer to caption
Figure 13: Finetuning from different pretrained models (B=4,6,8,16B=4,6,8,16). Only one demonstration per task can significantly improve OOD success. However, finetuning can also suffer from overfitting when the pretrained model is already strong.

Appendix E Q3: What Coverage Is Needed for Dependent Instructions?

Dependent-task policies train for 150,000 steps. The supplement gives failure-stage breakdowns, the expanded 48-instruction design, oracle tracking rules, and an experiment isolating ambiguous container choice.

Original figure
Figure 14: Failure case analysis on 2S-PP. Numbers indicate rollout counts.

36-task set.

The 36-instruction failure diagrams show that low dependent-pair coverage harms not only container choice but also grasping. Moving from four to six covered pairs reduces some early failures, while wrong-container placement remains common. These breakdowns show why a violation rate conditioned on successful first placement is insufficient by itself.

48-task set.

Allowing four final-container choices gives 48 partial instruction tuples. Sixteen selected training tasks cover sixteen, twelve, nine, or six dependent pairs, with five training seeds and ten evaluation trials per tuple. Broader coverage again improves held-out success and container separation, while narrower coverage concentrates supervision and can improve training-task metrics.

Ground-truth stage indicator.

The oracle tracker uses robot, object, and container state to detect grasps and valid placements, requiring two consecutive matching observations and permitting reversal if progress is lost. PPP exposes object, container, then button instructions. The dependent task exposes first object, second object, then final container.

The tracker uses the final-container constraint to validate the first placement, but the low-level modular controller does not receive that container during the first stage. Tracking poses are not added as policy inputs, and episode success is checked separately. This preserves the distinction between privileged stage supervision and the information available for action selection.

Refer to caption
Figure 15: Performance and same-container violations in 2S-PP (48-task set). Broader dependent-pair coverage improves success and reduces relational errors (e.g., placing object o1o_{1} into container c2c_{2}).
Original figure
Figure 16: Failure case analysis on 2S-PP (48-task set). Numbers indicate rollout counts.

Factors limiting task success in 2S-PP.

Omitting the first-container label leaves several valid placement modes. In an auxiliary PP experiment over five seeds, removing the container instruction lowers success from 71.05% ± 5.00% to 49.95% ± 1.14%. This supports multimodal ambiguity as another difficulty beyond pair coverage. Sequential grasp-and-place errors also accumulate; improved learning from these demonstrations is left open.

Concept demonstration

Open image ↗