Literature
Display
← All papers

Explicitly capturing demonstrator multi-modality for distillation into diffusion policies

Shao Yuan Chew Chia · Rebecca H. Jiang · Abhinav Agarwal · Andreas Hofmann · Leslie Pack Kaelbling · Russ Tedrake

Preprint posted · CoRL 2026 Pretrain to Adapt Workshop ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

Overview

Reading focus: a policy can achieve acceptable single-rollout success while failing to preserve the demonstrator’s alternative actions. This study tests a data-collection intervention for re-queryable demonstrators, with controlled synthetic analysis and simulated truck unloading. The edition keeps all nine figures, three tables, three numbered equations and the complete sampling algorithm.

At a glance

Sliding-window behavior-cloning data usually pair each continuous observation history with only one action sequence. A sufficiently expressive conditional diffusion model can memorize those pairings, collapsing its output at each observation even when the demonstrator is stochastic. Vine sampling instead records several independently queried actions at exactly the same history. At matched numbers of stored training pairs, this improves conditional-distribution matching and downstream search/self-improvement in the tested sparse-data settings. It trades fewer distinct observations and substantially more collection compute for explicit action alternatives; dense-data and hardware benefits are not established.

1 Introduction

Best-of-N search, expert iteration and reinforcement-learning post-training need useful alternatives to choose from, not merely a policy that succeeds once. Ordinary trajectories omit the actions a demonstrator could have taken at each visited state. With continuous high-dimensional observations, exact histories rarely recur, leaving ambiguity between a genuinely multimodal demonstrator and a rapidly changing unimodal function. Repeated queries disambiguate the training target. This requires either resettable environments for closed-loop demonstrators or planners that return actions without changing the world. The paper isolates actions-per-condition k while fixing stored-pair count and training budget, then tests the resulting policies as search and improvement proposals.

Refer to caption
Figure 1: Actions from the truck unloading demonstrator of sec. C.1 (top), the vine policy (middle), and the sliding window policy (bottom), queried at the same state of a held-out problem instance. Each panel shows the observation image with future arm poses overlaid; poses with suction on are shown in pink. Of 10 samples each, the vine policy’s reach three of the four grasps, while the sliding window policy’s all approach the same grasp.
Original figure
Figure 2: The choices that a demonstrator has the capacity to take branch along an episode. Sliding window datasets record one action per observation history, while vine datasets record several counterfactual actions at a random subset of observation histories. Gray actions are recorded in neither dataset.

2 Related Work

The analysis builds on conditional diffusion memorization results: at training conditions, the ideal denoising solution reproduces the empirical conditional sample distribution. Existing diversity interventions add noise during training/inference or alter latent exploration, while other work deliberately reduces expert entropy for single-rollout success. Here the aim is to preserve expert entropy for downstream search. Multiple labels per condition already appear in language, segmentation, grasping and motion planning; the contribution is a controlled study under the ordinary diffusion objective and a concrete behavior-cloning collection protocol. Prior sparse-data studies motivate the setting without proving that every large robot policy is in the same regime.

3 Problem Formulation

The formulation distinguishes the demonstrator’s conditional distribution, the finite set recorded at one observation and the learned policy distribution. Comparing all three separates a model-fitting problem from a collection-coverage problem.

3.1 Setting

An observation history contains Tₒ timesteps; an action output spans Tₐ future timesteps. Group the dataset by n distinct histories, each with k independently drawn demonstrator action sequences, equivalent to n×k ordinary training pairs. Standard collection uses k=1. Experiments increase k while holding n×k fixed, so explicit action diversity necessarily reduces the number of unique conditioning histories. The policy class is assumed expressive enough to represent the relevant multimodal distributions.

3.2 Study of the trained policy

Use Wasserstein-1 distance for distributional error and mode recall@N for whether N draws cover useful modes. Equation 1 bounds policy-to-demonstrator error by model-to-empirical error A plus empirical-to-demonstrator sampling error B; the reverse triangle inequality also gives a lower bound B−A. Thus the policy error lies within A of B. Near exact memorization, A is small, so better fitting alone cannot overcome poor conditional samples: lowering B requires more representative recorded alternatives. This argument is local to training conditions and the memorization regime; a model that generalizes away from training samples can escape the k=1 empirical bottleneck.

Equation 1
W1​(πθ,πE)⏟(LHS) policy error≤W1​(πθ,P^k)⏟(A) memorization error+W1​(P^k,πE)⏟(B) data-collection sampling error,\underbrace{W_{1}\big(\pi_{\theta},\,\pi_{E}\big)}_{\mathclap{\text{(LHS) policy error}}}\hskip 15.00002pt\leq\hskip 15.00002pt\underbrace{W_{1}\big(\pi_{\theta},\,\hat{P}_{k}\big)}_{\mathclap{\text{(A) memorization error}}}\hskip 15.00002pt+\hskip 15.00002pt\underbrace{W_{1}\big(\hat{P}_{k},\,\pi_{E}\big)}_{\mathclap{\text{(B) data-collection sampling error}}}, (1)

4 Analysis

The controlled experiment separates action-mode coverage from generalization across observations. It first holds total sample count at 1,000, then evaluates unseen conditions and larger datasets to identify where repeated queries help.

4.1 Synthetic task

Observations and actions are each 16-dimensional, with one informative observation coordinate z and 15 nuisance coordinates. The first action coordinate y must lie in four equally weighted intervals shifted by z; the other 15 coordinates are independent Gaussian nuisance actions. Widely separated high-dimensional observations mimic sparse visual histories. Random action coordinates make near-exact retrieval across all 16 dimensions strong evidence of memorization rather than merely choosing the same valid task mode. Equation 2 preserves the full distribution, including Gaussian standard deviation 0.3.

Original figure
Figure 3: (a) Coordinate yy of the synthetic task at two observations, with k=5k=5 actions sampled at each; arrows run from each observation to its four valid modes. (b) Visualization of synthetic datasets with fixed total dataset size n⋅k=1000n\cdot k=1000, but varied numbers of observations nn, and actions per observations kk; gray diagonals show the valid modes of the task. At fixed dataset size, more actions per condition comes at the cost of sparsity in zz.
Equation 2
y=z+u,u∼pR,𝐚[1,c]=εc(c=2,…,16),εc∼i.i.d.𝒩(0,0.32),y=z+u,\quad u\sim p_{R},\qquad\mathbf{a}[1,c]\;=\;\varepsilon_{c}\quad(c=2,\dots,16),\quad\varepsilon_{c}\overset{\text{i.i.d.}}{\sim}\mathcal{N}(0,0.3^{2}), (2)

4.2 Findings

At 1,000 stored pairs, k=1,10,100 gives 1,000,100,10 distinct conditions. Each policy is queried 1,000 times per condition. All three retrieve training actions frequently at 5%/25% distance tolerances, while fresh demonstrator draws almost never do. Model-to-empirical error stays small; sampling error and policy error fall as k grows. Conditional mode entropy rises from near zero to near one, mode recall and spread improve, and off-support mass changes little. More inference draws barely restore missing modes for k=1.

At held-out observations, k>1 retains lower error and better diversity, though its four-mode structure weakens farther from training observations. With 100,000 total pairs, even k=1 generalizes, retrieval approaches zero and its error/diversity largely match larger k. Repeated conditioning is therefore useful here in the sparse-data memorization regime, not a universal requirement for diffusion policies. Entropy only measures mass already within valid support, so the separate off-support metric is essential.

Original figure
Figure 4: Policies trained on the three datasets of fig. 3, queried at their training observations. (a) Each policy’s draws at a representative training observation. Dashed segments and gray bands are πE\pi_{E} and its support. Short black lines mark the training actions at each observation. (b)–(g) The metrics of sec. 4.2, averaged over training observations; (c) shows the terms of (1) on the primary coordinate yy. Reference lines are fresh draws from the demonstrator (black dashed, ℋ0gen\mathcal{H}^{\mathrm{gen}}_{0}) and resampled draws from the training actions (colored dashed, ℋ0mem\mathcal{H}^{\mathrm{mem}}_{0}).

5 Data collection method: vine sampling

Adapt the vine procedure to collect labels: the demonstrator follows one normal episode, replanning when necessary; independently at each visited timestep, with probability p, re-query k complete candidate action sequences from the identical observation/state. Reset between queries if querying changes the environment, and record only these selected histories with their k labels. Continue the original single trajectory rather than executing all branches. This preserves the episode’s temporal state distribution while exposing counterfactual action choices. Algorithm 1 retains the exact control flow and reset condition.

6 Truck unloading experiments

A simulated planar suction robot unloads four boxes before an 80-second timeout. Randomized grasp ordering and motion planning yield alternatives in box choice, grasp face, arm configuration and trajectory. Both methods use the same 2,000 problem instances and 370,000 stored pairs: sliding-window k=1 versus vine k=10,p=0.1, followed by 100,000 training steps. Equal stored-pair count does not equal collection compute. Tests compare raw execution, best-of-N search and expert iteration. Vine better matches the demonstrator’s conditional distribution and improves search success/throughput; later expert iterations retain throughput gains but slightly lose success, plausibly through imperfect distillation and declining diversity.

A 20,000-step early-stopped baseline preserves more spread than the 100,000-step sliding-window policy but matches expert choices less well. Raising sliding-window search from N=10 to 100 helps without closing the gap. SIME’s encoder noise approaches vine’s search success but yields lower throughput, more raw-policy failures and worsening later expert iterations; noise can diversify imprecise actions. PostBC widens actions using ensemble uncertainty but only modestly improves distributional coverage and worsens raw success here. These are task-specific comparisons; Figure 5 keeps all uncertainty intervals and raw/search results.

Original figure
Figure 5: Truck unloading experiment. (a), (b) Policy error W1​(πθ,πE)W_{1}(\pi_{\theta},\pi_{E}) and SD ratio per method at held-out conditions. (c), (d) Success rate and throughput, in best-of-NN search; N=10N=10 for solid bars except more search where N=100N=100. (e), (f) The same two metrics over expert iteration for the sliding window, SIME, and vine methods. Plotted values are means and error bars are 95% intervals.

7 Conclusions and limitations

Repeated demonstrator queries preserve alternatives that a memorizing policy cannot infer from one action per unique history. The improvement comes with costs: vine collection takes about 30 times more compute in truck unloading, and p/k were not optimized. The best k for search may differ from the best k for raw execution. All experiments use simulation, including candidate verification inside search and expert iteration. Non-state-changing planners could provide real-world re-queries without resets, but those unexecuted alternatives would not be physically verified. Whether much larger robot models/datasets remain sparse enough to benefit is open.

Acknowledgments

Support is acknowledged from Pickle Robot Company and the Charles Stark Draper Laboratory. Rebecca H. Jiang is a Draper Scholar; part of Shao Yuan Chew Chia’s contribution was made during a Pickle Robot internship.

Appendix A Synthetic Analysis

The appendix specifies the synthetic distribution, training recipe, metric conventions and held-out query construction, then repeats the analysis across four dataset scales. These details qualify the apparent entropy and generalization results.

A.1 Task, datasets and training

The four residual modes have width 0.1 and centers ±0.15,±0.45, leaving gaps within the overall [−0.5,0.5] range. Twelve policies combine k∈{1,10,100} with total sizes 10²,10³,10⁴,10⁵. Table 1 retains the shared 3.1-million-parameter one-level U-Net, 100-step DDPM, batch 256 and five million training steps, about 13.5 hours per policy on an RTX 4090. This extensive fitting budget matters for the observed memorization; it is not the truck policy’s training recipe.

Implementation LeRobot DiffusionPolicy
Architecture 1D convolutional U-Net, one level of width 256
Parameters 3.1M
Diffusion steps 100 for training and sampling (DDPM)
Noise schedule cosine
Prediction target ϵ\epsilon
Optimizer AdamW, learning rate 10−410^{-4}, weight decay 10−610^{-6}
Learning rate schedule 500-step warmup, then constant
Batch size 256
Training steps 5×1065\times 10^{6}
Hardware NVIDIA RTX 4090, 13.5 h per policy
Table 1: Architecture and training recipe of the twelve synthetic policies.

A.2 Metrics

Synthetic Wasserstein terms against the expert use its exact cumulative distribution; policy distributions use 1,000 draws. Retrieval searches all stored actions, not just labels at the queried condition. If fewer than 15 draws fall within expert support, conditional entropy is set to zero and that condition stays in the mean. This occurs for 7% of k=1 training conditions and 58% of its held-out queries at 1,000 total pairs, versus none/under 1% for larger k. Reported training means cover up to 128 conditions spaced in z, and reference distributions use the same 1,000-draw budget. This convention prevents selectively dropping failed conditions but affects interpretation of low entropy.

A.3 Held-out conditions

Held-out queries follow random-direction lines from training observations through the 16-dimensional space. Distance d is normalized by the mean pairwise training distance of the k=1,n=1,000 dataset. Use 24 distance bins from 0 to 1.18, with 96 queries per bin and no more than one query from each originating line in a bin. Figure 6 retains representative histograms at d=0.5/1.0 and all metric curves; this constructed probe distribution is not a robot deployment distribution.

Original figure
Figure 6: The policies of fig. 4, queried at held-out conditions at distance dd from the nearest training observation, in units of the mean distance between two training observations. (a), (b) Each policy’s draws at a representative held-out condition at d=0.5d=0.5 and d=1.0d=1.0. Dashed segments and gray bands are πE\pi_{E} and its support. Short black lines mark the nearest training observation’s actions, shifted into the query’s residual frame. (c)–(h) The metrics of fig. 4, averaged within bins of dd; diamonds at d=0d=0 are the values at training observations. Reference lines are fresh draws from the demonstrator (black dashed, ℋ0gen\mathcal{H}^{\mathrm{gen}}_{0}).

A.4 Results against total dataset size

Figure 7 compares all 12 policies over total sample count. At sparse scales, repeated labels improve conditional coverage; at 100,000 pairs, k=1 learns across nearby conditions rather than retrieving individual labels and approaches the other policies on the principal error/diversity measures. This supports a regime-dependent conclusion and does not establish an optimal k for arbitrary architectures or data budgets.

Original figure
Figure 7: The metrics of fig. 4 against the total dataset size n⋅kn\cdot k, one curve per kk, averaged over training observations. Reference lines are fresh draws from the demonstrator (black dashed, ℋ0gen\mathcal{H}^{\mathrm{gen}}_{0}).

Appendix B Vine Sampling

Algorithm 1 records selected histories along a single demonstrator rollout. Following every queried branch would overrepresent later-time states, so those branches are stored as alternatives and not all executed. Selected conditions occur at probability p across the original temporal distribution. The procedure assumes the expert can plan from any visited state; if partial observability or another restriction prevents that, selection must also test whether that state is queryable. The appendix’s HTML cross-reference to the conditional line is resolved by the preserved original algorithm numbering.

Algorithm 1 Vine sampling
Original Algorithm 1, input and all 14 numbered lines, preserving branch indentation.

Appendix C Truck Unloading

The truck appendix specifies simulator failures, reward, observations, dataset splits, policy/value architectures, distributional metrics, search and baseline implementations. It exposes both a substantial collection-compute imbalance and a plan-versus-executed-action distinction in the evaluation.

C.1 Task and demonstrator

Drake simulates randomized boxes and a planar robot viewed from the side, removing partial observability as a confound. Observations contain an image plus four joint positions, suction state and a grasped-box flag; two observations condition 32 future actions, each four joint positions plus suction. Excess penetration or excessive tracking error causes failure. Each timestep costs 0.1 reward; failure/timeout costs 100. Failed throughput uses the 80-second timeout duration. The expert shuffles candidate grasps, finds admissible motions with bidirectional RRT, then refines them by trajectory optimization; every visited state is re-queryable.

C.2 Problem instances and datasets

Training uses instances 0–1,999; expert iteration uses 2,000–6,999; other tests share 600 separate instances. The expert achieves 95.8% success and 766 pieces/hour on that test set. Failed demonstration episodes remain in both datasets: successful counts are 1,899 sliding-window and 1,878 vine episodes. Vine stores returned plans; sliding-window stores executed motion, which includes controller tracking error. Distributional evaluation also compares against re-queried plans, favoring the same action representation used for vine training; this is an explicit measurement distinction. Collection takes 55 versus 1.8 seconds per episode on a 96-core machine.

C.3 Policy and value networks

Tables 2/3 preserve the full architectures and optimization settings. Both policies share the CNN U-Net with widths 256/512/1024, from-scratch ResNet-18 encoder,100-step DDPM, batch 2,048,100,000 steps and exponential moving-average weights; only data differ. Policy training takes about 8 hours on two H100 GPUs. The shared initial value function uses one image plus six state coordinates, a ResNet/MLP architecture and tanh output, trained 100,000 steps with batch 1,024 and Adam learning rate 2×10⁻³, about 4.9 hours on one H100. PostBC changes training targets rather than this policy architecture.

Table 2: Diffusion policy settings.
Architecture CNN-based U-Net of Chi et al. [10], channels (256,512,1024)(256,512,1024)
Image encoder ResNet-18 trained from scratch, GroupNorm, spatial softmax
Diffusion steps 100 for training and sampling (DDPM)
Noise schedule cosine
Optimizer AdamW, learning rate 3×10−43\times 10^{-4} (constant), weight decay 10−610^{-6}
Batch size 2048
Training steps 100,000
EMA of weights yes
Hardware two NVIDIA H100 GPUs, about 8 h per policy
Table 3: Value network settings.
Input one image and the 6D state
Image encoder ResNet-18 trained from scratch
State encoder MLP
Head MLP with a tanh\tanh output
Optimizer Adam, learning rate 2×10−32\times 10^{-3}
Batch size 1024
Training steps 100,000
Hardware one NVIDIA H100 GPU, 4.9 h

C.4 Metrics

Select 576 states along expert test episodes with probability 0.005/timestep, then retain 507 where all 200 expert queries return plans. Split those queries into 100-action estimate/reference sets, and compare 100 draws from each policy. Equation 3 defines RMS trajectory distance after scaling each coordinate by expert standard deviation across all evaluation states/timesteps. Policy error is Wasserstein-1 under that distance; spread is RMS distance from the mean, normalized by expert spread. Filtering to fully plannable states limits the evaluated distribution. Error/spread/throughput intervals bootstrap problem instances 1,000 times, preserving within-instance dependence; success uses equal-tailed Beta(1+successes,1+failures) posterior intervals. All are 95%, but they are not the same interval construction.

Equation 3
d⁡(𝐚,𝐚′)=1Ta​da​∑h=1Ta∑c=1da(𝐚⁡[h,c]−𝐚′​[h,c]ςc)2.d(\mathbf{a},\mathbf{a}^{\prime})\;=\;\sqrt{\frac{1}{T_{a}d_{a}}\sum_{h=1}^{T_{a}}\sum_{c=1}^{d_{a}}\left(\frac{\mathbf{a}[h,c]-\mathbf{a}^{\prime}[h,c]}{\varsigma_{c}}\right)^{\!2}}. (3)

C.5 Best-of- N search and expert iteration

Best-of-N simulates each candidate’s first 20 timesteps, rejects failures, ranks survivors by simulated reward plus learned terminal value and executes the best; all-failed candidates end the episode. Planning time does not advance simulation time, so pieces/hour excludes search latency. Expert iteration uses best-of 100 to collect 1,000 fresh instances per iteration, then fine-tunes policy and value for 10,000 steps with older data weighted 0.8ʲ. Every method uses ordinary sliding-window collection during these later iterations: only initialization benefits from vine. Value networks begin shared but diverge under fine-tuning. Iteration curves report the 1,000 collection instances themselves, not a separately evaluated policy test set; iteration 0 is base-policy best-of 100 on new instances.

C.6 SIME and PostBC

SIME adds one image-encoder noise sample per action draw, linearly annealed to zero through denoising; scaleσ=4 is chosen from the sweep to approximately match expert spread and minimize policy error. Its absolute amplitude uses encoder-output RMS standard deviation over 576 states. During expert iteration, that base-policy noise level remains fixed. PostBC trains 10 bootstrapped conditional-mean regressors, each using the image encoder plus three 512-unit MLP layers for 6,000 steps at batch 256. Their prediction covariance spans the full 32×5 action sequence and supplies fresh Gaussian target noise with α=1. Ensemble size/α follow defaults without tuning, with fewer ensemble-training epochs because this dataset is larger.

Original figure
Figure 8: The SIME baseline’s noise sweep on the sliding window policy, against the noise scale σ\sigma; σ=0\sigma=0 is the unmodified policy. (a) Policy error and (b) SD ratio at the states of fig. 5, for the 100,000-step checkpoint (dark) and the 50,000-step one (light). (c) Success rate and (d) throughput on the 600 test instances at the 100,000-step checkpoint, under best-of-10 (filled) and for the raw policy (open). Reference lines are two independent sets of expert re-queries (black dashed) and the vine policy at 100,000 steps, under best-of-10 in (c) and (d) (green). The circled point is σ=4\sigma=4, the value used in fig. 5. Means, with 95% intervals as in fig. 5.

C.7 Results against training step

Figure 9 preserves checkpoints every 10,000 training steps. Sliding-window spread falls while its policy error remains high; vine maintains broader, better-matched conditionals and stronger best-of 10 success. Both raw-policy curves remain much lower than search-assisted success. The 20,000-step early-stopping comparison is selected for similar raw success/throughput to the fully trained sliding-window policy, not as a universal optimum.

Original figure
Figure 9: The sliding window and vine policies against training step. (a) Policy error and (b) SD ratio at the states of fig. 5. (c) Success rate under best-of-10 and (d) of the raw policy, on the 600 test instances. Reference lines in (a) and (b) are two independent sets of expert re-queries (black dashed).
Concept demonstration

Open image ↗