Literature
Display
← All papers

Efficient Data Collection for Robotic Manipulation via Compositional Generalization

Jensen Gao · Annie Xie · Ted Xiao · Chelsea Finn · Dorsa Sadigh

· Robotics: Science and Systems 2024 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

Visual imitation policies can reuse familiar environmental factors in unfamiliar combinations, allowing data collection to cover individual values without exhaustively covering their Cartesian product. Simulation and a real fork-placement task favor structured variation, especially the Stair strategy. Real-world composition depends strongly on prior robot data: Stair with BridgeData V2 transfers to new kitchens in 31/40 trials, versus 1/40 for equally sized in-domain data without variation.

I Introduction

The study treats environmental composition as a data-collection problem. If a policy can combine familiar objects, positions, surfaces, and heights, demonstrators can avoid recording every combination. It compares collection schedules at fixed demonstration budgets and approximately matched numbers of factor changes, then tests when prior robot data supports this generalization.

Refer to caption
Fig. 1: We investigate if robot policies can compose environmental factors of variation (e.g., object types, table heights) in their in-domain training data (1, 2), and if using prior data can be helpful for enforcing such generalization. We propose efficient in-domain data collection strategies guided by the ability of policies to reason about unseen combinations of factor values (3).

II Related Work

Previous work emphasizes larger datasets, pretrained visual representations, or semantic composition of skills and instructions. This paper instead varies concrete environmental factors that change images or required motions, and asks how their coverage should guide in-domain demonstration collection. It evaluates existing imitation policies rather than proposing a new compositional architecture. [38], [16], [5].

III Exploiting Compositional Generalization for Efficient Data Collection

The framework separates environmental factors, collection effort, and policy generalization.

III-A Problem Statement

A factor vector specifies environmental conditions and thereby initial-state and goal distributions. Goal-conditioned behavior cloning learns from expert trajectories labeled implicitly by these conditions; evaluation measures the probability of reaching the associated goal across factor combinations.

Equation 1
J⁡(π)=𝔼f∼p⁡(f)​[Pπ​(sH=μ⁡(f))],J(\pi)=\mathbb{E}_{f\sim p(f)}[P_{\pi}(s_{H}=\mu(f))], (1)

III-B Objective

With N factors and k values each, exhaustive coverage grows as k to the N, whereas covering every individual value requires only order kN changes. This saving is useful only when the learned policy can compose those values. The experiments therefore measure both overall coverage-based generalization and success specifically on unseen combinations of seen values.

Original figure
Fig. 2: Visualization of our data collection strategies with N=2N=2 factors. Each axis consists of possible values for a factor. Each green dot indicates that the strategy captures a specific combination of factor values represented by it, and each pink dot represents a combination that compositional generalization may address. We name our strategies based on the patterns in this visualization.

III-C Data Collection Strategies

Complete covers every combination; Random resamples combinations; Single Factor changes only one dimension; No Variation keeps the base scene. Diagonal introduces unseen values for all factors together. L varies one factor relative to the base scene. Stair cycles through factors while retaining previous changes, producing more diverse overlapping combinations. Factor-change counts approximate effort and do not model all practical costs of resetting scenes.

III-D Hypotheses

The hypotheses are that structured coverage can beat Random and approach Complete, that Stair benefits from its more varied combinations, and that prior robot data strengthens real-world composition.

IV Simulation Experiments

Factor World provides Pick Place and Door Open simulations with controlled environmental variation.

Refer to caption
Fig. 3: Visualization of the Pick Place task. We show combinations of 2 values each for the factors table texture and object position.
Original figure
Fig. 4: Simulation results of data collection strategies for Pick Place. We report results where ℱN\mathcal{F}^{N} consists of each possible factor pair (N=2N=2), average results across all pairs, and results where ℱN\mathcal{F}^{N} consists of all factors (N=5N=5). All points within the same subplot use the same amount of demonstrations. The strategies that exploit composition (Stair, L, Diagonal) generally outperform Random, and often approach Complete. Stair generally performs the best, especially in the N=5N=5 setting. Error bars represent standard error across 5 seeds.

IV-A Evaluation Protocol

Five factors each have ten values: object texture, table texture, camera position, distractors, and object position. Every factor pair is tested with 100 demonstrations; all five factors use 1,000. Policies are evaluated over 100 episodes and five random seeds. Complete is feasible only for pairs; exhaustive five-factor coverage would require at least 100,000 demonstrations.

IV-B Results

For pairs, Stair, L, and Diagonal generally outperform Random and approach Complete after roughly 20 factor changes rather than 100. With five factors, Stair is the only structured strategy with a substantial advantage over Random. Increasing demonstrations at fixed variation helps Stair most; more examples of an unchanged scene do not solve the generalization problem.

Original figure
Fig. 5: (left) Compositional success rate of different strategies.
(right) Generalization of strategies with increasing dataset sizes.

V Real Robot Experiments

Real experiments test whether these compositional effects survive visual and physical variability outside simulation.

V-A Evaluation Protocol

A WidowX 250 arm places a fork into a container. Five primary factors each have four values: fork type and position, container type, table height, and table texture. Stair and L each use 160 demonstrations covering 16 configurations; No Variation uses the same count. Models either train from scratch or pretrain and co-fine-tune with BridgeData V2. [47].

Refer to caption
Fig. 6: Our WidowX robot setup. We consider the task of putting a fork inside a container, in a real office kitchen.
Refer to caption
Fig. 7: Visualization of our primary real robot factors in BaseKitch. The top row shows our base factor values f∗f^{*}. The other rows show all deviations from f∗f^{*} by one factor value. We highlight the portion of each image affected by the specified factor value.
Data Strategy L No Variation
Train Method Bridge From Scratch Bridge
Object
Type
Container
Type
Table
Height
Table
Tex
Object
Type
Container
Type
Table
Height
Table
Tex
Object
Type
Container
Type
Table
Height
Table
Tex
Object Pos 8/9 5/9 2/9 5/9 0/9 0/9 0/9 1/9 3/9 2/9 1/9 1/9
Object Type 8/9 8/9 8/9 5/9 5/9 4/9 4/9 3/9 2/9
Container Type 5/9 6/9 4/9 6/9 2/9 3/9
Table Height 4/9 3/9 1/9
Overall 59/90 28/90 22/90
TABLE I: Real robot pairwise composition results for our “put fork in container” task. When leveraging BridgeData V2 as prior data, a policy is able to compose the factor values present in the L data to succeed on 59/90 compositional combinations of factor values. Without prior data, the model is unable to compose nearly as effectively, with compositional success rate dropping by roughly half. Prior data alone is also not enough to generalize to these situations, as a policy trained with prior data on No Variation also performs poorly.

V-B Pairwise Composition

Across 90 unseen pairwise combinations, L with BridgeData succeeds in 59, versus 28 from scratch and 22 for No Variation with BridgeData. Physically interacting changes, such as object position and table height, remain difficult. Frozen R3M and VC-1 encoders perform poorly in the tested 27-case subset; these results concern the reported diffusion-policy implementations, not every possible use of pretrained representations.

Data Strategy L No Variation
Train Method Bridge R3M VC-1 From Scratch Bridge R3M VC-1
Object Type +
Table Tex
8/9 1/9 0/9 4/9 2/9 0/9 0/9
Object Pos +
Table Tex
5/9 0/9 0/9 1/9 1/9 0/9 0/9
Object Pos +
Table Height
2/9 1/9 0/9 0/9 1/9 0/9 0/9
Overall 15/27 2/27 0/27 5/27 4/27 0/27 0/27
TABLE II: Additional real robot pairwise composition results for our “put fork in container” task with R3M and VC-1.
Refer to caption
Fig. 8: Visualization of BaseKitch with base factor values f∗f^{*} (left), compared to our transfer conditions in CompKitch (top right) and TileKitch (bottom right), which inherently have different factor values (e.g., table texture) and other characteristics (e.g., distractor objects, lighting). We consider setups both with no further changes from f∗f^{*} beyond these inherent differences, as well as with additional changes.
Data Strategy Stair L No Variation
Train Method Bridge
(Co-FT)
Bridge
(FT)
From
Scratch
Bridge
(Co-FT)
Bridge
(FT)
From
Scratch
Bridge
(Co-FT)
Bridge
(FT)
No Further
Changes
CompKitch 4/5 4/5 0/5 5/5 0/5 0/5 0/5 0/5
TileKitch 5/5 2/5 0/5 5/5 5/5 3/5 0/5 0/5
Fork Down +
Blue Plate
CompKitch 4/5 4/5 0/5 3/5 0/5 0/5 1/5 0/5
TileKitch 4/5 3/5 0/5 3/5 5/5 5/5 0/5 0/5
Wooden Fork +
Pink Bowl
CompKitch 3/5 5/5 0/5 2/5 0/5 0/5 0/5 0/5
TileKitch 4/5 3/5 0/5 3/5 5/5 2/5 0/5 0/5
Plastic Fork +
Higher Table
CompKitch 4/5 0/5 0/5 1/5 0/5 0/5 0/5 0/5
TileKitch 3/5 0/5 0/5 2/5 0/5 1/5 0/5 0/5
Overall CompKitch 15/20 13/20 0/20 11/20 0/20 0/20 1/20 0/20
TileKitch 16/20 8/20 0/20 13/20 15/20 11/20 0/20 0/20
Combined 31/40 21/40 0/40 24/40 15/40 11/40 1/40 0/40
TABLE III: Out-of-domain transfer results to new kitchens CompKitch and TileKitch. We find that varied in-domain data from BaseKitch, and BridgeData V2 as prior data, are both critical for effective transfer to these new kitchens. Stair outperforms L, although both achieve significant levels of transfer. Co-fine-tuning generally performs better than only fine-tuning.

V-C Out-Of-Domain Transfer

Two new kitchens introduce unseen factor values and additional uncontrolled shifts. Across 40 trials, co-fine-tuned Stair scores 31/40 and L 24/40, versus 1/40 for No Variation. Fine-tuning without retaining prior data falls to 21/40 and 15/40. From-scratch Stair scores 0/40 and L 11/40. Prior data and varied in-domain demonstrations both contribute.

V-D Additional Experiments

Held-out distractors can reduce robustness when structured datasets train from scratch; prior data restores 15/15 successes in this test. A separate camera/texture composition experiment reaches 10/10 with both complementary datasets and BridgeData, versus 1/10 or 6/10 using either dataset alone. None of the corresponding from-scratch models succeeds.

Data Strategy Stair L No Variation
Train Method Bridge From
Scratch
Bridge From
Scratch
Bridge From
Scratch
Tape Measure 5/5 1/5 5/5 5/5 5/5 5/5
Pink Bowl 5/5 3/5 5/5 4/5 5/5 5/5
Spoon 5/5 5/5 5/5 0/5 5/5 4/5
Overall 15/15 9/15 15/15 9/15 15/15 15/15
TABLE IV: Evaluation on held-out factor distractor objects. Data collection strategies that do not account for this factor can perform worse than No Variation when training from scratch, but this issue is alleviated completely by using prior data.
Refer to caption
(a) Tape Measure
Refer to caption
(b) Pink Bowl
Refer to caption
(c) Spoon
Fig. 9: Visualization of the different values for distractor objects we consider in our held-out factor evaluation.
Refer to caption
(a) Main Camera
Refer to caption
(b) Secondary Camera
Fig. 10: Visualization of the secondary camera view we use for studying composition of camera position (right), compared with our main camera view used in all other experiments (left).
Train Method Bridge From Scratch
No Camera Position 1/10 0/10
No Table Texture 6/10 0/10
Both 10/10 0/10
TABLE V: Our policy composes datasets missing either the correct camera position or table texture seen during evaluation, to outperform training on either dataset alone. Prior data is critical for composition.

VI Discussion

The findings motivate systematic variation without establishing universal compositional guarantees.

VI-A Summary

Stair is generally strongest in the reported comparisons, while L can be operationally convenient when factors are easier to vary separately. Prior robot data supports both composition and robustness to unmodeled changes. Physical interactions may still warrant denser coverage than purely visual variation.

VI-B Limitations and Future Work

Real experiments center on one robot, one main task, and one prior dataset. A preliminary attempt to compose a new removal task with existing factor variation fails. The results do not establish how to collect an optimal large-scale prior dataset, and factor interactions, different architectures, and other learning methods remain open questions.

Acknowledgments

The authors acknowledge DARPA, ONR, NSF, Toyota Research Institute, and colleagues who supported the experiments and robot setup.

Appendix A Data Collection Strategies

The appendix specifies the proposed collection schedules.

A-A Pseudocode

The three retained algorithms distinguish simultaneous unseen-value changes (Diagonal), one-factor deviations from the base scene (L), and cyclic changes that retain previous settings (Stair).

Algorithm 1 Diagonal
Input: scene S, factor values F (size NN factors ×\times kk values)
for j←1j\leftarrow 1 to kk do
  f←0Nf\leftarrow 0^{N}
  for i←1i\leftarrow 1 to NN do
   fi←Fi​jf_{i}\leftarrow\textbf{F}_{ij}
  end for
  SetFactors(S, ff)
  CollectData(S)
end for
Algorithm 2 L
Input: scene S, factor values F (size NN factors ×\times kk values), base factor values f∗f^{*} (size NN factors)
for i←1i\leftarrow 1 to NN do
  f←f∗f\leftarrow f^{*}
  for j←1j\leftarrow 1 to kk do
   fi←Fi​jf_{i}\leftarrow\textbf{F}_{ij}
   SetFactors(S, ff)
   CollectData(S)
  end for
end for
Algorithm 3 Stair
Input: scene S, factor values F (size NN factors ×\times kk values), base factor values f∗f^{*} (size NN factors)
f←f∗f\leftarrow f^{*}
for j←1j\leftarrow 1 to kk do
  for i←1i\leftarrow 1 to NN do
   fi←Fi​jf_{i}\leftarrow\textbf{F}_{ij}
   SetFactors(S, ff)
   CollectData(S)
  end for
end for

Appendix B Simulation Experiments

The simulation appendix reports additional results, sampling choices, augmentation, and effort accounting.

B-A Door Open Results

Door Open broadly reproduces the Pick Place pattern: strong pairwise composition and a larger Stair advantage when all five factors vary.

Original figure
Fig. 11: Simulation results of different data collection strategies for Door Open. We report results where ℱN\mathcal{F}^{N} consists of each possible factor pair (N=2N=2), average results across all pairs, and results where ℱN\mathcal{F}^{N} consists of all factors (N=5N=5). Results are similar as with Pick Place, with Stair generally performing the best, especially in the (N=5N=5) setting. Error bars represent standard error across 5 seeds.

B-B Factors

Textures and distractors are sampled from Factor World training values; positions and camera poses use bounded perturbations. Samples must remain solvable by the scripted expert. Each seed resamples the ten allowed values, and Pick Place retains small per-episode position noise.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
(a) Object Position
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(b) Object Texture
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(c) Table Texture
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(d) Camera Position
Refer to caption
Refer to caption
Refer to caption
Refer to caption
(e) Distractor Objects
Fig. 12: Visualization of our factors for the Pick Place (left) and Door Open (right) tasks from Factor World. We show two examples of values for each factor we consider.

B-C Training

Simulation policies use two 84×84 RGB views and proprioception, with end-effector displacement and gripper actions. Random-shift augmentation is always enabled. These single-task models omit goal conditioning.

B-D Color Jitter Augmentation

Adding color jitter slightly improves overall performance while preserving the main ranking among collection strategies.

Original figure
Fig. 13: Simulation results of data collection strategies for Pick Place, with additional color jitter augmentation. Overall performance across all strategies slightly improves, but the overall trends are similar to as without color jitter. Error bars represent standard error across 5 seeds.

B-E Accounting for Factor Value Similarity

For object and camera positions, selecting representative values by k-medoids improves Stair over random value selection at the same change budget. It still trails observing all values. Extending this approach to objects or textures requires useful similarity measures.

Original figure
Fig. 14: Performance of the Stair strategy for the Pick Place task, when composing object position and camera position. We find that selecting factor values using a similarity-aware strategy outperforms random selection (as done in our main simulation results).

B-F Computing Factor Changes

Effort counts include initial factor setup. Stair, L, and Single Factor count one change per subsequent configuration; Diagonal and Random count one per factor. This can underestimate L resets and overestimate Random when sampled values repeat. Demonstrations are divided across the configurations allowed by each budget.

Appendix C Real Robot Experiments

The real-world appendix retains extra kitchen views, model details, and factor analyses.

Refer to caption
(a) BaseKitch
Refer to caption
(b) CompKitch
Refer to caption
(c) TileKitch
Fig. 15: Additional views of each kitchen in our real robot experiments. We name CompKitch and TileKitch after their countertop materials of composite and tile, respectively. These images were taken after our evaluations, so there may be some slight differences from then.

C-A Robot Platform

Experiments use third-person webcams. A wrist camera helped some from-scratch policies but impaired transfer from BridgeData, which contains relatively little wrist-camera coverage; it is omitted from the reported experiments.

C-B Evaluation Protocol

One experienced operator collects demonstrations with a Quest 2 headset. Relative table height is changed by adjusting the robot platform. Stair follows a fixed cycle through fork position, fork type, container type, table height, and texture.

C-C Training

The ResNet-34 diffusion policy uses two 128×128 image observations plus a goal image and predicts four actions, usually executing only the first. BridgeData pretraining runs for two million steps; fine-tuning uses 300,000. Co-fine-tuning mixes 75% in-domain and 25% prior data. Additional-factor results are retained in Table VI.

Data Strategy L
Train Method Bridge From Scratch
Object
Pos
Object
Type
Container
Type
Table
Height
Table
Tex
Object
Pos
Object
Type
Container
Type
Table
Height
Table
Tex
Object/Container Pos N/A 7/9 6/9 2/9 6/9 N/A 5/9 2/9 0/9 3/9
Object Orientation 2/9 5/9 5/9 1/9 3/9 0/9 2/9 3/9 0/9 1/9
Overall 36/81 16/81
TABLE VI: Additional real robot pairwise composition results for our “put fork in container” task, with additional factors object/container position and object orientation. Similarly as in our main pairwise composition results in Table I, leveraging BridgeData V2 as prior data significantly improves composition for these factors compared to training from scratch.
Refer to caption
Fig. 16: Visualization of our additional real robot factors in BaseKitch. The top row shows our base factor values f∗f^{*}. The other rows show all deviations from f∗f^{*} by one factor value.

C-D Pairwise Composition for Additional Factors

Adding joint fork/container positions and fork orientations yields 36/81 compositional successes with prior data versus 16/81 without it. Orientation requires separate training with half the in-domain batch allocated to that factor and remains especially difficult.

Original figure
Fig. 17: Per-factor value success rates of policies from our pairwise composition results in Table I. Factor values are placed in tiers, where increasing tiers are more dissimilar from the base factor values f∗f^{*}. Composition is generally more challenging for factor values that are more dissimilar from the base factor values.

C-E Factor Similarity Analysis

Factor values are ranked by a No Variation policy’s success as a proxy for similarity to the base scene. More dissimilar values generally compose less well, although the ranking also reflects intrinsic task difficulty and trends are not strictly monotonic.

Factor Tier 1 Tier 2 Tier 3
Object Pos Down (3) Up (4) Left (2)
Object Type Wooden (3) Gray (4) Plastic (2)
Container Type Blue Plate (4) Pink Bowl (3) White Cup (2)
Table Height Higher 5cm (2) Lower 5cm (3) Lower 8cm (4)
Table Tex Brown Wood (2) Gift Wrap (4) White Marble (3)
TABLE VII: Tiers for each factor value used in Fig. 17, determined using our success rate-based similarity metric as described in Section C-E. In parentheses next to each factor, we provide the row number where the factor value is visualized in Fig. 7.

C-F R3M

R3M features use ResNet-50, dataset normalization, a diffusion head without goal conditioning, 500,000 training steps, and execution of all four predicted actions to reduce jitter. These changes make the frozen-encoder comparison distinct from the end-to-end setup.

C-G VC-1

VC-1 follows the R3M training and evaluation procedure, substituting its ViT-L visual encoder.

Concept demonstration

Open image ↗