Literature
Display
← All papers

Composable Part-Based Manipulation

Weiyu Liu · Jiayuan Mao · Joy Hsu · Tucker Hermans · Animesh Garg · Jiajun Wu

· CoRL 2023 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

Composable part-based manipulation (CPM) represents an action as several relations between functional object parts, such as aligning rims, tilting bodies, and keeping a handle facing upward during pouring. A separate diffusion model learns each relation from successful demonstrations; their predictions are combined at inference to generate start and end object poses.

Part-based composition improves several simulated transfers to new categories and supports qualitative pouring demonstrations on a real robot. It assumes symbolic skill descriptions and external part segmentation, uses two-object tasks with interpolated two-pose trajectories, and does not guarantee that every generated trajectory satisfies the intended constraints.

Task overview

Figure 1 illustrates learning from glasses, pans, and bowls in simulation and transferring the shared pouring relations to previously unseen mugs.

Refer to caption
Figure 1: CPM composes part-based diffusion models to predict target object poses directly from point clouds. In this example, we show that the “pouring” action is decomposed into three part-based correspondences, which generalize manipulation across object categories, and from simulation to the real world

1 Introduction

Different object categories can share functional parts even when their overall shapes differ. A mug and a pan both have a rim, body, and handle, making part relations a potential bridge between their pouring actions. CPM decomposes the skill through these relations rather than learning one undifferentiated whole-object pose distribution.

The main claim is that separately learned part-conditioned distributions can be reused and combined for new object instances and categories. The experiments test pouring and safe placing; they do not evaluate arbitrary skill discovery or long-horizon task planning.

2 Related Work

Prior manipulation representations include part segmentations, affordance maps, keypoints, implicit object descriptors, and pretrained visual features. Other methods model whole-object interactions or relational states. CPM connects these directions by learning distributions conditioned on pairs of functional parts.

The approach builds on SE(3)-DiffusionFields for rigid-pose diffusion and compositional diffusion for combining conditions. The cited MCMC composition work motivates a distributional interpretation; transferring the idea to object poses also requires coordinate-frame handling.

3 Composable Part-Based Manipulation

The anchor object remains fixed while the function object moves. Inputs are partial point clouds and a task description; outputs are two rigid transformations, the starting and ending object poses. SE(3) interpolation supplies the intervening motion. Thus the learned output is a short object-motion parameterization, not a full robot policy or unrestricted trajectory.

Object and part frames are centered at their point-cloud centroids and share the world frame’s orientation. The retained equations convert relative object poses to world motion and relative part poses. These conventions matter when composing models trained on different part pairs.

Refer to caption

Figure 2: (a) Given a task, the partial point clouds of the anchor and function objects, and their parts extracted from a learned segmentation model gϕg_{\phi}, we sample a sequence of transformations from a learned distribution pθp_{\theta} to parameterize the function object’s trajectory. (b) CPM can be generalized to novel object categories because it decomposes each action to a collection of functional correspondences between object parts. To sample the target transformations that satisfy all functional correspondences, CPM combines the noise predictions from a collection of primitive diffusion models at inference time. (c) Each primitive diffusion model learns a target pose distribution that satisfies a particular part-part correspondence, based on the point clouds of the object parts.
Equation𝑻𝑾=𝑻𝑾​𝑨​𝑻𝑨​𝑭​(𝑻𝑾​𝑭)−1{\bm{T}}_{{\bm{W}}}={\bm{T}}_{{\bm{W}}{\bm{A}}}{\bm{T}}_{{\bm{A}}{\bm{F}}}({\bm{T}}_{{\bm{W}}{\bm{F}}})^{-1}

3.1 Action as Part-Based Functional Correspondences

Each symbolic correspondence names a spatial relation and one part from each object. Pouring combines rim alignment, body tilt, and handle/body orientation; safe placing combines head containment, tip-to-bottom contact, and body placement. The names and required relations are supplied rather than discovered automatically.

An external segmentation model extracts each requested part’s point cloud. CPM then learns a conditional generative model for the relation’s relative pose sequence. The paper concentrates on this pose model; the quality and availability of the functional segmentations remain assumptions.

Equation⟨𝑺i,𝑷𝑨,j,𝑷𝑭,k⟩\langle{\bm{S}}_{i},{\bm{P}}_{{\bm{A}},j},{\bm{P}}_{{\bm{F}},k}\rangle

3.2 Generative Modeling of Functional Correspondences with Diffusion Models

For each relation, a Point Cloud Transformer separately encodes the two part clouds. Pose encoders and a diffusion-time embedding join those features in a transformer encoder that predicts noise for the two target transformations. Models for the primitive relations are trained independently on the successful trajectories.

The source describes noise in the tangent space of SE(3), adapting pose denoising to a DDPM. Its mathematical training objective is an MSE noise loss; Appendix A’s implementation table instead lists Huber loss. Both are retained rather than silently treated as identical.

Equation
ℒMSE=‖ϵ−ϵθ,𝑺i​(1−βt​𝒯𝑷j​k(0)+βt​ϵ|𝑿𝑷𝑨,j,𝑿𝑷𝑭,k,t)‖22,{\mathcal{L}}_{\text{MSE}}=\left\|{\epsilon}-{\epsilon}_{\theta,{\bm{S}}_{i}}\left(\sqrt{1-\beta_{t}}{\mathcal{T}}_{{\bm{P}}_{jk}}^{(0)}+\sqrt{\beta_{t}}{\epsilon}~|~{\bm{X}}_{{\bm{P}}_{{\bm{A}},j}},{\bm{X}}_{{\bm{P}}_{{\bm{F}},k}},t\right)\right\|_{2}^{2},

3.3 Inference-Time Composition of Diffusion Models

At inference, a sampled relative pose sequence is converted to each model’s part frame. The models’ denoising predictions are combined in the retained update equation and iterated to produce the final poses. The paper interprets this as a product of the individual relation distributions, favoring poses compatible with all of them.

This composition provides a sampling preference, not an exact constraint solver. Feasibility and physical success are tested through simulation or execution. The main denoising equation and the part-frame transform remain as printed, including the source’s cumulative-schedule notation.

Equationspθ,𝑺i(𝒯|⋅)∝exp(−Eθ,𝑺i(𝒯|⋅))p_{\theta,{\bm{S}}_{i}}({\mathcal{T}}|\cdot)\propto\exp(-E_{\theta,{\bm{S}}_{i}}({\mathcal{T}}|\cdot))∏𝑪∈𝒞𝑴pθ,𝑺i(𝒯|⋅)\prod_{{\bm{C}}\in{\mathcal{C}}_{\bm{M}}}p_{\theta,{\bm{S}}_{i}}({\mathcal{T}}|\cdot)
𝒯𝑨​𝑭(t−1)=1αt​(𝒯𝑨​𝑭(t)−1−αt1−α¯t​∑𝑪∈𝒞𝑴ϵθ,𝑺i​(fto_part​(𝒯𝑨​𝑭(t))|𝑿𝑷𝑨,j,𝑿𝑷𝑭,k,t))+σt​ϵ,{\mathcal{T}}^{(t-1)}_{{\bm{A}}{\bm{F}}}=\frac{1}{\alpha_{t}}\left({\mathcal{T}}^{(t)}_{{\bm{A}}{\bm{F}}}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\sum_{{\bm{C}}\in{\mathcal{C}}_{\bm{M}}}{\epsilon}_{\theta,{\bm{S}}_{i}}\left(f_{\texttt{to\_part}}({\mathcal{T}}^{(t)}_{{\bm{A}}{\bm{F}}})~~|~~{\bm{X}}_{{\bm{P}}_{{\bm{A}},j}},{\bm{X}}_{{\bm{P}}_{{\bm{F}},k}},t\right)\right)+\sigma_{t}\epsilon,
(𝒯𝑨​𝑷𝑨,j)−1​𝒯𝑨​𝑭(t)​𝒯𝑭​𝑷𝑭,k({\mathcal{T}}_{{\bm{A}}{\bm{P}}_{{\bm{A}},j}})^{-1}{\mathcal{T}}^{(t)}_{{\bm{A}}{\bm{F}}}{\mathcal{T}}_{{\bm{F}}{\bm{P}}_{{\bm{F}},k}}

4 Data Collection

Procedural policies generate 4,522 successful pouring demonstrations and 2,836 successful safe-placing demonstrations. Thirteen categories come from PartNet and ShapeNetSem objects cataloged in ACRONYM. PartNet supplies segmentations; ShapeNetSem parts are extracted from manually labeled 3D keypoints and canonical poses.

Simulation follows generated object trajectories and renders RGB-D views from multiple cameras. Only successful demonstrations train the models. The rendered part point clouds can be incomplete, providing some variation resembling partial real observations. Table A2 preserves the full generation and success counts.

Refer to caption
Figure 3: We generate task demonstrations using the PartNet and ShapeNetSem datasets for the “pouring” and “safe placing” tasks. We create demonstrations for a variety of function and anchor object combinations.

5 Experiments

Evaluation separates transfer within known categories, transfer to withheld categories, model ablations, and qualitative simulation-to-robot pouring.

5.1 Experimental Setup

PyBullet directly actuates the manipulated object’s center of mass, deliberately excluding robot grasp sampling and arm motion planning from the simulation comparison. Each reported experiment uses 100 test initializations, five sampled actions each, and three random seeds: 1,500 trials.

Main tables report a 0–100 completion score with partial credit, not binary success percentages. The seen-category protocol describes a random 80/20 data split; it does not give additional object-identity-disjoint split details. The category-transfer protocol explicitly withholds a target function-object category during training. Appendix E separately reports binary success.

5.2 Compared Methods

Baselines are an autoregressive Transformer-BC model, TAX-Pose, whole-object PC-DDPM, and Part-Aware PC-DDPM. The last baseline receives the same part information but concatenates segmentation channels to full point clouds instead of learning separate relation models.

Ablations compare SE(3) against a 6D-rotation-plus-translation representation, remove part-frame composition, train models with joint noise composition, or use individual primitive relation models. The representation ablation also changes frame handling, so it does not isolate only the rotation encoding.

Table 1: CPM demonstrates strong generalization to novel instances of objects within seen categories.
Model Pouring Safe Placing
Transformer-BC 19.21 37.11
TAX-Pose 21.71 76.97
PC-DDPM 75.83 51.55
Part-Aware PC-DDPM 75.28 42.68
CPM (ours) 80.00 70.99
Table 2: CPM demonstrates strong generalization to function objects from unseen object categories.
Model Pouring Safe Placing
Bowl Glass Mug Pan Fork Pen Scissors
Transformer-BC 10.23 20.91 6.06 32.04 26.15 31.93 26.44
TAX-Pose 23.32 3.82 8.64 46.14 50.90 67.60 36.80
PC-DDPM 63.02 75.95 71.39 64.39 40.60 46.63 32.34
Part-Aware PC-DDPM 58.98 72.11 67.11 66.17 39.76 48.04 28.15
CPM (ours) 79.32 81.44 77.57 62.13 55.94 59.45 63.35

5.3 Simulation Results

For the reported seen-category setting, CPM scores 80.00 on pouring and 70.99 on safe placing. It leads the pouring comparison, but TAX-Pose scores higher on safe placing at 76.97. Thus the results do not show CPM best on both tasks.

For withheld categories, CPM pouring scores are 79.32 for bowls, 81.44 for glasses, 77.57 for mugs, and 62.13 for pans. It leads the first three, while Part-Aware PC-DDPM reaches 66.17 on pans. For safe placing, CPM scores 55.94 on forks, 59.45 on pens, and 63.35 on scissors; TAX-Pose leads pens at 67.60. The larger pan rims are the authors’ proposed explanation for that transfer weakness.

Inference-time composition with part frames improves over joint training-time composition: 75.11 versus 69.89 on pouring and 59.58 versus 48.46 on safe placing. The 6D-rotation/no-part-frame variant instead scores 71.22 and 68.77, outperforming full CPM on safe placing. The ablation supports a task-dependent tradeoff rather than uniform superiority of the full configuration.

Composing primitive relations improves the listed aggregate scores over using each primitive alone. For pouring, alignment and tilt are the strongest individual constraints; for safe placing, containment and body placement are stronger than tip contact alone. Some alignment and handle-orientation evaluations omit examples whose partial point clouds lack the needed part, as Table 4’s footnote states. Figure 4 shows the corresponding sampled pose distributions.

Table 3: We ablate the contributions of CPM on the ability to generalize to novel categories of objects.
Target Pose Rep Part Frames Composition Pouring Safe Placing
6D Rot + 3D Trans No Inf-time 71.22 68.77
SE​(3)\textit{SE}(3) Yes Train-time 69.89 48.46
SE​(3)\textit{SE}(3) Yes Inf-time 75.11 59.58
Table 4: We explore the effect of composition, comparing to individual diffusion models, in generalization across both “pouring” and “safe placing” tasks. *We note that for the align and facing-up evaluation, a small percentage of examples were removed as they do not contain the involved parts in the partial object point clouds.
Pouring
⟨align,rim,rim⟩\langle\textit{align},\textit{rim},\textit{rim}\rangle 70.05*
⟨facing-up,handle,body⟩\langle\textit{facing-up},\textit{handle},\textit{body}\rangle 16.42*
⟨t​i​l​t,b​o​d​y,b​o​d​y⟩\langle tilt,body,body\rangle 68.69
CPM 75.11
Safe Placing
⟨c​o​n​t​a​i​n,h​e​a​d,b​o​d​y⟩\langle contain,head,body\rangle 41.22
⟨t​o​u​c​h,t​i​p,b​o​t​t​o​m⟩\langle touch,tip,bottom\rangle 9.34
⟨p​l​a​c​e,b​o​d​y,b​o​d​y⟩\langle place,body,body\rangle 39.86
CPM 59.58
Refer to caption
Figure 4: We illustrate the learned distribution of each primitive diffusion model, which generates diverse samples conforming to the specified constraints, as well as the distribution from the combined full CPM model. The highest-ranked sample is highlighted.
Refer to caption
Figure 5: We show sampled frames from trajectories of CPM’s policy. The model is trained only on demonstrations with pans, bowls, and wine glasses in simulation and generalizes to mugs in the real world.

5.4 Real-World Transfer

The physical experiment transfers models trained on simulated glasses, pans, and bowls to real mugs using a Franka Emika arm. Two calibrated Azure Kinect cameras supply the scene cloud; plane removal and k-means separate objects, and a simulation-trained Point Transformer segments their parts.

Contact-GraspNet proposes grasps, and an impedance-based operational-space controller follows end-effector waypoints derived from CPM’s object transformations. Figure 5 and the project videos show examples. The paper gives no physical-trial count or aggregate success rate, so this is qualitative transfer evidence.

6 Limitations and Conclusion

The demonstrated scope is two objects and two poses connected by interpolation. Multi-object piles, longer paths, temporal constraints, and loops are not modeled. The system relies on externally supplied part segmentations and symbolic correspondences; more composed models can also make sampling expensive.

CPM provides a useful structural bias for several cross-category transfers, with explicit exceptions in the numerical tables. Learning functional parts jointly, using noisier human demonstrations, and extending trajectory and task complexity remain future work.

Acknowledgments

The authors thank NVIDIA Seattle Robotics Lab, Georgia Tech RAIL, and Stanford Vision and Learning Lab. Support includes NSF, AFOSR, ONR MURI, Stanford HAI, MIT–IBM Watson Lab, MIT Quest for Intelligence, CBMM, Analog Devices, JPMC, and Salesforce; the stated views are the authors’ own.

Appendix A Network Architecture

Each primitive model encodes two 512-point part clouds with 200-dimensional Point Cloud Transformer features. A pose MLP also outputs 200 dimensions, learned position embeddings have 16 dimensions, and sinusoidal time embeddings have 40. The original token-construction formulas are retained.

The diffusion transformer uses four layers, four attention heads, hidden dimension 128, and no dropout. Table A1 lists 200 diffusion steps with a linear noise schedule from 0.0001 to 0.02, 2,000 training epochs, Adam at 10⁻⁴, and gradient clipping at 1.0. Its stated Huber loss differs from the main MSE objective.

Equation
χ𝑨(t)\displaystyle\chi^{(t)}_{{\bm{A}}} =[hp​(𝑿𝑷𝑨,j);hp​o​s​(0);ht​i​m​e​(t)]\displaystyle=[h_{p}({\bm{X}}_{{\bm{P}}_{{\bm{A}},j}});h_{pos}(0);h_{time}(t)]
χ𝑭(t)\displaystyle\chi^{(t)}_{{\bm{F}}} =[hp​(𝑿𝑷𝑭,k);hp​o​s​(1);ht​i​m​e​(t)]\displaystyle=[h_{p}({\bm{X}}_{{\bm{P}}_{{\bm{F}},k}});h_{pos}(1);h_{time}(t)]
τn(t)\displaystyle\tau^{(t)}_{n} =[hT​(𝑻𝑷j​k,n(t));hp​o​s​(n−2);ht​i​m​e​(t)]\displaystyle=[h_{T}({\bm{T}}^{(t)}_{{\bm{P}}_{jk},n});h_{pos}(n-2);h_{time}(t)]
Table A1: Model Parameters
Parameter Value
Number of 𝑷𝑨,j{\bm{P}}_{{\bm{A}},j} and 𝑷𝑭,k{\bm{P}}_{{\bm{F}},k} points 512
PCT point cloud encoder hph_{p} out dim 200
Position embedding hp​o​sh_{pos} learned embedding
Position embedding hp​o​sh_{pos} dim 16
Time embedding ht​i​m​eh_{time} Sinusoidal
Time embedding ht​i​m​eh_{time} dim 40
Pose encoder hTh_{T} out dim 200
Transformer number of layers 4
Transformer number of heads 4
Transformer hidden dim 128
Transformer dropout 0.0
Diffusion steps TT 200
Diffusion noise schedule βt\beta_{t} Linear
Start value β0\beta_{0} 0.0001
End value βT\beta_{T} 0.02
Loss Huber
Epochs 2000
Optimizer Adam
Learning rate 1e-4
Gradient clip value 1.0

Appendix B Implementation Details for Baselines

Transformer-BC independently encodes each full object cloud and autoregressively decodes the two poses. TAX-Pose follows the official implementation without pretrained object embeddings; separate models predict the start and end poses because each TAX-Pose model outputs only one relative pose.

PC-DDPM shares the primitive-model architecture but conditions on whole objects. The part-aware variant adds binary ground-truth segmentation channels for all known parts to each point. CPM instead uses those masks to extract the relevant part clouds for separate correspondence models, making the comparison about both representation and composition.

Appendix C Dataset Details

The thirteen categories are mug, pan, bowl, wine glass, knife, can opener, scissors, screwdriver, fork, spoon, marker, pen, and flashlight. Categories can serve different roles across tasks, such as a mug being moved for pouring or serving as a safe-placement container.

The dataset runs 1,000 simulations for each of nine safe-placement categories and 2,000 for each of four pouring categories. It retains only the 2,836 and 4,522 successful trajectories, respectively. Geometry-derived segmentations are mapped onto rendered partial point clouds using the known transformations and scales. Table A2 retains category-specific counts.

Table A2: Simulation and Demonstration Data
Task Object Source Number of Simulations Number of Success Demonstrations
Safe Placing Pen PartNet 1000 568
Fork PartNet 1000 390
ScrewDriver PartNet 1000 145
Spoon PartNet 1000 410
Knife Acronym 1000 496
Scissors PartNet 1000 354
Flashlight PartNet 1000 141
CanOpener PartNet 1000 101
Marker PartNet 1000 231
Pouring Mug PartNet 2000 1051
WineGlass PartNet 2000 1542
Bowl Acronym 2000 776
Pan PartNet 2000 1153

Appendix D Evaluation Details

Test states are drawn from held-out successful demonstrations, ensuring that the chosen object pair has a known feasible action. Each model is sampled five times per state to probe multimodal behavior; three seeds yield the 1,500-trial total.

Pouring receives zero if the object motion interpenetrates the container; otherwise its completion score is the percentage of particles transferred. Safe placing first rejects initial interpenetration and an incorrect orientation, such as an exposed blade. If valid, it scores the percentage of the proposed path executed until contact. These task-specific scores differ from end-to-end robot success.

Appendix E Additional Results

Binary pouring success requires no interpenetration and at least 70% of particles transferred. Safe-placing success additionally requires correct orientation and at least 70% of the proposed path executed without object collision. Tables A3–A4 retain the full success-rate results and dispersion.

For the seen-category setting, CPM achieves 76.87% pouring success and 68.87% safe-placing success; TAX-Pose reaches 74.00% on safe placing. The withheld-category exceptions remain: Part-Aware PC-DDPM performs better on pans, and TAX-Pose on pens. These rates should not be substituted for the partial-credit scores in the main tables.

Table A3: CPM shows strong generalization to novel instances of objects within seen categories.
Model Pouring Safe Placing
Transformer-BC 17.53±\pm3.13 33.27±\pm2.14
TAX-Pose 21.33±\pm0.58 74.00±\pm1.00
PC-DDPM 70.67±\pm1.27 48.73±\pm2.97
Part-Aware PC-DDPM 73.60±\pm2.60 36.53±\pm2.20
CPM (ours) 76.87±\pm1.70 68.87±\pm2.25
Table A4: CPM demonstrates strong generalization to function objects from unseen object categories.
Model Pouring Safe Placing
Bowl Glass Mug Pan Fork Pen Scissors
Transformer-BC 10.00±\pm2.51 19.20±\pm0.92 5.80±\pm1.93 29.33±\pm2.04 24.00±\pm1.51 27.33±\pm2.34 18.47±\pm2.93
TAX-Pose 21.00±\pm1.00 3.00±\pm1.00 8.00±\pm1.00 42.67±\pm2.08 47.67±\pm3.21 62.67±\pm4.04 33.33±\pm1.15
PC-DDPM 56.53±\pm2.00 70.67±\pm3.06 68.67±\pm4.31 59.93±\pm2.80 38.00±\pm3.83 43.47±\pm1.68 28.47±\pm0.83
Part-Aware PC-DDPM 54.87±\pm2.10 68.33±\pm2.97 65.20±\pm4.61 62.00±\pm3.56 28.67±\pm1.68 42.40±\pm3.12 17.67±\pm2.70
CPM (ours) 76.40±\pm1.78 78.93±\pm3.14 76.00±\pm5.26 54.67±\pm1.50 53.93±\pm2.91 56.53±\pm2.04 62.07±\pm1.72

Appendix F Assumptions

Training assumes a symbolic list of part correspondences, successful trajectories, and part segmentations. Testing still requires part segmentations. Humans supply the relation names without continuous constraint formulas; the data grounds those relations geometrically.

Programs used to generate the demonstrations are not required as training labels. Nevertheless, the evaluated data is procedurally generated and success-filtered. Extracting constraints with language models, learning from noisy demonstrations, and discovering functional parts from manipulation data are proposed extensions rather than capabilities established here.

Concept demonstration

Open image ↗