At a glance
Composable part-based manipulation (CPM) represents an action as several relations between functional object parts, such as aligning rims, tilting bodies, and keeping a handle facing upward during pouring. A separate diffusion model learns each relation from successful demonstrations; their predictions are combined at inference to generate start and end object poses.
Part-based composition improves several simulated transfers to new categories and supports qualitative pouring demonstrations on a real robot. It assumes symbolic skill descriptions and external part segmentation, uses two-object tasks with interpolated two-pose trajectories, and does not guarantee that every generated trajectory satisfies the intended constraints.
Task overview
Figure 1 illustrates learning from glasses, pans, and bowls in simulation and transferring the shared pouring relations to previously unseen mugs.
1 Introduction
Different object categories can share functional parts even when their overall shapes differ. A mug and a pan both have a rim, body, and handle, making part relations a potential bridge between their pouring actions. CPM decomposes the skill through these relations rather than learning one undifferentiated whole-object pose distribution.
The main claim is that separately learned part-conditioned distributions can be reused and combined for new object instances and categories. The experiments test pouring and safe placing; they do not evaluate arbitrary skill discovery or long-horizon task planning.
2 Related Work
Prior manipulation representations include part segmentations, affordance maps, keypoints, implicit object descriptors, and pretrained visual features. Other methods model whole-object interactions or relational states. CPM connects these directions by learning distributions conditioned on pairs of functional parts.
The approach builds on SE(3)-DiffusionFields for rigid-pose diffusion and compositional diffusion for combining conditions. The cited MCMC composition work motivates a distributional interpretation; transferring the idea to object poses also requires coordinate-frame handling.
3 Composable Part-Based Manipulation
The anchor object remains fixed while the function object moves. Inputs are partial point clouds and a task description; outputs are two rigid transformations, the starting and ending object poses. SE(3) interpolation supplies the intervening motion. Thus the learned output is a short object-motion parameterization, not a full robot policy or unrestricted trajectory.
Object and part frames are centered at their point-cloud centroids and share the world frame’s orientation. The retained equations convert relative object poses to world motion and relative part poses. These conventions matter when composing models trained on different part pairs.
Equation
3.1 Action as Part-Based Functional Correspondences
Each symbolic correspondence names a spatial relation and one part from each object. Pouring combines rim alignment, body tilt, and handle/body orientation; safe placing combines head containment, tip-to-bottom contact, and body placement. The names and required relations are supplied rather than discovered automatically.
An external segmentation model extracts each requested part’s point cloud. CPM then learns a conditional generative model for the relation’s relative pose sequence. The paper concentrates on this pose model; the quality and availability of the functional segmentations remain assumptions.
Equation
3.2 Generative Modeling of Functional Correspondences with Diffusion Models
For each relation, a Point Cloud Transformer separately encodes the two part clouds. Pose encoders and a diffusion-time embedding join those features in a transformer encoder that predicts noise for the two target transformations. Models for the primitive relations are trained independently on the successful trajectories.
The source describes noise in the tangent space of SE(3), adapting pose denoising to a DDPM. Its mathematical training objective is an MSE noise loss; Appendix A’s implementation table instead lists Huber loss. Both are retained rather than silently treated as identical.
Equation
3.3 Inference-Time Composition of Diffusion Models
At inference, a sampled relative pose sequence is converted to each model’s part frame. The models’ denoising predictions are combined in the retained update equation and iterated to produce the final poses. The paper interprets this as a product of the individual relation distributions, favoring poses compatible with all of them.
This composition provides a sampling preference, not an exact constraint solver. Feasibility and physical success are tested through simulation or execution. The main denoising equation and the part-frame transform remain as printed, including the source’s cumulative-schedule notation.
Equations
4 Data Collection
Procedural policies generate 4,522 successful pouring demonstrations and 2,836 successful safe-placing demonstrations. Thirteen categories come from PartNet and ShapeNetSem objects cataloged in ACRONYM. PartNet supplies segmentations; ShapeNetSem parts are extracted from manually labeled 3D keypoints and canonical poses.
Simulation follows generated object trajectories and renders RGB-D views from multiple cameras. Only successful demonstrations train the models. The rendered part point clouds can be incomplete, providing some variation resembling partial real observations. Table A2 preserves the full generation and success counts.
5 Experiments
Evaluation separates transfer within known categories, transfer to withheld categories, model ablations, and qualitative simulation-to-robot pouring.
5.1 Experimental Setup
PyBullet directly actuates the manipulated object’s center of mass, deliberately excluding robot grasp sampling and arm motion planning from the simulation comparison. Each reported experiment uses 100 test initializations, five sampled actions each, and three random seeds: 1,500 trials.
Main tables report a 0–100 completion score with partial credit, not binary success percentages. The seen-category protocol describes a random 80/20 data split; it does not give additional object-identity-disjoint split details. The category-transfer protocol explicitly withholds a target function-object category during training. Appendix E separately reports binary success.
5.2 Compared Methods
Baselines are an autoregressive Transformer-BC model, TAX-Pose, whole-object PC-DDPM, and Part-Aware PC-DDPM. The last baseline receives the same part information but concatenates segmentation channels to full point clouds instead of learning separate relation models.
Ablations compare SE(3) against a 6D-rotation-plus-translation representation, remove part-frame composition, train models with joint noise composition, or use individual primitive relation models. The representation ablation also changes frame handling, so it does not isolate only the rotation encoding.
| Model | Pouring | Safe Placing |
| Transformer-BC | 19.21 | 37.11 |
| TAX-Pose | 21.71 | 76.97 |
| PC-DDPM | 75.83 | 51.55 |
| Part-Aware PC-DDPM | 75.28 | 42.68 |
| CPM (ours) | 80.00 | 70.99 |
| Model | Pouring | Safe Placing | |||||
| Bowl | Glass | Mug | Pan | Fork | Pen | Scissors | |
| Transformer-BC | 10.23 | 20.91 | 6.06 | 32.04 | 26.15 | 31.93 | 26.44 |
| TAX-Pose | 23.32 | 3.82 | 8.64 | 46.14 | 50.90 | 67.60 | 36.80 |
| PC-DDPM | 63.02 | 75.95 | 71.39 | 64.39 | 40.60 | 46.63 | 32.34 |
| Part-Aware PC-DDPM | 58.98 | 72.11 | 67.11 | 66.17 | 39.76 | 48.04 | 28.15 |
| CPM (ours) | 79.32 | 81.44 | 77.57 | 62.13 | 55.94 | 59.45 | 63.35 |
5.3 Simulation Results
For the reported seen-category setting, CPM scores 80.00 on pouring and 70.99 on safe placing. It leads the pouring comparison, but TAX-Pose scores higher on safe placing at 76.97. Thus the results do not show CPM best on both tasks.
For withheld categories, CPM pouring scores are 79.32 for bowls, 81.44 for glasses, 77.57 for mugs, and 62.13 for pans. It leads the first three, while Part-Aware PC-DDPM reaches 66.17 on pans. For safe placing, CPM scores 55.94 on forks, 59.45 on pens, and 63.35 on scissors; TAX-Pose leads pens at 67.60. The larger pan rims are the authors’ proposed explanation for that transfer weakness.
Inference-time composition with part frames improves over joint training-time composition: 75.11 versus 69.89 on pouring and 59.58 versus 48.46 on safe placing. The 6D-rotation/no-part-frame variant instead scores 71.22 and 68.77, outperforming full CPM on safe placing. The ablation supports a task-dependent tradeoff rather than uniform superiority of the full configuration.
Composing primitive relations improves the listed aggregate scores over using each primitive alone. For pouring, alignment and tilt are the strongest individual constraints; for safe placing, containment and body placement are stronger than tip contact alone. Some alignment and handle-orientation evaluations omit examples whose partial point clouds lack the needed part, as Table 4’s footnote states. Figure 4 shows the corresponding sampled pose distributions.
| Target Pose Rep | Part Frames | Composition | Pouring | Safe Placing |
| 6D Rot + 3D Trans | No | Inf-time | 71.22 | 68.77 |
| Yes | Train-time | 69.89 | 48.46 | |
| Yes | Inf-time | 75.11 | 59.58 |
5.4 Real-World Transfer
The physical experiment transfers models trained on simulated glasses, pans, and bowls to real mugs using a Franka Emika arm. Two calibrated Azure Kinect cameras supply the scene cloud; plane removal and k-means separate objects, and a simulation-trained Point Transformer segments their parts.
Contact-GraspNet proposes grasps, and an impedance-based operational-space controller follows end-effector waypoints derived from CPM’s object transformations. Figure 5 and the project videos show examples. The paper gives no physical-trial count or aggregate success rate, so this is qualitative transfer evidence.
6 Limitations and Conclusion
The demonstrated scope is two objects and two poses connected by interpolation. Multi-object piles, longer paths, temporal constraints, and loops are not modeled. The system relies on externally supplied part segmentations and symbolic correspondences; more composed models can also make sampling expensive.
CPM provides a useful structural bias for several cross-category transfers, with explicit exceptions in the numerical tables. Learning functional parts jointly, using noisier human demonstrations, and extending trajectory and task complexity remain future work.
Acknowledgments
The authors thank NVIDIA Seattle Robotics Lab, Georgia Tech RAIL, and Stanford Vision and Learning Lab. Support includes NSF, AFOSR, ONR MURI, Stanford HAI, MIT–IBM Watson Lab, MIT Quest for Intelligence, CBMM, Analog Devices, JPMC, and Salesforce; the stated views are the authors’ own.
Appendix A Network Architecture
Each primitive model encodes two 512-point part clouds with 200-dimensional Point Cloud Transformer features. A pose MLP also outputs 200 dimensions, learned position embeddings have 16 dimensions, and sinusoidal time embeddings have 40. The original token-construction formulas are retained.
The diffusion transformer uses four layers, four attention heads, hidden dimension 128, and no dropout. Table A1 lists 200 diffusion steps with a linear noise schedule from 0.0001 to 0.02, 2,000 training epochs, Adam at 10⁻⁴, and gradient clipping at 1.0. Its stated Huber loss differs from the main MSE objective.
Equation
| Parameter | Value |
| Number of and points | 512 |
| PCT point cloud encoder out dim | 200 |
| Position embedding | learned embedding |
| Position embedding dim | 16 |
| Time embedding | Sinusoidal |
| Time embedding dim | 40 |
| Pose encoder out dim | 200 |
| Transformer number of layers | 4 |
| Transformer number of heads | 4 |
| Transformer hidden dim | 128 |
| Transformer dropout | 0.0 |
| Diffusion steps | 200 |
| Diffusion noise schedule | Linear |
| Start value | 0.0001 |
| End value | 0.02 |
| Loss | Huber |
| Epochs | 2000 |
| Optimizer | Adam |
| Learning rate | 1e-4 |
| Gradient clip value | 1.0 |
Appendix B Implementation Details for Baselines
Transformer-BC independently encodes each full object cloud and autoregressively decodes the two poses. TAX-Pose follows the official implementation without pretrained object embeddings; separate models predict the start and end poses because each TAX-Pose model outputs only one relative pose.
PC-DDPM shares the primitive-model architecture but conditions on whole objects. The part-aware variant adds binary ground-truth segmentation channels for all known parts to each point. CPM instead uses those masks to extract the relevant part clouds for separate correspondence models, making the comparison about both representation and composition.
Appendix C Dataset Details
The thirteen categories are mug, pan, bowl, wine glass, knife, can opener, scissors, screwdriver, fork, spoon, marker, pen, and flashlight. Categories can serve different roles across tasks, such as a mug being moved for pouring or serving as a safe-placement container.
The dataset runs 1,000 simulations for each of nine safe-placement categories and 2,000 for each of four pouring categories. It retains only the 2,836 and 4,522 successful trajectories, respectively. Geometry-derived segmentations are mapped onto rendered partial point clouds using the known transformations and scales. Table A2 retains category-specific counts.
| Task | Object | Source | Number of Simulations | Number of Success Demonstrations |
| Safe Placing | Pen | PartNet | 1000 | 568 |
| Fork | PartNet | 1000 | 390 | |
| ScrewDriver | PartNet | 1000 | 145 | |
| Spoon | PartNet | 1000 | 410 | |
| Knife | Acronym | 1000 | 496 | |
| Scissors | PartNet | 1000 | 354 | |
| Flashlight | PartNet | 1000 | 141 | |
| CanOpener | PartNet | 1000 | 101 | |
| Marker | PartNet | 1000 | 231 | |
| Pouring | Mug | PartNet | 2000 | 1051 |
| WineGlass | PartNet | 2000 | 1542 | |
| Bowl | Acronym | 2000 | 776 | |
| Pan | PartNet | 2000 | 1153 |
Appendix D Evaluation Details
Test states are drawn from held-out successful demonstrations, ensuring that the chosen object pair has a known feasible action. Each model is sampled five times per state to probe multimodal behavior; three seeds yield the 1,500-trial total.
Pouring receives zero if the object motion interpenetrates the container; otherwise its completion score is the percentage of particles transferred. Safe placing first rejects initial interpenetration and an incorrect orientation, such as an exposed blade. If valid, it scores the percentage of the proposed path executed until contact. These task-specific scores differ from end-to-end robot success.
Appendix E Additional Results
Binary pouring success requires no interpenetration and at least 70% of particles transferred. Safe-placing success additionally requires correct orientation and at least 70% of the proposed path executed without object collision. Tables A3–A4 retain the full success-rate results and dispersion.
For the seen-category setting, CPM achieves 76.87% pouring success and 68.87% safe-placing success; TAX-Pose reaches 74.00% on safe placing. The withheld-category exceptions remain: Part-Aware PC-DDPM performs better on pans, and TAX-Pose on pens. These rates should not be substituted for the partial-credit scores in the main tables.
| Model | Pouring | Safe Placing |
| Transformer-BC | 17.533.13 | 33.272.14 |
| TAX-Pose | 21.330.58 | 74.001.00 |
| PC-DDPM | 70.671.27 | 48.732.97 |
| Part-Aware PC-DDPM | 73.602.60 | 36.532.20 |
| CPM (ours) | 76.871.70 | 68.872.25 |
| Model | Pouring | Safe Placing | |||||
| Bowl | Glass | Mug | Pan | Fork | Pen | Scissors | |
| Transformer-BC | 10.002.51 | 19.200.92 | 5.801.93 | 29.332.04 | 24.001.51 | 27.332.34 | 18.472.93 |
| TAX-Pose | 21.001.00 | 3.001.00 | 8.001.00 | 42.672.08 | 47.673.21 | 62.674.04 | 33.331.15 |
| PC-DDPM | 56.532.00 | 70.673.06 | 68.674.31 | 59.932.80 | 38.003.83 | 43.471.68 | 28.470.83 |
| Part-Aware PC-DDPM | 54.872.10 | 68.332.97 | 65.204.61 | 62.003.56 | 28.671.68 | 42.403.12 | 17.672.70 |
| CPM (ours) | 76.401.78 | 78.933.14 | 76.005.26 | 54.671.50 | 53.932.91 | 56.532.04 | 62.071.72 |
Appendix F Assumptions
Training assumes a symbolic list of part correspondences, successful trajectories, and part segmentations. Testing still requires part segmentations. Humans supply the relation names without continuous constraint formulas; the data grounds those relations geometrically.
Programs used to generate the demonstrations are not required as training labels. Nevertheless, the evaluated data is procedurally generated and success-filtered. Extracting constraints with language models, learning from noisy demonstrations, and discovering functional parts from manipulation data are proposed extensions rather than capabilities established here.