Literature
Display
← All papers

Compositional Diffusion-Based Continuous Constraint Solvers

Zhutian Yang · Jiayuan Mao · Yilun Du · Jiajun Wu · Joshua B. Tenenbaum · Tomás Lozano-Pérez · Leslie Pack Kaelbling

· CoRL 2023 ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

The complete supplementary material covers data construction, implementation, failure cases, sampler comparisons, task-and-motion planning, and learned fields. Project materials are linked from the original paper.

At a glance

Diffusion-CCSP learns reusable diffusion models for individual continuous constraint types and composes their predictions on a factor graph. Shared variables receive contributions from all relevant constraints, allowing a single sampling process to seek placements satisfying containment, collision avoidance, stability, and qualitative spatial relations.

Four simulated domains test new constraint combinations and object counts. An outer task-and-motion planner selects discrete action structures; diffusion supplies continuous pose proposals that external checkers verify. The method improves several difficult compositional cases, but success is empirical and finite-budget sampling neither certifies feasibility nor proves infeasibility.

1 Introduction

Robot placement choices interact: a pose can fit inside a box yet block the gripper or make the stack unstable. Solving each constraint sequentially can require repeated rejection and backtracking. Joint optimization addresses these dependencies, but analytic differentiable formulas are difficult to obtain for some qualitative relations.

The proposed alternative learns local constraint models and composes them at inference. Conditioning variables such as geometry stay fixed, while shared decision variables are denoised together. Reuse is over known constraint types and compatible representations; larger or differently connected graphs need not match the training graphs.

Refer to caption
Figure 1: Solving Continuous Constraint Satisfaction Problems by Composing Diffusion Models. Our approach combines diffusion models, representing individual constraints, to generate object placement poses. The choice of an object’s placement depends on both qualitative constraints about object placement and collision avoidance constraints on the object and the robot gripper.

2 Related Work

Scene-level methods such as StructDiffusion [14] generate arrangements under a global condition. Compositional energy models, including Gkanatsios et al. [12], combine learned goals. Diffusion-CCSP uses an explicit constraint graph to combine local geometric, physical, and qualitative relations. Specialized packing systems solve narrower combinations, so the paper does not treat them as equivalent general-purpose baselines.

3 Compositional Diffusion Constraint Solvers

A continuous constraint satisfaction problem separates unknown assignments from fixed conditioning data and specifies which constraint types connect them. Qualitative relations such as “close to” are intended to be learnable from context, although the experiments generate their labels with hand-written rules.

3.1 Formulation and Representation of Constraint Satisfaction Problems

The graph contains decision variables V, conditioning variables U, and typed constraint instances C. Each instance names the variables it consumes. A valid assignment must pass every corresponding test, which may be a rule, classifier, or simulator. “Collision-free” here means nonpenetration.

For a bowl placement, coupled decisions include grasp, target pose, and trajectory. Constraints express containment, proximity to another bowl, collision avoidance, and a valid motion between initial and final poses. This is the general formulation; the robot experiment uses learned pose proposals alongside separate grasp enumeration and motion planning.

Refer to caption
Figure 2: Continuous Constraint Satisfaction Problem (CCSP) in Robot Planning. It unifies geometric, physical, and qualitative constraints. To place A into the tray, we need to generate the grasping pose graspA\textit{grasp}_{A}, placement pose poseA\textit{pose}_{A}, and the robot arm trajectory. We omit collision-free constraints with robots in (b) for brevity.
Equations𝒢=⟨𝒱,𝒰,𝒞⟩{\mathcal{G}}=\langle{\mathcal{V}},{\mathcal{U}},{\mathcal{C}}\rangle⟨cfree,(poseA,poseB,geomA,geomB)⟩\langle\text{cfree},(\textit{pose}_{\text{A}},\textit{pose}_{\text{B}},\textit{geom}_{\text{A}},\textit{geom}_{\text{B}})\rangletest​(tc,(Vc,Uc))=1\textit{test}(t_{c},(V^{c},U^{c}))=1

3.2 Compositional Diffusion Models

An ideal component distribution is uniform over the bounded feasible set for one constraint. Multiplying the components corresponds to summing negative-log energies. Annealed unadjusted Langevin sampling combines noisy updates from the learned scores, seeking assignments compatible with all factors. Approximate learned scores and finite sampling do not guarantee an exact product distribution or a valid solution.

Training corrupts a solved graph assignment with Gaussian noise and minimizes the error between that noise and the combined prediction of instantiated constraint modules. Although separate component training is possible, the experiments train the modules jointly on compositional examples. The retained objective and Algorithms 1–2 give the published equations; Algorithm 2 describes the reverse-diffusion update, while the full ULA variant adds the sampling procedure described in Appendix B.

Equationspc​(𝒱c=Vc∣𝒰c=Uc)∝𝟙⁡[test​(tc,(Vc,Uc))]p_{c}({\mathcal{V}}^{c}=V^{c}\mid{\mathcal{U}}^{c}=U^{c})\propto\mathbb{1}[\textit{test}(t_{c},(V^{c},U^{c}))]p⁡(𝒱c∣𝒰c)∝e−E⁡(𝒱c∣𝒰c)p({\mathcal{V}}^{c}\mid{\mathcal{U}}^{c})\propto e^{-E({\mathcal{V}}^{c}\mid{\mathcal{U}}^{c})}
V0=arg​maxV∏c∈𝒞pc(𝒱c∣𝒰c)=arg​minV∑c∈𝒞E(𝒱c∣𝒰c).V_{0}=\argmax_{V}\prod_{c\in{\mathcal{C}}}p_{c}({\mathcal{V}}^{c}\mid{\mathcal{U}}^{c})=\argmin_{V}\sum_{c\in{\mathcal{C}}}E({\mathcal{V}}^{c}\mid{\mathcal{U}}^{c})\;\;.
ℒMSE=𝔼⟨𝒢=(𝒱,𝒰,𝒞),U,V0⟩∈𝒟ϵ∼𝒩⁡(𝟎,𝐈)t∼Uniform​(1,⋯,T)[‖ϵ−∑c∈𝒞ϵtc​(α¯t​V0c+1−α¯t​ϵ,Uc,t)‖2],\mathcal{L}_{\text{MSE}}=\mathop{\mathbb{E}}\limits_{\begin{subarray}{c}\langle{\mathcal{G}}=({\mathcal{V}},{\mathcal{U}},{\mathcal{C}}),U,V_{0}\rangle\in{\mathcal{D}}\\ {\bm{\epsilon}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ t\sim\text{Uniform}({1,\cdots,T})\end{subarray}}\left[\Big\|\mathbf{\epsilon}-\sum_{c\in{\mathcal{C}}}\epsilon_{t_{c}}\left(\sqrt{\bar{\alpha}_{t}}V^{c}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon},U^{c},t\right)\Big\|^{2}\right],
Algorithm 1 Training
1: repeat
2:  𝒢,V0∼𝒟{\mathcal{G}},V_{0}\sim{\mathcal{D}}
3:  t∼Uniform⁡({1,…,T})t\sim\mathrm{Uniform}(\{1,\dotsc,T\}); ϵ∼𝒩⁡(𝟎,𝐈){\bm{\epsilon}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
4:  Take gradient descent step on
5:    ∇θ∥ϵ−∑c∈𝒞ϵtc(α¯tV0c+1−α¯tϵ,t))∥2\nabla_{\theta}\Big\|\mathbf{\epsilon}-\sum\limits_{c\in{\mathcal{C}}}\epsilon_{t_{c}}(\sqrt{\bar{\alpha}_{t}}V_{0}^{c}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon},t))\Big\|^{2}
6: until converged
Algorithm 2 Sampling
1: VT∼𝒩⁡(𝟎,𝐈)V_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
2: for t=T,…,1t=T,\dotsc,1 do
3:  if t>1t>1 then 𝐳∼𝒩⁡(𝟎,𝐈)\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) else 𝐳=𝟎\mathbf{z}=\mathbf{0}  
4:  Vt−1=1αt​(Vt−1−αt1−α¯t​∑c∈𝒞ϵtc​(Vtc,Uc,t))+σt​𝐳V_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\Big(V_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\sum\limits_{c\in{\mathcal{C}}}\epsilon_{t_{c}}\left(V^{c}_{t},U^{c},t\right)\Big)+\sigma_{t}\mathbf{z}
5: return V0V_{0}
Refer to caption
Figure 3: Illustration of the denoising neural network for satisfying constraint cfree (collision-free) and the gradient map learned by Diffusion-CCSP for the centroid of B at (x,y)(x,y) when A is at (0,0)(0,0).

3.3 Geometric and Physical Constraint Solving with Diffusion-CCSP

The illustrated network encodes geometry and pose, concatenates the arguments of each constraint with a diffusion-time embedding, and predicts noise for the participating poses. Geometry encoders are shared; pose encoders and interaction modules are constraint-specific. Fixed variables can be conditioned on by discarding their predicted updates.

The general example uses axis-aligned bounding boxes and three-dimensional translation plus yaw, encoded by sine and cosine. The planar tasks use their own compact shape and pose encodings. Point-cloud inputs and arbitrary SE(3) poses are proposed extensions. Figure 3 illustrates a collision-avoidance field, and Appendix F retains all qualitative-field visualizations.

The solver assumes a constraint graph is supplied. Appendix E explains how a symbolic action skeleton generates that graph and how the outer planner retries when sampled continuous assignments fail validation.

Equation

ϵcfreet​(geomA,geomB,poseA,poseB)=gcfree​(fgeom​(geomA)⊕fgeom​(geomB)⊕fpose​(poseA)⊕fpose​(poseB)⊕t),{\epsilon}_{\text{cfree}}^{t}(\textit{geom}_{\text{A}},\textit{geom}_{\text{B}},\textit{pose}_{\text{A}},\textit{pose}_{\text{B}})=g_{\text{cfree}}\left(f_{\text{geom}}(\textit{geom}_{\text{A}})\oplus f_{\text{geom}}(\textit{geom}_{\text{B}})\oplus f_{\text{pose}}(\textit{pose}_{\text{A}})\oplus f_{\text{pose}}(\textit{pose}_{\text{B}})\oplus t\right),

4 Experiments

The four tasks compare ULA sampling with direct reverse diffusion, sequential rejection sampling, and a task-adapted StructDiffusion baseline where applicable. Sequential sampling tries at most 50 poses per object. StructDiffusion is tested on triangle packing and robot packing, using a single scene-level constraint. Main success curves count problems solved within ten samples and show variation across five training seeds.

Refer to caption
Figure 4: Illustration of four domains. They contain geometric, physical, and qualitative constraints. (a) Triangle packing. (b) Dense 2D packing with qualitative constraints. The figure shows a subset of 45 constraints of 13 types. (c) 3D object stacking. The arrows show the support relationships. (d) 3D object packing with a panda robot.
Refer to caption
Figure 5: Quantitative Comparisons of Constraint Solvers. Accumulated number of problems solved in 10 runs of different models. OOD=Out of training distribution. The shaded area indicates the standard deviation of various models across five different seeds. The sequential sampling baseline completely failed for task (c) and hard tasks in (b) and (d). Our full model performs better than a variant without ULA sampling and better than StructDiffusion in more complex problems (d).

4.1 2D Triangle Packing

Training uses 30,000 solutions with two to four rotatable triangles; tests include five-triangle problems. Random tray triangulations generate collision-free examples. Rejection sampling becomes weak as packing gets dense. StructDiffusion performs comparably in distribution and generalizes better than Diffusion-CCSP to five triangles, so the compositional model does not dominate every task.

4.2 2D Shape Arrangement with Qualitative Constraints

Rectangles must satisfy containment, noncollision, and selected relations such as relative position, proximity, and alignment. Training uses 30,000 two-to-four-object examples. The plotted quantitative test extends to five objects; the illustrative six-object scene contains 45 constraints of 13 types. The text also discusses six-object rejection failure, but that example should not be conflated with the plotted test range.

Diffusion-CCSP outperforms the sequential baseline on these combinations. Success still declines sharply on the five-object distribution shift, and narrow or disconnected feasible regions remain difficult.

Refer to caption
Figure 6: Visualization of learned energy functions of selected geometric and qualitative constraints.

4.3 3D Object Stacking with Stability Constraints

The stacking task places prism-like objects on a shelf under support and stability constraints. Every example contains a bridge supported by multiple objects, and both intermediate placements and the final stack must remain stable. Training uses 24,000 five-to-seven-object examples; evaluation also includes four-object graphs.

An outer planner samples placement orders and support structures before calling the continuous solver. A failed call may reflect an infeasible structure or unsuccessful sampling. Table 1(a) reports 11.9 calls for four objects and 33.33 for five; the narrative’s separate six-object example is not a tabulated measurement. Batched throughput figures are reported separately from complete planning latency.

Objects N=4 N=5
# of Calls to Diffusion-CCSP 11.9 33.33
(a) 3D Object Stacking.
Objects N=2 N=3 N=4 N=5 N=6
# of Calls to Diffusion-CCSP 1.33 1.97 2.96 4.37 7.26
(b) Task 3: 3D Object Packing with Robots.
Table 1: Average number of calls to Diffusion-CCSP in order to solve the full TAMP problem.

4.4 Task 4: 3D Object Packing with Robot Trajectory Constraints

A simulated Franka Panda packs 53 assets from PartNet Mobility and ShapeNet, including 15 assets restricted to side grasps. Training has 10,000 examples with three to five objects, and testing extends to six. Candidate grasps are generated beforehand; an external motion planner checks trajectories.

Diffusion predicts placements likely to admit feasible grasps and motions; it does not directly synthesize arm trajectories. Within ten samples, the six-object result is approximately 60–80% across the two diffusion variants, exceeding the listed rejection and StructDiffusion baselines. The source accidentally calls these “six-triangle” cases; Figure 5(d) is the robot-packing experiment. Table 1(b) reports the full planner’s increasing solver-call counts.

5 Limitation and Conclusion

The demonstrated reuse is over fixed-arity known constraints, fully observed states, compact geometry encodings, and simulated rule-labeled examples. Variable-arity factors, natural-language conditioning, richer shapes, and human-derived qualitative labels remain future work. The system also depends on an outer symbolic planner for action order.

The contribution is a modular learned proposal mechanism for continuous assignments that can be inserted into a planning system. It offers useful empirical generalization, while external validity checks and repeated samples remain essential. The authors acknowledge government, university, and industry research support.

Appendix A Problem Domains and Data Generation

The datasets balance examples across their included object counts. Construction starts from satisfiable arrangements, then derives constraints or validates action sequences, which shapes the distribution the models learn.

A.1 2D Triangle Packing

Bowyer–Watson triangulation splits sampled tray points into triangles, which are slightly shrunk to leave clearance. A canonical triangle places one vertex at the origin and its longest incident edge on the positive x-axis. Geometry uses that edge length and the other vertex vector; pose uses translation and sine/cosine rotation.

Refer to caption
Figure 7: Example collision-free configurations of triangles

A.2 2D Shape Arrangement with Qualitative Constraints

Recursive tray subdivision to depth three generates rectangles, with random padding and rejection of tiny regions. Hand-written rules label eleven qualitative relations; for example, “close to” uses a distance threshold tied to object widths. The full list and sample arrangements remain visible in the figures.

Rectangles use width and length in a horizontal resting pose. Decision variables are centroid position and sine/cosine yaw; some constraint combinations require vertical orientation.

Refer to caption
Figure 8: Example rearrangements of rectangles with 11 types of qualitative constraints.

A.3 3D Object Stacking with Stability Constraints

A vertical region is split, shrunk, and cut into shapes that settle in PyBullet. Configurations are retained only if stable, and removing objects one at a time checks stability of intermediate stages. The upper shelf allows only slight clearance. The reported encoding uses width, length, centroid coordinates, and sine/cosine roll, restricting the pose search despite the three-dimensional simulation.

Refer to caption
Figure 9: Example stable configurations of rectangles, with in, cfree, and supported-by constraints. There is at least one object that’s supported by multiple objects.

A.4 3D Robot Packing with Robots

The 53 assets span dispensers, bowls, flat staplers, eyeglasses, pliers, scissors, cameras, bottles, opened bottles, and mugs. Prespecified grasps approach one of five object faces. Random tray subdivisions determine object scale and candidate placement regions; a planner checks grasp combinations, collision-free approach poses, and arm paths for a sampled placement order.

Geometry is an object bounding box, grasp is a five-entry face indicator, and placement pose contains tray-frame translation plus sine/cosine yaw. This finite grasp representation is narrower than arbitrary continuous grasp generation.

Refer to caption
Figure 10: Example configuration of 3D shapes that enables the robot place each object in a given sequence without colliding into other objects already placed. All the bottles, dispensers, and bowls affords only side grasps.

A.5 Discussion: Sample Complexity of Diffusion-CCSP

Each task needs approximately 10,000–30,000 solved examples. Simulation supplies geometric and physical examples, while realistic qualitative relations would require costly human annotations or another data source. The authors argue that geometric learning should transfer readily, but this paper does not establish sim-to-real performance with physical robot trials.

A.6 Discussion: Partial Observability

All experiments assume known shapes and initial poses. Completing meshes from partial point clouds, detecting collisions during execution, and replanning are suggested ways to relax that assumption; no partial-observation experiment is reported.

Appendix B Implementation and Training Details

Geometry, pose, grasp, and time embeddings have width 256, with inputs normalized relative to the container. A pose decoder maps combined module outputs to predicted noise. The implementation description averages contributions for each object, whereas the main mathematical objective writes a sum; both original descriptions are retained without assuming they are numerically identical.

Most models use 1,000 diffusion timesteps; Diffusion-CCSP uses 500 for robot packing. ULA uses ten sampling steps with step size twice the cosine-scheduled beta. Multiple random seeds are batched for throughput.

The StructDiffusion adaptation concatenates geometry and pose into 512-dimensional tokens, or 768 dimensions with grasp features. It uses four residual attention blocks with two heads and normalization. Language embeddings are omitted because these tasks provide no language input. The rejection baseline samples each object at most 50 times.

B.1 Discussion: Individual Constraint Diffusion Training and Finetuning

Joint training fits the constraint modules through examples containing several simultaneous factors. Independent training and later composition are supported by the formulation but are not the main experimental protocol. Finetuning for new shapes is a proposed use.

B.2 Discussion: Weights for Energy Function Composition

The authors use unit weights for all constraint types. They attribute the workable composition to steep learned fields and joint training on combined constraints. This is an empirical choice, not a guarantee that unit weighting works for arbitrary factors; the separate per-object averaging detail also matters.

B.3 Discussion: Local Optima in Optimization

Parallel initial-noise seeds reduce sensitivity to bad local optima but do not remove it. The cited five-object stacking case averages 33.3 calls. Simulator checks validate collision and stability; learned checkers are suggested when a direct test is unavailable. Repeated invalid samples do not certify an empty feasible set.

Appendix C Failure Case Analysis

The failure analysis runs 4,000 solver calls on 400 two-to-four-shape qualitative problems. Narrow feasible regions, rare training constraints, and disconnected valid sets correlate with more failures. A separate two-object rejection estimate measures feasible area; it does not characterize multi-object collision-free volume, so collision avoidance is excluded from that particular comparison.

Figures 11–14 retain both failed and successful samples, including triangle overlap, missing qualitative relations, penetration-induced simulator instability, falling stacks, and incorrect support structures. Slight penetration is tolerated by the simulator, as the original figure notes.

Refer to caption
Figure 11: Failure and success samples for Task 2: 2D Shape Rearrangement with Qualitative Constraints. All solutions are generated by Diffusion-CCSP (ULA), trained on 30k data up to 4 objects.
Original figure
(a) Failure rate of qualitative constraints. The failure rate of each constraint is annotated on the bars. The red bars denote the number of failures, while the green bars denote success. Constraints are sorted by failure rate.
Original figure
(b) Estimated area occupied by qualitative constraints. The order is highly similar to the order in (a), supporting the hypothesis that the smaller the defining region of a constraint, the harder it is to satisfy.
Refer to caption
(c) Training data distribution for each constraint, in the order of increasing area of the defining region. Each point is the centroid for an object A that appeared in one constraint​(A,B)\textit{constraint}(\text{A},\text{B}) inside the training dataset.
Figure 12: Analysis of how failure rate of qualitative constraints correlates with area of its defining region.
Refer to caption
Figure 13: Failure and success samples for Task 1: 2D Triangle Packing. All solutions are generated by Diffusion-CCSP (ULA), trained on 30k data up to 4 objects.
Refer to caption
Figure 14: Failure and success samples for Task 3: 3D Object Stacking with Stability Constraints. All solutions are generated by Diffusion-CCSP (ULA), trained on 24k data up to 7 objects.

Appendix D Additional Sampler Comparison

The expanded sampler comparison allows up to 100 samples per problem. Figure 15 includes an explicit unsolved bin, preventing failed cases from disappearing into a mean. Table 2 reports amortized per-problem time in batches of 100: roughly 0.01–0.04 s for reverse diffusion and 0.06–0.19 s for ULA. These are batched sampling costs, not complete robot-plan runtimes.

Original figure
(a) Task 1: 2D Triangle Packing
Original figure
(b) Task 2: 2D Shape Arrangement with Qualitative Constraints
Original figure
(c) Task 3: 3D Object Stacking with Stability Constraints
Original figure
(d) Task 4: 3D Object Packing with Robots
Figure 15: Number of Diffusion-CCSP runs it takes to solve 100 CCSPs. OOD means problems that are out of training distribution (i.e. include more objects than trained on). On the xx-axis, tick x means problems not solved within 100 samples.
2 Objects 3 Objects 4 Objects 5 Objects
Diffusion-CCSP (ULA) 0.10 0.10 0.11 0.12
Diffusion-CCSP (Reverse) 0.01 0.01 0.01 0.01
StructDiffusion 0.08 0.08 0.08 0.08
(a) Task 1. 2D Triangle Packing
2 Objects 3 Objects 4 Objects 5 Objects
Diffusion-CCSP (ULA) 0.17 0.18 0.19 0.19
Diffusion-CCSP (Reverse) 0.04 0.04 0.04 0.04
(b) Task 2. 2D Shape Arrangement with Qualitative Constraints
4 Objects 5 Objects 6 Objects 7 Objects
Diffusion-CCSP (ULA) 0.15 0.16 0.17 0.19
Diffusion-CCSP (Reverse) 0.01 0.01 0.01 0.02
(c) Task 3: 3D Object Stacking with Stability Constraints
3 Objects 4 Objects 5 Objects 6 Objects
Diffusion-CCSP (ULA) 0.06 0.06 0.07 0.07
Diffusion-CCSP (Reverse) 0.01 0.01 0.01 0.01
StructDiffusion 0.09 0.09 0.08 0.08
(d) Task 4: 3D Object Packing with Robots
Table 2: Average run time (sec) of models in a batch of 100 CCSPs.

Appendix E Integration with Task and Motion Planning Algorithms

A task-and-motion planning problem specifies initial state, goals, parameterized actions, and a transition model. Action preconditions constrain grasps, placements, and trajectories; effects update the object state. The retained formal semantics and Figures 16–17 show the pick-and-place domain and a two-action constraint graph.

Following the bi-level approach discussed in the TAMP survey [36] and PDDLStream [4], the outer search chooses discrete objects and action order, then a continuous solver proposes the remaining values. A validated assignment instantiates the skeleton; failure prompts another sample or skeleton. The examples use a simplified planner, not a demonstration of solving arbitrary TAMP domains.

Equations⟨𝒮,s0,𝒢,𝒜,𝒯⟩\langle{\mathcal{S}},s_{0},{\mathcal{G}},{\mathcal{A}},{\mathcal{T}}\rangle𝒯:𝒮×𝒜→𝒮{\mathcal{T}}:{\mathcal{S}}\times{\mathcal{A}}\rightarrow{\mathcal{S}}∀s∈𝒮.∀a∈𝒜.prea​(s)⇒(𝒯⁡(s,a)=effa​(s))\forall s\in{\mathcal{S}}.\forall a\in{\mathcal{A}}.\textit{pre}_{a}(s)\Rightarrow\left({\mathcal{T}}(s,a)=\textit{eff}_{a}(s)\right)
Refer to caption
Figure 16: Illustration of a simple task and motion planning problem. The domain contains only one action: pick-and-place of objects. It contains three preconditions: g should be a valid grasp pose on object xx, t should be a valid robot trajectory that moves xx from its current pose to the target pose pp, and during the movement, the robot and the object xx should not collide with any other objects yy. If successful, the object will be moved to the new location. The goal of the task is to pack both objects into the target box without any collisions.
Original figure
Figure 17: Example of a partially specified plan skeleton in task and motion planning, together with the corresponding set of constraints. pA​0p_{A0}, pB​0p_{B0}, and pC​0p_{C0} corresponds to the initial pose of objects.

Appendix F Learned Gradient Fields

Figures 18–19 show all thirteen geometric and qualitative fields. Unary visualizations move a 0.15 × 0.15 square inside a unit container; binary visualizations fix a second square at the origin. Columns show an energy proxy, update field, sampled pose, and training distribution. The training plots contain different normalized shapes, so they are not identical slices of the visualized test geometry.

The paper constructs its energy visualization from the difference between input noise and reconstructed output at the final diffusion step. These qualitative plots illustrate low-value feasible regions and update directions; they do not independently establish exact calibrated energies or convergence guarantees.

Refer to caption
Figure 18: Visualization of energy and gradient functions of trained diffusion models for qualitative constraints (Part 1. Constraints between a shape and the container). Column 2: Energy fields inside a container of size 1 by 1. Column 3: Gradient functions that update poses in different time steps. Column 4: Poses sampled according to the given qualitative constraint. Column 5: Poses in the training data set with the given constraint, for 30,000 CCSPs that involve 2 - 4 objects that have different sizes. The number of constraints that appeared in the dataset is shown at the top right corner.
Refer to caption
Figure 19: Visualization of energy and gradient functions of trained diffusion models for qualitative constraints (Part 2. Constraints between two shapes).
Concept demonstration

Open image ↗