Literature
Display
← All papers

PDSketch: Integrated Domain Programming, Learning, and Planning

Jiayuan Mao · Tomás Lozano-Pérez · Joshua B. Tenenbaum · Leslie Pack Kaelbling

· NeurIPS 2022 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

The complete paper and appendices A–D are included, with all domain-language examples, derived heuristic rules, model variants, figures, tables, and equations.

At a glance

PDSketch lets a programmer specify which object properties an action depends on and changes, while neural networks learn unspecified functions. These structured transition models support planning for novel goals and provide the basis for automatically derived search heuristics. Experiments in navigation and simulated manipulation show benefits from useful locality and sparsity priors. The approach depends on supplied object structure, low-level controllers, and a suitable hand-written domain sketch.

1 Introduction

Physical actions usually affect only a few properties of a few nearby objects. PDSketch exposes these dependencies in a symbolic model while leaving perception and detailed update functions learnable. Training fits the model from trajectories; execution combines it with a planner to form a goal-conditioned policy. The central question is how much explicit structure improves data efficiency, compositional transfer, and search effort compared with less structured models.

Refer to caption
Figure 1: The life cycle of a PDSketch model. A PDSketch model is composed of a model structure definition and a collection of trainable neural modules. The model parameters can be learned from data. During performance time, the model is used by a domain-independent planner to form a policy that directly interacts with the environment.
Original figure
Figure 2: Defining both the transition model structure and implementation in Python (a) vs. defining only the structure while leaving details (the ?? functions) to be learned (b).

2 PDSketch

Training data contains states, actions, a logical goal, and per-state goal-success signals. The learned components are an object-state encoder, grounded goal predicates, and latent dynamics. External perception supplies object detection and tracking; object identities are fixed within an episode. Predicates may return Boolean scores or vectors. Action schemas combine parameters, controller preconditions, effects, and a supplied primitive policy. The intended dynamics are local and sparse: unmentioned factors remain unchanged. Although real execution can be noisy, planning uses a deterministic latent model.

Refer to caption
Figure 3: A factorized state representation and transition model for the robot painting domain. The raw observation (a) is first processed by an external perception module into an object-centric representation (b). This representation is further transformed into a fine-grained factorization (c). The transition function 𝒯θ{\mathcal{T}}_{\theta} can be defined over this factored state representation: each action may only change a few factors of the state. Executing a specific action move-into(item#1, item#6) produces the predicted factored state at the next timestep. During training time, we will be using the object-centric observation from the next timestep to supervise the learning of 𝒯θ{\mathcal{T}}_{\theta}.

2.1 Representation Language

PDSketch extends PDDL-style predicates and actions (Fox and Long, 2003) with vector values and named blanks, written ??, that are filled by learned functions, adapting Solar-Lezama (2008)’s program-sketch idea. The programmer defines dependencies and parameter sharing; neural modules learn their numerical meaning. Conditional effects use soft gates so gradients can flow. For example, moving an object changes its pose and may change wetness depending on the container, while leaving unrelated properties untouched. A minimal sketch can fall back to a largely unstructured object-centric dynamics model.

Refer to caption
Figure 4: A minimal example of defining derived features and predicates with blanks “??”.

2.2 Model Learning and Planning with PDSketch

Learning combines goal-truth classification with latent next-state prediction; the paper also describes a lookahead term to discourage degenerate representations. Logical operators use soft truth values, with min/max reductions for conjunction/disjunction and finite quantification. Planning searches the learned latent model and accepts states whose learned goal score exceeds 0.5. Success is therefore conditional on model, perception, goal-grounding, and controller accuracy rather than guaranteed by symbolic syntax alone.

Equation
ℒ⁡(θ)=∑⟨s¯,a¯,g,succ¯⟩∈𝒟{∑iBCE​(evalθ​(g,ℰθ​(si)),succi)+∑iL1​(𝒯θ​(ℰθ​(si),ai),ℰθ​(si+1))},\small\mathcal{L}(\theta)=\sum_{\langle\overline{s},\overline{a},g,\overline{\textit{succ}}\rangle\in{\mathcal{D}}}\left\{\sum_{i}\text{BCE}\left(\textit{eval}_{\theta}(g,{\mathcal{E}}_{\theta}(s_{i})),\textit{succ}_{i}\right)+\sum_{i}\text{L1}\left({\mathcal{T}}_{\theta}({\mathcal{E}}_{\theta}(s_{i}),a_{i}),{\mathcal{E}}_{\theta}(s_{i+1})\right)\right\},
Refer to caption
Figure 5: A computation graph of a partial definition of the action “move-into.”

2.3 Inducing Domain-Independent Heuristics

Search guidance comes from a relaxed planning problem that allows a variable to hold several values simultaneously. A Fast Forward-style heuristic estimates a useful action set under that relaxation. Continuous features can be discretized and neural functions approximated by first-order rules. This avoids training a separate task-specific value heuristic, but the detailed compilation still performs quantizer fitting/fine-tuning and rule learning. Approximate heuristics guide search; the original learned transition model remains responsible for simulated action effects.

3 Experiments

The evaluation compares structure levels within PDSketch and against behavior cloning and Decision Transformer baselines. Input encoders and available prior knowledge are specified in the appendix.

Refer to caption
Figure 6: A screenshot of the BabyAI environment.
Model Inductive Biases Succ. Rate
Obj-Centric.
Facing
Rob. Dyn.
Obj. Prop.
Basic
#Obj.Gen.
BC Y N N N 0.93 0.79
DT (S) Y N N N 0.91 0.82
DT (S+F) Y N N N 0.32 0.19
DreamerV2 N N N N 0.96 0.79
PDS-Base Y N N N 0.82 0.62
PDS-Abs Y Y N N 0.99 0.98
PDS-Rob Y Y Y N 1.00 1.00
Table 1: The planning success rate of different models on BabyAI.
Original figure
Figure 7: (a) and (b) Data efficiency comparison for model learning. We compare three structures with different levels of abstractness. (c) Planning efficiency, measured as the number of expanded nodes for different heuristic computation methods.

3.1 BabyAI

BabyAI uses a 7×7 grid with navigation, pickup, and door goals. Training has four doors and four other objects; generalization increases those counts to six and eight. PDS-Base keeps holistic object features, PDS-Abs separates pose from appearance and introduces a learned facing relation, and PDS-Rob supplies movement rules. More structure improves learning and transfer: the main table reports 0.98 generalized success for PDS-Abs and 1.00 for PDS-Rob, versus 0.62 for PDS-Base. Derived heuristics reach the reported 0.8-success operating point with about one-fifth the node expansions of the learned graph heuristic. This is a search-node comparison under the tested domain, not a universal runtime guarantee.

Original figure
Figure 8: The PDSketch definition of the “forward” action and the corresponding computation graph for the PDS-Abs model. The mentioned predicate facing is an derived predicate represented as a neural network that is jointly learned (omitted in the graph for brevity).
Original figure
Figure 9: Our encoding of the painting factory domain. Each object has a pose and a state. Action move-to directly changes the object pose. move-into moves an object into a container, which possible changes the state (e.g., color) of the object. Goals are represented using relational predicates.
Model
Succ. Rate
@100
Succ Rate
@1000
BC 0.70 0.95
DT 0.56 0.97
PDS 0.91 0.99
Table 2: Model performance on Painting Factory, measured as plan success rate. The number after @ is the number of demonstration trajectories. PDSketch models show stronger data efficiency than baselines.
Refer to caption
Figure 10: From only one training task (left), a model specified in PDSketch learns primitive concepts including object colors and spatial relations. They can be used in planning for unseen tasks given new first-order logic descriptions of the goal. The natural language descriptions are shown for readability.

3.2 Robot Painting

Painting Factory is a simulated tabletop domain with blocks, colored bowls, and pick/place controllers. Training goals paint and arrange two blocks. Object inputs include positions and image crops; actions move into a container or to a position. PDSketch is more data-efficient than the compared policies and transfers to goals with more objects, new spatial arrangements, or universal quantification. The three displayed transfer tasks score 0.99, 0.98, and 0.87 success; stacking is less reliable because of controller and physics noise. Closed-loop recovery is left for future work.

4 Related Work

Related model-learning approaches range from pixel dynamics to fully symbolic transition rules. PDSketch occupies an intermediate position: object-centric input is supplied, finer factors and groundings are learned within a programmed dependency structure. It also connects goal-conditioned planning, object-oriented models, and differentiable symbolic computation. The distinguishing prior is explicit locality and sparsity rather than object-centricity alone.

5 Conclusions and Limitations

The experiments support structured model learning followed by planning for new goals. Current limits include dependence on programmer expertise, no automatic discovery of new factorizations, and no hierarchical domain representation. Supplied perception and controllers, deterministic model approximation, and finite tested task domains further constrain generalization claims. The authors acknowledge the listed research support.

Appendix A Additional BabyAI Results

The appendix separates final planning success from transition-prediction accuracy and tests an interactive data-collection variant.

A.1 Data Efficiency and Variance Analysis

Data-efficiency experiments use 100, 1,000, or 10,000 demonstrations, with means and standard deviations over three runs. Structure is especially helpful with little data. The strongest-prior PDS-Rob model reaches about 0.72 success from 100 episodes, while less structured models need substantially more data and may still mislearn blocked movement.

Model 100 episodes 1000 episodes 10000 episodes
BC 0.24 ±\pm 0.01 0.24 ±\pm 0.01 0.39 ±\pm 0.03
DT 0.23 ±\pm 0.02 0.24 ±\pm 0.01 0.30 ±\pm 0.05
PDS-Base 0.04 ±\pm 0.00 0.11 ±\pm 0.01 0.80 ±\pm 0.07
PDS-Abs 0.04 ±\pm 0.02 0.98 ±\pm 0.03 0.99 ±\pm 0.01
PDS-Rob 0.72 ±\pm 0.09 0.99 ±\pm 0.01 0.99 ±\pm 0.01
Table 3: The planning success rate for different models using different amount of training data.

A.2 On-Policy Learning

Interactive learning alternates planning with the current model, simulator execution, and model updates from the resulting success signals; it does not reuse past data in this implementation. PDS-Rob reaches 0.99 success within 10,000 episodes, compared with the cited PPO result requiring 200,000. Directly applying the same loop to PDS-Abs or PDS-Base stalls near 6% success because exploration rarely finds successful plans. The favorable sample count therefore relies on strong supplied dynamics knowledge.

Appendix B PDSketch Language

The appendix gives the complete Lisp-style domain language, distinguishing reusable types, predicates, and action schemas from episode-specific states and goals.

B.1 Predicate and Action Definition

Types distinguish world objects from scalar or vector values. Input predicates read observations; derived predicates compose other features and learned blanks. Blank names identify externally registered implementations, with shared parameters across object instances. The examples progressively define item encoders, color and type tests, holding and facing relations, logical goals, and turn/move/pickup actions. More detailed sketches constrain which features may change and which objects matter; the final unrestricted example lets every update depend on the whole state. Every grammar and code block is retained.

definition:      "(" "define" "domain" domain-name-def content-def* ")"
domain-name-def: "(" "domain" string ")"
content-def:     type-def | predicate-def | derived-def | action-def
(define domain
  (domain my-domain-name)
)
type-def:        "(" ":types" single-type-def ")"
single-type-def: type-name-list - base-type
type-name-list:  type-name+
type-name:       string
base-type:       type-name | prim-type | vector-type
prim-type:       "bool" | "int64" | "float32"
vector-type:     "vector" "[" prim-type (, int)? "]"
(:types
  robot item - object
  pose       - vector[float32, 2]
  direction  - vector[int64, 1]
)
predicate-def:        "(" ":predicates" single-predicate-def ")"
single-predicate-def: "(" predicate-name kwargs? variable-list ")"
predicate-name:       string
kwargs:               "[" kwarg "]"
kwarg:                kwarg-key "=" kwarg-value
kwarg-value:          "\"" string "\"" | int | bool | float | base-type
variable-list:        typed-variable*
typed-variable:       variable-name "-" type-name
variable-name:        string
(:predicates
  (robot-pose      [return_type=pose]            ?r - robot)
  (robot-direction [return_type=direction]       ?r - robot)
  (item-pose       [return_type=pose]            ?o - item)
  (item-image      [return_type=vector[float32]] ?o - item)
)
derived-def: "(" ":derived" single-predicate-def expr ")"
(:derived (item-feature [return_type=vector[float32, 64]] ?o - item)
  (??f (item-image ?o))
)
(:derived (is-red  ?o - item) (??f (item-feature ?o)))
(:derived (is-blue ?o - item) (??f (item-feature ?o)))
; ... more colors
(:derived (is-ball ?o - item) (??f (item-feature ?o)))
(:derived (is-door ?o - item) (??f (item-feature ?o)))
; ... more shapes
(:derived (is-open ?o - item) (??f (item-feature ?o)))  ; for doors
(:derived (robot-holding ?r - robot ?o - item)
  (??f (item-pose ?o))
)
(:derived (robot-facing ?r - robot ?o - item)
  (??f (robot-pose ?r) (item-pose ?o))
)
(exists (?o - item) (and
  (robot-facing agent ?o) (is-red ?o) (is-box ?o)
))
(exists (?o - item) (and
  (robot-holding agent ?o) (is-red ?o) (is-box ?o)
))
(exists (?o - item) (and
  (is-red ?o) (is-door ?o) (is-open ?o)
))
action-def: "(" ":action" (action-name kwargs)
  ":parameter" "(" variable-list ")"
  ":precondition" expr
  ":effect" expr
")"
(:action lturn
 :parameters (?r - robot)
 :precondition (and )
 :effect (assign (robot-direction ?r) (??f (robot-direction ?r)))
 ; could also be written as:
 ; :effect (robot-direction::assign ?r (??f (robot-direction ?r)))
)
(:action forward
 :parameters (?r - robot)
 :precondition (and )
 :effect (robot-pose::assign ?r (??f
   (robot-pose ?r)
   (robot-direction ?r)
   (foreach (?o - item)          ; enumerate all items in the map
     (when (robot-facing ?r ?o)  ; if the robot is facing this item
       (item-feature ?o)         ; consider the item-feature of ?o
     )
   )
 ))
)
; define a helper derived predicate.
(:derived (is-obstacle ?o - item) (??f (item-feature ?o)))
(:action forward-detail
 :parameters (?r - robot)
 :precondition (and )
 :effect (when
   ; when there is no item ?o such that
   ;   ?o is an obstacle and the robot is facing ?o
   (not (exists (?o - item) (and (is-obstacle ?o) (robot-facing ?r ?o)) ))
   ; the robot pose will move forward and the new pose
   ;   will be computed by the current pose and the facing direction.
   (assign (robot-pose ?r)
     (??f (robot-pose ?r) (robot-direction ?r))
   )
 )
)
; define a helper predicate indicating whether the object
;   can be picked up.
(:derived (can-pickup ?o) (??f (item-feature ?o)))
(:action pickup
 :parameters (?r - robot)
 :precondition (and )
 :effect (foreach (?o - item)
   (when (and (robot-facing ?r ?o) (can-pickup ?o) )
     (assign (item-pose ?o)
       ; a dummy function, that should be implemented to return
       ;   [-1, -1], indicating the item is in robot’s inventory.
       (??f )
     )
   )
 )
)
(:action mysterious-action
 :parameter (?r - robot)
 :precondition (and )
 :effect (and
   (robot-pose::assign ?r (??f1
     (robot-pose ?r) (robot-direction ?r) (item-pose ??) (item-feature ??)
   ))
   (robot-direction::assign ?r (??f2
     (robot-pose ?r) (robot-direction ?r) (item-pose ??) (item-feature ??)
   ))
   (foreach (?o - item) (item-pose::assign ?o (??f3
     (robot-pose ?r) (robot-direction ?r) (item-pose ??) (item-feature ??)
   )))
   (foreach (?o - item) (item-feature::assign ?o (??f4
     (robot-pose ?r) (robot-direction ?r) (item-pose ??) (item-feature ??)
   )))
 )
)

Remark: Preconditions vs . conditional effects.

A precondition states when a low-level controller is executable; a conditional effect states when execution changes a particular variable. The former needs controller-applicability feedback, while the latter is learned from observed outcomes. Primitive actions in many reinforcement-learning environments have empty preconditions even when their effects can be blocked.

(:predicates
  (p ?o - item)
  (q ?o - item)
)
(:action op1
 :parameters (?o - item)
 :precondition (p ?o)
 :effect (q:assign (??f (q ?o)))
)
(:action op2
 :parameters (?o - item)
 :precondition (and )
 :effect (when
   (p ?o)
   (q:assign (??f (q ?o)))
 )
)

B.2 Expressions

Propositional logic operations: and , or , not , and implies .

Boolean expressions implement negation, conjunction, disjunction, and implication using soft Gödel operations. In action effects, an and block groups updates and a bare Boolean predicate sets it true. The source’s sentence describing a negated Boolean effect repeats a true assignment; that appears inconsistent with its intended negation semantics, so the original code/text examples are retained without silently correcting them.

and-expr-def:     "(" "and"     expr* ")"
or-expr-def:      "(" "or"      expr* ")"
not-expr-def:     "(" "not"     expr  ")"
implies-expr-def: "(" "implies" expr expr ")"
(:action set-pqr
 :parameters (?r - robot)
 :precondition (and )
 :effect (and
   (assign (p ?r) (??f))
   (assign (q ?r) (??f))
   (assign (r ?r) (??f))
 )
)

Boolean quantification: forall and exists .

Forall and exists quantify over the finite objects of a declared type, using minimum and maximum soft truth values respectively.

forall-expr-def: "(" "forall" (typed-variable) expr ")"
exists-expr-def: "(" "exists" (typed-variable) expr ")"
(exists (?o - item) (is-red ?o))

Assignment.

Assign updates a grounded state variable with a computed expression. Learned functions can supply the new scalar or vector value.

assign-expr-def: "(" "assign" "(" predicate-name variable-list ")" expr ")"

Foreach and condition.

Foreach enumerates typed objects. In expressions, when selects relevant inputs; in an effect, it gates an update. The same syntax therefore has context-dependent selection or mutation semantics.

foreach-expr-def: "(" "foreach" (typed-variable) expr ")"
when-expr-def:    "(" "when" expr expr ")"

Syntax sugars.

The source lists compact forms for assignment, conditional assignment, and conditional selection, together with their expanded equivalents.

Blank notation ?? .

A named ?? blank stands for an externally implemented function whose listed inputs define its dependency structure. Learning supplies its parameters, not a new dependency graph.

slot-expr-def: "(" "??" predicate-name kwargs expr* ")"

Appendix C Domain-Independent Heuristic

The heuristic implementation compiles vector-valued models into a relaxed discrete representation. Optimistic compilation is coarse and inexpensive; And-Or compilation is more detailed and computationally heavier. Both guide search without replacing the original latent transition model used to expand actions.

Optimistic compilation: leveraging locality.

Optimistic compilation marks an affected feature as potentially satisfying any later test once an action changes it. This retains object-level locality but ignores fine distinctions among possible values. The example needs only two optimistic setters, versus the richer dependency chain in Figure 11.

Original figure
Figure 11: A graphical illustration of the hFF heuristics. The goal is to set the r value of all objects to 4. The heuristic value hFF=6\text{hFF}=6, which is the number of selected operators (highlighted in red).

And-Or compilation.

And-Or compilation quantizes continuous features, initializes codebooks by clustering, fine-tunes for one epoch, and uses FOIL to extract first-order rules from neural functions. First-order rules accommodate variable object counts. The retained examples show quantized color tests and a movement rule that quantifies over blocking objects.

(is-red ?o - item) = (or
  (item-feature@3  ?o)
  (item-feature@14 ?o)
  (item-feature@16 ?o)
  (item-feature@10 ?o)
)
(((robot-pose ?r - robot) <- (SAS
  37 <- (or
  (and
    (not (robot-direction@2 ?r - robot))
    (not (robot-direction@3 ?r - robot))
    (not (robot-direction@1 ?r - robot))
    (robot-pose@37 ?r - robot)
  )
  (and (robot-direction@3 ?r - robot) (robot-pose@45 ?r - robot))
  (and (robot-pose@36 ?r - robot) (robot-direction@0 ?r - robot))
  (and (robot-pose@29 ?r - robot) (robot-direction@1 ?r - robot))
)
  4 <- (and (robot-pose@12 ?r - robot) (robot-direction@3 ?r - robot))
  40 <- (and (robot-pose@41 ?r - robot) (robot-direction@2 ?r - robot))
  21 <- (and (robot-pose@29 ?r - robot) (robot-direction@3 ?r - robot))
  32 <- (and (robot-pose@33 ?r - robot) (robot-direction@2 ?r - robot))
  3 <- (and
    (robot-pose@11 ?r - robot) (robot-direction@3 ?r - robot))
    (and
      (not (robot-pose@27 ?r - robot))
      (not (exists (_t0 - item) (and (robot-is-facing ?r - robot _t0 - item)
                                     (item-feature@9 _t0 - item))))
      (not (exists (_t0 - item) (and (robot-is-facing ?r - robot _t0 - item)
                                     (item-feature@0 _t0 - item))))
      (not (exists (_t0 - item) (and (robot-is-facing ?r - robot _t0 - item)
                                     (item-feature@6 _t0 - item))))
      (not (exists (_t0 - item) (and (robot-is-facing ?r - robot _t0 - item)
                                     (item-feature@11 _t0 - item))))
; ... more rules omitted.

The role of sparsity and locality structures in heuristics.

Factorization reduces the number of distinct codes needed for relaxed reasoning. The example separates two seven-valued coordinates, six colors, and four shapes into 24 factor values instead of 1,176 joint combinations. This illustrates why a useful programmed factorization matters for heuristic efficiency.

Appendix D Experimental Setup.

D.1 BabyAI

Setup.

The BabyAI agent starts at the center of a 7×7 grid; object positions are randomized.

Offline data.

Successful offline trajectories come from grid A*. Failed trajectories approach an incorrect object and then take five random-walk steps, so failure examples follow a specific generation procedure.

Baselines.

Shared inputs encode object state, raw positions, and facing direction. Goals concatenate verb, adjective, and noun embeddings. A two-layer graph network supplies a 128-dimensional world embedding for both behavior cloning and Decision Transformer, with the latter additionally processing history and return information.

Models.

The complete code distinguishes holistic PDS-Base updates, PDS-Abs updates routed through learned facing relations, and PDS-Rob’s supplied movement and obstacle rules. The strongest variant therefore receives materially more prior dynamics information.

(:action forward
 :parameters (?r - robot)
 :precondition (and )
 :effect (and
   (robot-feature::assign ?r (??f (robot-feature ?r) (item-feature ??)))
 )
)
(:action forward
 :parameters (?r - robot)
 :precondition (and )
 :effect (robot-pose::cond-assign ?r
 (??f
   (robot-pose ?r) (robot-direction-feature ?r)
   (foreach (?o - item)
     (item-feature::cond-select ?o (robot-is-facing ?r ?o))
   )
 )
 (??g (robot-pose ?r) (robot-direction-feature ?r))
 )
)
; The first function returns the 2D position that the robot is facing
;   when the robot is at ?p and facing ?d.
;   This function has been implemented externally.
; The second function returns whether an object is an obstacle (so
;   it will block the robot’s movement.
;   This information has also been given to the agent as a input.
(:predicates
  (facing [return_type=pose] ?p - pose ?d - direction)
  (is-obstacle ?o - item)
)
(:derived (_is-facing ?p - pose ?d - direction ?t - pose)
  (equal (facing ?p ?d) ?t)
)
(:derived (_robot-facing [return_type=pose] ?r - robot)
  (facing (robot-pose ?r) (robot-direction ?r))
)
(:derived (_robot-is-facing ?r - robot ?o - item)
  (_is-facing (robot-pose ?r) (robot-direction ?r) (item-pose ?o))
)
(:derived (_robot-facing-clear ?r - robot)
  (not (exists (?o - item) (and
    (_robot-is-facing ?r ?o)
    (is-obstacle ?o)
  ))
)
(:action forward
 :parameters (?r - robot)
 :precondition (_robot-facing-clear ?r)
 :effect (and (robot-pose::assign ?r (_robot-facing ?r)) )
)

D.2 Painting Factory

State space.

Painting inputs pair normalized three-dimensional positions with 32×32 object image crops. A three-layer convolutional encoder and linear layer produce 128-dimensional image features.

Action space.

Move-into updates an item’s pose and conditionally paints it to a bowl’s color. Move-to places an item at a specified pose. Both complete action definitions remain below.

(:action move-into
 :parameters (?o - item ?c - container)
 :precondition (and )
 :effect (and
   (item-pose::assign ?o (container-pose ?c))
   (item-feature::cond-assign ?o
     (??g (container-feature ?c))  ; when the container is a bowl,
     (??h (container-feature ?c))  ; paint ?o to be the same color as ?c.
   )
 )
)
(:action move-to
 :parameters (?o - item ?p - pose)
 :precondition (and )
 :effect (and (item-pose::assign ?o ?p))
)

Offline data.

Oracle demonstrations choose two blocks, paint and place the first centrally, then paint and place the second in the requested relation to it.

Baselines.

The policy baselines score block/container pairs and direct block moves, normalize those scores into action probabilities, and predict a target pose for direct placement.

Models.

Continuous planning uses a task-and-motion-planning sampling strategy. Learned geometric relation classifiers have a structured pose-offset form that permits sampling poses on, left of, or right of existing objects. These samples instantiate concrete planning actions. The approach searches this generated candidate set rather than exhaustively optimizing over all continuous poses.

Equation
p⁡(a​on ​b)=σ⁡(‖pose​(a)−pose​(b)−θ‖22−γτ),p(a~\text{\tt on }b)=\sigma\left(\frac{\|\textit{pose}(a)-\textit{pose}(b)-\theta\|^{2}_{2}-\gamma}{\tau}\right),
Concept demonstration

Open image ↗