Literature
Display
← All papers

Learning Compositional Behaviors from Demonstration and Language

Liu, Weiyu · Nie, Neil · Zhang, Ruohan · Mao, Jiayuan · Wu, Jiajun

· Conference on Robot Learning 2024 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

The retained overview figure illustrates how language-annotated demonstrations become symbolic behavior descriptions, visual state classifiers, and executable policies. The appendices include complete prompts, operator definitions, algorithms, and evaluation examples.

At a glance

BLADE combines learned robot skills with symbolic planning. Language models propose each skill’s preconditions and effects from annotated demonstrations and contact sequences; these descriptions generate supervision for visual state classifiers. A planner composes the skills and replans after execution. Simulation and physical manipulation tests examine new goals, changed states, obstacles, and hidden objects.

1 Introduction

Useful long-horizon skills need more than executable policies: a planner must know when a behavior is applicable and what it changes. BLADE learns these dependencies from language and demonstrations, then grounds the resulting predicates in perception. This separates reusable low-level behavior from task-level composition.

Refer to caption
Figure 1: blade, a robot manipulation framework combining imitation learning and model-based planning. (a) blade takes language-annotated demonstrations as training data. (b) It generalizes to unseen initial conditions, state perturbations, and geometric constraints. (c) In the depicted scenarios, blade recovers from perturbations such as moving the kettle out of the sink, and resolves geometric constraints including a blocked stove.

2 Related Work

The approach combines imitation, symbolic action-model learning, and language-model-assisted planning. Unlike direct language-model action generation, planning searches an explicit transition model. Unlike fully hand-written symbolic systems, action dependencies and classifier labels are inferred from demonstrations, although the main experiments still provide predicate names.

3 Problem Formulation

Inputs are demonstrations segmented and annotated with a finite behavior vocabulary. Each behavior has arguments, binary predicate preconditions and effects, and a learned controller. Humans provide the predicate names in the main implementation; a later appendix explores generating them automatically. The abstract action semantics describe intended effects, not guarantees that every learned controller will succeed.

Refer to caption
Figure 2: Overview of blade. (a) blade receives language-annotated human demonstrations, (b) segments demonstrations into contact primitives, and learns a structured behavior representation. (c) It generalizes to novel conditions by leveraging bi-level planning and execution to achieve goal states.

4 Behavior from Language and Demonstration

The pipeline generates behavior descriptions, trains predicate classifiers and policies, and then uses symbolic planning with repeated execution feedback.

4.1 Behavior Description Learning

Gripper state and contact information segment demonstrations into seven primitive types: open, close, move-to, grasp, place, move, and push. A language model proposes a PDDL-style operator and its primitive sequence using behavior labels, neighboring behaviors, scene context, and one example. Abstract verification checks consistency across demonstrations and resamples invalid definitions. The inferred primitive sequence also corrects noisy demonstration boundaries.

Refer to caption
Figure 3: Behavior Descriptions Learning. (a) A demonstration is provided along with corresponding language annotations. (b) The demonstration is segmented into a sequence of contact primitives. (c) A large language model interprets the annotation and contact sequence, generating a symbolic behavior definition. (d) The system automatically generates data to learn classifiers for state predicates.

4.2 Classifier and Policy Learning

Operator preconditions, effects, and persistence create positive and negative predicate labels across demonstration frames, rather than using only segment endpoints. Classifiers learn the resulting visual predicates; diffusion policies learn each behavior. Real-world classifiers focus on relevant objects to reduce spurious correlations, while the policies use multiple camera views.

4.3 Bi-Level Planning and Execution

A language model translates the requested goal to a logical formula. The planner searches abstract states, executes the first behavior, observes again, and replans. Geometric feasibility is checked just before an action, and hidden objects are handled through optimistic state assumptions and subsequent replanning. This is not full belief-space planning.

5 Experiments

Experiments compare composition and state estimation in simulation and in two physical manipulation domains.

5.1 Simulation Experimental Setup

Simulation uses CALVIN split D: about six hours of play data and 34 behavior types. Six new tasks cover abstract goals, geometric constraints, and partial observability. Each task uses twenty initial states and three seeds. Baselines include HULC, SayCan, Robot-VILA, and Text2Motion shooting, with both learned and ground-truth state variants where applicable.

Refer to caption
Figure 4: Generalization Tasks in CALVIN. Examples from the three generalization tasks in the CALVIN simulation environment. Successfully completing these tasks require planning for and executing 3-7 actions.
Table 1: Generalization results in CALVIN. Mean success rates with STD from three seeds are reported. blade outperforms latent planning, LLM, and VLM baselines in completing novel long-horizon tasks.
Method State Classifier Latent Feasibility Generalization Task
Abstract Goal Geometric Constraint Partial Observability
HULC [72] N/A N/A 2.78±3.472.78\pm 3.47 11.67±11.5511.67\pm 11.55 0.00±0.000.00\pm 0.00
SayCan [6] N/A Short 23.89±1.9223.89\pm 1.92 1.67±2.891.67\pm 2.89 1.67±2.891.67\pm 2.89
VILA [57] N/A N/A 18.38±2.4818.38\pm 2.48 0.00±0.000.00\pm 0.00 4.17±5.204.17\pm 5.20
T2M-Shooting [51] Learned Long 57.78±12.2957.78\pm 12.29 0.00±0.000.00\pm 0.00 13.33±1.4413.33\pm 1.44
Ours Learned N/A 68.33±10.14\mathbf{68.33\pm 10.14} 26.67±7.64\mathbf{26.67\pm 7.64} 75.83±3.82\mathbf{75.83\pm 3.82}
T2M-Shooting [51] GT Long 61.67±5.0061.67\pm 5.00 0.00±0.000.00\pm 0.00 0.83±1.440.83\pm 1.44
Ours GT N/A 76.11±6.74\mathbf{76.11\pm 6.74} 56.67±16.07\mathbf{56.67\pm 16.07} 70.00±5.00\mathbf{70.00\pm 5.00}

5.2 Results in Simulation

With learned classifiers, BLADE reports success rates of 68.33% on abstract goals, 26.67% on geometric constraints, and 75.83% on partial observability. HULC reaches 2.78% on abstract goals—a 65.55 percentage-point gap, rather than a 65% relative improvement. The geometric setting remains difficult despite outperforming the tested baselines. Ground-truth state information improves some categories but does not uniformly improve every result. Propagating predicate labels across trajectories outperforms the endpoint-only labeling comparison; the paper reports a 20.7% F1 improvement, with the full planning ablation retained in Table 2.

Table 2: Ablation on state classifier learning in CALVIN.
Method Abstract Geometric Partial Obs.
[66] 33.89±5.8533.89\pm 5.85 9.17±5.209.17\pm 5.20 3.33±2.893.33\pm 2.89
blade 68.33±10.14\mathbf{68.33\pm 10.14} 26.67±7.64\mathbf{26.67\pm 7.64} 75.83±3.82\mathbf{75.83\pm 3.82}

5.3 Real World Experiments

A Franka arm and five cameras perform Boil Water and Make Tea. Each domain has 85 teleoperated demonstrations. Tests include the demonstrated sequences and seven listed generalization cases across initial-state changes, human perturbations, blocking objects, and hidden objects. Each case has six trials; the paper reports at least four successful trials in each generalization condition. Robot-VILA often proposes actions inconsistent with the current scene. These small controlled experiments do not establish broad household reliability.

Refer to caption
Figure 5: Domains and Results in Real World. Make Tea features a toy kitchen designed to simulate boiling water on a stove. The robot must assess the available space on the stove for the kettle. It also needs to manage the dependencies between actions, such as the faucet must be turned away before the kettle can be placed into the sink to avoid collisions. Boil Water involves a tabletop task aimed at preparing tea, incorporating a cabinet, a drawer, and a stove. The robot must locate the kettle, potentially hidden within the cabinet, and a teabag in the drawer. Additionally, it must consider geometric constraints by removing obstacles that block the cabinet doors. In both environments, our model significantly outperforms the VLM-based planner Robot-VILA.
Refer to caption
Figure 6: Real World Planning and Execution. We show the execution traces from blade and Robot-VILA for two generalization tasks: (a) partial observability and (b) geometric constraints.

6 Conclusion and Discussion

Explicit behavior models and grounded predicates improve composition in the tested domains. Limitations include gripper-based segmentation, supplied predicate names in the main system, classifier errors, and low-level policy failures in unfamiliar states. Abstract verification checks consistency with the demonstrations, not universal correctness of the learned action model.

Acknowledgments

The authors acknowledge research sponsors, collaborating laboratories, colleagues, and reviewers.

Appendix A blade Details

The appendix provides the operator-generation prompts, labeling algorithms, model architectures, and planning assumptions.

A.1 Behavior Description Generation with LLMs

Listings 1–6 preserve the generated CALVIN operators, primitive definitions, predicate vocabulary and environment context, an example, reasoning instructions, and a sample behavior query. Scene context is important; replacing it with automatically generated scene descriptions is suggested rather than part of the main comparison.

A.2 Predicate Generation with LLMs

An additional language-model procedure proposes predicates, grounds their arguments, and removes redundant names. Across three domains, 28 of 30 predicates closely match expert-designed ones; left/right pushing is misrepresented by a next-to relation. The authors check that the generated real-world action descriptions can produce correct plans for the nine tested tasks, which is distinct from rerunning all physical executions with automatically invented predicates.

Table A1: Comparison of Manually Defined and Automatically Generated Predicates for CALVIN.
Manually Defined Automatically Generated
rotated-left(?​x?x) rotated-left(?​x?x)
rotated-right(?​x?x) rotated-right(?​x?x)
lifted(?​x?x) holding(?​x?x)
is-open(?​x?x) is-open(?​x?x)
is-close(?​x?x) is-closed(?​x?x)
is-turned-on(?​x?x) light-on(?​x?x)
is-turned-off(?​x?x) light-off(?​x?x)
is-slider-left(?​x?x) slider-left(?​x?x)
is-slider-right(?​x?x) slider-right(?​x?x)
is-on(?​x,?​y?x,?y) on-table(?​x?x)
is-in(?​x,?​y?x,?y) in-slider(?​x?x), in-drawer(?​x?x)
stacked(?​x,?​y?x,?y) on(?​x,?​y?x,?y)
unstacked(?​x,?​y?x,?y) clear(?​x?x)
pushed-left(?​x?x) -
pushed-right(?​x?x) -
- next-to(?​x,?​y?x,?y)
Table A2: Comparison of Manually Defined and Automatically Generated Predicates for Boil Water.
Manually Defined Automatically Generated Automatically Generated (Grounded)
is-placed-on(?​x,?​y?x,?y) on-table(?​x?x) on-table(kettle), on-table(pot)
is-placed-in(?​x,?​y?x,?y) in-sink(?​x?x) in-sink(kettle), in-sink(pot)
is-blocked(?​x?x) on-stove(?​x?x) on-stove(pot)
is-turned-away(?​x?x) faucet-over-sink(?​x,?​y?x,?y) faucet-over-sink(faucet, sink)
is-aligned(?​x,?​y?x,?y) faucet-over-sink(?​x,?​y?x,?y) faucet-over-sink(faucet, sink)
is-turned-on(?​x?x) faucet-on(?​x?x) faucet-on(faucet)
is-turned-off(?​x?x) faucet-off(?​x?x) faucet-off(faucet)
is-filled(?​x?x) filled(?​x?x) filled(kettle), filled(pot)
holding(?​x?x) holding(?​x?x) holding(pot)
Table A3: Comparison of Manually Defined and Automatically Generated Predicates for Make Tea.
Manually Defined Automatically Generated Automatically Generated (Grounded)
is-placed-on(?​x,?​y?x,?y) is-on(?​x,?​y?x,?y) is-on(kettle, stove)
is-cabinet-door-open(?​x?x) is-open(?​x?x) is-open(left-door), is-open(right-door)
is-placed-inside(?​x,?​y?x,?y) is-in-drawer(?​x?x) is-in-drawer(teabag)
is-in-kettle(?​x?x) is-in-kettle(teabag)
is-in-cabinet-left(?​x?x) is-in-cabinet-left(kettle)
is-in-cabinet-right(?​x?x) is-in-cabinet-right(kettle)
is-drawer-open(?​x?x) is-open(?​x?x) is-open(drawer)
is-left-cabinet-door-blocked(?​x?x) is-blocking(?​x,?​y?x,?y) is-blocking(pot, left-door)
is-right-cabinet-door-blocked(?​x?x) is-blocking(?​x,?​y?x,?y) is-blocking(pot, right-door)
- is-closed(?​x?x) is-closed(left-door), is-closed(right-door)
is-closed(drawer)
- is-moved-away(?​x?x) is-moved-away(pot)

A.3 Temporal Segmentation

Segmentation assumes demonstrators move freely with fully open grippers and push with fully closed grippers. Simulator contacts provide arguments in simulation; real-world arguments come from language annotations. Re-segmentation with the inferred operator body removes irrelevant actions from noisy labels. These instructed demonstration conventions and contact cues are substantive inputs.

A.4 Abstract Verification

Algorithm 1 checks whether accumulated effects can satisfy each demonstrated behavior’s preconditions and whether the primitive sequence is consistent. Descriptions failing this abstract check are regenerated.

Algorithm 1 Abstract Verification
1: Dataset 𝒟\mathcal{D}, Behavior descriptions 𝒜\mathcal{A}
2: error_counter←\textit{error\_counter}\leftarrow a counter for sequencing errors related to each behavior
3: counter←\textit{counter}\leftarrow a counter for storing the occurrences of each behavior
4: for i←1i\leftarrow 1 to KK do
5:   obtain a behavior sequence 𝒟i←{a1i,…,aNi}\mathcal{D}_{i}\leftarrow\{a^{i}_{1},...,a^{i}_{N}\}
6:   initialize a dictionary for predicate state pred←{}\textit{pred}\leftarrow\{\}
7:   for t←1t\leftarrow 1 to NN do
8:    for each exp in preati\textit{pre}_{a^{i}_{t}} do
9:      (p,v)←ExtractPredicateAndBool​(exp)(p,v)\leftarrow\textsc{ExtractPredicateAndBool}(\textit{exp})
10:      if pp not in pred then
11:       pred​[p]←v\textit{pred}[p]\leftarrow v
12:      else
13:       if pred​[p]≠v\textit{pred}[p]\neq v then
14:         increment error_counter​[ati]\textit{error\_counter}[a^{i}_{t}]               
15:    for each exp in effati\textit{eff}_{a^{i}_{t}} do
16:      (p,v)←ExtractPredicateAndBool​(exp)(p,v)\leftarrow\textsc{ExtractPredicateAndBool}(\textit{exp})
17:      pred​[p]←v\textit{pred}[p]\leftarrow v    
18:    increment counter​[ati]\textit{counter}[a^{i}_{t}]   
19: for each aa in error_counter do
20:   if error_counter​[a]/counter​[a]>threshold\textit{error\_counter}[a]/\textit{counter}[a]>\textit{threshold} then
21:    regenerate the behavior description for aa   

A.5 Automatic Predicate Annotation

Algorithm 2 initializes predicates as unknown, infers values from preconditions and effects, and propagates them until a later action changes them. It supplies both positive and negative examples without separately annotated state labels.

Algorithm 2 Predicate Annotation
1: Behavior sequence {a1,…,aN}\{a_{1},...,a_{N}\}, Observation sequence {o1,…,oH}\{o_{1},...,o_{H}\}, Descriptions 𝒜\mathcal{A}
2: propagated←\textit{propagated}\leftarrow an empty list of propagated predicates
3: prev_effs←\textit{prev\_effs}\leftarrow a list for storing effects from previous step
4: timed_preds←\textit{timed\_preds}\leftarrow an empty list of predicates associated with time steps
5: pred_obs←\textit{pred\_obs}\leftarrow an empty list for storing predicates paired with observations
6: for t←1t\leftarrow 1 to NN do
7:   // Precondition
8:   timed_preds←timed_preds∪GetTimedPredicates​(preat,t)\textit{timed\_preds}\leftarrow\textit{timed\_preds}\cup\textsc{GetTimedPredicates}(\textit{pre}_{a_{t}},t)
9:   timed_preds←timed_preds∪GetTimedPredicates​(¬effat,t)\textit{timed\_preds}\leftarrow\textit{timed\_preds}\cup\textsc{GetTimedPredicates}(\neg\textit{eff}_{a_{t}},t)
10:   // Propagated
11:   for each pp in propagated do
12:    if not altered(pp, ata_{t}) then
13:      UpdateTime(pp, tt)
14:    else
15:      propagated.remove(pp)
16:      timed_preds.add(pp)      
17:   // Previous effects
18:   for each pp in prev_effs do
19:    if not altered(pp, ata_{t}) then
20:      propagated.add(pp)
21:    else
22:      timed_preds.add(pp)      
23:   // Store effects for next step
24:   prev_effs←GetTimedPredicates​(effat,t)\textit{prev\_effs}\leftarrow\textsc{GetTimedPredicates}(\textit{eff}_{a_{t}},t)
25: timed_preds.update​(propagated)\textit{timed\_preds.update}(\textit{propagated})
26: timed_preds.update​(prev_effs)\textit{timed\_preds.update}(\textit{prev\_effs})
27: for each pp in timed_preds do
28:   pred_obs.update​(MatchTimedPredicateWithObservation​(p,{o1,…,oH}))\textit{pred\_obs.update}(\textsc{MatchTimedPredicateWithObservation}(p,\{o_{1},...,o_{H}\}))
29: return pred_obs

A.6 Classifier Implementation

Simulation classifiers use frozen CLIP patch features with learned predicate heads and mask the arm to reduce overfitting. Physical classifiers instead use Grounding-DINO detections and segmented 3D point clouds with a Point Cloud Transformer. Substituting GPT-4o as the classifier yields 0% success on the tested Boil Water generalization cases. Adding automatically labeled visual examples improves its classification accuracy by about 5% on 220 held-out examples in a preliminary test.

A.7 Policy Implementation

Skill training randomizes initial observation histories, balances starting states, and appends zero actions to teach termination. Simulation policies use segmented object point clouds with ground-truth simulator masks. Physical policies use four stationary views plus a wrist camera, separate image encoders, and diffusion-based gripper-pose prediction.

A.8 Planner Implementation

The planner first satisfies semantic prerequisites, then checks immediate geometric prerequisites and recursively moves obstructions if needed. This assumes enough free space for rearrangement and does not predict exact future object poses. Hidden-object search uses deterministic optimistic effects and retries after observations; richer uncertainty and tightly packed rearrangement need additional models.

Appendix B Simulation Experiment Details

Evaluation retains CALVIN’s original language annotations, skills, action space, and environments while adding task-level generalization tests.

B.1 Task Design

The six simulation cases turn off both lights; put all blocks in an initially closed drawer; put them in an open drawer; retrieve a hidden block from a closed drawer; retrieve one from behind a slider; and open a slider blocked by another object. Random initial states and object/color variants distinguish composition from replaying one fixed trajectory.

B.2 Baseline Implementation

SayCan uses object prompts and demonstration-trained feasibility classifiers rather than reinforcement-learned value functions. Robot-VILA uses GPT-4 Turbo with accumulated dialogue and images. Text2Motion uses a longer-horizon feasibility model and symbolic state descriptions, comparing learned and ground-truth states. These adaptations define what the reported baseline results measure.

Appendix C Qualitative Examples

The retained qualitative examples show complete plans and execution traces for abstract goals, hidden objects, and obstructed motion. They illustrate particular successes and baseline failures rather than adding independent aggregate evaluations.

Appendix D Real World Experiment Details

The physical setup records end-effector pose, gripper width, and five RGB-D camera streams during SpaceMouse teleoperation. Each of the two domains contributes 85 demonstrations.

Refer to caption
Figure A1: We use a 7-degree of freedom (DOF) Franka Emika robotic arm with a parallel jaw gripper for our real-world experiment. A total of Five Intel RealSense RGB-D cameras are used to provide observation for our policies and state classifiers. Four cameras are mounted on the frame and an additional one is mounted to the robot’s wrist.

D.1 Task Design

Seven generalization cases are listed: three Boil Water cases change the starting state, move the kettle mid-task, or block the stove; four Make Tea cases change the starting arrangement, reclose a drawer, block cabinet doors, or hide the kettle. Logical goals require the final object relations and, for Boil Water, a turned-off faucet. The retained figures show each setting.

Refer to caption
Figure A2: Training and testing states for the Boil Water domain.
Refer to caption
Figure A3: Training and testing states for the Make Tea domain.

D.2 Qualitative Examples of Novel States

Visual comparisons indicate that more than half of the illustrated initial or perturbed states were absent from demonstrations. This supports novelty within the constructed test cases, without proving that all underlying low-level motions are unseen.

Appendix E Prompts for Baselines

Listings 8–11 retain the full SayCan, Robot-VILA initial and follow-up, and Text2Motion shooting prompts. Together with the earlier operator-generation listings, these specify the model inputs used in the comparisons.

Refer to caption
Figure A4: blade and baseline performance on an Abstract Goal generalization task in the CALVIN environment.
Refer to caption
Figure A5: blade and baseline performance on the Partial Observability generalization task in the CALVIN environment.
Refer to caption
Figure A6: blade and baseline performance on the Geometric Constraint generalization task in the CALVIN environment.
Listing 1: Behavior descriptions generated by the LLM for the CALVIN domain.
;; lift_block_table
(:action lift-block-table
:parameters (?block - item ?table - item)
:precondition (and (is-block ?block) (is-table ?table) (is-on ?block ?table) (not (is-lifted ?block)))
:effect (and (lifted ?block) (not (is-on ?block ?table)))
:body (then
(grasp ?block ?table)
(move ?block)
)
)
;; lift_block_slider
(:action lift_block_slider
:parameters (?block - item ?slider - item)
:precondition (and (is-block ?block) (is-slider ?slider) (is-in ?block ?slider))
:effect (and (lifted ?block) (not (is-in ?block ?slider)))
:body (then
(grasp ?block ?slider)
(move ?block)
)
)
;; lift_block_drawer
(:action lift-block-drawer
:parameters (?block - item ?drawer - item)
:precondition (and (is-block ?block) (is-drawer ?drawer) (is-in ?block ?drawer) (is-open ?drawer))
:effect (and (lifted ?block) (not (is-in ?block ?drawer)))
:body (then
(grasp ?block ?drawer)
(move ?block)
)
)
;; place_in_slider
(:action place-in-slider
:parameters (?block - item ?slider - item)
:precondition (and (is-block ?block) (is-slider ?slider) (is-lifted ?block))
:effect (and (is-in ?block ?slider) (not (is-lifted ?block)))
:body (then
(place ?block ?slider)
)
)
;; place_in_drawer
(:action place-in-drawer
:parameters (?block - item ?drawer - item)
:precondition (and (is-block ?block) (is-drawer ?drawer) (is-lifted ?block) (is-open ?drawer))
:effect (and (is-in ?block ?drawer) (not (is-lifted ?block)))
:body (then
(place ?block ?drawer)
)
)
;; place_on_table
(:action place-on-table
:parameters (?block - item ?table - item)
:precondition (and (is-block ?block) (is-table ?table) (is-lifted ?block))
:effect (and (is-on ?block ?table) (not (is-lifted ?block)))
:body (then
(place ?block ?table)
)
)
;; stack_block
(:action stack_block
:parameters (?block - item ?target - item)
:precondition (and (is-block ?block) (is-block ?target) (is-lifted ?block))
:effect (and (stacked ?block ?target) (not (is-lifted ?block)))
:body (then
(place ?block ?target)
)
)
;; unstack_block
(:action unstack_block
:parameters (?block1 - item ?block2 - item)
:precondition (and (is-block ?block1) (is-block ?block2) (stacked ?block1 ?block2))
:effect (and (unstacked ?block1 ?block2) (is-lifted ?block1) (not (stacked ?block1 ?block2)))
:body (then
(grasp ?block1 ?block2)
(move ?block1)
)
)
;; rotate_block_right
(:action rotate-block-right
:parameters (?block - item ?table - item)
:precondition (and (is-block ?block) (is-table ?table) (is-on ?block ?table))
:effect (and
(rotated-right ?block)
(not (rotated-left ?block)))
:body (then
(grasp ?block ?table)
(move ?block)
(place ?block ?table)
)
)
;; rotate_block_left
(:action rotate_block_left
:parameters (?block - item ?table - item)
:precondition (and (is-block ?block) (is-table ?table) (is-on ?block ?table))
:effect (and (rotated-left ?block))
:body (then
(grasp ?block)
(move ?block)
(place ?block)
)
)
;; push_block_right
(:action push_block_right
:parameters (?block - item ?table - item)
:precondition (and (is-block ?block) (is-table ?table) (is-on ?block ?table))
:effect (and (pushed-right ?block) (not (pushed-left ?block)))
:body (then
(close)
(push ?block)
(open)
)
)
;; push_block_left
(:action push-block-left
:parameters (?block - item)
:precondition (and (is-block ?block))
:effect (and (pushed-left ?block))
:body (then
(close)
(push ?block)
(open)
)
)
;; move_slider_left
(:action move_slider_left
:parameters (?slider - item)
:precondition (and (is-slider ?slider) (is-slider-right ?slider))
:effect (and (is-slider-left ?slider) (not (is-slider-right ?slider)))
:body (then
(grasp ?slider)
(move ?slider)
(place ?slider)
)
)
;; move_slider_right
(:action move-slider-right
:parameters (?slider - item)
:precondition (and (is-slider ?slider) (not (is-slider-right ?slider)))
:effect (and (is-slider-right ?slider))
:body (then
(grasp ?slider)
(move ?slider)
(place ?slider)
)
)
;; open_drawer
(:action open-drawer
:parameters (?drawer - item)
:precondition (and (is-drawer ?drawer) (is-close ?drawer))
:effect (and (is-open ?drawer) (not (is-close ?drawer)))
:body (then
(close)
(push ?drawer)
(open)
)
)
;; close_drawer
(:action close-drawer
:parameters (?drawer - item)
:precondition (and (is-drawer ?drawer) (is-open ?drawer))
:effect (and (is-close ?drawer) (not (is-open ?drawer)))
:body (then
(close)
(push ?drawer)
(open)
)
)
;; turn_on_lightbulb
(:action turn-on-lightbulb
:parameters (?lightbulb - item)
:precondition (and (is-lightbulb ?lightbulb) (is-turned-off ?lightbulb))
:effect (and (is-turned-on ?lightbulb) (not (is-turned-off ?lightbulb)))
:body (then
(close)
(push ?lightbulb)
(open)
)
)
;; turn_off_lightbulb
(:action turn-off-lightbulb
:parameters (?lightbulb - item)
:precondition (and (is-lightbulb ?lightbulb) (is-turned-on ?lightbulb))
:effect (and (is-turned-off ?lightbulb) (not (is-turned-on ?lightbulb)))
:body (then
(close) (push ?lightbulb) (open)
)
)
;; turn_on_led
(:action turn-on-led
:parameters (?led - item)
:precondition (is-led ?led)
:effect (and (is-turned-on ?led) (not (is-turned-off ?led)))
:body (then
(close)
(push ?led)
(open)
)
)
;; turn_off_led
(:action turn-off-led
:parameters (?led - item)
:precondition (and (is-led ?led) (is-turned-on ?led))
:effect (and (is-turned-off ?led) (not (is-turned-on ?led)))
:body (then
(close)
(push ?led)
(open)
)
)
;; push_into_drawer
(:action push-into-drawer
:parameters (?block - item ?drawer - item)
:precondition (and (is-block ?block) (is-drawer ?drawer) (is-open ?drawer))
:effect (and (is-in ?block ?drawer))
:body (then
(close)
(push ?block)
(open)
)
)
Listing 2: Example Prompt for CALVIN–Contact Primitives.
**Primitive Actions:**
There are seven primitive actions that the robot can perform. They are:
- (grasp ?x ?y): ?x and ?y are two object variables. ?x is the object that the robot will be grasping, ?y is the object that ?x is currently on or in.
- (place ?x ?y): ?x and ?y are two object variables. ?x is the object that the robot is currently holding, ?y is the object that ?x will be placed on or in.
- (move ?x): ?x is the object that the robot is currently holding and will be moved by the robot.
- (push ?x): ?x is the object that the robot will be pushing.
- (move-to ?x): the robot arm will move without holding any object or pushing any object.
- (open): the robot gripper will open fully.
- (close): the robot gripper will close without grasping any object.
**Combined Primitives:**
The primitive actions can be combined into a high-level routine. For example, (then (grasp ?x ?y) (move ?x) (place ?x ?y)) means the robot will pick up ?x from ?y, move ?x, and place ?x to ?z. The possible combination of primitives are:
A. (then (grasp ?x ?y) (move ?x))
B. (then (place ?x ?y))
C. (then (grasp ?x ?y) (move ?x) (place ?x ?z))
D. (then (close) (push ?x) (open))
Listing 3: Example Prompt for CALVIN–Environment.
**Predicates for Preconditions and Effects:**
The list of all possible predicates for defining the preconditions and effects of the high-level routine are listed below:
For specifying the type of the object:
- (is-table ?x - item): ?x is a table.
- (is-slider ?x - item): ?x is a slider.
- (is-drawer ?x - item): ?x is a drawer.
- (is-lightbulb ?x - item): ?x is a lightbulb.
- (is-led ?x - item): ?x is a led.
- (is-block ?x - item): ?x is a block.
For specifying the attributes of the object:
- (is-red ?x - item): ?x is red. This predicate applies to a block.
- (is-blue ?x - item): ?x is blue. This predicate applies to a block.
- (is-pink ?x - item): ?x is pink. This predicate applies to a block.
For specifying the state of the object:
- (rotated-left ?x - item): ?x is rotated left. This predicate applies to a block.
- (rotated-right ?x - item): ?x is rotated right. This predicate applies to a block.
- (pushed-left ?x - item): ?x is pushed left. This predicate applies to a block.
- (pushed-right ?x - item): ?x is pushed right. This predicate applies to a block.
- (lifted ?x - item): ?x is lifted in the air. This predicate applies to a block.
- (is-open ?x - item): ?x is open. This predicate applies to a drawer.
- (is-close ?x - item): ?x is close. This predicate applies to a drawer.
- (is-turned-on ?x - item): ?x is turned on. This predicate applies to a lightbulb or a led.
- (is-turned-off ?x - item): ?x is turned off. This predicate applies to a lightbulb or a led.
- (is-slider-left ?x - item): the sliding door of the slider ?x is on the left.
- (is-slider-right ?x - item): the sliding door of the slider ?x is on the right.
For specifying the relationship between objects:
- (is-on ?x - item ?y - item): ?x is on top of ?y. This predicate applies when ?x is a block and ?y is a table.
- (is-in ?x - item ?y - item): ?x is inside of ?y. This predicate applies when ?x is a block and ?y is a drawer or a slider.
- (stacked ?x - item ?y - item): ?x is stacked on top of ?y. This predicate applies when ?x and ?y are blocks.
- (unstacked ?x - item ?y - item): ?x is unstacked from ?y. This predicate applies when ?x and ?y are blocks.
**Task Environment:**
In the environment where the demonstrations are being performed, there are the following objects:
- A table. Objects can be placed on the table.
- A drawer that can be opened. Objects can be placed into the drawer when it is open.
- A slider which is a cabinet with a sliding door. The sliding door can be moved to the left or to the right. Objects can be placed into the slider no matter the position of the sliding door.
- A lightbulb that be can turned on/off with a button.
- A led that can be turned on/off with a button.
- Three blocks that can be rotated, pushed, lifted, and placed.
Listing 4: Example Prompt for CALVIN–In-Context Example.
**Demonstration Parsing:**
Now, you will help to parse several human demonstrations of the robot performing a task and generate a lifted description of how to accomplish this task.
For each demonstration, a sequence of performed primitives will be given, with actual object names. Three demonstrations for the task of "place_in_slider" is:
<code name="primitive_sequence">
primitives = [
{"name": "grasp", "arguments": ["red_block", "table"]}
{"name": "move", "arguments": ["red_block"]}
{"name": "place", "arguments": ["red_block", "slider"]}
{"name": "move-to", "arguments": [""]}
]
</code>
<code name="primitive_sequence">
primitives = [
{"name": "grasp", "arguments": ["blue_block", "table"]}
{"name": "move", "arguments": ["blue_block"]}
{"name": "place", "arguments": ["blue_block", "slider"]}
{"name": "move-to", "arguments": [""]}
]
</code>
<code name="primitive_sequence">
primitives = [
{"name": "grasp", "arguments": ["pink_block", "table"]}
{"name": "move", "arguments": ["pink_block"]}
{"name": "place", "arguments": ["pink_block", "slider"]}
{"name": "move-to", "arguments": [""]}
]
</code>
**Previous Tasks:**
A list of tasks that can be performed before the current task will also be provided as context. For the task of "place_in_slider", the possible previous tasks are:
lift_block_table, lift_block_drawer, move_slider_right
**Example Output:**
You should generate a lifted description, treating all objects as variables. For example, the lifted description for "place_in_slider" is:
<code name="mechanism">
(:mechanism place-in-slider
:parameters (?block - item ?slider - item)
:precondition (and (is-block ?block) (is-slider ?slider) (is-lifted ?block))
:effect (and (is-in ?block ?slider) (not (is-lifted ?block)))
:body (then
(place ?block ?slider)
)
)
</code>
Listing 5: Example Prompt for CALVIN–Instructions.
**Think Step-by-Step:**
To generate the lifted description, you should think through the task in natural language in the following steps. Be EXTREMELY CAREFUL to think through step 3a, 3b, and 4a, 4b.
1. Parse the goal. For example "place_in_slider", the goal is to place a block into the slider.
2. Think about the possible effects achieved by previous tasks and the previous actions that have been performed. For "lift_block_table", a block is lifted from the table and the effect is that the block is lifted. For "lift_block_drawer", a block is lifted from the drawer and the effect is that the block is lifted. For "move_slider_right", the sliding door of the slider is moved to the right and the effect is that the sliding door is on the right.
3. Parse the demonstrations and choose the combination of primitives for the current task. The demonstrations are noisy so that the demonstrated primitive sequences may include extra primitive actions that are not necessary for the current task at the beginning or end. The extra primitive actions can be for the previous tasks. Combining with the understanding of the task and previous task to infer the correct combination of primitives for the current task.
3a. In this case, the previous tasks are relevant to the current task. We should think about how to sequence the previous tasks with the current task. The primitive combination for the current task should not include primitive actions that have been performed. The above example for "place_in_slider" is this case. We can infer that "grasp" in the demonstrated sequences is likely to be for the previous tasks and should not be included in the primitive combination for the current task. We therefore choose B. (then (place ?x ?y)). The semantics is that the robot place the lifted block in the slider.
3b. In this case, the previous tasks are not relevant to the current task.
4. Think about the preconditions. Also specify the types of all relevant objects in the preconditions.
4a. In this case, previous tasks are relevant to the current task. We should think about the effects of the previous tasks. For "place_in_slider", the effects of previous tasks include the block is already lifted. So we should specify that the block is lifted in the preconditions for the current task.
4b. In this case, previous tasks are not relevant to the current task.
5. Think about the effects. For "place_in_slider", the effects are that the block is in the slider and the block is not lifted.
6. Write down the mechanism in the format of the example.
**Additional Instructions:**
1. Make sure the generated lifted description starts with <code name="mechanism"> and ends with </code>.
2. Please do not invent any new predicates for the precondition and effect. You can only use the predicates listed above.
3. Consider the physical constraints of the objects. For example, a robot arm can not go through a closed door.
4. For each parameter in :parameters, you should use one of the predicates for specifying the type of the object to indicate its type (e.g., is-drawer, is-block, and etc).
Listing 6: Example Prompt for CALVIN–Task Input.
**Current Task:** place_in_drawer
**Example Sequences:**
<code name="primitive_sequence">
primitives = [
{"name": "grasp", "arguments": ["blue_block", "table"]}
{"name": "move", "arguments": ["blue_block"]}
{"name": "place", "arguments": ["blue_block", "drawer"]}
{"name": "move-to", "arguments": [""]}
]
</code>
<code name="primitive_sequence">
primitives = [
{"name": "grasp", "arguments": ["red_block", "table"]}
{"name": "move", "arguments": ["red_block"]}
{"name": "place", "arguments": ["red_block", "drawer"]}
{"name": "move-to", "arguments": [""]}
]
</code>
<code name="primitive_sequence">
primitives = [
{"name": "grasp", "arguments": ["pink_block", "table"]}
{"name": "move", "arguments": ["pink_block"]}
{"name": "place", "arguments": ["pink_block", "drawer"]}
{"name": "move-to", "arguments": [""]}
]
</code>
**Previous Tasks:** push_into_drawer, lift_block_table, lift_block_slider
Listing 7: Example Prompt for Predicate Generation.
You are a helpful agent in helping a robot interpret human demonstrations and discover a generalized high-level routine to accomplish a given task.
**Primitive Actions:**
There are seven primitive actions that the robot can perform. They are:
- (grasp ?x ?y): ?x and ?y are two object variables. ?x is the object that the robot will be grasping, ?y is the object that ?x is currently on or in.
- (place ?x ?y): ?x and ?y are two object variables. ?x is the object that the robot is currently holding, ?y is the object that ?x will be placed on or in.
- (move ?x): ?x is the object that the robot is currently holding and will be moved by the robot.
- (push ?x): ?x is the object that the robot will be pushing.
- (move-to ?x): the robot arm will move without holding any object or pushing any object.
- (open): the robot gripper will open fully.
- (close): the robot gripper will close without grasping any object.
**Task Environment:**
In the environment where the demonstrations are being performed, there are the following objects:
- A table. Objects can be placed on the table.
- A drawer that can be opened. Objects can be placed into the drawer when it is open.
- A slider which is a cabinet with a sliding door. The sliding door can be moved to the left or to the right. Objects can be placed into the slider no matter the position of the sliding door.
- A lightbulb that be can turned on/off with a button.
- A led that can be turned on/off with a button.
- Three blocks that can be rotated, pushed, lifted, and placed.
**Task**
You will help the robot to write PDDL definitions for the following actions:
1. lift_red_block_table
2. lift_red_block_slider
3. lift_red_block_drawer
4. lift_blue_block_table
5. lift_blue_block_slider
6. lift_blue_block_drawer
7. lift_pink_block_table
8. lift_pink_block_slider
9. lift_pink_block_drawer
10. stack_block
11. unstack_block
12. place_in_slider
13. place_in_drawer
14. place_on_table
15. rotate_red_block_right
16. rotate_red_block_left
17. rotate_blue_block_right
18. rotate_blue_block_left
19. rotate_pink_block_right
20. rotate_pink_block_left
21. push_red_block_right
22. push_red_block_left
23. push_blue_block_right
24. push_blue_block_left
25. push_pink_block_right
26. push_pink_block_left
27. move_slider_left
28. move_slider_right
29. open_drawer
30. close_drawer
31. turn_on_lightbulb
32. turn_off_lightbulb
33. turn_on_led
34. turn_off_led
Before writing the operators, define the predicates that should be used to write the preconditions and effects of the operators. Group the predicates into unary predicates that define the states of objects and binary relations that specify relations between two objects. For each predicate, list actions that are relevant.
Listing 8: Prompt for SayCan.
**Objective:**
You are a helpful agent in helping a robot plan a sequence of actions to accomplish a given task.
I will first provide context and then provide an example of how to perform the task.
**Task Environment:**
In the robot’s environment, there are the following objects:
- A table. Objects can be placed on the table.
- A drawer that can be opened. Objects can be placed into the drawer when it is open.
- A slider which is a cabinet with a sliding door. The sliding door can be moved to the left or to the right. Objects can be placed into the slider no matter the position of the sliding door.
- A lightbulb that be can turned on/off with a button.
- A led that can be turned on/off with a button.
- Three blocks that can be rotated, pushed, lifted, and placed.
**Actions:**
There are the following actions that the robot can perform. They are:
- lift_red_block_table: lift the red block from the table.
- lift_red_block_slider: lift the red block from the slider.
- lift_red_block_drawer: lift the red block from the drawer.
- lift_blue_block_table: lift the blue block from the table.
- lift_blue_block_slider: lift the blue block from the slider.
- lift_blue_block_drawer: lift the blue block from the drawer.
- lift_pink_block_table: lift the pink block from the table.
- lift_pink_block_slider: lift the pink block from the slider.
- lift_pink_block_drawer: lift the pink block from the drawer.
- stack_block: stack the blocks.
- place_in_slider: place the block in the slider.
- place_in_drawer: place the block in the drawer.
- place_on_table: place the block on the table.
- rotate_red_block_right: rotate the red block to the right.
- rotate_red_block_left: rotate the red block to the left.
- rotate_blue_block_right: rotate the blue block to the right.
- rotate_blue_block_left: rotate the blue block to the left.
- rotate_pink_block_right: rotate the pink block to the right.
- rotate_pink_block_left: rotate the pink block to the left.
- push_red_block_right: push the red block to the right.
- push_red_block_left: push the red block to the left.
- push_blue_block_right: push the blue block to the right.
- push_blue_block_left: push the blue block to the left.
- push_pink_block_right: push the pink block to the right.
- push_pink_block_left: push the pink block to the left.
- move_slider_left: move the slider to the left.
- move_slider_right: move the slider to the right.
- open_drawer: open the drawer.
- close_drawer: close the drawer.
- turn_on_lightbulb: turn on the lightbulb.
- turn_off_lightbulb: turn off the lightbulb.
- turn_on_led: turn on the led.
- turn_off_led: turn off the led.
- do_nothing: do nothing.
**Example Task:**
Now, you will help to parse the goal predicate and generate a list of candidate actions the robot can potentially take to accomplish the task. You should rank the actions in terms of how likely they are to be performed next.
Goal predicate: (is-turned-off led)
Task output:
‘‘‘python
[’turn_off_led’, ’do_nothing’]
‘‘‘
In this example above, if the led is on, the robot should turn it off. If the led is already off, the robot should do nothing.
When the robot successfully completes an action, the robot will ask for the next action to take.
Considering the executed task: turn_off_led
Your output should be:
‘‘‘python
[’do_nothing’]
‘‘‘
Since the led is already off, the robot should do nothing.
**Additional Instructions:**
1. Make sure the generated plan is a list of actions. Place the list between ‘‘‘python and ends with ‘‘‘.
2. Think Step-by-Step.
Goal predicate: {Based on the given task}
Current symbolic state: {Based on the simulator state}
Executed actions: {Based on the previously executed actions}
Listing 9: Initial Prompt for Robot-VILA.
You are highly skilled in robotic task planning, breaking down intricate and long-term tasks into distinct primitive actions.
If the object is in sight, you need to directly manipulate it. If the object is not in sight, you need to use primitive skills to find the object
first. If the target object is blocked by other objects, you need to remove all the blocking objects before picking up the target object. At
the same time, you need to ignore distracters that are not related to the task. And remember your last step plan needs to be "done".
Consider the following skills a robotic arm can perform.
- lift_red_block_table: lift the red block from the table.
- lift_red_block_slider: lift the red block from the slider.
- lift_red_block_drawer: lift the red block from the drawer.
- lift_blue_block_table: lift the blue block from the table.
- lift_blue_block_slider: lift the blue block from the slider.
- lift_blue_block_drawer: lift the blue block from the drawer.
- lift_pink_block_table: lift the pink block from the table.
- lift_pink_block_slider: lift the pink block from the slider.
- lift_pink_block_drawer: lift the pink block from the drawer.
- stack_block: stack the blocks.
- place_in_slider: place the block in the slider.
- place_in_drawer: place the block in the drawer.
- place_on_table: place the block on the table.
- rotate_red_block_right: rotate the red block to the right.
- rotate_red_block_left: rotate the red block to the left.
- rotate_blue_block_right: rotate the blue block to the right.
- rotate_blue_block_left: rotate the blue block to the left.
- rotate_pink_block_right: rotate the pink block to the right.
- rotate_pink_block_left: rotate the pink block to the left.
- push_red_block_right: push the red block to the right.
- push_red_block_left: push the red block to the left.
- push_blue_block_right: push the blue block to the right.
- push_blue_block_left: push the blue block to the left.
- push_pink_block_right: push the pink block to the right.
- push_pink_block_left: push the pink block to the left.
- move_slider_left: move the slider to the left.
- move_slider_right: move the slider to the right.
- open_drawer: open the drawer.
- close_drawer: close the drawer.
- turn_on_lightbulb: turn on the lightbulb.
- turn_off_lightbulb: turn off the lightbulb.
- turn_on_led: turn on the led.
- turn_off_led: turn off the led.
- done: the goal has reached.
You are only allowed to use the provided skills. You can first itemize the task-related objects to help you plan.
For the actions you choose, list them as a list in the following format.
<code>
[’turn_off_led’, ’open_drawer’, ’done’]
</code>
Listing 10: Follow-Up Prompt for Robot-VILA.
This image displays a scenario after you have executed some steps from the plan generated earlier. When interacting with people, sometimes the robotic arm needs to wait for the person’s action. If you do not find the target object in the current image, you need to continue searching elsewhere. Continue to generate the plan given the updated environment state.
Listing 11: Prompt for Text2Motion.
**Objective:**
You are a helpful agent in helping a robot plan a sequence of actions to accomplish a given task.
I will first provide context and then provide an example of how to perform the task.
**Task Environment:**
In the robot’s environment, there are the following objects:
- A table. Objects can be placed on the table.
- A drawer that can be opened. Objects can be placed into the drawer when it is open.
- A slider which is a cabinet with a sliding door. The sliding door can be moved to the left or to the right. Objects can be placed into the slider no matter the position of the sliding door.
- A lightbulb that be can turned on/off with a button.
- A led that can be turned on/off with a button.
- Three blocks that can be rotated, pushed, lifted, and placed.
**Predicates for symbolic state:**
The list of all possible predicates for defining the symbolic state are listed below:
- (rotated-left ?x - item): ?x is rotated left. This predicate applies to a block.
- (rotated-right ?x - item): ?x is rotated right. This predicate applies to a block.
- (pushed-left ?x - item): ?x is pushed left. This predicate applies to a block.
- (pushed-right ?x - item): ?x is pushed right. This predicate applies to a block.
- (lifted ?x - item): ?x is lifted in the air. This predicate applies to a block.
- (is-open ?x - item): ?x is open. This predicate applies to a drawer.
- (is-close ?x - item): ?x is close. This predicate applies to a drawer.
- (is-turned-on ?x - item): ?x is turned on. This predicate applies to a lightbulb or a led.
- (is-turned-off ?x - item): ?x is turned off. This predicate applies to a lightbulb or a led.
- (is-slider-left ?x - item): the sliding door of the slider ?x is on the left.
- (is-slider-right ?x - item): the sliding door of the slider ?x is on the right.
- (is-on ?x - item ?y - item): ?x is on top of ?y. This predicate applies when ?x is a block and ?y is a table.
- (is-in ?x - item ?y - item): ?x is inside of ?y. This predicate applies when ?x is a block and ?y is a drawer or a slider.
- (stacked ?x - item ?y - item): ?x is stacked on top of ?y. This predicate applies when ?x and ?y are blocks.
- (unstacked ?x - item ?y - item): ?x is unstacked from ?y. This predicate applies when ?x and ?y are blocks.
**Actions:**
There are the following actions that the robot can perform. They are:
- lift_red_block_table: lift the red block from the table.
- lift_red_block_slider: lift the red block from the slider.
- lift_red_block_drawer: lift the red block from the drawer.
- lift_blue_block_table: lift the blue block from the table.
- lift_blue_block_slider: lift the blue block from the slider.
- lift_blue_block_drawer: lift the blue block from the drawer.
- lift_pink_block_table: lift the pink block from the table.
- lift_pink_block_slider: lift the pink block from the slider.
- lift_pink_block_drawer: lift the pink block from the drawer.
- stack_block: stack the blocks.
- place_in_slider: place the block in the slider.
- place_in_drawer: place the block in the drawer.
- place_on_table: place the block on the table.
- rotate_red_block_right: rotate the red block to the right.
- rotate_red_block_left: rotate the red block to the left.
- rotate_blue_block_right: rotate the blue block to the right.
- rotate_blue_block_left: rotate the blue block to the left.
- rotate_pink_block_right: rotate the pink block to the right.
- rotate_pink_block_left: rotate the pink block to the left.
- push_red_block_right: push the red block to the right.
- push_red_block_left: push the red block to the left.
- push_blue_block_right: push the blue block to the right.
- push_blue_block_left: push the blue block to the left.
- push_pink_block_right: push the pink block to the right.
- push_pink_block_left: push the pink block to the left.
- move_slider_left: move the slider to the left.
- move_slider_right: move the slider to the right.
- open_drawer: open the drawer.
- close_drawer: close the drawer.
- turn_on_lightbulb: turn on the lightbulb.
- turn_off_lightbulb: turn off the lightbulb.
- turn_on_led: turn on the led.
- turn_off_led: turn off the led.
**Example Task:**
Now, you will help to parse the goal predicate and generate a sequence of actions to accomplish this task.
Goal predicate: (is-turned-off led)
Symbolic state: is-turned-on(led), is-turned-on(lightbulb), not(is-turned-off(led)), not(is-turned-off(lightbulb))
Task output:
‘‘‘python
[’turn_off_led’]
‘‘‘
**Example Task:**
Goal predicate: (is-turned-on led)
Symbolic state: is-turned-on(led), is-turned-on(lightbulb), not(is-turned-off(led)), not(is-turned-off(lightbulb))
Task output:
‘‘‘python
[]
‘‘‘
**Example Task:**
Goal predicate: (is-in red_block drawer)
Symbolic state: not(is-in(red_block, drawer)), not(is-in(red_block, slider)), is-on(red_block, table), not(is-open(drawer)), is-close(drawer), is-slider-left(slider), not(is-slider-right(slider)), not(lifted(red_block))
Task output:
‘‘‘python
[’open_drawer’, ’lift_red_block_table’, ’place_in_drawer’]
‘‘‘
**Example Task:**
Goal predicate: (is-in red_block drawer)
Symbolic state: not(is-in(red_block, drawer)), not(is-in(red_block, slider)), not(is-on(red_block, table)), is-open(drawer), not(is-close(drawer)), is-slider-left(slider), not(is-slider-right(slider)), lifted(red_block)
Task output:
‘‘‘python
[’place_in_drawer’]
‘‘‘
**Example Task:**
Goal predicate: (and (is-turned-on lightbulb) (is-slider-right slider))
Symbolic state: is-slider-left(slider), not(is-slider-right(slider)), is-turned-off(lightbulb), not(is-turned-on(lightbulb))
Task output:
‘‘‘python
[’turn_on_lightbulb’, ’move_slider_right’]
‘‘‘
**Additional Instructions:**
1. Make sure the generated plan is a list of actions. Place the list between ‘‘‘python and ends with ‘‘‘.
2. Think Step-by-Step.
Concept demonstration

Open image ↗