Overview
The retained overview figure illustrates how language-annotated demonstrations become symbolic behavior descriptions, visual state classifiers, and executable policies. The appendices include complete prompts, operator definitions, algorithms, and evaluation examples.
At a glance
BLADE combines learned robot skills with symbolic planning. Language models propose each skill’s preconditions and effects from annotated demonstrations and contact sequences; these descriptions generate supervision for visual state classifiers. A planner composes the skills and replans after execution. Simulation and physical manipulation tests examine new goals, changed states, obstacles, and hidden objects.
1 Introduction
Useful long-horizon skills need more than executable policies: a planner must know when a behavior is applicable and what it changes. BLADE learns these dependencies from language and demonstrations, then grounds the resulting predicates in perception. This separates reusable low-level behavior from task-level composition.
2 Related Work
The approach combines imitation, symbolic action-model learning, and language-model-assisted planning. Unlike direct language-model action generation, planning searches an explicit transition model. Unlike fully hand-written symbolic systems, action dependencies and classifier labels are inferred from demonstrations, although the main experiments still provide predicate names.
3 Problem Formulation
Inputs are demonstrations segmented and annotated with a finite behavior vocabulary. Each behavior has arguments, binary predicate preconditions and effects, and a learned controller. Humans provide the predicate names in the main implementation; a later appendix explores generating them automatically. The abstract action semantics describe intended effects, not guarantees that every learned controller will succeed.
4 Behavior from Language and Demonstration
The pipeline generates behavior descriptions, trains predicate classifiers and policies, and then uses symbolic planning with repeated execution feedback.
4.1 Behavior Description Learning
Gripper state and contact information segment demonstrations into seven primitive types: open, close, move-to, grasp, place, move, and push. A language model proposes a PDDL-style operator and its primitive sequence using behavior labels, neighboring behaviors, scene context, and one example. Abstract verification checks consistency across demonstrations and resamples invalid definitions. The inferred primitive sequence also corrects noisy demonstration boundaries.
4.2 Classifier and Policy Learning
Operator preconditions, effects, and persistence create positive and negative predicate labels across demonstration frames, rather than using only segment endpoints. Classifiers learn the resulting visual predicates; diffusion policies learn each behavior. Real-world classifiers focus on relevant objects to reduce spurious correlations, while the policies use multiple camera views.
4.3 Bi-Level Planning and Execution
A language model translates the requested goal to a logical formula. The planner searches abstract states, executes the first behavior, observes again, and replans. Geometric feasibility is checked just before an action, and hidden objects are handled through optimistic state assumptions and subsequent replanning. This is not full belief-space planning.
5 Experiments
Experiments compare composition and state estimation in simulation and in two physical manipulation domains.
5.1 Simulation Experimental Setup
Simulation uses CALVIN split D: about six hours of play data and 34 behavior types. Six new tasks cover abstract goals, geometric constraints, and partial observability. Each task uses twenty initial states and three seeds. Baselines include HULC, SayCan, Robot-VILA, and Text2Motion shooting, with both learned and ground-truth state variants where applicable.
5.2 Results in Simulation
With learned classifiers, BLADE reports success rates of 68.33% on abstract goals, 26.67% on geometric constraints, and 75.83% on partial observability. HULC reaches 2.78% on abstract goals—a 65.55 percentage-point gap, rather than a 65% relative improvement. The geometric setting remains difficult despite outperforming the tested baselines. Ground-truth state information improves some categories but does not uniformly improve every result. Propagating predicate labels across trajectories outperforms the endpoint-only labeling comparison; the paper reports a 20.7% F1 improvement, with the full planning ablation retained in Table 2.
5.3 Real World Experiments
A Franka arm and five cameras perform Boil Water and Make Tea. Each domain has 85 teleoperated demonstrations. Tests include the demonstrated sequences and seven listed generalization cases across initial-state changes, human perturbations, blocking objects, and hidden objects. Each case has six trials; the paper reports at least four successful trials in each generalization condition. Robot-VILA often proposes actions inconsistent with the current scene. These small controlled experiments do not establish broad household reliability.
6 Conclusion and Discussion
Explicit behavior models and grounded predicates improve composition in the tested domains. Limitations include gripper-based segmentation, supplied predicate names in the main system, classifier errors, and low-level policy failures in unfamiliar states. Abstract verification checks consistency with the demonstrations, not universal correctness of the learned action model.
Acknowledgments
The authors acknowledge research sponsors, collaborating laboratories, colleagues, and reviewers.
Appendix A blade Details
The appendix provides the operator-generation prompts, labeling algorithms, model architectures, and planning assumptions.
A.1 Behavior Description Generation with LLMs
Listings 1–6 preserve the generated CALVIN operators, primitive definitions, predicate vocabulary and environment context, an example, reasoning instructions, and a sample behavior query. Scene context is important; replacing it with automatically generated scene descriptions is suggested rather than part of the main comparison.
A.2 Predicate Generation with LLMs
An additional language-model procedure proposes predicates, grounds their arguments, and removes redundant names. Across three domains, 28 of 30 predicates closely match expert-designed ones; left/right pushing is misrepresented by a next-to relation. The authors check that the generated real-world action descriptions can produce correct plans for the nine tested tasks, which is distinct from rerunning all physical executions with automatically invented predicates.
| Manually Defined | Automatically Generated |
| rotated-left() | rotated-left() |
| rotated-right() | rotated-right() |
| lifted() | holding() |
| is-open() | is-open() |
| is-close() | is-closed() |
| is-turned-on() | light-on() |
| is-turned-off() | light-off() |
| is-slider-left() | slider-left() |
| is-slider-right() | slider-right() |
| is-on() | on-table() |
| is-in() | in-slider(), in-drawer() |
| stacked() | on() |
| unstacked() | clear() |
| pushed-left() | - |
| pushed-right() | - |
| - | next-to() |
| Manually Defined | Automatically Generated | Automatically Generated (Grounded) |
| is-placed-on() | on-table() | on-table(kettle), on-table(pot) |
| is-placed-in() | in-sink() | in-sink(kettle), in-sink(pot) |
| is-blocked() | on-stove() | on-stove(pot) |
| is-turned-away() | faucet-over-sink() | faucet-over-sink(faucet, sink) |
| is-aligned() | faucet-over-sink() | faucet-over-sink(faucet, sink) |
| is-turned-on() | faucet-on() | faucet-on(faucet) |
| is-turned-off() | faucet-off() | faucet-off(faucet) |
| is-filled() | filled() | filled(kettle), filled(pot) |
| holding() | holding() | holding(pot) |
| Manually Defined | Automatically Generated | Automatically Generated (Grounded) |
| is-placed-on() | is-on() | is-on(kettle, stove) |
| is-cabinet-door-open() | is-open() | is-open(left-door), is-open(right-door) |
| is-placed-inside() | is-in-drawer() | is-in-drawer(teabag) |
| is-in-kettle() | is-in-kettle(teabag) | |
| is-in-cabinet-left() | is-in-cabinet-left(kettle) | |
| is-in-cabinet-right() | is-in-cabinet-right(kettle) | |
| is-drawer-open() | is-open() | is-open(drawer) |
| is-left-cabinet-door-blocked() | is-blocking() | is-blocking(pot, left-door) |
| is-right-cabinet-door-blocked() | is-blocking() | is-blocking(pot, right-door) |
| - | is-closed() | is-closed(left-door), is-closed(right-door) |
| is-closed(drawer) | ||
| - | is-moved-away() | is-moved-away(pot) |
A.3 Temporal Segmentation
Segmentation assumes demonstrators move freely with fully open grippers and push with fully closed grippers. Simulator contacts provide arguments in simulation; real-world arguments come from language annotations. Re-segmentation with the inferred operator body removes irrelevant actions from noisy labels. These instructed demonstration conventions and contact cues are substantive inputs.
A.4 Abstract Verification
Algorithm 1 checks whether accumulated effects can satisfy each demonstrated behavior’s preconditions and whether the primitive sequence is consistent. Descriptions failing this abstract check are regenerated.
A.5 Automatic Predicate Annotation
Algorithm 2 initializes predicates as unknown, infers values from preconditions and effects, and propagates them until a later action changes them. It supplies both positive and negative examples without separately annotated state labels.
A.6 Classifier Implementation
Simulation classifiers use frozen CLIP patch features with learned predicate heads and mask the arm to reduce overfitting. Physical classifiers instead use Grounding-DINO detections and segmented 3D point clouds with a Point Cloud Transformer. Substituting GPT-4o as the classifier yields 0% success on the tested Boil Water generalization cases. Adding automatically labeled visual examples improves its classification accuracy by about 5% on 220 held-out examples in a preliminary test.
A.7 Policy Implementation
Skill training randomizes initial observation histories, balances starting states, and appends zero actions to teach termination. Simulation policies use segmented object point clouds with ground-truth simulator masks. Physical policies use four stationary views plus a wrist camera, separate image encoders, and diffusion-based gripper-pose prediction.
A.8 Planner Implementation
The planner first satisfies semantic prerequisites, then checks immediate geometric prerequisites and recursively moves obstructions if needed. This assumes enough free space for rearrangement and does not predict exact future object poses. Hidden-object search uses deterministic optimistic effects and retries after observations; richer uncertainty and tightly packed rearrangement need additional models.
Appendix B Simulation Experiment Details
Evaluation retains CALVIN’s original language annotations, skills, action space, and environments while adding task-level generalization tests.
B.1 Task Design
The six simulation cases turn off both lights; put all blocks in an initially closed drawer; put them in an open drawer; retrieve a hidden block from a closed drawer; retrieve one from behind a slider; and open a slider blocked by another object. Random initial states and object/color variants distinguish composition from replaying one fixed trajectory.
B.2 Baseline Implementation
SayCan uses object prompts and demonstration-trained feasibility classifiers rather than reinforcement-learned value functions. Robot-VILA uses GPT-4 Turbo with accumulated dialogue and images. Text2Motion uses a longer-horizon feasibility model and symbolic state descriptions, comparing learned and ground-truth states. These adaptations define what the reported baseline results measure.
Appendix C Qualitative Examples
The retained qualitative examples show complete plans and execution traces for abstract goals, hidden objects, and obstructed motion. They illustrate particular successes and baseline failures rather than adding independent aggregate evaluations.
Appendix D Real World Experiment Details
The physical setup records end-effector pose, gripper width, and five RGB-D camera streams during SpaceMouse teleoperation. Each of the two domains contributes 85 demonstrations.
D.1 Task Design
Seven generalization cases are listed: three Boil Water cases change the starting state, move the kettle mid-task, or block the stove; four Make Tea cases change the starting arrangement, reclose a drawer, block cabinet doors, or hide the kettle. Logical goals require the final object relations and, for Boil Water, a turned-off faucet. The retained figures show each setting.
D.2 Qualitative Examples of Novel States
Visual comparisons indicate that more than half of the illustrated initial or perturbed states were absent from demonstrations. This supports novelty within the constructed test cases, without proving that all underlying low-level motions are unseen.
Appendix E Prompts for Baselines
Listings 8–11 retain the full SayCan, Robot-VILA initial and follow-up, and Text2Motion shooting prompts. Together with the earlier operator-generation listings, these specify the model inputs used in the comparisons.