Literature
Display
← All papers

Learning Planning Abstractions from Language

Weiyu Liu · Geng Chen · Joy Hsu · Jiayuan Mao · Jiajun Wu

· ICLR 2024 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

PARL learns a discrete action vocabulary from language and a latent dynamics model from segmented demonstrations, then searches for abstract actions that make a requested final action feasible.

At a glance

Planning Abstraction from Language combines language parsing, object-centered latent states, learned action feasibility/transitions, and low-level execution. Tests in grid worlds and simulated kitchens examine longer plans, new scenes, and withheld color–object combinations. Planning is substantially stronger with oracle low-level execution than with the learned controller. The method assumes segmented language-aligned trajectories, success/failure examples, and structured or instance-segmented observations; it does not demonstrate unrestricted language learning or physical robot deployment.

1 Introduction

Symbolic action terms make search manageable, while a latent state can represent geometric constraints without manually naming every predicate. PARL parses language into verbs and object attributes, learns how these actions change latent object states and when they are feasible, and plans at test time. This separates task-level search from short-horizon execution. The demonstrated generalization concerns recombinations of known concepts and new layouts, rather than recognizing arbitrary unseen object concepts.

Refer to caption
Figure 1: The overview of our training and testing paradigm, and different types of generalizations supported by our framework. (a) Given paired demonstration trajectories and language descriptions, our framework discovers an abstract action space and a latent state abstraction that supports planning for diverse language goals. (b) The example illustrates that our model can generalize to a new kitchen environment, generalize to a goal that requires reasoning about the geometries of the sink and grey pan, and generalize to the combination of concepts red and pan that is unseen during training.

2 Related Work

Language-guided hierarchical policies use instructions to structure skills; model-based methods learn transitions for planning. PARL combines both by discovering discrete action concepts from instructions and learning planning-compatible latent transitions. Its abstraction differs from systems that explicitly learn logical state predicates, such as Konidaris and colleagues, and from continuous-parameter task-and-motion search. The low-level executor handles refinement after discrete abstract planning.

3 Problem Formulation

The formulation assumes full observability, object-centered raw states, and instance segmentation. Training demonstrations are already segmented so language actions align with trajectory intervals, and failures label action infeasibility. Test instructions specify a final action, such as placing a bowl in a crowded sink; the planner must insert prerequisite actions. Four learned components provide state encoding, abstract transitions, feasibility, and low-level execution. Successful parsing and usable object instances are supplied assumptions, not outputs of a wholly unsupervised pipeline.

4 Planning Abstraction from Language

A pretrained language model first extracts action and object symbols. Demonstrations then ground those symbols in learned state, transition, and feasibility models. At inference, a parsed goal action defines the end of a searched abstract plan, which is executed one step at a time with replanning.

Refer to caption
Figure 2: The overall framework of PARL. PARL takes paired natural language instructions and demonstration trajectories as inputs. It recovers object-level concepts such as shapes and colors, and action concepts from the natural language. It then learns a planning-compatible model for object and action concepts. At test time, given novel instructions, it performs a combined high-level planning and low-level policy unrolling to output the next action to take.
Refer to caption
Figure 3: PARL prompts a pretrained large language model (LLM) to parse instructions into symbolic formulas. Next, we extract the object-level and action concepts from the formulas.

4.1 Symbol Discovery

Instructions are decomposed into a verb and ordered object arguments, each described by attribute symbols. Recombining a finite vocabulary makes candidate actions enumerable. The GPT-4 prompt provides demonstrations of the desired representation, so discovery is guided by a fixed symbolic schema and examples rather than inventing arbitrary executable operators or formal preconditions from nothing.

4.2 Planning-Compatible Model Learning

In three dimensions, a Point Cloud Transformer independently encodes each segmented colored object cloud. A transformer reads object latents and action-symbol tokens, predicts next object latents, and uses an extra query token to classify feasibility. This can represent whether a placement fits without an explicit hand-coded “space available” predicate.

Training samples suffixes of successful and failed abstract trajectories, rolls the latent transition model forward, and combines latent-state prediction error with feasibility supervision. The main formula uses binary cross-entropy, while the implementation table specifies focal loss with gamma two; both original descriptions are retained. The feasibility-label prose also has an endpoint indexing inconsistency relative to the displayed sum. The operational intent is to label successful steps feasible and a failed final step infeasible.

Refer to caption
Figure 4: Neural network architectures for our planning-compatible models, composed of (a) an object-level PCT encoder for extracting state abstractions and (b) an abstract transition Transformer for abstract transition and the feasibility prediction.
Equation
ℒ=∑i=k+1ℓ‖si′−ϕ⁡(si)‖2+∑i=kℓ−1BCE​(feas¯i,feasi),{\mathcal{L}}=\sum_{i=k+1}^{\ell}\|s^{\prime}_{i}-\phi(s_{i})\|_{2}+\sum_{i=k}^{\ell-1}\text{BCE}(\overline{\textit{feas}}_{i},\textit{feas}_{i}),

4.3 Planning with Learned Abstractions

Search expands possible abstract actions to a fixed horizon using predicted transitions and feasibility. A sequence is scored by its lowest step-feasibility prediction, not a calibrated product of success probabilities. Pruning retains promising sequences, making this a beam-like search rather than exhaustive breadth-first search with completeness guarantees. The selected sequence ends in the requested action. After each executed abstract action, the system re-encodes the observed state and replans.

4.4 Low-Level Policy

In BabyAI, a FiLM-conditioned network and actor–critic learning produce primitive navigation/interactions. In the kitchen, the low-level model selects an object to pick or a supporting object/location on which to place the held item. It does not directly learn arbitrary robot joint trajectories; simulator pick/place machinery handles refinement. Results must therefore distinguish abstract planning from the capabilities of the supplied executor.

5 Experiments

Evaluation uses BabyAI and PyBullet-based Kitchen-Worlds, with one hundred test instances per reported setting. The paper reports success proportions without multi-seed confidence intervals. No physical deployment experiment is included.

5.1 BabyAI

Grid-world tasks require finding keys, unlocking intermediate doors, retrieving varied objects, or choosing the correct disconnected corridor. Training goals need at most three abstract actions, while long-horizon tests need at least five. A separate test withholds the red-key combination.

With a learned controller, PARL succeeds on 45%, 39%, and 52% of the three long-horizon settings and 27% of the held-out-combination setting. With oracle execution, these become 87%, 89%, 94%, and 91%. High-level actor–critic reaches 27%, 0%, 44%, and 0%; cloning and end-to-end baselines score zero in the table. The large oracle gap identifies low-level execution as a major bottleneck, and the learned system’s 27% compositional score should not be described as a negligible generalization drop.

Removing replanning reduces oracle-controller All-Objects success from 89% to 65%; replacing factorwise FiLM conditioning with a multilayer perceptron reduces it to zero. The appendix also gives the end-to-end reinforcement baseline a partial 7-by-7 view while high-level models use a 13-by-13 full map, limiting an architecture-only interpretation of that baseline comparison.

Longer Steps Novel Concept Combinations
Key-Door All-Objects Two-Corridors All-Objects
End-to-End BC 0.00 0.00 0.00 0.00
End-to-End A2C 0.00 0.00 0.00 0.00
Ours (high + low) 0.45 0.39 0.52 0.27
High-Level BC 0.00 0.00 0.00 0.00
High-Level A2C 0.27 0.00 0.44 0.00
Ours (high only) 0.87 0.89 0.94 0.91
Table 1: Success rate of different models on BabyAI. We compare with behavior cloning and A2C baselines on two settings: (End-to-End) directly predicting low-level actions, and (High-Level) predicting high-level action tokens with oracle low-level policies.
Refer to caption
Figure 5: We evaluate our models on diverse task settings created in the BabyAI environments.
Novel Environments   Novel Concept Combinations
All-Objects Sink All-Objects
Goal-Conditioned BC 0.82 0.39 0.14
Ours 0.86 0.61 0.57
Table 2: Success rate of different models on Kitchen-Worlds. We compare Goal-Conditioned BC with our model on generalization to novel concept combinations and environments.

5.2 Kitchen-Worlds

Six simulated cameras and provided instance masks produce object point clouds. Kitchen tasks either place an object at a target or first clear a crowded sink. New layouts/assets and a withheld red-bowl combination test grounding and composition. PARL reaches 86% versus 82% for goal-conditioned cloning on new-environment All-Objects, 61% versus 39% on Sink, and 57% versus 14% on the new-combination test.

Both systems replan, but PARL learns predicted feasibility and transitions rather than directly imitating the next action. A scene-level FiLM ablation loses six percentage points on Sink. These results support object-centered latent planning under the available geometry and execution primitives, without establishing open-world perception or guaranteed geometric reasoning.

Refer to caption
Figure 6: The kitchen environments with 3D furniture and objects.
Refer to caption
Figure 7: Given the language goal of placing blue mug in the sink, our model can reason about the geometry of the environment and perform planning with learned abstract latent space to predict the sequence of abstract actions. Each abstract action is expressed as a verb term and object arguments. For example, (pick-up, blue, pan, sink) corresponds to picking up the blue pan from the sink.

6 Conclusion

Language-derived action factors and learned feasibility/dynamics support useful abstract search and several tested generalizations. Current limitations include presegmented aligned trajectories, complete parseable instructions, provided object segmentation, and a finite learned concept vocabulary. Automatic segmentation and pretrained visual-language grounding are proposed extensions, not demonstrated capabilities.

Acknowledgments

The acknowledgments list United States research-agency and industry funding, including NSF, ONR, Samsung, and Analog Devices. The original funding acronym and award identifiers remain in the source.

Appendix A PARL

The appendix documents the GPT-4 parsing prompt and the distinct grid-world and point-cloud architectures used for abstraction, transition, and feasibility learning.

A.1 Prompts for LLMs

The original colored prompt listing is preserved with exact indentation and all four examples. One example instructs placement into the sink but its displayed symbolic output says cabinet; this source inconsistency is not silently corrected. The examples also use varying argument counts, while the kitchen implementation table lists a fixed four-token action length. The listing illustrates the parsing convention rather than a comprehensive validated language grammar.

Original GPT-4 parsing prompt, preserving gray context, green examples, purple outputs, and code indentation.
Listing 1: Example GPT-4 prompt to map natural language instructions to symbolic formulas that our model consumes. Prompt context is in gray, input language descriptions are green, and generated word tokens are purple.

A.2 Model Architecture for BabyAI

BabyAI represents each 13-by-13 cell with integer object, color, and state indices, so it is structured input rather than natural-image perception. A convolutional encoder yields a latent feature map. One FiLM modulation per action-symbol token conditions transition and feasibility predictions; the latter ends in a two-layer classifier. Symbol-factor conditioning supplies the intended compositional structure.

A.3 Model Architecture for Kitchen-Worlds

Kitchen objects use 512 colored points each and 352-dimensional point features. Word features share that width; sixteen-dimensional position and type embeddings yield a 384-dimensional transformer input. A twelve-layer, twelve-head transformer supports up to eleven objects, six-step plans, and a twenty-three-token vocabulary in the reported configuration. Feasibility reads a query output; transition prediction masks the query and retains object outputs. Padding and masks handle variable lengths within configured limits. Training uses Adam, batch thirty-two, and two thousand epochs with warmup and cosine decay.

Equation
ei\displaystyle e_{i} =[hi;hp​o​s​(i);ht​y​p​e​(0)],\displaystyle=[h_{i};h_{pos}(i);h_{type}(0)],
ci\displaystyle c_{i} =[hw​(wi);hp​o​s​(i−N);ht​y​p​e​(1)],\displaystyle=[h_{w}(w_{i});h_{pos}(i-N);h_{type}(1)],
q\displaystyle q =[hq;hp​o​s​(i−N−K);ht​y​p​e​(2)],\displaystyle=[h_{q};h_{pos}(i-N-K);h_{type}(2)],
Parameter Value
Number of points for object point cloud MM 512
Max plan lengths 6
Max number of objects Nm​a​xN_{max} 11
Length of factorized action KK 4
PCT point cloud encoder hxh_{x} out dim 352
Word embedding vocab size 23
Word embedding hwh_{w} learned embedding
Word embedding hwh_{w} dim 352
Position embedding hp​o​sh_{pos} learned embedding
Position embedding hp​o​sh_{pos} dim 16
Type embedding ht​y​p​eh_{type} learned embedding
Type embedding ht​y​p​eh_{type} dim 16
Transformer number of layers 12
Transformer number of heads 12
Transformer hidden dim 384
Transformer dropout 0.1
Transformer activation ReLU
Loss Focal (γ=2\gamma=2)
Epochs 2000
Optimizer Adam
Learning rate 1e-4
Gradient clip value 1.0
Batch size 32
Learing rate warmup Linear (Te​n​d=20T_{end}=20)
Learning rate decay Cosine annealing (Tm​a​x=2000T_{max}=2000)
Table 3: Model Parameters

Appendix B Baselines

The baselines compare end-to-end or abstract-action policies against explicit model learning and search. Differences in observation and execution access matter when interpreting their scores.

B.1 RL Models for BabyAI

The end-to-end actor–critic observes a partial egocentric 7-by-7 map and chooses among seven primitive commands. The high-level version observes the full 13-by-13-by-three state and selects an abstract action, executed by an oracle controller. Both receive goal-reaching rewards adjusted for primitive-step count. Thus the high-level comparison is the closer test of planning versus direct abstract-policy learning.

B.2 Goal-Conditioned Behavior Cloning for Kitchen-Worlds

Goal-conditioned kitchen cloning shares PARL’s point-cloud encoding, tokenization, and auxiliary embeddings. A transformer encoder reads scene and goal; an autoregressive decoder predicts the next abstract action. Hindsight-style goal relabeling augments training, and execution replans after each step. Its difference from PARL is direct action prediction rather than a learned transition/feasibility model used in search.

Appendix C Environments

Dataset construction randomizes scene layouts and records successful and failed interactions. Task budgets and available concepts constrain what the generalization tests measure.

C.1 BabyAI

Each BabyAI setting has one hundred thousand demonstrations generated with a high-level task planner and oracle controller. Nine-room layouts vary locked doors, matching keys, and distractors. Key-Door and All-Objects use three keys/doors; Two-Corridors uses two disconnected three-door chains and six keys. Evaluation requires reaching the goal within two hundred primitive steps.

Type Value
Verb Pick-up, Open
Object Key, Door, Ball, Box
Color Red, Green, Blue, Yellow, Grey, Purple
Table 4: BabyAI Concepts

C.2 Kitchen-Worlds

Kitchen All-Objects contains interactions from 3,920 scenes and Sink from 3,360 scenes, with up to four movable objects. Each scene contributes three six-action feasible trajectories plus sampled failure attempts; Sink biases half the choices toward sink interactions. Evaluation allows five abstract actions. Optimal plans use two actions for simple placement and up to four for Sink, so these kitchen tests emphasize geometry and composition rather than exceeding the six-action training trajectory length.

Type Value
Verb Pick, Place
Object Bowl, Mug, Pan, Medicine Bottle, Water Bottle, Food
Location Left Counter, Right Counter, Top Shelf, Sink, Cabinet
Color Red, Green, Blue, Yellow, Grey, Brown
Table 5: Kitchen-Worlds Concepts

Appendix D Additional Results

Additional results show concrete grid-world plans and feasibility scores at progressively farther latent predictions. Planning success and feasibility-classification accuracy remain different metrics.

D.1 BabyAI Qualitative Examples

The retained plan diagrams show prerequisite key/door sequences and the final requested action across all three grid-world settings. They illustrate the planner’s output structure alongside the aggregate success table.

Refer to caption
Figure 8: Plan predictions for different task settings. Each figure visualizes the initial state of the environment and the location of the agent (represented by the red arrow). On the right of each figure, the sequence of planned abstract actions is listed. The last abstract action in each list is the goal abstract action in the given language instruction.

D.2 Feasibility Prediction

On new environments, average feasibility F1 is 98.17% for All-Objects and 98.01% for Sink. Withheld concept combinations average 90.08%, but individual reported horizons range from 87.67% to 99.01%; not every step exceeds ninety percent. These classification results do not imply equally high end-to-end task success, as the kitchen success table and controller ablations show.

Task Setting Generalization Train Test
Avg. Avg. kk k+1k+1 k+2k+2 k+3k+3 k+4k+4 k+5k+5
All-Objects New Env. 99.70 98.17 99.00 98.51 97.79 97.32 96.45 98.89
Sink New Env. 99.84 98.01 98.13 98.55 98.10 97.38 97.63 97.01
All-Objects New Concept Comb. 91.97 90.08 88.74 90.36 87.67 90.24 88.68 99.01
Table 6: Feasibility prediction F1 scores for Kitchen-Worlds environments. For each setting, the scores on the training data and testing data are presented. For the testing data, we further present the feasibility prediction performance for abstract actions at different future time step aia_{i} given only current environment state sks_{k}, for i=k,⋯,k+5i=k,\cdots,k+5.

Appendix E Limitations

The method assumes provided 3D instance segmentation, a fixed symbolic language format, complete instructions, and language-aligned segmented demonstrations. Expanding concepts would require additional grounding mechanisms beyond the learned finite vocabulary. Suggested directions include object discovery, richer instruction handling, automatic trajectory segmentation, and pretrained vision–language representations.

Concept demonstration

Open image ↗