Literature
Display
← All papers

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

Manling Li · Shiyu Zhao · Qineng Wang · Kangrui Wang · Yu Zhou · Sanjana Srivastava · Cem Gokmen · Tony Lee · Li Erran Li · Ruohan Zhang · Weiyu Liu · Percy Liang · Li Fei-Fei · Jiayuan Mao · Jiajun Wu

· NeurIPS 2024 Datasets and Benchmarks Track ↗

AI agents25 min text · CondensedOriginal paper ↗
Contents

Overview

Refer to caption
Figure 1: Embodied Agent Interface unifies a broad set of tasks involving both state and temporally extended goals and four LLM-based modules for decision-making.
[Uncaptioned image]

At a glance

Embodied Agent Interface standardizes four language-model roles in household decision making: interpreting goals, decomposing them into subgoals, sequencing actions, and predicting action preconditions/effects. A shared object-centric temporal-logic representation connects these roles and supports diagnosis beyond final success. Eighteen contemporary models are evaluated in symbolic versions of BEHAVIOR and VirtualHome. The benchmark exposes missing prerequisites, redundant actions, relational errors, and mismatches between a planner’s internal model and the environment. Its primary scope is reasoning over supplied symbolic state, rather than end-to-end robot perception and control.

1 Introduction

Previous systems ask language models to produce different artifacts under different assumptions, so their success rates are difficult to compare. This work makes goal specifications, module boundaries, and error categories explicit. The resulting interface can evaluate one model component while holding the others fixed, or compose components into a pipeline. The central empirical finding is that coherent language and valid syntax are insufficient: models often omit necessary physical relationships, confuse intermediate states with final goals, or issue actions whose prerequisites are false. Model rankings vary across simulator and ability; aggregate scores should not replace the module-level results.

Refer to caption
Figure 2: The input and output formulation of four ability modules.
Refer to caption
Figure 3: Embodied Agent Interface supports a collection of fine-grained metrics and provides automatic toolkits for error analysis and benchmarking different LLMs on various embodied decision-making tasks.

2 Embodied Agent Interface Based on LTL

The representation separates a finite object universe, Boolean relational state, parameterized actions, task goals, subgoals, and trajectories. Four typed input/output interfaces sit above this shared representation. The notation table is retained in full.

Table 1: Summary of notations used in Embodied Agent Interface.
Notation Symbol

Description

Environment Representations

Object u∈𝒰u\in{\mathcal{U}}

An object, which has relational features ff

State s=⟨𝒰,ℱ⟩∈𝒮s=\langle{\mathcal{U}},{\mathcal{F}}\rangle\in{\mathcal{S}}

A tuple of the universe of objects and relational features

Action a=⟨name,args⟩∈𝒜a=\langle\textit{name},\textit{args}\rangle\in{\mathcal{A}}

A tuple of the action name and arguments

Operator o=⟨name,vars⟩∈𝒪o=\langle\textit{name},\textit{vars}\rangle\in{\mathcal{O}}

An action schema: a tuple of the name and a list of parameters. Each oo can be instantiated into an action aa

Transition Model ℳ:𝒮×𝒜→𝒮{\mathcal{M}}:{\mathcal{S}}\times{\mathcal{A}}\rightarrow{\mathcal{S}}

The deterministic transition function of the environment

Natural Language Goal lgl_{g}

A sentence in English

LTL Goal gg

An LTL formula. Here, we only consider formulas containing a sequence of action items and a conjunction of propositions (for the final state): g=a1​ then ​…​ then ​ak​ then ​(p1∧…∧pℓ)g=a_{1}\textbf{~then~}\ldots\textbf{~then~}a_{k}\textbf{~then~}\left(p_{1}\land\ldots\land p_{\ell}\right).

Action Trajectory a¯={ai}i=1n\bar{a}=\{a_{i}\}_{i=1}^{n}

A sequence of nn actions

Subgoal Trajectory ϕ¯={ϕi}i=1m\bar{\phi}=\{\phi_{i}\}_{i=1}^{m}

A sequence of LTL subgoals ϕi\phi_{i} connected by “then”

State-action Trajectory t¯=⟨{si}i=0n,{ai}i=1n⟩\bar{t}=\langle\{s_{i}\}_{i=0}^{n},\{a_{i}\}_{i=1}^{n}\rangle

A sequence of state-action pairs. ∀t.st+1=ℳ⁡(st,at)\forall t.s_{t+1}={\mathcal{M}}(s_{t},a_{t})

Task ⟨s0,g,lg⟩\langle s_{0},g,l_{g}\rangle

A tuple of the initial state and the LTL/Natural Language goals

Abilities

Goal Interpretation 𝒢:⟨s0,lg⟩→g{\mathcal{G}}:\langle s_{0},l_{g}\rangle\rightarrow{g}

Initial State & Natural Language Goal →\rightarrow LTL Goal

Subgoal Decomposition Φ:⟨s0,g⟩→ϕ¯{\Phi}:\langle s_{0},g\rangle\rightarrow\bar{\phi}

Initial State & Goal →\rightarrow Subgoal Trajectory

Action Sequencing 𝒬:⟨s0,g⟩,ℳ→a¯{\mathcal{Q}}:\langle s_{0},g\rangle,{\mathcal{M}}\rightarrow\bar{a}

Initial State & Goal & Transition Model →\rightarrow Action Trajectory

Transition Modeling 𝒯:⟨s0,g⟩,o→⟨pre,eff⟩{\mathcal{T}}:\langle s_{0},g\rangle,o\rightarrow\langle\textit{pre},\textit{eff}\rangle

Initial State & Goal & Operator →\rightarrow Preconditions & Effects

2.1 Representation for Objects, States and Actions

A state contains a fixed finite set of objects and Boolean features over object tuples. Unary features represent object properties; higher-arity features represent relations. Actions are named primitives with ordered object arguments. This abstracts continuous geometry, sensing uncertainty, and motor execution into simulator-recognized symbols.

2.2 Representation for Goals, Subgoals, Action Sequences, and State-Action Trajectories

Goals, subgoals, and ordered action descriptions use a finite-trajectory fragment of linear temporal logic. It permits state/action predicates, Boolean connectives, universal and existential quantifiers, exact-count quantification, and a then operator. A state formula holds if it is true at some trajectory step; then requires a prefix/suffix split with the first formula before the second. Globally and until are not implemented. This distinction matters: the general temporal semantics are not simply a test of the last state, although several benchmark metrics explicitly inspect final-state conditions.

Original figure
Figure 4: The overview of evaluation pipeline for four abilities. For each ability module, to provide a comprehensive evaluation for it, we isolate this single module to be handled by the LLMs while using existing data or tools for the other modules. Note that the pipeline consists of goal interpretation, action sequencing to achieve the goal, and transition modeling that predicts how each action operate the environment’s state. Evaluating subgoal decomposition presents a challenge since it cannot be evaluated directly with no unified annotation strategy. To address this, we employ breadth-first search (BFS) to identify potential action sequences that accomplish each subgoal, allowing us to convert state trajectories into action sequences that can be executed in the simulator (Figure 21 in Appendix). Transition modeling evaluation poses another challenge, we first annotate transition models in PDDL for F1F_{1} evaluation followed with a PDDL planner to validate the feasibility of supporting potential plans. We also conduct a pipeline-based vs modularized analysis, detailed in the Appendix G.

2.3 Ability Module 1: Goal Interpretation 𝒢 : ⟨ s 0 , l g ⟩ → g

Goal interpretation maps an initial state and natural-language instruction into a symbolic goal. Predictions are restricted to an ordered action sequence followed by a conjunction of final-state predicates. Quantified reference goals are grounded into alternative simple goals, and the prediction receives the best set-matching F1 among valid alternatives. The ordered action sequence is treated as a single matching element. This evaluates formal translation, not execution of a plan.

2.4 Ability Module 2: Subgoal Decomposition Φ : ⟨ s 0 , g ⟩ → ϕ ¯

Subgoal decomposition outputs declarative intermediate conditions. A breadth-first planner searches for actions connecting those conditions, after which the same feasibility and goal metrics used for action sequencing apply. Because several action realizations can satisfy one decomposition, the benchmark reports the maximum score over considered realizations. This provides an optimistic assessment of whether the subgoal plan can be grounded by the supplied planner, not the success of a unique directly predicted action sequence.

2.5 Ability Module 3: Action Sequencing 𝒬 : ⟨ s 0 , g ⟩ , ℳ → a ¯

Action sequencing outputs executable primitives given the initial state, goal, and transition information. Simulation tests both whether actions execute and whether the resulting trajectory satisfies the task. Partial credit counts satisfied state conditions and required action subsequences, taking the best valid grounding of quantified goals. Execution can terminate at the first blocking error, so a fluent or partly correct plan need not complete the task.

2.6 Ability Module 4: Transition Modeling 𝒯 : ⟨ s 0 , g ⟩ , o → ⟨ pre , eff ⟩

Transition modeling asks the model to generate PDDL preconditions and effects for supplied operator names and arguments. Relevant operators are selected using the reference trajectory. Evaluation combines structural F1 against expert definitions with whether an external planner finds a plan under the predicted model. The latter considers state goals and omits action subgoals. A plan found under incorrect dynamics need not execute in the real simulator; this metric must be read alongside logical accuracy and error analysis.

3 Dataset Annotations and Benchmark Implementations

The authors annotate goals, task-relevant actions, and transition models for two household simulators: BEHAVIOR [4] and VirtualHome [5]. VirtualHome includes action goals such as touching an object when no persistent state change establishes completion. BEHAVIOR has quantified goals with many valid object assignments: the paper reports 6.7 grounded conditions and 4,164.4 goal options per task on average. A new symbolic EvalGibson layer provides 30 actions for BEHAVIOR, and VirtualHome is revised for evaluation. These benchmark implementations define the action semantics being tested.

Table 2: Simulator dataset statistics. New annotations collected in this paper are highlighted in color.
VirtualHome BEHAVIOR
#task name 26 100
#task instruction 338 100
#goal 801 673
- #state 340 153
- #relation 299 520
- #action 162 -
#trajectory 338 100
- #step 2960 1460
- avg. step 8.76 14.6
#transition model 33 30
- #precondition 99 84
- #effect 57 51

4 Results

Across eighteen models, performance depends on both ability and simulator. The reported BEHAVIOR aggregate favors o1-preview at 74.9%, compared with 64.2% for the next result discussed by the authors; this aggregate combines different metrics and is not a single task-success rate. BEHAVIOR trajectories are longer and harder to execute, while VirtualHome instructions can make goal grounding more ambiguous. Relations are often harder than unary object states. Examples include placing food directly on a table instead of on a plate on the table, and omitting an implicit floor relation. These illustrate disagreement with the benchmark’s physically explicit target annotations, rather than a universal interpretation of every natural-language instruction.

Table 3: Results (%) overview. V: VirtualHome, B: BEHAVIOR. Full results in Appendix E.
Model Goal Interpretation Action Sequencing Subgoal Decomposition Transition Modeling Average Perf.
F1F_{\textsubscript{1}} Task SR Execution SR Task SR Execution SR F1F_{\textsubscript{1}} Planner SR Module SR
V B V B V B V B V B V B V B V B
Claude-3 Haiku 28.0 52.5 54.8 26.0 60.7 32.0 78.4 30.0 82.8 35.0 42.3 51.6 30.4 64.0 49.4 41.6
Claude-3 Sonnet 29.4 69.4 58.0 44.0 63.3 60.0 83.1 39.0 86.4 43.0 41.2 56.2 13.2 80.0 49.4 55.1
Claude-3 Opus 31.4 77.0 64.6 51.0 69.5 59.0 86.7 41.0 89.9 47.0 48.8 63.4 61.8 82.0 59.5 60.4
Claude-3.5 Sonnet 33.0 82.7 76.1 60.0 81.3 69.0 89.1 39.0 92.0 44.0 48.9 67.9 80.5 82.0 65.7 64.2
Cohere Command R 36.7 36.0 44.9 16.0 44.3 19.0 71.3 15.0 78.1 25.0 11.7 24.1 51.1 41.0 46.1 24.9
Cohere Command R+ 22.4 51.2 54.1 27.0 65.2 35.0 77.8 25.0 83.7 37.0 30.8 49.7 37.2 59.0 47.1 39.1
Gemini 1.0 Pro 23.8 60.0 45.6 27.0 56.7 32.0 70.4 24.0 84.6 33.0 41.8 45.8 11.8 16.0 41.7 35.5
Gemini 1.5 Flash 26.8 74.8 69.5 40.0 75.4 52.0 89.1 34.0 94.1 42.0 45.7 53.4 46.6 66.0 57.9 52.1
Gemini 1.5 Pro 36.2 79.6 76.7 42.0 83.6 54.0 87.0 31.0 91.1 37.0 34.1 45.8 91.9 39.0 65.7 48.8
GPT-3.5-turbo 22.7 50.4 24.9 16.0 40.7 20.0 69.2 24.0 81.4 36.0 30.0 42.1 0.7 41.0 33.0 33.0
GPT-4-turbo 33.2 77.2 60.0 38.0 65.2 45.0 85.5 38.0 94.1 47.0 42.9 44.2 56.1 46.0 57.1 49.6
GPT-4o 36.5 79.2 71.5 47.0 81.3 53.0 87.6 49.0 91.1 55.0 46.7 60.9 68.2 67.0 63.3 59.8
Llama 3 8B Instruct 22.6 28.3 21.3 10.0 23.6 16.0 48.8 22.0 58.0 29.0 12.9 35.0 28.7 29.0 28.4 23.1
Llama 3 70B Instruct 26.9 70.9 59.0 34.0 66.6 42.0 78.4 21.0 87.3 30.0 37.4 55.1 12.2 78.0 47.3 48.1
Mistral Large 26.8 74.3 78.4 33.0 84.6 50.0 84.3 31.0 92.0 38.0 36.1 49.5 31.1 77.0 55.8 50.4
Mixtral 8x22B MoE 26.6 54.7 63.3 30.0 67.9 40.0 80.5 28.0 90.2 33.0 42.0 52.4 37.5 55.0 52.5 41.6
o1-mini 31.2 76.4 71.5 56.0 76.4 65.0 79.3 31.0 84.6 39.0 41.5 56.4 69.0 77.0 59.3 57.5
o1-preview 42.7 81.6 65.2 81.0 72.5 91.0 89.4 60.0 93.2 62.0 48.0 69.5 72.4 89.0 64.4 74.9
Table 4: Logic form accuracy for goal interpretation (%). Full results in Table 9.
Model State Goal Relation Goal Action Goal Overall
Precision Recall F1F_{\textsubscript{1}} Precision Recall F1F_{\textsubscript{1}} Precision Recall F1F_{\textsubscript{1}} Precision Recall F1F_{\textsubscript{1}}
V B V B V B V B V B V B V B V B V B V B V B V B
Claude-3.5 Sonnet 25.3 74.0 60.9 94.8 35.8 83.1 31.1 84.4 63.8 81.3 41.8 82.9 14.0 - 98.8 - 24.5 - 21.7 81.1 69.6 84.4 33.0 82.7
Gemini 1.5 Pro 47.2 94.0 47.5 92.8 47.3 93.4 42.0 74.4 7.2 76.7 12.4 75.6 24.1 - 81.4 - 37.2 - 33.6 78.8 39.3 80.4 36.2 79.6
GPT-4o 29.0 67.1 60.0 94.8 39.1 78.6 31.5 81.1 43.6 78.5 36.6 79.8 20.5 - 85.8 - 33.1 - 26.4 76.5 59.1 82.2 36.5 79.2
Llama 3 70B 23.9 69.5 61.2 95.4 34.3 80.4 22.6 70.0 37.5 73.3 28.2 71.6 11.2 - 88.8 - 19.8 - 17.5 64.7 58.0 78.3 26.9 70.9
o1-mini 26.3 63.8 58.6 90.8 36.3 74.9 30.4 77.3 39.9 76.5 34.5 76.9 13.5 - 56.8 - 21.8 - 22.4 73.3 51.3 79.8 31.2 76.4
o1-preview 28.2 66.8 60.3 94.8 38.5 78.4 44.9 82.9 62.4 82.7 52.2 82.8 26.0 - 81.5 - 39.5 - 31.8 78.1 65.4 85.4 42.7 81.6
Refer to caption
Figure 5: Examples of different types of errors in trajectory feasibility, logic form parsing (e.g., in subgoals decomposition and transition modeling), and goal satisfaction rates.
Table 5: Goal satisfaction rates (%) for action sequencing and subgoal decomposition. Full results in Appendix E.2. Behavior does not include action goals.
Model Action Sequencing Subgoal Decomposition
State Goal Relation Goal Action Goal Total State Goal Relation Goal Action Goal Total
V B V B V B V B V B V B V B V B
Claude-3.5 Sonnet 87.8 63.0 83.3 62.4 60.8 - 79.9 62.6 92.9 41.0 88.6 39.5 87.0 - 90.1 39.9
Claude-3 Opus 57.2 45.0 77.8 53.0 54.7 - 62.7 50.8 92.4 43.0 88.6 41.6 83.3 - 89.1 42.0
Gemini 1.5 Pro 85.6 41.0 76.7 43.2 62.2 - 77.2 42.6 91.2 31.0 72.5 37.1 89.5 - 83.9 35.4
GPT-4o 87.1 49.0 76.1 45.5 56.1 - 76.2 46.5 92.1 50.0 84.2 53.2 93.2 - 89.4 52.3
Llama 3 70B 61.5 31.0 68.3 45.5 42.6 - 58.9 41.5 93.2 25.0 63.4 27.7 82.7 - 80.0 27.0
o1-mini 88.5 64.0 72.2 66.9 57.4 - 76.1 66.1 89.7 28.0 68.8 38.0 81.5 - 80.3 35.3
o1-preview 80.9 89.5 65.0 84.4 46.6 - 67.8 85.8 91.8 56.5 88.3 69.4 92.6 - 90.6 65.9

4.1 Ability Module Analysis

Goal interpretation frequently includes temporary states in the final goal. Planning errors mostly arise during execution, especially overlooked prerequisites and redundant actions. Subgoal decomposition still requires detailed reasoning, despite expressing states rather than commands. Transition models often omit necessary preconditions or invent effects, allowing an external planner to succeed in an inaccurate model. The appendix studies sensitivity to individual operators, module composition, replanning, and prompt choices. Statements of numerical error prevalence differ between the introduction and Appendix A; the complete result tables are retained instead of treating one prose percentage as a universal rate.

Table 6: Trajectory evaluation results (%) for action sequencing and subgoal decomposition. Full results in Appendix E.3.
Model Goal Evaluation Trajectory Evaluation
Task SR Execution SR Grammar Error (↓\downarrow) Runtime Error (↓\downarrow)
Parsing Hallucination Predicate-Arg Num Wrong Order Missing Step Affordance Additional Step
V B V B V B V B V B V B V B V B V B
Action Sequencing
Claude-3.5 Sonnet 76.7 60.0 81.3 69.0 0.0 0.0 2.0 0.0 0.3 0.0 0.3 5.0 14.4 25.0 1.6 1.0 1.3 2.0
Claude-3 Opus 64.9 51.0 69.5 59.0 0.0 0.0 17.0 0.0 0.0 0.0 0.3 3.0 12.8 35.0 0.3 3.0 2.3 2.0
Gemini 1.5 Pro 76.7 42.0 83.6 54.0 0.0 0.0 1.3 0.0 0.7 0.0 0.3 6.0 14.1 39.0 0.0 1.0 3.0 2.0
GPT-4o 71.5 47.0 81.3 53.0 1.0 0.0 2.0 1.0 0.7 0.0 0.3 9.0 15.1 36.0 0.0 1.0 2.3 0.0
Llama 3 70B 59.0 34.0 66.6 42.0 0.0 0.0 14.1 2.0 8.2 0.0 2.0 15.0 9.2 38.0 0.0 3.0 6.2 6.0
o1-mini 71.5 56.0 76.4 65.0 4.9 0.0 2.0 3.0 0.0 0.0 1.0 7.0 17.7 17.0 0.3 6.0 2.6 5.0
o1-preview 65.2 81.0 72.5 91.0 6.6 0.0 11.5 0.0 0.0 0.0 0.0 0.0 12.1 6.0 0.3 2.0 2.0 3.0
Subgoal Decomposition
Claude-3.5 Sonnet 89.1 39.0 92.0 44.0 0.0 0.0 1.8 1.0 0.0 0.0 1.5 11.0 2.7 44.0 2.1 0.0 24.6 4.0
Claude-3 Opus 87.0 39.0 89.9 47.0 0.3 0.0 3.3 3.0 0.0 0.0 1.2 5.0 3.0 45.0 2.4 0.0 16.0 5.0
Gemini 1.5 Pro 87.0 31.0 91.1 37.0 0.0 1.0 1.5 0.0 1.8 1.0 0.0 3.0 5.6 59.0 0.0 0.0 16.0 2.0
GPT-4o 88.8 48.0 90.2 55.0 0.0 0.0 6.2 3.0 0.0 0.0 1.2 5.0 2.4 37.0 0.0 0.0 15.7 5.0
Llama 3 70B 78.4 20.0 87.3 30.0 0.0 1.0 2.4 5.0 0.9 1.0 2.4 8.0 5.3 51.0 1.8 4.0 20.4 4.0
o1-mini 79.3 31.0 84.6 39.0 0.0 0.0 1.5 3.0 0.6 3.0 0.3 7.0 8.9 46.0 4.1 2.0 21.9 1.0
o1-preview 89.4 57.0 93.2 62.0 0.0 2.0 1.5 3.0 0.0 0.0 0.3 5.0 2.7 25.0 2.4 3.0 12.1 7.0
Table 7: Logic form accuracy (F1) and planner success rate (SR) for transition modeling (%). Full results in Appendix E.4.
Model Object States Object Orientation Object Affordance Spatial Relations Non-Spatial Relations
F1F_{\textsubscript{1}} SR F1F_{\textsubscript{1}} SR F1F_{\textsubscript{1}} SR F1F_{\textsubscript{1}} SR F1F_{\textsubscript{1}} SR
V B V B V B V B V B V B V B V B V B V B
Claude-3.5 Sonnet 60.5 78.8 67.4 86.7 95.3 - 96.4 - 76.6 - 67.7 - 42.4 58.6 96.6 80.9 5.9 73.6 91.9 80.3
Claude-3 Opus 63.0 71.9 63.5 84.4 62.6 - 71.4 - 75.5 - 58.7 - 38.7 54.6 64.8 80.9 7.0 68.8 55.4 82.0
Gemini 1.5 Pro 18.8 55.9 94.4 35.6 90.9 - 89.3 - 77.7 - 95.8 - 38.7 35.9 89.0 40.4 7.8 52.8 83.8 39.3
GPT-4o 54.6 71.3 71.9 68.9 52.8 - 78.6 - 74.9 - 63.5 - 40.8 45.9 66.9 64.9 7.5 73.0 68.9 68.9
Llama-3 70B 32.5 66.3 10.1 68.9 56.6 - 3.6 - 57.0 - 6.6 - 27.0 47.2 15.2 77.7 3.0 58.9 18.9 85.2
o1-mini 59.0 41.3 63.5 77.8 56.3 - 82.1 - 58.5 - 59.3 - 32.5 53.1 75.9 77.7 4.5 67.5 71.6 75.4
o1-preview 58.5 78.3 69.1 86.7 78.4 - 100.0 - 77.5 - 67.1 - 38.8 56.3 76.6 89.4 11.8 83.5 78.4 90.2

5 Related Work

Earlier embodied systems use language models for one or several of the four roles, often coupled to planners, skill libraries, or feedback. This benchmark contributes a common representation and diagnostic protocol for comparing those roles. The retained survey table maps prior systems to their interfaces; the contribution is evaluation and composition support rather than a new universal robot controller.

Table 8: Existing work in leveraging LLMs for embodied agents.
Goal Interpretation Subgoal Decomposition Action Sequencing Transition Modeling
[2, 7, 46, 47, 21, 48, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 13, 14, 15, 49, 50, 51, 52, 53] [2, 27, 34, 18, 33, 35, 54, 55] [6, 8, 35, 56, 57, 58, 59, 16, 43, 60, 17, 19, 61, 20, 42, 62, 14, 63, 64, 65, 66, 3, 15, 45, 67, 68, 69] [8, 28, 32, 36, 70, 37, 13, 38]

6 Conclusions and Future Work

The interface makes symbolic decision-making failures easier to localize and suggests selective use of language models with external tools. The principal scope limit is that objects, states, actions, and goals are already expressible in a supplied symbolic vocabulary. Sensory grounding, actuation, memory, geometric reasoning, navigation, and uncertain world models require further integration. The later visual and perturbation experiments probe some of these gaps without removing them.

Acknowledgments and Disclosure of Funding

The authors disclose academic, US research-agency, Amazon, and Microsoft support.

Checklist

The checklist reports released code, annotations, evaluation settings, and asset information. Main model comparisons do not include repeated-run error bars because of inference cost. The authors state that existing public data contain no personally identifiable or offensive material and discuss limitations and negative impacts in Appendix S.

Appendix

Appendix A Summary of Empirical Findings

The empirical summary reiterates that models confuse intermediate and final goals, struggle with relational predicates, and generate infeasible action sequences. Reported BEHAVIOR action task/execution success is 81%/91% for o1-preview versus 47%/53% for GPT-4o; VirtualHome rankings differ, with Mistral Large and Gemini 1.5 Pro ahead of o1-preview on the cited action metrics. Learned subgoal sequences are not uniformly easier than actions. Operator consistency matters when predicted and reference transitions are mixed. Composition can preserve executability while degrading task achievement, and feedback can improve both. Some claims in this summary reflect different model subsets or conflict with main-text rankings; consult the complete tables for the relevant metric and simulator.

Appendix B Embodied Agent Interface Design

This appendix relates the diagnostic design, module interfaces, and symbolic decision process.

B.1 Motivation

The design separates goal-directed environmental interaction from general language generation. Each task package contains instructions, symbolic goals, a reference action trajectory, and transition models. The retained diagrams show how standardized goals, interfaces, and error metrics connect these annotations to simulator evaluation.

Refer to caption
Figure 6: Compared to general language models, the embodied agent has three key new abilities, including interacting with environments, being goal-driven, and performing decision-making to achieve the goal. We believe a systematic evaluation for embodied decision-making should cover three aspects by standardizing the interface, the goal representation, and the evaluation metrics. Our Embodied Agent Interface addresses the limitations of traditional evaluations by focusing on goal-driven evaluation, standard interface, and modules, as well as broad coverage of evaluation and fine-grained metrics.
Refer to caption
Figure 7: The Embodied Agent Interface aims to design a standard interface for LLMs to perform tasks in the embodied environment.
Original figure
Figure 8: Comparison with existing benchmarks on LLMs for embodied decision making.
Original figure
Figure 9: VirtualHome dataset structure example.
Refer to caption
Figure 10: BEHAVIOR dataset structure example.

B.2 Input and Output Details

Inputs combine task instructions, module instructions/examples, and an object-state-relation description. Goal interpretation grounds names into object identifiers; decomposition proposes intermediate states; action sequencing realizes them; transition modeling supplies operator preconditions and effects. The cleaning example illustrates that one state transition may require several actions. The text’s description of transition modeling as a low-level controller refers here to symbolic transitions, not continuous motor control.

Refer to caption
Figure 11: Example input and output for the ability modules.
Refer to caption
Figure 12: An example of successful execution in BEHAVIOR .

B.3 Grounding to Markov Decision Process

The Markov decision-process account uses objects, states, actions, a transition function, reward, and goal. Reward is illustrated as a binary goal-satisfaction indicator. The four modules correspond to interpreting the target, producing state or action trajectories, and modeling transitions. This is a useful symbolic decomposition; temporal goals require trajectory context, so the simple state-only reward expression is not by itself a complete realization of every temporal specification.

Refer to caption
Figure 13: Embodied Decision Making is a Markov Decision Process.
Refer to caption
Figure 14: Our four ability modules are fundamental modules of the MDP process.

B.4 The Relationship between Ability Modules

Isolating a module with reference data or external tools diagnoses its contribution. Decomposition outputs declarative conditions; sequencing outputs imperative commands; transition models enable symbolic search over those commands. Pipeline experiments then measure how errors interact. Isolated scores should therefore not be read as end-to-end performance of one autonomous agent.

Appendix C LTL Representation and Implementation

C.1 Why LTL

Temporal logic gives a compact common language for final-state constraints, required actions, ordering, and alternatives. The decomposition pipeline parses model output into an expression tree, samples a concrete subgoal path, searches for actions, and tests execution and goal achievement. Representation unification does not make all module metrics equivalent.

Refer to caption
Figure 15: Pipeline of subgoal decomposition based on LTL in Embodied Agent Interface

C.2 Comparision with Traditional LTL Representation

The implemented language extends relational predicates with quantifiers and exact counts, and uses a custom then connective over finite trajectories. It shares recursive logical structure with LTL but is a particular supported fragment, not the full standard temporal language.

C.3 Syntax and Semantics of LTL Formulas

Atomic propositions describe a state relation or an action with object arguments. Boolean operators and quantifiers are evaluated over the finite object universe; exact-count quantification requires exactly the specified number of satisfying objects. A non-temporal formula holds at some state-action pair in the trajectory. Then holds when the trajectory can be divided into a prefix satisfying the first formula and a strictly later suffix satisfying the second. Parsing and dynamic programming evaluate nested expressions. The complete displayed syntax, semantics, and example tree remain available.

Equation
ϕ::=p​∣¬ϕ∣​ϕ1∧ϕ2​∣ϕ1∨ϕ2∣​ϕ1⇒ϕ2|∀x​ϕ​(x)​∣∃x​ϕ​(x)​∣∃=nx​ϕ​(x)∣​(ϕ)∣​ϕ1​ then ​ϕ2\phi::=p\mid\neg\phi\mid\phi_{1}\land\phi_{2}\mid\phi_{1}\lor\phi_{2}\mid\phi_{1}\Rightarrow\phi_{2}\mid\forall x\,\phi(x)\mid\exists x\,\phi(x)\mid\exists^{=n}x\,\phi(x)\mid(\phi)\mid\phi_{1}\text{ then }\phi_{2}
LTL Grammar Definition
?start: stmt
primitive: VARNAME "(" [args] ")“  # primitive format looks like varname(param)
object_name: VARNAME               # object_name can be an object name (eg. pants)
           | VARNAMEWITHID         # or object name with ID (eg. pants.1000)
args: object_name ("," object_name)*  # definition of arguments

?stmt: then_stmt | primitive_stmt    # a formula is a Boolean stmt
then_stmt: or_stmt ("then" or_stmt)* # connective priority order:
or_stmt: and_stmt ("or" and_stmt)*   # then < or < and < not < forall = forn = exists
and_stmt: primitive_stmt ("and" primitive_stmt)*
primitive_stmt: "not" primitive_stmt -> not_stmt
              | primitive
              | "(" stmt ")"
              | "forall" VARNAME "." "(" stmt ")" -> forall_stmt
              | "forn" VARNAME "." "(" stmt ")" -> forn_stmt
              | "exists" VARNAME "." "(" stmt ")" -> exists_stmt

VARNAME: /[a-zA-Z_]\w*/
VARNAMEWITHID: /[a-zA-Z_]\w*\.[0-9]+/
    
Refer to caption
Figure 16: An example of LTL representation.
Equation
eval​(ϕ1​then​ϕ2,T)=∃k.ϕ1​(T≤k)∧ϕ2​(T>k),\textit{eval}(\phi_{1}~\textbf{then}~\phi_{2},T)=\exists k.\phi_{1}(T_{\leq k})\land\phi_{2}(T_{>k}),

Appendix D Fine-Grained Metrics and Automatic Error Detection

The following protocols distinguish formal correctness, execution, and task achievement.

D.1 Goal Interpretation: State Goal, Relation Goal and Action Goal

Goal predictions are checked for format and invented symbols, then compared with reference unary state goals, binary relation goals, and action goals. Precision penalizes extra predicates, recall measures omitted requirements, and F1 combines them. Action goals are used for VirtualHome tasks without a persistent completion effect; BEHAVIOR does not include them.

Refer to caption
Figure 17: Evaluation pipeline of goal interpretation. We evaluate the LLM’s output for the Goal Interpretation in three key dimensions: single-object states, object-object relations, and agent-action goals. For each dimension, we calculate precision, recall, and F1 score to measure the LLM’s false positive predictions, likelihood of missing goals, and overall capability, respectively.

D.2 Action Sequencing: Trajectory Error Detection for Missing Step, Additional Step, Wrong Temporal Order, Affordance Error

Action plans are checked against simulator vocabulary, state transitions, and task conditions.

Refer to caption
Figure 18: Evaluation pipeline of action sequencing.

Input Instruction.

The model receives task-relevant objects and initial states, the action/predicate vocabulary, and goal conditions rephrased in natural language. Relevance selection uses reference initial/final differences, so this input is already curated rather than a raw, unfiltered scene.

Trajectory Evaluation.

Grammar checks distinguish malformed output, invented action/object names, and wrong argument counts. Runtime categorization first tests object affordances, then whether the intended effect is already true. Unsatisfied prerequisites become missing-step errors if they were never true in the history, or wrong-order errors if they were true earlier. This deterministic hierarchy makes diagnosis reproducible, while its labels remain dependent on the simulator’s transition rules.

Refer to caption
Figure 19: Automatic Error Categorization for Trajectory: (1) Check if the action is affordable based on the properties of objects in the current state. If not, return "Affordance Error". (2) If the action is affordable, check if the intended effect of the action is already satisfied in the current state. If it is, return "Additional Step". (3) If the effect is not redundant, check whether the precondition of the action is satisfied in the current state. If not, proceed to the next step. Otherwise, return "No Runtime Error". (4) If the precondition is not satisfied, check if it has ever been satisfied in any of the historical states. If the precondition has never been satisfied, return "Missing Step". (5) If the precondition has been satisfied in a historical state, return "Wrong Order".
Trajectory Error Detection: Missing Step, Additional Step, Wrong Temporal Order, Affordance Error
def check_runtime_errors(action, current_state, historical_states):
    # Check affordance error
    if not is_affordable(action, current_state):
        return "Affordance Error"

    # Check if action effect is redundant (Additional Step)
    if is_effect_redundant(action, current_state):
        return "Additional Step"

    # Check if precondition for action is met
    if not is_precondition_satisfied(action, current_state):
        # Check if the effect has not been satisfied
        if not is_precondition_ever_satisfied(action, historical_states):
            # Check historical states for precondition satisfaction
            if not precondition_satisfied_in_history(action, historical_states):
                return "Missing Step"
            else:
                return "Wrong Order"

    return "No Runtime Error"
    

Goal Satisfaction

Execution proceeds until a blocking action or the end of the sequence. The resulting state and executed actions are checked against node, relation, and action goals. Full task success requires all specified conditions; partial goal recall is reported separately. A partially satisfied task is not counted as complete.

D.3 Subgoal Decomposition: Converting Subgoal Trajectory to Action Trajectory with BFS Searching

Breadth-first search connects each generated subgoal to an action trajectory, carrying the reached state forward. Several refinements may be possible; evaluation uses the best considered realization. The paper notes that breadth-first search returns shortest paths in its discrete search space and can be expensive at scale. The main text describes a per-subgoal action restriction, while this appendix says no additional search-space control is needed for the benchmark’s small vocabulary; neither implies scalability to unconstrained robot action spaces.

Refer to caption
Figure 20: Evaluation pipeline of subgoal decomposition. The subgoal decomposition module generates a sequence of declarative states, which need to be transformed into an actionable trajectory for execution in the simulator.
BFS Searching to Transform State Trajectory to Action Trajectory
Initialize an empty state-action sequence t¯\bar{t}

Set the current state scurrs_{\text{curr}} to the initial state s0s_{0}

For each subgoal ϕi\phi_{i} in the subgoal sequence ϕ¯\bar{\phi}:

For each subgoal ϕi\phi_{i} in the subgoal sequence ϕ¯\bar{\phi}:

   Perform BFS starting from scurrs_{\text{curr}} to find a state sgoals_{\text{goal}} that satisfies ϕi\phi_{i}.

   Extract the path from scurrs_{\text{curr}} to sgoals_{\text{goal}} and append the corresponding state-action pairs to t¯\bar{t}.

   Set scurrs_{\text{curr}} to sgoals_{\text{goal}}.

Return the complete state-action sequence t¯\bar{t}.
Original figure
Figure 21: The pipeline of evaluating a generated subgoal plan.

D.4 Transition Modeling: Evaluating with PDDL Planners

Transition-model evaluation supplies object/predicate information and operator signatures, then scores generated bodies. Recursive clause parsing and bipartite matching compare preconditions and effects with reference logical structure, producing precision, recall, and F1. A PDDL planner additionally tests whether the predicted domain admits a plan. Surface/structural matching can penalize alternative formulations, and a satisfiable predicted domain can still encode physically incorrect dynamics.

Original figure
Figure 22: Transition Model evaluation metrics based on Bipartite Graph Matching for pre-conditions and post-effects.
Logic Form Accuracy function(expr1, expr2)
if expr1 and expr2 are literals:
    return expr1 == expr2
if type(expr1) != type(expr2):
    return 0
if isinstance(expr1, Not):
    return match_expressions(expr1.expression, expr2.expression)
if isinstance(expr1, When):
    if match_expressions(expr1.condition, expr2.condition)
    and match_expressions(expr1.consequence, expr2.consequence):
        return 1
    return 0
if isinstance(expr1, (Exists, Forall)):
    if quantifier are the same:
        return match_expressions(expr1.body, expr2.body)
    return 0
if isinstance(expr1, (And, Or)):
    adj_matrix = np.zeros((len(expr1.args), len(expr2.args)), dtype=int)
    for i in range(size1):
        for j in range(size2):
            adj_matrix[i, j] = match_expressions(sub1[i], sub2[j])
    match_result = maximum_bipartite_matching(sparse_matrix, perm_type=’column’)
    total_match = sum(adj_matrix[i, match_result[i]]
    for i in range(size1) if match_result[i] != -1)
    max_possible_match = min(size1, size2)
    return total_match / max_possible_match if max_possible_match > 0 else 0

Equations
SuccessRate​(cj)=∑p∈PcjIsSuccessful​(p)|Pcj|\text{SuccessRate}(c_{j})=\frac{\sum_{p\in P_{c_{j}}}\text{IsSuccessful}(p)}{|P_{c_{j}}|}
OverallSuccessRate=∑j=1k∑p∈PcjIsSuccessful​(p)∑j=1k|Pcj|\text{OverallSuccessRate}=\frac{\sum_{j=1}^{k}\sum_{p\in P_{c_{j}}}\text{IsSuccessful}(p)}{\sum_{j=1}^{k}|P_{c_{j}}|}

D.5 Average Performance

The overall score averages goal-interpretation F1, subgoal task success, action task success, and an average of transition-model F1 and planner success. This mixes complementary measures; it is a diagnostic aggregate, not a probability that a full robot pipeline succeeds.

Equation
AveragePerf=14​(GoalF1+SubgoalTask SR+ActSeqTask SR+0.5∗(TrsF1+TrsPlanner SR))\text{AveragePerf}=\frac{1}{4}(\text{Goal}_{F_{1}}+\text{Subgoal}_{\text{Task SR}}+\text{ActSeq}_{\text{Task SR}}+0.5*(\text{Trs}_{F_{1}}+\text{Trs}_{\text{Planner SR}}))

Appendix E Full Results with 18 models

Complete results for all eighteen models and both simulators are preserved, alongside qualitative error cases. Scores and model names are those of the evaluated versions, not a current model ranking.

Refer to caption
Figure 23: Performance Overview: Tasks in BEHAVIOR require longer sequences of actions and complex subgoal decomposition, while VirtualHome presents greater challenges in understanding environmental states.

E.1 Goal Interpretation

Goal-interpretation results separate unary state, relation, and VirtualHome action goals, with micro-averaged overall precision, recall, and F1.

E.1.1 Result Analysis

The detailed analysis identifies o1-preview as the highest overall F1 model on VirtualHome and Claude-3.5 Sonnet on BEHAVIOR, with Gemini 1.5 Pro competitive and Llama-3 70B strongest among the discussed open models. Category leaders differ. This supersedes the appendix summary’s blanket claim that Gemini leads both simulators; the retained table provides the exact per-category values.

E.1.2 Case Studies

Cases distinguish malformed syntax and invented symbols from semantically missing or extra goal conditions. Strong proprietary models rarely fail basic formatting under these prompts, but still mistake temporary states for final requirements and omit spatial links such as food-on-plate. The authors suggest implicit reasoning may leak intermediate states into outputs; this is an interpretation of errors, not a demonstrated mechanism inside the models.

Takeaways of Goal Interpretation
(1)

Top-performing Models: o1-preview and Claude 3.5 Sonnet achieves top overall goal interpretation performance (F1-score) in VirtualHome and BEHAVIOR simulators, respectively. Whereas Gemini 1.5 Pro achieves the highest accuracy in VirtualHome .

(2)

Grammar Errors: Current SOTA proprietary LLMs generally make very little to no grammar errors in goal interpretation, whereas top open-source LLMs including Llama 3 70B Instruct tend to suffer more from format/parsing errors and object/state hallucination.

(3)

Goal Prediction Errors: Both open-source and proprietary LLMs suffer significantly from two common types of goal prediction errors:

(a)

Generating intermediate goals in place of final goals (when asked to directly output final goals), resulting in false positive goal predictions.

(b)

Omitting conversationally uncommon spatial relationship goals that are essential for precise robotic control, resulting in missing goals (reporting bias).

Table 9: All goal evaluation results (%) for goal interpretation
Model Name Goal Interpretation
State Spatial Action Overall
Precision Recall F1 Precision Recall F1 Precision Recall F1 Precision Recall F1
V B V B V B V B V B V B V B V B V B V B V B V B
Claude-3 Haiku 21.8 22.8 58.9 93.5 31.8 36.7 24.2 64.5 50.8 64.6 32.8 64.6 12.2 - 95.7 - 21.6 - 18.0 41.5 63.2 71.2 28.0 52.5
Claude-3 Sonnet 23.3 36.8 57.1 88.9 33.1 52.0 26.6 76.2 53.0 79.8 35.5 77.9 12.4 - 85.8 - 21.7 - 19.3 60.2 61.5 81.9 29.4 69.4
Claude-3 Opus 27.0 72.6 66.9 93.5 38.5 81.7 22.6 75.2 46.8 79.2 30.5 77.1 14.5 - 92.6 - 25.1 - 20.7 72.2 65.0 82.5 31.4 77.0
Claude-3.5 Sonnet 25.3 74.0 60.9 94.8 35.8 83.1 31.1 84.4 63.8 81.3 41.8 82.9 14.0 - 98.8 - 24.5 - 21.7 81.1 69.6 84.4 33.0 82.7
Cohere Command R 51.1 7.7 69.6 31.4 58.9 12.4 34.5 56.8 21.3 55.0 26.3 55.9 3.6 - 38.9 - 6.5 - 27.4 28.2 55.7 49.6 36.7 36.0
Cohere Command R+ 20.9 23.3 52.0 79.1 29.8 36.0 17.9 66.7 15.2 61.5 16.4 64.0 10.4 - 82.6 - 18.5 - 14.9 42.0 44.5 65.5 22.4 51.2
Gemini 1.0 Pro 25.3 27.4 57.9 81.1 34.9 41.0 17.0 75.2 20.6 70.4 18.6 72.7 9.9 - 68.7 - 17.2 - 16.2 51.0 45.2 72.8 23.8 60.0
Gemini 1.5 Flash 23.6 55.8 57.9 94.1 33.5 70.1 19.8 76.6 21.1 76.7 20.5 76.7 13.5 - 90.1 - 23.5 - 18.2 69.7 50.8 80.7 26.8 74.8
Gemini 1.5 Pro 47.2 94.0 47.5 92.8 47.3 93.4 42.0 74.4 7.2 76.7 12.4 75.6 24.1 - 81.4 - 37.2 - 33.6 78.8 39.3 80.4 36.2 79.6
GPT-3.5-turbo 22.4 52.0 50.0 66.7 30.9 58.5 8.5 51.5 18.8 46.9 11.7 49.1 15.2 - 60.5 - 24.4 - 15.7 49.5 40.5 51.4 22.7 50.4
GPT-4-turbo 28.6 70.4 58.5 86.9 38.4 77.8 24.7 77.5 32.9 76.4 28.2 76.9 19.0 - 82.1 - 30.9 - 24.0 75.6 53.8 78.8 33.2 77.2
GPT-4o 29.0 67.1 60.0 94.8 39.1 78.6 31.5 81.1 43.6 78.5 36.6 79.8 20.5 - 85.8 - 33.1 - 26.4 76.5 59.1 82.2 36.5 79.2
Llama 3 8B Instruct 21.7 17.3 54.4 80.4 31.0 28.4 14.0 51.4 7.4 20.8 9.7 29.6 11.1 - 79.4 - 19.4 - 15.5 24.1 41.9 34.3 22.6 28.3
Llama 3 70B Instruct 23.9 69.5 61.2 95.4 34.3 80.4 22.6 70.0 37.5 73.3 28.2 71.6 11.2 - 88.8 - 19.8 - 17.5 64.7 58.0 78.3 26.9 70.9
Mistral Large 23.6 63.5 59.1 92.2 32.8 75.2 23.7 75.1 40.3 76.2 29.8 75.6 11.2 - 84.0 - 19.7 - 17.5 69.6 57.1 79.8 26.8 74.3
Mixtral 8x22B MoE 23.6 22.9 56.9 83.7 33.4 36.0 22.2 70.7 36.3 67.7 27.5 69.2 11.2 - 94.8 - 20.0 - 17.4 44.4 56.2 71.3 26.6 54.7
o1-mini 26.3 63.8 58.6 90.8 36.3 74.9 30.4 77.3 39.9 76.5 34.5 76.9 13.5 - 56.8 - 21.8 - 22.4 73.3 51.3 79.8 31.2 76.4
o1-preview 28.2 66.8 60.3 94.8 38.5 78.4 44.9 82.9 62.4 82.7 52.2 82.8 26.0 - 81.5 - 39.5 - 31.8 78.1 65.4 85.4 42.7 81.6
Refer to caption
(a) VirtualHome
Refer to caption
(b) BEHAVIOR
Figure 24: Goal evaluation error examples for goal interpretation.
Refer to caption
(a) VirtualHome
Refer to caption
(b) BEHAVIOR
Figure 25: Goal satisfaction error examples for goal interpretation

E.2 Subgoal Decomposition

E.2.1 Result Analysis

The complete decomposition tables separate executable refinements, full task success, grammar/runtime failures, and partial goal satisfaction.

VirtualHome

VirtualHome subgoal task success reaches 89.4% for o1-preview and 89.1% for Gemini 1.5 Flash and Claude-3.5 Sonnet. Execution success can be higher, illustrating that feasible plans may miss goals. Strong models usually follow syntax but can invent predicates such as POUR. Redundant subgoals are common when the initial scene already satisfies them; weaker models also miss container-opening and standing prerequisites. State conditions are generally easier than relations or required actions.

BEHAVIOR

BEHAVIOR is substantially harder: o1-preview reaches 57% task success and 62% execution success in this comparison. Missing prerequisites dominate; even that model has a 25% missing-step rate. The simulator enforces tool possession and other conditions absent from analogous VirtualHome actions. Quantified requirements are scored as combined conditions, so one unmet instance can fail the whole quantified goal. Differences in these semantics and initial states contribute to the cross-simulator gap.

Table 10: All trajectory evaluation results (%) for subgoal decomposition.
Model Goal Evaluation Trajectory Evaluation
Task SR Execution SR Grammar Error (↓\downarrow) Runtime Error (↓\downarrow)
Parsing Hallucination Predicate-Arg Num Wrong Order Missing Step Affordance Additional Step
V B V B V B V B V B V B V B V B V B
Claude-3 Haiku 78.4 30.0 82.8 35.0 0.3 0.0 2.4 1.0 1.8 0.0 1.8 3.0 2.7 58.0 8.3 3.0 20.4 3.0
Claude-3 Sonnet 83.1 39.0 86.4 43.0 0.0 0.0 1.8 2.0 0.0 2.0 0.6 3.0 2.7 51.0 8.6 1.0 33.7 3.0
Claude-3 Opus 87.0 41.0 90.0 47.0 0.3 0.0 3.6 3.0 0.0 0.0 1.2 5.0 3.0 45.0 2.4 0.0 16.0 6.0
Claude-3.5 Sonnet 89.1 39.0 92.0 44.0 0.0 0.0 1.8 1.0 0.0 0.0 1.5 11.0 2.7 44.0 2.1 0.0 24.6 4.0
Gemini 1.0 Pro 70.4 24.0 84.6 33.0 0.6 2.0 3.3 4.0 2.4 0.0 1.2 3.0 2.7 51.0 5.3 7.0 10.4 3.0
Gemini 1.5 Flash 89.1 34.0 94.1 42.0 0.0 2.0 1.5 1.0 0.0 0.0 0.6 2.0 3.9 53.0 0.0 0.0 13.3 3.0
Gemini 1.5 Pro 87.0 31.0 91.1 37.0 0.0 1.0 1.5 0.0 1.8 1.0 0.0 3.0 5.6 59.0 0.0 0.0 16.0 2.0
GPT-3.5-turbo 69.2 24.0 81.4 36.0 1.5 2.0 0.0 3.0 0.6 0.0 1.5 4.0 11.8 51.0 3.3 4.0 20.4 3.0
GPT-4-turbo 85.5 38.0 94.1 47.0 0.0 0.0 1.8 3.0 0.0 0.0 1.5 9.0 2.4 40.0 0.3 1.0 22.2 6.0
GPT-4o 88.8 49.0 90.2 55.0 0.0 0.0 6.2 3.0 0.0 0.0 1.2 6.0 2.4 36.0 0.0 0.0 15.7 5.0
Cohere Command R 71.3 15.0 79.6 25.0 2.1 23.0 3.9 10.0 0.9 0.0 1.5 0.0 6.2 37.0 5.9 5.0 14.5 4.0
Cohere Command R+ 79.0 25.0 83.7 37.0 1.5 2.0 4.5 4.0 2.1 0.0 0.9 4.0 7.7 52.0 2.7 1.0 16.0 6.0
Mistral Large 84.3 31.0 92.0 38.0 0.3 1.0 1.8 3.0 0.3 0.0 2.1 4.0 3.3 52.0 0.3 2.0 11.0 1.0
Mixtral 8x22B MoE 80.5 28.0 90.2 33.0 0.3 0.0 2.4 4.0 0.0 0.0 3.0 2.0 3.9 59.0 0.3 2.0 11.2 0.0
Llama 3 8B 48.8 21.0 58.0 29.0 0.6 2.0 2.4 11.0 0.6 0.0 6.8 6.0 5.0 44.0 26.6 8.0 18.3 7.0
Llama 3 70B 78.4 20.0 87.3 30.0 0.0 1.0 2.4 5.0 0.9 1.0 2.4 8.0 5.3 51.0 1.8 4.0 20.4 4.0
o1-mini 79.3 31.0 84.6 39.0 0.0 0.0 1.5 3.0 0.6 3.0 0.3 7.0 8.9 46.0 4.1 2.0 21.9 1.0
o1-preview 89.4 57.0 93.2 62.0 0.0 2.0 1.5 3.0 0.0 0.0 0.3 5.0 2.7 25.0 2.4 3.0 12.1 7.0
Table 11: All goal success results (%) for action sequencing and subgoal decomposition.
Model Action Sequencing Subgoal Decomposition
State Goal Relation Goal Action Goal Total State Goal Relation Goal Action Goal Total
V B V B V B V B V B V B V B V B
Claude-3 Haiku 71.2 27.0 53.9 38.7 41.9 - 58.9 35.5 89.4 26.0 82.2 34.8 71.6 - 83.1 32.4
Claude-3 Sonnet 80.2 41.0 68.3 59.8 38.5 - 66.5 54.6 89.1 37.0 89.3 49.8 83.3 - 88.0 46.3
Claude-3 Opus 57.2 45.0 77.8 53.0 54.7 - 62.7 50.8 92.4 43.0 88.6 41.6 83.3 - 89.1 42.0
Claude-3.5 Sonnet 87.8 63.0 83.3 62.4 60.8 - 79.9 62.6 92.9 41.0 88.6 39.5 87.0 - 90.1 39.9
Gemini 1.0 Pro 70.5 28.0 41.7 32.0 34.5 - 53.1 30.9 84.4 26.0 61.5 31.1 72.8 - 73.5 29.7
Gemini 1.5 Flash 86.0 34.0 73.3 50.0 47.3 - 72.8 45.6 93.5 44.0 88.3 36.0 92.0 - 91.3 38.2
Gemini 1.5 Pro 85.6 41.0 76.7 43.2 62.2 - 77.2 42.6 91.2 31.0 72.5 37.1 89.5 - 83.9 35.4
GPT-3.5-turbo 49.3 20.0 25.0 22.6 33.8 - 38.3 21.9 84.7 28.0 54.4 28.5 64.8 - 69.4 28.3
GPT-4-turbo 75.9 39.0 75.0 39.5 48.6 - 69.0 39.3 93.5 45.0 84.2 46.1 90.7 - 89.5 45.8
GPT-4o 87.1 49.0 76.1 45.5 56.1 - 76.2 46.5 92.1 50.0 84.2 53.2 93.2 - 89.4 52.3
Cohere Command R 46.8 20.0 38.9 25.9 48.6 - 44.9 24.3 85.3 20.0 67.4 21.4 60.5 - 73.6 21.0
Cohere Command R+ 68.7 28.0 54.4 32.0 50.7 - 60.1 30.9 89.4 34.0 66.8 29.6 75.9 - 78.3 30.8
Mistral Large 88.5 38.5 81.1 41.2 70.9 - 82.0 40.4 92.9 33.0 71.5 35.6 90.1 - 84.4 34.9
Mixtral 8x22B MoE 69.4 30.0 62.2 36.8 57.4 - 64.4 35.0 92.1 30.0 74.8 34.1 87.7 - 84.8 33.0
Llama 3 8B 34.2 16.0 16.7 23.7 11.5 - 23.4 21.6 68.8 21.0 54.7 23.6 50.0 - 59.8 22.9
Llama 3 70B 61.5 31.0 68.3 45.5 42.6 - 58.9 41.5 93.2 25.0 63.4 27.7 82.7 - 80.0 27.0
o1-mini 88.5 64.0 72.2 66.9 57.4 - 76.1 66.1 89.7 28.0 68.8 38.0 81.5 - 80.3 35.3
o1-preview 80.9 89.5 65.0 84.4 46.6 - 67.8 85.8 91.8 56.5 88.3 69.4 92.6 - 90.6 65.9

E.2.2 Case Studies

The preserved case figures show how formal, temporal, affordance, and goal errors differ.

Grammar Errors

Examples include duplicated punctuation, unmatched closing parentheses, incorrect predicate arity, and object identifiers absent from the scene. Such failures are detected before executing a subgoal realization.

Refer to caption
(a) VirtualHome
Refer to caption
(b) BEHAVIOR
Figure 26: Parsing error examples for subgoal decomposition.

Runtime Errors

Runtime cases include filling both hands before a plug action, dropping soap before cleaning, retrieving items from closed containers, trying to hold a fridge or slice a knife, and opening an already open container. The distinctions rely on both present state and historical prerequisite satisfaction.

Refer to caption
(a) VirtualHome
Refer to caption
(b) BEHAVIOR
Figure 27: Trajectory runtime error examples for subgoal decomposition.

Goal Satisfaction Errors

A plan may execute yet omit an intended light/window state, required placement, or DRINK action. Another example satisfies one adjacency relation and later destroys it while moving a shared object. Satisfying each subgoal once is therefore insufficient when final conditions must hold together.

Refer to caption
(a) VirtualHome
Refer to caption
(b) BEHAVIOR
Figure 28: Goal satisfaction error examples for subgoal decompositions
Takeaways of Subgoal Decomposition
(1)

Top-performing Models: o1-preview demonstrates superior performance in both VirtualHome and BEHAVIOR simulator compared with other SOTA LLMs, achieving the highest task success rates of 89.4% and 57.0%. Claude-3.5 Sonnet and Gemini 1.5 Flash also demonstrate high performance in VirtualHome with a success rate close to 90%.

(2)

Grammar and Runtime Errors: State-of-the-art models generally avoid grammar errors but can hallucinate actions and objects. The most common runtime errors in VirtualHome are additional steps, whereas most of runtime errors in BEHAVIOR are missing steps.

(3)

Goal Satisfaction: Stronger LLMs like o1-preview and GPT-4o show higher accuracy in action goal success rates in VirtualHome compared to weaker models like Llama 3 8B. However, achieving state and relation goals in BEHAVIOR is challenging due to more complex task representations and stricter precondition checks.

(4)

Impact of Task Complexity In Different Simulators: Overall LLM performance is lower in BEHAVIOR compared with VirtualHome due to complex task representations involving quantifiers like forall and forpairs, which articulate complex temporal and spatial requirements.

E.3 Action Sequencing

Action-sequencing rankings differ between the relatively short VirtualHome plans and longer BEHAVIOR tasks. The text alternates Mistral Large and Mixtral Large naming and retains an outdated claim that the best BEHAVIOR model completes only half the tasks, whereas its updated results report o1-preview at 81%; the tables should govern quantitative comparisons. Strong models make fewer syntax errors but still omit tool/sink/container prerequisites, act in a destructive order, or repeat completed actions. State-versus-relation difficulty also depends on simulator semantics. Explanations involving overthinking, model training quality, or memory failures are hypotheses rather than controlled causal findings.

Takeaways of Action Sequencing
1.

Larger models did not always outperform smaller ones, especially for shorter and simpler tasks.

2.

o1-preview takes the lead. Other LLMs (including powerful ones like GPT-4-turbo, Gemini 1.5 Pro, and Claude-3 Opus) consistently outperformed others across all runtime error metrics.

3.

Generally, larger model sizes and higher quality training data correlated with lower grammar error rates.

4.

Commonsense reasoning about object properties, agent states, and preconditions for actions remains a significant challenge for current LLMs, particularly when given limited context.

5.

Errors in goal satisfaction can be attributed to: 1) Missed goals: Failing to generate actions to achieve the goal. 2) Wrong actions: Incorrectly mapping goals to inappropriate actions.

Table 12: Trajectory evaluation results (%) for action sequencing. Full results.
Model Goal Evaluation Trajectory Evaluation
Task SR Execution SR Grammar Error (↓\downarrow) Runtime Error (↓\downarrow)
Parsing Hallucination Predicate-Arg Num Wrong Order Missing Step Affordance Additional Step
V B V B V B V B V B V B V B V B V B
Claude-3 Haiku 54.8 26.0 60.7 32.0 0.0 0.0 6.6 6.0 0.0 0.0 0.7 7.0 30.5 54.0 1.6 1.0 1.6 1.0
Claude-3 Sonnet 58.4 44.0 63.3 57.0 0.0 0.0 4.9 1.0 0.0 7.9 4.6 11.0 26.9 19.0 0.3 11.0 3.0 2.0
Claude-3 Opus 64.9 51.0 69.5 59.0 0.0 0.0 17.0 0.0 0.0 0.0 0.3 3.0 12.8 35.0 0.3 3.0 2.3 2.0
Claude-3.5 Sonnet 76.7 60.0 81.3 69.0 0.0 0.0 2.0 0.0 0.3 0.0 0.3 5.0 14.4 25.0 1.6 1.0 1.3 2.0
Gemini 1.0 Pro 45.6 27.0 56.7 32.0 0.0 7.0 3.6 3.0 2.6 6.0 2.0 13.0 33.4 35.0 1.6 4.0 8.2 4.0
Gemini 1.5 Flash 69.5 40.0 75.4 52.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 5.0 22.3 42.0 0.3 1.0 1.6 2.0
Gemini 1.5 Pro 76.7 42.0 83.6 54.0 0.0 0.0 1.3 0.0 0.7 0.0 0.3 6.0 14.1 39.0 0.0 1.0 3.0 2.0
GPT-3.5-turbo 24.9 16.0 40.7 20.0 10.2 4.0 1.0 7.0 2.0 23.0 0.0 1.0 41.6 36.0 4.6 8.0 3.0 1.3
GPT-4-turbo 60.0 38.0 65.2 45.0 0.0 0.0 2.0 0.0 1.6 0.0 0.0 7.0 30.8 47.0 0.3 1.0 1.3 0.0
GPT-4o 71.5 47.0 81.3 53.0 1.0 0.0 2.0 1.0 0.7 0.0 0.3 9.0 15.1 36.0 0.0 1.0 2.3 0.0
Cohere Command R 44.9 16.0 44.3 19.0 2.0 5.0 23.3 13.0 2.6 0.0 2.0 8.0 17.0 43.0 9.8 12.0 3.6 4.0
Cohere Command R+ 54.1 27.0 65.2 35.0 6.9 0.0 12.5 1.0 2.0 15.0 0.3 10.0 11.8 39.0 1.6 0.0 3.9 15.0
Mistral Large 78.4 33.0 84.6 50.0 0.0 0.0 2.0 0.0 0.3 0.0 0.0 8.0 12.5 35.0 0.7 6.0 3.9 7.0
Mixtral 8x22B MoE 63.3 30.0 67.9 40.0 0.0 3.0 13.1 6.0 1.0 0.0 1.3 10.0 14.1 32.0 2.6 9.0 5.6 2.0
Llama 3 8B 21.3 10.0 23.6 16.0 1.0 0.0 34.1 15.0 8.2 9.0 1.0 6.0 30.8 44.0 1.3 9.0 2.6 5.0
Llama 3 70B 59.0 34.0 66.6 42.0 0.0 0.0 14.1 2.0 8.2 0.0 2.0 15.0 9.2 38.0 0.0 3.0 6.2 6.0
o1-mini 71.5 56.0 76.4 65.0 4.9 0.0 2.0 3.0 0.0 0.0 1.0 7.0 17.7 17.0 0.3 6.0 2.6 5.0
o1-preview 65.2 81.0 72.5 91.0 6.6 0.0 11.5 0.0 0.0 0.0 0.0 0.0 12.1 6.0 0.3 2.0 2.0 3.0
Refer to caption
Figure 29: Grammar error examples for action sequencing.
Refer to caption
Figure 30: Trajectory runtime error examples for action sequencing.
Refer to caption
Figure 31: Goal satisfaction error examples for action sequencing.

E.4 Transition Modeling

Full transition-model results are retained for logical matching and planner success, including category breakdowns.

Table 13: Full results of logic form accuracy for transition modeling in VirtualHome
Model Object States Object Orientation Object Affordance Spatial Relations Non-Spatial Relations
Precision Recall F1 Precision Recall F1 Precision Recall F1 Precision Recall F1 Precision Recall F1
Claude-3 Haiku 76.0 40.1 52.5 19.0 34.4 24.4 67.8 73.9 70.7 37.7 38.7 38.2 2.0 1.5 1.7
Claude-3 Opus 87.4 49.2 63.0 46.3 96.9 62.6 76.8 74.3 75.5 37.6 39.9 38.7 10.4 5.2 7.0
Claude-3 Sonnet 76.6 37.4 50.3 48.1 78.1 59.5 60.7 74.3 66.8 32.3 39.9 35.7 6.2 4.1 4.9
Claude-3.5 Sonnet 86.1 46.7 60.5 93.9 96.9 95.3 77.7 75.5 76.6 45.3 39.8 42.4 7.1 5.1 5.9
Cohere Command R 18.0 6.8 9.9 38.7 90.6 54.2 40.2 23.0 29.2 12.6 6.7 8.8 3.3 0.9 1.4
Cohere Command R+ 44.9 19.0 26.3 34.6 68.8 45.9 51.0 62.1 56.0 30.1 34.8 32.4 7.6 3.1 4.4
Gemini 1.0 Pro 68.4 12.3 20.4 16.3 62.5 27.9 55.3 20.1 29.6 45.0 16.5 24.3 7.7 2.5 3.8
Gemini 1.5 Flash 82.3 37.6 51.6 2.0 3.1 2.5 54.4 74.7 62.9 47.4 42.9 45.0 16.3 5.2 7.9
Gemini 1.5 Pro 45.3 11.9 18.8 88.2 93.8 90.9 79.9 75.5 77.7 42.2 35.8 38.7 15.5 5.2 7.8
GPT-3.5-turbo 63.5 21.9 32.5 11.4 15.6 13.2 57.2 53.1 54.9 35.2 21.7 26.8 1.7 0.3 0.6
GPT-4-turbo 79.3 44.2 56.7 10.1 31.3 15.3 65.9 71.0 68.4 31.8 34.2 32.9 3.8 1.0 1.6
GPT-4o 80.2 41.5 54.6 48.0 59.4 52.8 76.2 73.7 74.9 40.8 40.7 40.8 14.8 5.1 7.5
Llama 3 8b 30.8 13.7 18.9 0.0 0.0 0.0 1.6 3.2 2.1 15.5 18.2 16.8 0.0 0.0 0.0
Llama 3 70b 63.5 21.9 32.5 49.0 66.3 56.6 65.0 50.0 57.0 27.0 27.0 27.0 5.0 2.0 3.0
Mistral Large 30.0 8.0 13.0 48.0 88.0 62.0 72.0 29.0 41.0 35.0 18.0 24.0 3.0 1.0 1.0
Mixtral 8x22B MoE 72.0 33.0 45.0 43.0 83.0 57.0 64.0 74.0 69.0 40.0 38.0 39.0 12.0 4.0 6.0
o1-mini 82.5 45.9 59.0 51.3 62.5 56.3 59.8 57.1 58.5 32.1 32.8 32.5 5.0 4.1 4.5
o1-preview 83.0 45.1 58.5 69.0 90.6 78.4 84.7 71.4 77.5 39.8 37.8 38.8 17.1 9.0 11.8
Table 14: Full results of logic form accuracy for transition modeling in BEHAVIOR
Model Object States Spatial Relations Non-Spatial Relations
Precision Recall F1 Precision Recall F1 Precision Recall F1
Claude-3.5 Sonnet 83.3 74.8 78.8 73.3 48.8 58.6 82.9 66.2 73.6
Claude-3 Haiku 64.1 55.2 59.3 54.7 37.4 44.4 63.3 51.4 56.7
Claude-3 Opus 74.6 69.4 71.9 70.4 44.6 54.6 68.5 69.1 68.8
Claude-3 Sonnet 66.2 68.7 67.5 62.8 39.8 48.7 68.8 52.0 59.2
Cohere Command R 59.7 43.9 50.6 29.1 11.6 16.6 27.2 15.3 19.6
Cohere Command R+ 58.0 58.4 58.2 54.2 33.6 41.5 53.0 56.6 54.7
Gemini 1.0 Pro 67.2 55.2 60.6 47.5 35.3 40.5 43.8 48.3 45.9
Gemini 1.5 Flash 73.9 57.2 64.5 54.5 40.7 46.6 60.7 53.8 57.0
Gemini 1.5 Pro 69.6 46.7 55.9 52.9 27.2 35.9 59.6 47.4 52.8
GPT-3.5-turbo 67.1 46.1 54.6 57.6 31.6 40.9 40.8 36.1 38.3
GPT-4-turbo 58.2 59.4 58.8 50.3 27.8 35.8 58.5 38.4 46.4
GPT-4o 73.1 69.6 71.3 63.9 35.8 45.9 84.7 64.2 73.0
Llama 3 70b 68.1 64.6 66.3 60.3 38.8 47.2 65.1 53.8 58.9
Llama 3 8b 40.3 32.4 35.9 29.6 22.7 25.7 48.9 43.9 46.2
Mistral Large 67.5 66.5 67.0 54.9 32.3 40.7 59.7 44.6 51.1
Mixtral 8x22B MoE 60.2 60.0 60.1 53.2 39.9 45.6 57.9 55.8 56.8
o1-mini 46.3 37.2 41.3 71.1 42.3 53.1 80.1 58.3 67.5
o1-preview 85.5 72.3 78.3 72.4 46.1 56.3 88.0 79.5 83.5
Table 15: Full results of planner success rate for transition modeling (%\%)
Model Object States Object Orientation Object Affordance Spatial Relations Non-Spatial Relations
V B V B V B V B V B
Claude-3 Haiku 13.5 68.9 3.6 - 19.8 - 46.9 62.8 73.0 62.3
Claude-3 Opus 63.5 84.4 71.4 - 58.7 - 64.8 80.9 55.4 82.0
Claude-3 Sonnet 11.2 80.0 3.6 - 10.8 - 20.0 79.8 13.5 80.3
Claude-3.5 Sonnet 67.4 86.7 96.4 - 67.8 - 96.6 80.8 91.9 80.3
Cohere Command R 44.6 48.9 82.1 - 40.1 - 62.6 38.3 58.3 39.3
Cohere Command R+ 36.5 77.8 46.4 - 35.3 - 40.7 57.4 31.1 47.5
Gemini 1.0 Pro 10.7 22.2 0.0 - 10.2 - 14.5 13.8 2.7 14.8
Gemini 1.5 Flash 34.8 55.6 7.1 - 46.7 - 61.4 68.1 60.8 70.5
Gemini 1.5 Pro 94.4 35.6 89.3 - 95.8 - 89.0 40.4 83.8 39.3
GPT-3.5-turbo 1.1 26.7 25.0 - 1.2 - 0.0 39.4 0.0 54.1
GPT-4-turbo 51.7 40.0 50.0 - 47.9 - 67.6 44.7 64.9 52.5
GPT-4o 71.9 68.9 78.6 - 63.5 - 66.9 64.9 68.9 68.9
Llama 3 8b 27.0 35.6 0.0 - 26.4 - 37.9 27.7 31.1 26.2
Llama 3 70b 10.1 68.9 3.6 - 6.6 - 15.2 77.7 18.9 85.2
Mistral Large 15.7 73.3 7.1 - 14.4 - 17.9 76.6 8.1 80.3
Mixtral 8x22B MoE 36.5 57.8 50.0 - 28.1 - 44.1 52.1 43.2 57.4
o1-mini 63.5 77.8 82.1 - 59.3 - 75.9 77.7 71.6 75.4
o1-preview 69.1 86.7 100.0 - 67.1 - 76.6 89.4 78.4 90.2

E.4.1 Logic Form Accuracy

Accuracy varies across object states, affordances, orientation, spatial relations, and non-spatial relations. Complex possession and container conditions are difficult to predict exactly. Claude-3.5 Sonnet performs strongly in several categories, while o1-preview is notably strong on BEHAVIOR non-spatial relations. Some narrative category-leader statements conflict with updated tables; they do not establish architectural or training causes for performance differences.

E.4.2 Planner Success Rate Analysis

Planner success measures whether the predicted operators collectively admit a solution. Simplified operators may score well despite missing real-world prerequisites, so high success can reflect internal consistency rather than faithful dynamics. The reported 100% o1-preview VirtualHome orientation result and 90.2% BEHAVIOR non-spatial result must be interpreted in this sense.

Original figure
Figure 32: Transition modeling error examples

E.4.3 Error categorization

Missing predicates omit necessary conditions; extra predicates may overconstrain actions, put effects on the wrong object, or introduce unsupported properties. A detailed plug-in example shows that an entirely predicted domain can remain mutually consistent while mixing one predicted operator into reference definitions breaks planning. Thus operator replacement tests both individual correctness and compatibility with the surrounding domain. The three full planning examples preserve the exact differences.

Transition Models all from the Ground Truth - EXCUTION SUCCESS
(:action plug_in
  :parameters (?char - character ?obj - object)
  :precondition (or (and (next_to ?char ?obj) (has_plug ?obj)
                (plugged_out ?obj)) (and (next_to ?char ?obj)
                (has_switch ?obj) (plugged_out ?obj)))
  :effect (and (plugged_in ?obj) (not (plugged_out ?obj)))
)

(:action walk_towards
  :parameters (?char - character ?obj - object)
  :precondition (and (not (sitting ?char)) (not (lying ?char))
              (next_to ?char ?obj) (forall (?far_obj - object)
              (when (not (obj_next_to ?far_obj ?obj))
              (not (next_to ?char ?far_obj)))) (forall (?close_obj - object)
              (when (obj_next_to ?close_obj ?obj)
              (next_to ?char ?close_obj))))
  :effect (and (next_to ?char ?obj))
)

(:action switch_on
  :parameters (?char - character ?obj - object)
  :precondition (and (has_switch ?obj) (off ?obj)
              (plugged_in ?obj) (next_to ?char ?obj))
  :effect (and (on ?obj) (not (off ?obj)))
)
Figure 33: If we use a high-level planner (PDDL planner) over Ground Truth transition models of plug_in, walk_towards and switch_on, the output action sequence can be executed successfully.
Transition Models all from LLMs - EXECUTION SUCCESS
(:action plug_in
  :parameters (?char - character ?obj - object)
  :precondition (and (has_plug ?obj) (plugged_out ?obj)
              (next_to ?char ?obj))
  :effect (and (plugged_in ?obj) (not (plugged_out ?obj)))
)

(:action walk_towards
  :parameters (?char - character ?obj - object)
  :precondition (and (inside ?char ?room) (inside ?obj ?room))
  :effect (next_to ?char ?obj)
)

(:action switch_on
  :parameters (?char - character ?obj - object)
  :precondition (and (has_switch ?obj) (off ?obj)
              (next_to ?char ?obj))
  :effect (and (on ?obj) (not (off ?obj)))
)
Figure 34: If we use a high-level planner (PDDL planner) over GPT-4o predicted transition models of plug_in, walk_towards and switch_on, the output action sequence can be executed successfully.
plug_in from LLMs and walk_towards and switch_on from Ground Truth - EXECUTION FAILURE
(:action plug_in
  :parameters (?char - character ?obj - object)
  :precondition (and (has_plug ?obj) (plugged_out ?obj)
              (next_to ?char ?obj))
  :effect (and (plugged_in ?obj) (not (plugged_out ?obj)))
)

(:action walk_towards
  :parameters (?char - character ?obj - object)
  :precondition (and (not (sitting ?char)) (not (lying ?char))
              (next_to ?char ?obj) (forall (?far_obj - object)
              (when (not (obj_next_to ?far_obj ?obj))
              (not (next_to ?char ?far_obj)))) (forall (?close_obj - object)
              (when (obj_next_to ?close_obj ?obj)
              (next_to ?char ?close_obj))))
  :effect (and (next_to ?char ?obj))
)

(:action switch_on
  :parameters (?char - character ?obj - object)
  :precondition (and (has_switch ?obj) (off ?obj)
              (plugged_in ?obj) (next_to ?char ?obj))
  :effect (and (on ?obj) (not (off ?obj)))
)
Figure 35: If we use a high-level planner (PDDL planner) over the mixture of transtion models, i.e., GPT-4o predicted transition models of plug_in with ground truth walk_towards and switch_on, the output action sequence cannot be executed successfully.
Takeaways of Transition Modeling
(1)

Specialized Performance: Models such as o1-preview and Claude-3 Opus excel in specific categories like object states and object orientation, respectively, suggesting that targeted training or specialized architectures enhance LLM capabilities in understanding different types of tasks in transition modeling.

(2)

Difficulty with Non-Spatial Relations: Across various models, non-spatial relations consistently pose a challenge, highlighting a gap in the ability of LLMs to grasp complex relational dynamics that are crucial for realistic scenario modeling.

(3)

Consistency in Action Space: The effectiveness of planning relies heavily on the consistency of the predicted action space by LLMs; discrepancies between mixed predicted and ground truth actions lead to reduced planner success, emphasizing the need for coherent action definitions in LLM outputs.

E.5 Correlection with Action Length and Goal Complexity

For GPT-4o action sequencing in BEHAVIOR, success generally falls with more goals and longer reference plans. Tool-dependent actions such as soaking, cleaning, and slicing are difficult. The zero slicing success is based on only two cases, so it is not a precise estimate of general slicing competence. These are associations within this task set, not controlled evidence that one complexity factor alone causes failure.

Original figure
(a) Success Rate vs Action Length
Original figure
(b) Success Rate vs Total Goals
Original figure
(c) Success Rate vs Edge Goals
Original figure
(d) Success Rate vs Node Goals
Figure 36: BEHAVIOR Success Rate as a Function of Different Cofactors
Original figure
(a) Success Rate per Action
Original figure
(b) Success Rate per Predicate
Figure 37: BEHAVIOR Success Rate Analysis for Actions and Predicates
Takeaways of Error Factor Analysis
1.

Task complexity adversely affects goal success rates, including the number of total goals, state goals, relation goals, and ground-truth action sequence length.

2.

Complexity of action preconditions adversely affects success rates, especially for actions requiring tool usage, e.g., Soak, Slice, and Clean.

3.

Actions performed with the right hand have a slightly higher success rate than those with the left hand.

Appendix F Sensitivity Analysis

F.1 Motivation and Problem Formulation

Sensitivity analysis replaces one reference operator with its predicted counterpart and asks whether the reference planning problem remains solvable. This localizes disruptive definitions that an all-predicted, mutually consistent domain can conceal.

F.2 Implementation Details

For every task-relevant operator, the evaluator changes only that definition, runs a PDDL planner, and aggregates results by object/relational task category.

F.3 Result Analysis

VirtualHome plug_in and spatial movement definitions are particularly sensitive; plug_in must handle the benchmark’s switch-related cases as well as explicit plugs. BEHAVIOR slicing and some possession/placement definitions also disrupt planning. All ten sensitivity charts are retained. These tests assess symbolic domain compatibility and the selected task distribution; the narrative’s references to motor precision or deficiencies in training data are not directly tested by this experiment.

Refer to caption
Figure 38: VirtualHome sensitivity analysis per action
Refer to caption
Figure 39: Sensitivity analysis for object states tasks in VirtualHome
Refer to caption
Figure 40: Sensitivity analysis for object affordance tasks in VirtualHome
Refer to caption
Figure 41: Sensitivity analysis for object orientation tasks in VirtualHome
Refer to caption
Figure 42: Sensitivity analysis for spatial relations tasks in VirtualHome
Refer to caption
Figure 43: Sensitivity analysis for non-spatial relations tasks in VirtualHome
Refer to caption
Figure 44: BEHAVIOR sensitivity analysis per action
Refer to caption
Figure 45: Sensitivity analysis for object states tasks in BEHAVIOR
Refer to caption
Figure 46: Sensitivity analysis for spatial relations tasks in BEHAVIOR
Refer to caption
Figure 47: Sensitivity analysis for non-spatial relations tasks in BEHAVIOR
Takeaways of Sensitivity Analysis
(1)

Action-Specific Sensitivities: The analysis reveals significant variations in the success rates for specific actions across tasks, with actions like "plug_in" and "walk_towards" consistently showing low success due to complex preconditions and spatial requirements. This underscores the need for LLMs to better understand and generate actions with multi-faceted constraints.

(2)

Challenges with Complex Interactions: Complex interactions involving detailed object manipulation, such as "slice_carvingknife" and "place_inside," present notable challenges, highlighting deficiencies in the model’s training on precise movements and tool use. Enhancing data representation for such scenarios in training sets is crucial for improving performance.

(3)

Robustness in Basic Tasks vs. Nuanced Deficiencies: While the model performs well in straightforward tasks, it struggles with nuanced actions that require fine motor skills or advanced spatial awareness. This contrast points to the necessity of incorporating more complex and varied interaction scenarios in training to refine the model’s capabilities in handling real-world complexities.

(4)

Training and Fine-Tuning Needs: The consistent issues across different types of tasks suggest that the current training regimens may not fully capture the diversity of real-world interactions, especially in spatial and object-oriented tasks. A targeted approach to fine-tuning the model with specialized datasets that focus on underperforming actions could lead to significant improvements in model robustness and applicability in practical settings.

Appendix G Pipeline-Based vs Modularized

G.1 Motivation and Problem Formulation

Two-stage pipelines test the more realistic input condition in which the system starts with a natural-language instruction rather than a reference symbolic goal.

G.2 Implementation Details

In BEHAVIOR, predicted goal interpretation feeds either action sequencing or subgoal decomposition. These are compared with the corresponding isolated modules supplied with reference goals.

G.3 Result Analysis

Executability can remain similar while full goal success falls, because a downstream module can produce a feasible plan for the wrong interpreted goal. For example, the appendix summary reports GPT-4o decomposition task success falling from 48% to 38% under composition. This supports separate measurement of goal correctness and action feasibility rather than assuming one implies the other.

Takeaways of Pipeline-Based vs Modularized
(1)

Similar Trajectory Execution Rates: Both modularized and pipeline-based methods have similar trajectory executable rates for pipelines 𝒢+𝒬{\mathcal{G}}+{\mathcal{Q}} and 𝒢+Φ{\mathcal{G}}+\Phi.

(2)

Error Accumulation in Pipelines: Pipeline-based methods suffer from error accumulation, particularly due to the composition of two modules, which impacts their performance negatively.

(3)

Grammar Error Trends: State-of-the-art (SOTA) LLMs generally avoid grammar errors for both pipeline-based and modularized methods, unlike less advanced models such as Gemini 1.0 Pro, GPT-3.5-turbo, and some open-sourced LLMs.

(4)

Runtime Errors Persist: All LLMs, regardless of their level of advancement, are prone to runtime errors, missing necessary steps in their generation process.

Table 16: Pipeline-based evaluation results for (1)𝒢+𝒬{\mathcal{G}}+{\mathcal{Q}} and (2) 𝒢+Φ{\mathcal{G}}+\Phi in BEHAVIOR . 𝒢{\mathcal{G}}: Goal Interpretation. 𝒬{\mathcal{Q}}: Action Sequencing. Φ\Phi: Subgoal Decomposition. In this table, M means ‘modularized’, whereas P means ‘pipeline-based’.
Model Goal Evaluation Trajectory Evaluation
Task SR Execution SR Grammar Error (↓\downarrow) Runtime Error (↓\downarrow)
Parsing Hallucination Predicate-Arg Num Wrong Order Missing Step Affordance Additional Step
M P M P M P M P M P M P M P M P M P
Goal Interpretation + Action Sequencing
Claude-3 Haiku 26.0 21.0 32.0 29.0 0.0 0.0 6.0 6.0 0.0 0.0 7.0 6.0 54.0 52.0 1.0 7.0 1.0 17.0
Claude-3 Sonnet 44.0 41.0 57.0 53.0 0.0 0.0 1.0 3.0 0.0 0.0 11.0 14.0 19.0 21.0 11.0 9.0 2.0 12.0
Claude-3 Opus 51.0 46.0 59.0 54.0 0.0 1.0 0.0 1.0 0.0 0.0 3.0 6.0 35.0 35.0 3.0 3.0 2.0 4.0
Gemini 1.0 Pro 27.0 26.0 32.0 35.0 7.0 5.0 3.0 3.0 6.0 6.0 13.0 14.0 35.0 38.0 4.0 2.0 4.0 11.0
Gemini 1.5 Flash 40.0 35.0 52.0 49.0 0.0 0.0 0.0 2.0 0.0 0.0 5.0 10.0 42.0 41.0 1.0 0.0 2.0 7.0
Gemini 1.5 Pro 42.0 37.0 54.0 55.0 0.0 1.0 0.0 1.0 0.0 0.0 6.0 7.0 39.0 35.0 1.0 1.0 2.0 0.0
GPT-3.5-turbo 16.0 14.0 20.0 32.0 4.0 1.0 7.0 3.0 23.0 15.0 1.0 5.0 36.0 39.0 8.0 6.0 1.0 3.0
GPT-4-turbo 38.0 32.0 45.0 47.0 0.0 1.0 0.0 1.0 0.0 0.0 7.0 9.0 47.0 41.0 1.0 1.0 0.0 0.0
GPT-4o 47.0 42.0 53.0 55.0 0.0 0.0 1.0 3.0 0.0 0.0 9.0 6.0 36.0 35.0 1.0 1.0 0.0 4.0
Cohere Command R 16.0 5.0 19.0 9.0 5.0 3.0 13.0 38.0 0.0 1.0 8.0 8.0 43.0 31.0 12.0 12.0 4.0 8.0
Cohere Command R+ 27.0 15.0 35.0 29.0 0.0 0.0 1.0 8.0 15.0 14.0 10.0 30.0 39.0 31.0 0.0 2.0 15.0 22.0
Mistral Large 33.0 31.0 50.0 38.0 0.0 0.0 0.0 3.0 0.0 0.0 8.0 14.0 35.0 37.0 6.0 8.0 7.0 5.0
Mixtral 8x22B MoE 30.0 26.0 40.0 36.0 3.0 3.0 6.0 13.0 0.0 0.0 10.0 14.0 32.0 21.0 9.0 13.0 2.0 15.0
Llama 3 8B 10.0 0.0 16.0 5.0 0.0 2.0 15.0 25.0 9.0 6.0 6.0 11.0 44.0 34.0 9.0 17.0 5.0 14.0
Llama 3 70B 34.0 26.0 42.0 40.0 0.0 1.0 2.0 3.0 0.0 0.0 15.0 18.0 38.0 35.0 3.0 5.0 6.0 9.0
Goal Interpretation + Subgoal Decomposition
Claude-3 Haiku 29.0 21.0 35.0 40.0 0.0 0.0 1.0 5.0 0.0 0.0 2.0 2.0 59.0 46.0 3.0 7.0 3.0 16.0
Claude-3 Sonnet 38.0 31.0 43.0 45.0 0.0 0.0 2.0 3.0 0.0 0.0 3.0 2.0 51.0 47.0 1.0 3.0 3.0 18.0
Claude-3 Opus 39.0 35.0 47.0 45.0 0.0 0.0 3.0 8.0 0.0 0.0 5.0 4.0 45.0 42.0 0.0 1.0 5.0 7.0
Gemini 1.0 Pro 23.0 14.0 33.0 30.0 2.0 0.0 4.0 10.0 0.0 1.0 3.0 1.0 51.0 45.0 7.0 13.0 3.0 17.0
Gemini 1.5 Flash 34.0 32.0 42.0 44.0 2.0 1.0 1.0 3.0 0.0 0.0 2.0 2.0 53.0 48.0 0.0 2.0 3.0 7.0
Gemini 1.5 Pro 31.0 26.0 37.0 38.0 0.0 1.0 1.0 3.0 0.0 0.0 3.0 2.0 59.0 56.0 0.0 0.0 2.0 1.0
GPT-3.5-turbo 24.0 14.0 36.0 27.0 2.0 0.0 3.0 12.0 0.0 22.0 3.0 1.0 52.0 32.0 4.0 6.0 3.0 5.0
GPT-4-turbo 37.0 37.0 47.0 49.0 0.0 0.0 3.0 4.0 0.0 0.0 9.0 8.0 40.0 37.0 1.0 2.0 6.0 6.0
GPT-4o 48.0 38.0 55.0 52.0 0.0 0.0 3.0 4.0 0.0 0.0 5.0 6.0 37.0 35.0 0.0 3.0 5.0 9.0
Cohere Command R 15.0 8.0 25.0 15.0 21.0 13.0 11.0 32.0 0.0 1.0 0.0 1.0 38.0 32.0 4.0 6.0 4.0 12.0
Cohere Command R+ 24.0 17.0 37.0 31.0 2.0 6.0 4.0 10.0 0.0 2.0 5.0 7.0 51.0 40.0 1.0 4.0 6.0 14.0
Mistral Large 30.0 22.0 38.0 29.0 1.0 1.0 3.0 12.0 0.0 1.0 4.0 5.0 52.0 50.0 2.0 2.0 1.0 5.0
Mixtral 8x22B MoE 27.0 22.0 33.0 29.0 0.0 0.0 4.0 9.0 0.0 2.0 2.0 2.0 59.0 45.0 2.0 13.0 0.0 17.0
Llama 3 8B 21.0 3.0 29.0 14.0 2.0 7.0 11.0 29.0 0.0 2.0 6.0 3.0 44.0 30.0 8.0 15.0 7.0 7.0
Llama 3 70B 20.0 19.0 30.0 31.0 1.0 1.0 5.0 22.0 1.0 1.0 8.0 7.0 51.0 35.0 4.0 3.0 4.0 7.0

Appendix H Replanning and Feedback

H.1 Motivation and Problem Formulation

Replanning tests whether a model can use explicit simulator feedback after a failed initial plan.

H.2 Implementation Details

GPT-4o receives its previous plan, categorized execution errors or unmet goal conditions, and the accumulated retry history. At most three replans are allowed. This supplies detailed symbolic feedback, including which requirements remain unsatisfied.

H.3 Result Analysis

In BEHAVIOR action sequencing, the reported task/execution success improves from 47%/53% to 59%/63%. Redundant actions can become more frequent as the model edits prior plans. The gain includes additional attempts and feedback, so it is not directly comparable to a single-call budget.

Table 17: Replanning evaluation results (%) for action sequencing
Model Goal Evaluation Trajectory Evaluation
Task SR Execution SR Grammar Error (↓\downarrow) Runtime Error (↓\downarrow)
Parsing Hallucination Action-Arg Num Wrong Order Missing Step Affordance Additional Step
GPT-4o 65.2 71.8 0.0 1.3 0.7 0.0 25.3 1.0 0.3
GPT-4o w/ replanning 77.4 83.3 0.0 1.3 0.0 0.0 14.1 0.3 0.7
Takeaways of Replanning and Feedback
(1)

Enhanced Adaptability through Replanning: Incorporating the ability for models to replan based on feedback significantly improves their performance, demonstrating over a 10% increase in success rates. This shows that LLMs can effectively learn from their errors and adjust their action sequences to better align with the required task outcomes, enhancing overall adaptability. Replanning also helps to reduce the repetition of similar errors, showcasing the model’s ability to evolve its strategy over successive iterations.

(2)

Risk of Over-Correction: While replanning generally leads to improvements in task execution, it can sometimes result in the over-generation of actions, as indicated by an increased rate of additional step errors. This suggests that while the model is adept at adjusting to feedback, it may also benefit from more precise guidance to avoid introducing unnecessary actions in its revised plans.

H.4 Replanning with Stochastic Actions

A further BEHAVIOR experiment injects stochastic action failures and allows the same three retries. Replanning helps across the tested failure probabilities, with a generally larger gap as failures increase. The complete table preserves the tested probabilities and scores; this is simulated action unreliability rather than measured hardware failure.

Table 18: Replanning for Stochastic Actions on BEHAVIOR (Exp.6)
Fail Probability Method Execution SR (%) Goal SR (%)
0.05 w/o Replanning 60 50.0
w/ Replanning 85 (↑25) 65 (↑15)
0.1 w/o Replanning 25 15.0
w/ Replanning 70 (↑45) 55 (↑40)
0.2 w/o Replanning 10 5.0
w/ Replanning 65 (↑55) 45 (↑40)

Appendix I Prompt and Analysis

I.1 Prompt of Goal Interpretation

The full goal-interpretation template is preserved, including the expected structured goal output and supplied environment vocabulary.

# Behavior Goal Interpretation Prompt Template

SYSTEM_PROMPT = "For this task, please only output a parsable json string inside brackets. Please start your answer with { and end your answer with }. Don't include any notes or explanations with the output json string."
    
USER_PROMPT = "You are a helpful assistant for goal interpretation in an embodied environment. You should only output in json format. Your task is to understand natural language goals for a household robot, reason about the object states and relationships, and turn natural language goals into symbolic goal states in the designated format. The goals include: unary goals describing one object's own unary states, and binary goals describing object-object binary relationships. The input will be the goal's name, the goal's description, relevant objects as well as their possible unary states, and all initial unary and binary states. The output should be the symbolic version of the goal states. 
    
    Relevant objects in the scene indicates those objects involved in the action execution initially. It will include the object name, and the object's all possible unary states (In goal conditions, each state can be set to true or not true). It follows the format: object name including object id, possible unary states: ...(all possible unary states). Your proposed unary object states should be within the following set: {'Cooked', 'Open', 'Frozen', 'Dusty', 'Stained', 'Sliced', 'Soaked', 'Toggled_On'}
    
    Relevant objects in the scene are:
    <object_in_scene>
    
    All initial states in the scene are:
    <all_initial_states>
        
    Symbolic goals format:
    
    Node goal states should be a set indicating the desired final goal states of single objects. Each goal in the list should be a list with two elements: the first element is the state name, which comes from the set {'Cooked', 'Open', 'Frozen', 'Dusty', 'Stained', 'Sliced', 'Soaked', 'Toggled_On'}; the second element is the object name, which comes from the list of relevant objects in the scene provided above.

    Edge goal states should be a set indicating the desired binary relationships between two objects. Each goal state in the set is a list of three elements: the first element is the state name, which comes from the set {'NextTo', 'Inside', 'OnFloor', 'Touching', 'Under'}, the second and third elements are the object names, with relationship as indicated by the first element. 

    Here is an example of desired node and edge goal states:
    <in_context_example>
    
    Task Name and Goal Instructions:
    <instructions_str>
    
    Now using json format, output just the symbolic version of the goal states without any explanation. Output a single json object string, whose keys are 'node goals' and 'edge goals', and values are your output of symbolic node goals and symbolic edge goals, respectively. That is, your output should be of the format: {'node goals': SYMBOLIC_NODE_GOALS, 'edge goals': SYMBOLIC_EDGE_GOALS}. Also, please strictly follow the aforementioned symbolic goal format."
Figure 48: Examples prompt of goal interpretation.

I.2 Prompt of Subgoal Decomposition

The full decomposition template is preserved, including its declarative subgoal syntax and ordering requirements.

# Behavior Subgoal Decomposition Prompt Template

SYSTEM_PROMPT = '''# Background Introduction
    You are determining the complete state transitions of a household task solving by a robot. The goal is to list all intermediate states (which is called subgoals as a whole) to achieve the final goal states from the initial states. The output consists of a list of boolean expression, which is a combination of the state predicates. Note that your output boolean expression list is in temporal order, therefore, it must be consistent and logical. In short, your task is to output the subgoal plan in the required format.

    # Data Vocabulary Introduction
    Below we introduce the detailed data vocabulary and their format that you can use to generate the subgoal plan.
    ## Available States
    The state is represented as a first-order predicate, which is a tuple of a predicate name and its arguments. Its formal definition looks like this "<PredicateName>(Params)", where <PredicateName> is the state name and each param should be ended with an id. An example is "inside(coin.1, jar.1)". Below is a list of available states and their descriptions.
    <available_behavior_states>
    ## Available Connectives
    The connectives are used to satisfy the complex conditions. They are used to combine the state predicates.
    <available_behavior_connectives>
    # Rules You Must Follow
    - The initial states are the states that are given at the beginning of the task.
    - Your output must be a list of boolean expressions that are in temporal order.
    - You must follow the data format and the available states and connectives defined above.
    - The output must be consistent, logical and as detailed as possible. View your output as a complete state transition from the initial states to the final goal states.
    
    - Please note that the robot can only hold one object in one hand. Also, the robot needs to have at least one hand free to perform any action other than put, place, take, hold.
    - Use holds_rh and holds_lh in your plan if necessary. For example, stained(shoe) cannot directly change to not stained(shoe), but needs intermediate states like [soaked(rag), holds_rh(rag), not stained(shoe)] or [holds_rh(detergent), not stained(shoe)].
    - Your output follows the temporal order line by line. If you think there is no temporal order requirement for certain states, you can use connective "and" to combine them. If you think some states are equivalent, you can use connective "or" to combine them.
    - Please use provided relevant objects well to help you achieve the final goal states. Note that inside(obj1, agent) is an invalid state, therefore you cannot output it in your plan.
    - Do not output redundant states. A redundant state means a state that is either not necessary or has been satisfied before without broken. 
    - Your must strictly follow the json format like this {"output": [<your subgoal plan>]}, where <your subgoal plan> is a list of boolean expressions presented in the temporal order.
    - Start your output with "{" and end with "}". For each line of the output,  DO NOT INCLUDE IRRELEVANT INFORMATION (like number of line, explanation, etc.).
    
    # Example: Task is bottling_fruit
    Below we provide an example for your better understanding.
    <in_context_example>'''
    
USER_PROMPT = '''Now, it is time for you to generate the subgoal plan for the following task.
    # Target Task: <task_name>
    ## Relevant objects in this scene
    <relevant_objects>
    ## Initial States
    <initial_states>
    ## Goal States
    <goal_states>
    ## Output: Based on initial states in this task, achieve final goal states logically and reasonably. It does not matter which state should be satisfied first, as long as all goal states can be satisfied at the end. Make sure your output follows the json format, and do not include irrelevant information, do not include any explanation. Output concrete states and do not use quantifiers like "forall" or "exists".'''
Figure 49: Examples prompt of subgoal decomposition.

I.3 Prompt of Action Sequencing

The full action-sequencing template is preserved, including action names, argument conventions, and structured output instructions.

The reproduced prompt contains several left/right-hand inconsistencies in action descriptions. These are preserved as published and are relevant when interpreting instruction-following failures.

SYSTEM_PROMPT = """For this task, please only output a parsable json string inside brackets. Please start your answer with [ and end your answer with ]. Don't include any notes or explanations with the output json string."""

USER_PROMPT="""

Problem:
You are designing instructions for a household robot. 
The goal is to guide the robot to modify its environment from an initial state to a desired final state. 
The input will be the initial environment state, the target environment state, the objects you can interact with in the environment. 
The output should be a list of action commands so that after the robot executes the action commands sequentially, the environment will change from the initial state to the target state. 

Data format: After # is the explanation.

Format of the states:
The environment state is a list starts with a uniary predicate or a binary prediate, followed by one or two obejcts.
You will be provided with multiple environment states as the initial state and the target state.
For example:
['inside', 'strawberry_0', 'fridge_97'] #strawberry_0 is inside fridge_97
['not', 'sliced', 'peach_0'] #peach_0 is not sliced
['ontop', 'jar_1', 'countertop_84'] #jar_1 is on top of countertop_84

Format of the action commands:
Action commands is a dictionary with the following format:
{{
        'action': 'action_name', 
        'object': 'target_obj_name',
}}

or 

{{
        'action': 'action_name', 
        'object': 'target_obj_name1,target_obj_name2',
}}

The action_name must be one of the following:
LEFT_GRASP # the robot grasps the object with its left hand, to execute the action, the robot's left hand must be empty, e.g. {{'action': 'LEFT_GRASP', 'object': 'apple_0'}}.
RIGHT_GRASP # the robot grasps the object with its right hand, to execute the action, the robot's right hand must be empty, e.g. {{'action': 'RIGHT_GRASP', 'object': 'apple_0'}}.
LEFT_PLACE_ONTOP # the robot places the object in its left hand on top of the target object and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_ONTOP', 'object': 'table_1'}}.
RIGHT_PLACE_ONTOP # the robot places the object in its right hand on top of the target object and release the object in its left hand, e.g. {{'action': 'RIGHT_PLACE_ONTOP', 'object': 'table_1'}}.
LEFT_PLACE_INSIDE # the robot places the object in its left hand inside the target object and release the object in its left hand, to execute the action, the robot's left hand must hold an object, and the target object can't be closed e.g. {{'action': 'LEFT_PLACE_INSIDE', 'object': 'fridge_1'}}.
RIGHT_PLACE_INSIDE # the robot places the object in its right hand inside the target object and release the object in its left hand, to execute the action, the robot's right hand must hold an object, and the target object can't be closed, e.g. {{'action': 'RIGHT_PLACE_INSIDE', 'object': 'fridge_1'}}.
RIGHT_RELEASE # the robot directly releases the object in its right hand, to execute the action, the robot's left hand must hold an object, e.g. {{'action': 'RIGHT_RELEASE', 'object': 'apple_0'}}.
LEFT_RELEASE # the robot directly releases the object in its left hand, to execute the action, the robot's right hand must hold an object, e.g. {{'action': 'LEFT_RELEASE', 'object': 'apple_0'}}.
OPEN # the robot opens the target object, to execute the action, the target object should be openable and closed, also, toggle off the target object first if want to open it, e.g. {{'action': 'OPEN', 'object': 'fridge_1'}}.
CLOSE # the robot closes the target object, to execute the action, the target object should be openable and open, e.g. {{'action': 'CLOSE', 'object': 'fridge_1'}}.
COOK # the robot cooks the target object, to execute the action, the target object should be put in a pan, e.g. {{'action': 'COOK', 'object': 'apple_0'}}.
CLEAN # the robot cleans the target object, to execute the action, the robot should have a cleaning tool such as rag, the cleaning tool should be soaked if possible, or the target object should be put into a toggled on cleaner like a sink or a dishwasher, e.g. {{'action': 'CLEAN', 'object': 'window_0'}}.
FREEZE # the robot freezes the target object e.g. {{'action': 'FREEZE', 'object': 'apple_0'}}.
UNFREEZE # the robot unfreezes the target object, e.g. {{'action': 'UNFREEZE', 'object': 'apple_0'}}.
SLICE # the robot slices the target object, to execute the action, the robot should have a knife in hand, e.g. {{'action': 'SLICE', 'object': 'apple_0'}}.
SOAK # the robot soaks the target object, to execute the action, the target object must be put in a toggled on sink, e.g. {{'action': 'SOAK', 'object': 'rag_0'}}.
DRY # the robot dries the target object, e.g. {{'action': 'DRY', 'object': 'rag_0'}}.
TOGGLE_ON # the robot toggles on the target object, to execute the action, the target object must be closed if the target object is openable and open e.g. {{'action': 'TOGGLE_ON', 'object': 'light_0'}}.
TOGGLE_OFF # the robot toggles off the target object, e.g. {{'action': 'TOGGLE_OFF', 'object': 'light_0'}}.
LEFT_PLACE_NEXTTO # the robot places the object in its left hand next to the target object and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_NEXTTO', 'object': 'table_1'}}.
RIGHT_PLACE_NEXTTO # the robot places the object in its right hand next to the target object and release the object in its right hand, e.g. {{'action': 'RIGHT_PLACE_NEXTTO', 'object': 'table_1'}}.
LEFT_TRANSFER_CONTENTS_INSIDE # the robot transfers the contents in the object in its left hand inside the target object, e.g. {{'action': 'LEFT_TRANSFER_CONTENTS_INSIDE', 'object': 'bow_1'}}.
RIGHT_TRANSFER_CONTENTS_INSIDE # the robot transfers the contents in the object in its right hand inside the target object, e.g. {{'action': 'RIGHT_TRANSFER_CONTENTS_INSIDE', 'object': 'bow_1'}}.
LEFT_TRANSFER_CONTENTS_ONTOP # the robot transfers the contents in the object in its left hand on top of the target object, e.g. {{'action': 'LEFT_TRANSFER_CONTENTS_ONTOP', 'object': 'table_1'}}.
RIGHT_TRANSFER_CONTENTS_ONTOP # the robot transfers the contents in the object in its right hand on top of the target object, e.g. {{'action': 'RIGHT_TRANSFER_CONTENTS_ONTOP', 'object': 'table_1'}}.
LEFT_PLACE_NEXTTO_ONTOP # the robot places the object in its left hand next to target object 1 and on top of the target object 2 and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_NEXTTO_ONTOP', 'object': 'window_0, table_1'}}.
RIGHT_PLACE_NEXTTO_ONTOP # the robot places the object in its right hand next to object 1 and on top of the target object 2 and release the object in its right hand, e.g. {{'action': 'RIGHT_PLACE_NEXTTO_ONTOP', 'object': 'window_0, table_1'}}.
LEFT_PLACE_UNDER # the robot places the object in its left hand under the target object and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_UNDER', 'object': 'table_1'}}.
RIGHT_PLACE_UNDER # the robot places the object in its right hand under the target object and release the object in its right hand, e.g. {{'action': 'RIGHT_PLACE_UNDER', 'object': 'table_1'}}.

Format of the interactable objects:
Interactable object will contain multiple lines, each line is a dictionary with the following format:
{{
    'name': 'object_name',
    'category': 'object_category'
}}
object_name is the name of the object, which you must use in the action command, object_category is the category of the object, which provides a hint for you in interpreting initial and goal condtions.

Please pay specail attention:
1. The robot can only hold one object in each hand.
2. Action name must be one of the above action names, and the object name must be one of the object names listed in the interactable objects.
3. All PLACE actions will release the object in the robot's hand, you don't need to explicitly RELEASE the object after the PLACE action.
4. For LEFT_PLACE_NEXTTO_ONTOP and RIGHT_PLACE_NEXTTO_ONTOP, the action command are in the format of {{'action': 'action_name', 'object': 'obj_name1, obj_name2'}}
5. If you want to perform an action to an target object, you must make sure the target object is not inside a closed object.
6. For actions like OPEN, CLOSE, SLICE, COOK, CLEAN, SOAK, DRY, FREEZE, UNFREEZE, TOGGLE_ON, TOGGLE_OFF, at least one of the robot's hands must be empty, and the target object must have the corresponding property like they're openable, toggleable, etc.
7. For PLACE actions and RELEASE actions, the robot must hold an object in the corresponding hand.
8. Before slicing an object, the robot can only interact with the object (e.g. peach_0), after slicing the object, the robot can only interact with the sliced object (e.g. peach_0_part_0).


Examples: after# is the explanation.

Example 1:
Input:
initial environment state:
['stained', 'sink_7']
['stained', 'bathtub_4']
['not', 'soaked', 'rag_0']
['onfloor', 'rag_0', 'room_floor_bathroom_0']
['inside', 'rag_0', 'cabinet_1']
['not', 'open', 'cabinet_1']


target environment state:
['not', 'stained', 'bathtub_4']
['not', 'stained', 'sink_7']
['and', 'soaked', 'rag_0', 'inside', 'rag_0', 'bucket_0']


interactable objects:
{{'name': 'sink_7', 'category': 'sink.n.01'}}
{{'name': 'bathtub_4', 'category': 'bathtub.n.01'}}
{{'name': 'bucket_0', 'category': 'bucket.n.01'}}
{{'name': 'rag_0', 'category': 'rag.n.01'}}
{{'name': 'cabinet_1', 'category': 'cabinet.n.01'}}


Please output the list of action commands (in the given format) so that after the robot executes the action commands sequentially, the current environment state will change to target environment state. Usually, the robot needs to execute multiple action commands consecutively to achieve final state. Please output multiple action commands rather than just one. Only output the list of action commands with nothing else.

Output:
[
    {{
        'action': 'OPEN',
        'object': 'cabinet_1'
    }}, # you want to get the rag_0 from cabinet_1, should open it first
    {{
        'action': 'RIGHT_GRASP',
        'object': 'rag_0'
    }}, # you want to clean the sink_7 and bathtub_4, you found them stained, so you need to soak the rag_0 first
    {{
        'action': 'RIGHT_PLACE_INSIDE',
        'object': 'sink_7'
    }}, # to soak the rag_0, you need to place it inside the sink_7
    {{
        'action': 'TOGGLE_ON',
        'object': 'sink_7'
    }}, # to soak the rag_0, you need to toggle on the sink_7
    {{
        'action': 'SOAK',
        'object': 'rag_0'
    }}, # now you can soak the rag_0
    {{
        'action': 'TOGGLE_OFF',
        'object': 'sink_7'
    }}, # after soaking the rag_0, you need to toggle off the sink_7
    {{
        'action': 'LEFT_GRASP',
        'object': 'rag_0'
    }}, # now you can grasp soaked rag_0 to clean stain
    {{
        'action': 'CLEAN',
        'object': 'sink_7'
    }}, # now you clean the sink_7
    {{
        'action': 'CLEAN',
        'object': 'bathtub_4'
    }}, # now you clean the bathtub_4
    {{
        'action': 'LEFT_PLACE_INSIDE',
        'object': 'bucket_0'
    }} # after cleaning the sink_7, you need to place the rag_0 inside the bucket_0
]

Your task:
Input:
initial environment state:
{init_state}

target environment state:
{target_state}

interactable objects:
{obj_list}

Please output the list of action commands (in the given format) so that after the robot executes the action commands sequentially, the current environment state will change to target environment state. Usually, the robot needs to execute multiple action commands consecutively to achieve final state. Please output multiple action commands rather than just one. Only output the list of action commands with nothing else.

Output:
"""
Figure 50: Examples prompt of action sequencing.

I.4 Prompt of Transition Modeling

The full transition-model template is preserved, including operator signatures and the requested precondition/effect definitions.

# Virtualhome Transition Modeling Prompt Template

SYSTEM_PROMPT = "You are a software engineer who will be writing action definitions for a household robot in the PDDL planning language given the problem file and predicates in domain file. For this task, please only output a parsable json string inside brackets. Please start your answer with { and end your answer with }. Don't include any notes or explanations with the output json string."

USER_PROMPT = "The following is predicates defined in this domain file. Pay attention to the types for each predicate.
(define (domain virtualhome)
(:requirements :typing)
    ;; types in virtualhome domain
    (:types 
        object character  ; Define 'object' and 'character' as types
    )

    ;; Predicates defined on this domain. Note the types for each predicate.
    (:predicates
        (closed ?obj - object)  ; obj is closed
        (open ?obj - object)  ; obj is open
        (on ?obj - object)  ; obj is turned on, or it is activated
        (off ?obj - object)  ; obj is turned off, or it is deactivated
        (plugged_in ?obj - object)  ; obj is plugged in
        (plugged_out ?obj - object)  ; obj is unplugged
        (sitting ?char - character)  ; char is sitting, and this represents a state of a character
        (lying ?char - character)  ; char is lying
        (clean ?obj - object)  ; obj is clean
        (dirty ?obj - object)  ; obj is dirty
        (obj_ontop ?obj1 ?obj2 - object)  ; obj1 is on top of obj2
        (ontop ?char - character ?obj - object)  ; char is on obj
        (on_char ?obj - object ?char - character) ; obj is on char
        (inside_room ?obj ?room - object) ; obj is inside room
        (obj_inside ?obj1 ?obj2 - object)  ; obj1 is inside obj2
        (inside ?char - character ?obj - object)  ; char is inside obj
        (obj_next_to ?obj1 ?obj2 - object)  ; obj1 is close to or next to obj2
        (next_to ?char - character ?obj - object) ; char is close to or next to obj
        (between ?obj1 ?obj2 ?obj3 - object)  ; obj1 is between obj2 and obj3
        (facing ?char - character ?obj - object)  ; char is facing obj
        (holds_rh ?char - character ?obj - object)  ; char is holding obj with right hand
        (holds_lh ?char - character ?obj - object)  ; char is holding obj with left hand
        (grabbable ?obj - object)  ; obj can be grabbed
        (cuttable ?obj - object)  ; obj can be cut
        (can_open ?obj - object)  ; obj can be opened
        (readable ?obj - object)  ; obj can be read
        (has_paper ?obj - object)  ; obj has paper
        (movable ?obj - object)  ; obj is movable
        (pourable ?obj - object)  ; obj can be poured from
        (cream ?obj - object)  ; obj is cream
        (has_switch ?obj - object)  ; obj has a switch
        (lookable ?obj - object)  ; obj can be looked at
        (has_plug ?obj - object)  ; obj has a plug
        (drinkable ?obj - object)  ; obj is drinkable
        (body_part ?obj - object)  ; obj is a body part
        (recipient ?obj - object)  ; obj is a recipient
        (containers ?obj - object)  ; obj is a container
        (cover_object ?obj - object)  ; obj is a cover object
        (surfaces ?obj - object)  ; obj has surfaces
        (sittable ?obj - object)  ; obj can be sat on
        (lieable ?obj - object)  ; obj can be lied on
        (person ?obj - object)  ; obj is a person
        (hangable ?obj - object)  ; obj can be hanged
        (clothes ?obj - object)  ; obj is clothes
        (eatable ?obj - object)  ; obj is eatable
        )
    ;; Actions to be predicted
)

Objective: Given the problem file of pddl, which defines objects in the task (:objects), initial conditions (:init) and goal conditions (:goal), write the body of PDDL actions (:precondition and :effect) given specific action names and parameters, so that after executing the actions in some order, the goal conditions can be reached from initial conditions.

Each PDDL action definition consists of four main components: action name, parameters, precondition, and effect. Here is the general format to follow:
(:action [action name]
  :parameters ([action parameters])
  :precondition ([action precondition])
  :effect ([action effect]) 
)

The :parameters is the list of variables on which the action operates. It lists variable names and variable types. 

The :precondition is a first-order logic sentence specifying preconditions for an action. The precondition consists of predicates and 4 possible logical operators: or, and, not, exists! 
1. The precondition should be structured in Disjunctive Normal Form (DNF), meaning an OR of ANDs. 
2. The not operator should only be used within these conjunctions. For example, (or (and (predicate1 ?x) (predicate2 ?y)) (and (predicate3 ?x)))
3. Exists operator is followed by two parts, variable and body. It follows the format: exists (?x - variable type) (predicate1 ?x), which means there exists an object ?x of certain variable type, that predicate1 ?x satisfies.

The :effect lists the changes which the action imposes on the current state. The precondition consists of predicates and 6 possible logical operators: or, and, not, exists, when, forall. 
1. The effects should generally be several effects connected by AND operators. 
2. For each effect, if it is a conditional effect, use WHEN to check the conditions. The semantics of (when [condition] [effect]) are as follows: If [condition] is true before the action, then [effect] occurs afterwards. 
3. If it is not a conditional effect, use predicates directly. 
4. The NOT operator is used to negate a predicate, signifying that the condition will not hold after the action is executed. 
5. Forall operator is followed by two parts, variable and body. It follows the format: forall (?x - variable type) (predicate1 ?x), which means there for all objects ?x of certain variable type, that predicate1 ?x satisfies.

6. An example of effect is (and (when (predicate1 ?x) (not (predicate2 ?y))) (predicate3 ?x))

Formally, the preconditions and effects are all clauses <Clause>.
<Clause> := (predicate ?x)
<Clause> := (and <Clause1> <Clause2> ...)
<Clause> := (or <Clause1> <Clause2> ...)
<Clause> := (not <Clause>)
<Clause> := (when <Clause1> <Clause2>)
<Clause> := (exists (?x - object type) <Clause>)
<Clause> := (forall (?x - object type) <Clause>)

In any case, the occurrence of a predicate should agree with its declaration in terms of number and types of arguments defined in DOMAIN FILE at the beginning.

Here is an example of the input problem file and unfinished action. Observe carefully how to think step by step to write the action body of hang_up_clothes:
Input:
Problem file:
(define (problem hang-clothes-problem)
  (:domain household)
  (:objects
    character - character
    shirt - object
    hanger - object
  ) ; This section declares the instances needed for the problem: character is an instance of a character; shirt is an instance of an object classified as clothes; hanger is an object that is suitable for hanging clothes.
  (:init
    (clothes shirt)
    (hangable hanger)
    (holds_rh alice shirt)
    (next_to alice hanger)
  ) ; This section declares the initial conditions. (clothes shirt) and (hangable hanger) tells the properties of objects; (holds_rh alice shirt) indicates that Alice is holding the shirt in her right hand; (next_to alice hanger) means Alice is next to the hanger, ready to hang the shirt.
  (:goal
    (and
      (ontop shirt hanger)
    )
  ) ; This section declares the goal.  (ontop shirt hanger) is the goal, where the shirt should end up hanging on the hanger.
)
Action to be finished:
(:action hang_up_clothes
  :parameters (?char - character ?clothes - object ?hang_obj - object)
  :precondition ()
  :effect ()
)

Example output: 
Given the objects in the problem file, and what typically needs to be true to perform an action like hanging up clothes: 1. clothes must indeed be a type of clothing. 2. hang_obj should be something on which clothes can be hung (hangable). 3. char should be holding the clothes, either in the right or left hand. 4. char needs to be next to the hanging object to hang the clothes. Besides, we need to write preconditions in Disjunctive Normal Form.
These insights guide us to write:
:precondition (or
                (and
                    (clothes ?clothes)  ; the object must be a piece of clothing
                    (hangable ?hang_obj)  ; the target must be an object suitable for hanging clothes
                    (holds_rh ?char ?clothes)  ; character is holding clothes in the right hand
                    (next_to ?char ?hang_obj)  ; character is next to the hanging object
                )
                (and
                    (clothes ?clothes)  ; the object must be a piece of clothing
                    (hangable ?hang_obj)  ; the target must be an object suitable for hanging clothes
                    (holds_lh ?char ?clothes)  ; character is holding clothes in the left hand
                    (next_to ?char ?hang_obj)  ; character is next to the hanging object
                )
            )
Effects describe how the world state changes due to the action. After hanging up clothes, you'd expect: 1. char is no longer holding the clothes. 2. clothes is now on the hang_obj.
These expectations convert into effects:
:effect (and
             (when (holds_rh ?char ?clothes)(not (holds_rh ?char ?clothes)))  ; if clothes are held in the right hand, they are no longer held
             (when (holds_lh ?char ?clothes)(not (holds_lh ?char ?clothes)))  ; if clothes are held in the left hand, they are no longer held
             (ontop ?clothes ?hang_obj)  ; clothes are now hanging on the object
           )

Combining these parts, the complete hang_up_clothes action becomes:
(:action hang_up_clothes
  :parameters (?char - character ?clothes - object ?hang_obj - object)
  :precondition (or
                   (and
                     (clothes ?clothes)
                     (hangable ?hang_obj)
                     (holds_rh ?char ?clothes)
                     (next_to ?char ?hang_obj)
                   )
                   (and
                     (clothes ?clothes)
                     (hangable ?hang_obj)
                     (holds_lh ?char ?clothes)
                     (next_to ?char ?hang_obj)
                   )
                 )
  :effect (and
             (when (holds_rh ?char ?clothes)(not (holds_rh ?char ?clothes))) 
             (when (holds_lh ?char ?clothes)(not (holds_lh ?char ?clothes))) 
             (ontop ?clothes ?hang_obj) 
           )
)

Above is a good example of given predicates in domain file, problem file, action names and parameters, how to reason step by step and write the action body in PDDL. Pay attention to the usage of different connectives and their underlying logic.

Here are some other commonly used actions and their PDDL definition:
(:action put_to
    :parameters (?char - character ?obj - object ?dest - object)
    :precondition (or
      (and
          (hold_lh ?obj)        ; The character should hold either with left hand or right hand
          (next_to ?char ?dest) ; The character should be close to destination
      )
      (and
          (hold_rh ?obj)        ; The character should hold either with left hand or right hand
          (next_to ?char ?dest) ; The character should be close to destination
      )
    )
    :effect (obj_ontop ?obj ?dest)        ; The object is now on the destination
)
This case illustrates the use of OR to include all possible preconditions of an action.

(:action pick_and_place
    :parameters (?char - character ?obj - object ?dest - object)
    :precondition (and
        (grabbable ?obj)        ; The object must be grabbable
        (next_to ?char ?obj)    ; The character must be next to the object
        (not (obj_ontop ?obj ?dest)) ; Ensure the object is not already on the destination
    )
    :effect (and
        (obj_ontop ?obj ?dest)        ; The object is now on the destination
        (next_to ?char ?dest)         ; The character is now next to the destination
    )
)
This case illustrates a plain case with only AND operator.

(:action bow
    :parameters (?char - character ?target - character)
    :precondition (and
        (next_to ?char ?target)  ; The character must be next to the target to perform the bow
    )
    :effect ()
)
This case illustrates the action can have no effect (or no precondition.)

hint: 
1. Don't enforce the use of WHEN everywhere.

2. You MUST only use predicates and object types exactly as they appear in the domain file at the beginning. 

3. Use and only use the arguments provided in :parameters for each action. Don't propose additional arguments, unless you are using exists or forall.

4. It is possible that action has no precondition or effect.

5. The KEY of the task is to ensure after executing your proposed actions in some order, the intial state (:init) in problem file can reach the goals (:goal)!!! Pay attention to the initial state and final goals in problem file.

6. Preconditions and effects are <Clause> defined above. When there is only one predicate, do not use logic connectives.

For actions to be finished, write their preconditions and effects, and return in standard PDDL format:
(:action [action name]
  :parameters ([action parameters])
  :precondition ([action precondition])
  :effect ([action effect]) 
)
Concatenate all actions PDDL string into a single string. Output in json format where key is 'output' and value is your output string: {'output': YOUR OUTPUT STRING}

Input:
<problem_file>
<action_handlers>

Output:"



Figure 51: Examples prompt of transition modeling.

I.5 Prompt of Environment Representation

The complete environment-representation example shows how objects, initial states, and relations are supplied as text.

# Behavior Environment Prompt Template
AVAILABLE_STATES = '''The state is represented as a first-order predicate, which is a tuple of a predicate name and its arguments. Its formal definition looks like this "<PredicateName>(Params)", where <PredicateName> is the state name and each param should be ended with an id. An example is "inside(coin.1, jar.1)". Below is a list of available states and their descriptions.
    | State Name | Arguments | Description |
    | --- | --- | --- |
    | inside | (obj1.id, obj2.id) | obj1 is inside obj2. If we have state inside(A, B), and you want to take A out of B while B is openable and stayed at "not open" state, please open B first. Also, inside(obj1, agent) is invalid.|
    | ontop | (obj1.id, obj2.id) | obj1 is on top of obj2 |
    | nextto | (obj1.id, obj2.id) | obj1 is next to obj2 |
    | under | (obj1.id, obj2.id) | obj1 is under obj2 |
    | onfloor | (obj1.id, floor2.id) | obj1 is on the floor2 |
    | touching | (obj1.id, obj2.id) | obj1 is touching or next to obj2 |
    | cooked | (obj1.id) | obj1 is cooked |
    | burnt | (obj1.id) | obj1 is burnt |
    | dusty | (obj1.id) | obj1 is dusty. If want to change dusty(obj1.id) to "not dusty(obj1.id)", there are two ways to do it, depending on task conditions. Here, all objects other than obj1 are types but not instances: 1. [inside(obj1.id, dishwasher or sink), toggledon(dishwasher or sink)] 2. holding other cleanning tool |
    | frozen | (obj1.id) | obj1 is frozen |
    | open | (obj1.id) | obj1 is open |
    | sliced | (obj1.id) | obj1 is sliced. If want to change "not sliced(obj1.id)" to "sliced(obj1.id)", one must have a slicer. | 
    | soaked | (obj1.id) | obj1 is soaked |
    | stained | (obj1.id) | obj1 is stained. If want to change stained(obj1.id) to "not stained(obj1.id)", there are three ways to do it, depending on task conditions. Here, all objects other than obj1 are types but not instances: 1. [inside(obj1.id, sink), toggledon(sink)] 2. [soaked(cleaner)] 3. holding detergent. |
    | toggledon | (obj1.id) | obj1 is toggled on |
    | holds_rh | (obj1.id) | obj1 is in the right hand of the robot |
    | holds_lh | (obj1.id) | obj1 is in the left hand of the robot |'''

AVAILABLE_CONNECTIVES = '''The connectives are used to satisfy the complex conditions. They are used to combine the state predicates.
    | Connective Name | Arguments | Description |
    | --- | --- | --- |
    | and | exp1 and exp2 | evaluates to true if both exp1 and exp2 are true |
    | or | exp1 or exp2 | evaluates to true if either exp1 or exp2 is true |
    | not | not exp | evaluates to true if exp is false |
    | forall | forall(x, exp) | evaluates to true if exp is true for all x |
    | exists | exists(x, exp) | evaluates to true if exp is true for at least one x |
    | forpairs | forpairs(x, y, exp) | evaluates to true if exp is true for all pairs of x and y. For example, forpairs(watch, basket, inside(watch, basket)) means that for each watch and basket, the watch is inside the basket. |
    | forn | forn(n, x, exp) | evaluates to true if exp is true for exactly n times for x. For example, forn(2, jar_n_01, (not open(jar_n_01)) means that there are exactly two jars that are not open. |
    | fornpairs | fornpairs(n, x, y, exp) | evaluates to true if exp is true for exactly n times for pairs of x and y. For example, fornpairs(2, watch, basket, inside(watch, basket)) means that there are exactly two watches inside the basket. |'''

EXAMPLE_RELEVANT_OBJECTS = '''## Relevant objects in this scene
    {'name': 'strawberry.0', 'category': 'strawberry_n_01'}
    {'name': 'fridge.97', 'category': 'electric_refrigerator_n_01'}
    {'name': 'peach.0', 'category': 'peach_n_03'}
    {'name': 'countertop.84', 'category': 'countertop_n_01'}
    {'name': 'jar.0', 'category': 'jar_n_01'}
    {'name': 'jar.1', 'category': 'jar_n_01'}
    {'name': 'carving_knife.0', 'category': 'carving_knife_n_01'}
    {'name': 'bottom_cabinet_no_top.80', 'category': 'cabinet_n_01'}
    {'name': 'room_floor_kitchen.0', 'category': 'floor_n_01'}'''

INITIAL_STATES = '''inside(strawberry.0, fridge.97)
    inside(peach.0, fridge.97)
    not sliced(strawberry.0)
    not sliced(peach.0)
    ontop(jar.0, countertop.84)
    ontop(jar.1, countertop.84)
    ontop(carving_knife.0, countertop.84)
    onfloor(agent_n_01.1, room_floor_kitchen.0)'''

GOAL_STATES = '''exists(jar_n_01, (inside(strawberry.0, jar_n_01) and (not inside(peach.0, jar_n_01))))
    exists(jar_n_01, (inside(peach.0, jar_n_01) and (not inside(strawberry.0, jar_n_01))))
    forall(jar_n_01, (not open(jar_n_01)))
    sliced(strawberry.0)
    sliced(peach.0)'''
Figure 52: Examples prompt of object-centric representation for the embodied environment.

I.6 Prompt Analysis and Learned Lessons

Prompts favor structured JSON and Python-list outputs over unconstrained prose, because extra explanations and malformed strings can make plans unparsable. A common prompt is used across models. The authors’ prompting recommendations arise from qualitative development experiments, rather than a broad independently held-out prompt-optimization study.

Takeaways of Prompting Techniques
•

System Prompts are preferred over user prompts for specifying output structure and overall output style as it is more strongly enforced.

•

Stop Sequences are provided to prevent repetitive output and end-of-output explanations. This can improve format following and save token usage.

•

Shorter Strings are encouraged to reduce grammar errors. For example, in the goal interpretation, we ask models to generate [“cond1”, “and”, “cond2”, “or”, “cond3”] instead of “cond1 and cond2 or cond3".

•

Start / End Tokens are specified to discourage models from generating additional explanations and self-repetition of answers / prompt instructions.

I.7 Further Consideration about Prompt Variability

A limited prompt-variation study evaluates Claude-3 Opus, GPT-4o, and Gemini 1.5 Pro on VirtualHome goal interpretation. Improvements often transfer between models, supporting a shared prompt in that setting. The source’s statement that no model exceeds a 3.8% format-error rate is not a blanket result across every module: its BEHAVIOR decomposition discussion reports 23% parsing errors for Cohere Command R. Prompt robustness is therefore task- and model-dependent.

Refer to caption
(a) Model Performance ranked by Claude-3 Opus.
Refer to caption
(b) Model Performance ranked by GPT-4o.
Refer to caption
(c) Model Performance ranked by Gemini-1.5 Pro.
Figure 53: Heatmap results showing minor differences between general and model-specific prompts across the three tested models.

Appendix J Human Performance Comparison

The human comparison reports better human goal interpretation and action execution, while GPT-4o scores higher on the displayed decomposition and transition-planning measures. A perfect planner success score still refers to solvability under predicted operators. The table and brief description do not establish broad human-versus-model superiority, especially without detailed participant, training, and uncertainty information; the source also alternates GPT-4 and GPT-4o naming.

Table 19: Comparison of GPT-4 and Human Performance on Different Tasks
Method Goal Interpretation Action Sequencing Subgoal Decomposition Transition Modeling Average Perf.
F1F_{\textsubscript{1}} Task SR Execution SR Task SR Execution SR F1F_{\textsubscript{1}} Planner SR Module SR
GPT-4o 37.6 42.9 57.1 70.0 90.0 50.0 100.0 56.8
Human Annotation 80.6 57.1 85.7 60.0 80.0 52.9 66.7 64.4

Appendix K Further Discussion on Visual Information in Our Benchmark

K.1 Integration of Visual Inputs in Long-Horizon Decision Making

Supplementary experiments examine how adding visual inputs changes the symbolic-planning setting.

K.1.1 Challenges with Existing VLMs

Complex multi-room scenes can contain more information than a single image or present visual model reliably captures. The authors distinguish this perception challenge from the decision-making abilities isolated by the main benchmark.

K.1.2 Comparative Experiments with LLMs and VLMs

The controlled comparison uses Llama 3 and a LLaVA model based on Llama 3 in BEHAVIOR.

Experiment Settings

Three inputs are compared: a scene graph for the language model, images alone for the vision-language model, and images plus a scene graph for the vision-language model.

Results and Analysis

Image-only LLaVA performs worse than scene-graph-conditioned Llama 3. Adding the graph helps LLaVA but does not close the reported gap. This demonstrates a limitation of these tested configurations; it does not establish that visual input is inherently unhelpful or that all vision-language models share the same ranking. End-to-end scores also combine perception and planning errors.

Table 20: Performance Comparison between LLMs and VLMs on BEHAVIOR
Model Goal Interpretation (F1\text{F}_{\text{1}}, %) Action Sequencing (Success Rate %)
Llama 3 31.5 11.1
LLaVA (w/o scene graph) 9.1 2.5
LLaVA (w/ scene graph) 25.8 11.0

K.1.3 Why LLMs Benchmarking is Useful

Structured language-model evaluation remains relevant to modular systems that combine perception, planning, and control.

Experiment Settings

Two further settings add broader context to the scene-graph language model and to the image-plus-graph vision-language model.

Table 21: Insights from LLMs (Exp.3) can be applied to VLMs (Exp.4)
Model Goal Interpretation (F1,\text{F}_{\text{1}}, %) Improvement Action Sequencing (SR%) Improvement
LLama 3 31.5 → 31.9 11.1 → 13.9
LLaVA (w/ scene graph) 25.8 → 25.9 11.0 → 12.3

Result Analysis

Additional context raises the reported Llama goal F1 from 31.5 to 31.9 and action success from 11.1 to 13.9; LLaVA changes from 25.8 to 25.9 and 11.0 to 12.3, respectively. These small improvements suggest some transfer of input-design choices, with no reported uncertainty establishing statistical significance.

K.1.4 Implications for Future Research

The authors advocate better visual/text integration and evaluations that separate perception from planning. Transfer of prompt and representation improvements remains a hypothesis to test across broader model families.

K.2 Impact of Perception and State Estimation Errors

A further experiment supplies language planners with scene graphs generated from images, focusing on subgoal decomposition.

K.2.1 Experiment Setup (Exp.5)

Different vision-language models produce BEHAVIOR scene graphs, which then become planner inputs. This directly exposes downstream decisions to perception errors.

K.2.2 Results Analysis

The retained table and examples distinguish extra or missing state predicates, incorrect spatial relations, and invented objects. Higher predicate error rates accompany lower decomposition success; Claude-3.5 Sonnet reaches the reported best 75% in this experiment. This supplementary score uses a different input/evaluation setting from the main decomposition table and should not be interchanged with it.

Table 22: Effect of Perception Errors on Subgoal Decomposition
VLM Object Error Rate (%) Predicate Error Rate (%) Steps Generated Num Success Rate (%)
Claude-3.5-Sonnet 0.0 8.0 13 75.0
ChatGPT-4 0.0 20.0 8 50.0
LLaVA-Vicuna-13B 0.0 83.0 6 12.5
Refer to caption
Figure 54: Possible VLM scene graph perception error types.

K.3 Assumptions on Scene Graphs in Our Benchmark

Scene graphs are a practical abstraction for modular robot reasoning. In the main benchmark they are supplied perception results, so benchmark success does not establish that those graphs can be recovered reliably from real-world images.

Appendix L Dataset Statistics and Analysis

L.1 Dataset Structure

Each task instance combines a name, natural-language instruction, symbolic/LTL goal, symbolic action trajectory, and relevant transition models. BEHAVIOR also includes demonstration-video references. VirtualHome goals are annotated to exclude incidental final-state changes, making evaluation less dependent on one reference execution. Complete JSON examples for both simulators are preserved.

Data Format Examples for VirtualHome
# Below is an example of VirtualHome
"1057_1": {
  "task_name": "Watch TV",
  "natural_language_description": "Go to the living room, sit on the couch, find the
    remote, switch on the TV and watch",
  "vh_goal": {
    "actions": [
      "LOOKAT|WATCH"
    ],
    "goal": [{
        "id": 410,
        "class_name": "television",
        "state": "ON"
      }, {
        "id": 410,
        "class_name": "television",
        "state": "PLUGGED_IN"
      }, {
        "from_id": 65,
        "relation_type": "FACING",
        "to_id": 410
    }]
  },
  "tl_goal": "(exists x0. ((LOOKAT(x0) or WATCH(x0))) then (ON(television.410) and
    PLUGGED_IN(television.410) and FACING(character.65, television.410)))",
  "action_trajectory": [
    "[WALK] <home_office> (319)",
    "[WALK] <couch> (352)",
    "[FIND] <couch> (352)",
    "[SIT] <couch> (352)",
    "[FIND] <remote_control> (1000)",
    "[FIND] <television> (410)",
    "[SWITCHON] <television> (410)",
    "[TURNTO] <television> (410)",
    "[WATCH] <television> (410)"
  ],
  "transition_model": <pddl_definition>
}
Figure 55: We release the task annotations in JSON format. Here is the data instance example for VirtualHome .
Data Format Examples for BEHAVIOR
# Below is an example of BEHAVIOR
"cleaning_high_chair_0_Wainscott_0_int_0_2021-06-05_18-03-15": {
  "task_name": "cleaning_high_chair",
  "natural_description": "Clean the high chair.",
  "raw_bddl_goal": "(define (problem cleaning_high_chair_0)\n    (:domain igibson)\n\n
      (:objects\n        highchair.n.01_1 - highchair.n.01\n
      piece_of_cloth.n.01_1 - piece_of_cloth.n.01\n
      cabinet.n.01_1 - cabinet.n.01\n
      ...",
  "tl_goal": "not dusty(highchair.0)",
  "action_trajectory": [
    {
      "action": "OPEN",
      "object": "bottom_cabinet_no_top_80"
    },
    {
      "action": "RIGHT_GRASP",
      "object": "paper_towel_0"
    },
    {
      "action": "LEFT_GRASP",
      "object": "highchair_0"
    },
    {
      "action": "CLEAN",
      "object": "highchair_0"
    }
  ],
  "transition_model": <pddl_definition>,
  "demo": <demo_link>
}
Figure 56: We release the task annotations in JSON format. Here is the data instance example for BEHAVIOR .

L.2 Data Statistics and Distribution

The statistics describe the selected benchmark subset, not every task available in the underlying simulators.

VirtualHome

VirtualHome contains 801 annotated goal conditions: 340 state, 299 relation, and 162 action goals. Reference trajectories average 8.76 actions and span 2–54; about one third exceed ten actions. Counts by action, object, and action–object pair remain available in the distribution plots.

Refer to caption
Figure 57: Action sequence length distribution in VirtualHome
Refer to caption
Figure 58: Action counts distribution in VirtualHome
Refer to caption
Figure 59: Object counts distribution in VirtualHome
Refer to caption
Figure 60: Action object pair counts distribution in VirtualHome

BEHAVIOR

BEHAVIOR reference trajectories average 14.6 actions. The average 3.7 goals counts top-level BDDL expressions, which can each contain quantified requirements. It differs from the average number of grounded atomic goals.

Original figure
(a) Action Lengths
Original figure
(b) Number of Goals
Figure 61: BEHAVIOR Task Complexity

L.3 Goal Complexity Analysis

Quantified goals can be satisfied by many object assignments. The benchmark distinguishes one valid assignment’s atomic conditions from the number of alternative valid assignments. BEHAVIOR averages 6.7 grounded conditions and 4,164.4 goal options per task; distributions, rather than these averages alone, show their variability.

Original figure
(a) Number of Grounded Goals
Original figure
(b) Number of Goal Options
Figure 62: BEHAVIOR Goal Complexity

L.4 Task List

BEHAVIOR selection uses tasks with human demonstrations. VirtualHome selection uses executable tasks in a chosen scene. The full task lists and action lengths are retained. This supports diverse household-task evaluation within a constrained selection, rather than a representative sample of all robot environments or proof of cross-domain deployment.

Table 23: Task List on VirtualHome
Task Name Task Number Task Name Task Number
Wash hands 22 Pet cat 27
Drink 28 Work 35
Turn on light 34 Pick up phone 28
Go to toilet 11 Get some water 1
Wash teeth 11 Read book 28
Make coffee 10 Listen to music 22
Wash dishes by hand 11 Relax on sofa 35
Wash clothes 21 Wash dishes with dishwasher 27
Browse internet 31 Put groceries in Fridge 23
Set up table 8 Write an email 21
Take shower 2 Watch TV 32
Cook some food 7 Change TV channel 29
Go to sleep 9
Brush teeth 5
Table 24: Task List on BEHAVIOR . Length is the action trajectory length.
Task Name Length Task Name Length
assembling_gift_baskets 32 brushing_lint_off_clothing 13
boxing_books_up_for_storage 16 collecting_aluminum_cans 12
mopping_floors 15 preserving_food 20
re-shelving_library_books 8 polishing_silver 6
packing_boxes_for_household_move_or_trip 20 cleaning_freezer 15
installing_a_modem 3 cleaning_up_after_a_meal 31
putting_away_Christmas_decorations 17 setting_up_candles 12
cleaning_shoes 10 cleaning_sneakers 30
locking_every_window 4 washing_cars_or_other_vehicles 8
washing_dishes 11 putting_away_Halloween_decorations 15
collect_misplaced_items 11 cleaning_high_chair 4
washing_floor 8 sorting_mail 12
cleaning_bathrooms 12 cleaning_kitchen_cupboard 18
preparing_salad 34 putting_leftovers_away 18
preparing_a_shower_for_child 8 cleaning_stove 14
putting_dishes_away_after_cleaning 17 packing_child_s_bag 13
cleaning_windows 13 bottling_fruit 21
thawing_frozen_food 20 setting_mousetraps 8
washing_pots_and_pans 24 laying_wood_floors 8
putting_away_toys 16 cleaning_garage 19
watering_houseplants 14 installing_alarms 6
organizing_boxes_in_garage 12 putting_up_Christmas_decorations_inside 18
cleaning_bathtub 8 clearing_the_table_after_dinner 14
loading_the_dishwasher 13 filling_an_Easter_basket 26
cleaning_the_pool 8 opening_packages 6
packing_bags_or_suitcase 15 cleaning_up_refrigerator 23
making_tea 13 picking_up_trash 10
polishing_furniture 4 organizing_school_stuff 15
cleaning_cupboards 24 installing_a_printer 3
packing_food_for_work 12 storing_the_groceries 23

L.5 Task Categorization

Transition predicates are grouped into object affordance, object state, orientation, spatial relation, and non-spatial relation categories. Inverse-document-frequency weights emphasize rare predicates; each program is assigned its two highest-scoring categories. Counts can therefore overlap across categories. The complete scoring equations and category-count table are preserved.

Equations
IDF​(t)=log⁡(|P|df​(t))\text{IDF}(t)=\log\left(\frac{|P|}{\text{df}(t)}\right)
S⁡(cj,pi)=∑t∈pi,t∈cjIDF​(t)S(c_{j},p_{i})=\sum_{t\in p_{i},t\in c_{j}}\text{IDF}(t)
Categories​(pi)=topK​{(cj,S⁡(cj,pi))∣cj∈C}\text{Categories}(p_{i})=\text{top}_{K}\{(c_{j},S(c_{j},p_{i}))\mid c_{j}\in C\}
Table 25: Results on Task Categorization, showing task number in each task category.
VirtualHome BEHAVIOR
Object States 178 45
Object Orientation 167 -
Object Affordance 28 -
Spatial Relations 145 94
Non-spatial Relations 74 61

Appendix M Annotation Details

M.1 Simulator Comparison and Selection

The authors choose VirtualHome and BEHAVIOR to emphasize long plans and complex household goals, while discussing AI2-THOR as an alternative. This survey mixes platform-wide and benchmark-specific action/scene counts and contains inconsistent descriptions of BEHAVIOR physics. The concrete benchmark vocabulary and evaluator behavior in Appendix N are the relevant implementation specification.

M.2 BEHAVIOR

BEHAVIOR annotations add action sequences and natural-language descriptions to the existing task definitions and demonstrations.

M.2.1 Annotating Action Sequence based on VR Human Demonstrations

Object-state tracking and segmentation of VR demonstrations provide candidate changes, which authors manually map to symbolic action sequences for the 100 tasks.

M.2.2 Complicated Goals with Quantifiers

Quantified BDDL conditions are grounded into atomic alternatives. Different pairings can satisfy the same task, so reference evaluation must permit these alternatives rather than insist on one arbitrary assignment.

M.2.3 Annotating Transition Models

For transition-planner evaluation, a solvable BDDL goal option is sampled and a bounded subset of its grounded conditions is selected to keep planning tractable. Relevant objects and initial conditions form the PDDL problem. An existing BEHAVIOR domain is modified in names, predicates, preconditions, and effects. Thus these planner problems may cover fewer conditions than the full original BDDL task.

M.3 VirtualHome

VirtualHome extends RobotHow with manually curated symbolic goals for 338 instructions. The source describes 30 task categories here but 26 in its statistics discussion; the retained lists expose the actual scope. Annotation separates task-essential outcomes from incidental states in demonstrations.

M.3.1 Wildcard Representations: Deriving Common Task-Related Final States

Common outcomes across instructions are expressed with wildcards and object properties, allowing interchangeable items such as different kinds of clothing or books.

M.3.2 Extracting and Annotating Abstract Goals

Annotators derive state, relation, and action goals and add missing conditions. Action goals capture required behaviors such as touching a cat even when the simulator records no lasting effect.

M.3.3 Grounding to Concrete Goals from Abstract Goals

Abstract wildcard goals are grounded into actual object identifiers in the selected scene. Multiple grounded targets can remain valid, preserving alternative task solutions.

M.3.4 Annotating Transition Models

PDDL action definitions are derived from VirtualHome’s implementation, including quantified and conditional effects. Roughly ten body-action operators have no environmental effect and can be irrelevant to state-goal planning; they are nevertheless included in logical-accuracy evaluation. This further explains why planner success alone does not cover every action requirement.

M.4 Quality Verification

Annotation verification combines simulator execution checks with human ratings of action accuracy, coverage, and preference.

M.4.1 Annotation Quality Evaluation

The complete quality table reports means and weighted mean-squared disagreement for these three attributes.

Table 26: Annotation Quality Evaluation Metrics
Attribute Mean Score Weighted MSE
Action Accuracy 3.73 0.4062
Action Coverage 4.07 1.8438
Human Preference 4.27 0.8125

Explanation of Metrics:

Ratings average above 3.7 out of 5. Per-task squared deviations from the mean are averaged with weights reflecting evaluation frequency. This measures rater agreement and perceived annotation quality; it is not an exhaustive proof that every symbolic definition is correct. The full rating distributions are preserved.

Refer to caption
Figure 63: Score distribution visualization.

Appendix N Simulator Implementation Details

N.1 BEHAVIOR Implementation Details

BEHAVIOR has both an iGibson/PyBullet rendering/physics implementation and a graph-based symbolic implementation, sharing actions and states.

N.1.1 Building the Symbolic Simulator and Transition Model Implementation

The full action table defines arguments, prerequisites, and effects. Interactability excludes enclosed objects and ordinarily requires reachability; automatic NAVIGATE_TO removes the reachability burden from the language model. Consequently, benchmark planning does not solve explicit navigation or continuous reachability.

Original figure
Figure 64: Implementation of Kinetics Tree for the Transition Model in BEHAVIOR .
Table 27: BEHAVIOR Symbolic Simulator Implementation: Action Transition Models.
Action Name Arguments Preconditions Post Effects
NAVIGATE_TO tar_obj1 tar_obj1 is interactable. The agent is next to tar_obj1.
LEFT_GRASP tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is small enough. tar_obj1 is in the agent’s left hand.
RIGHT_GRASP tar_obj1 tar_obj1 is interactable; The agent’s right hand is empty; tar_obj1 is small enough. tar_obj1 is in the agent’s right hand.
LEFT_RELEASE tar_obj1 tar_obj1 is interactable; The agent’s left hand holds tar_obj1. tar_obj1 is released to the floor.
RIGHT_RELEASE tar_obj1 tar_obj1 is interactable; The agent’s right hand holds tar_obj1. tar_obj1 is released to the floor.
LEFT_PLACE_ONTOP tar_obj1 tar_obj1 is interactable; The agent’s left hand holds an object. The object in the agent’s left hand is on top of tar_obj1.
RIGHT_PLACE_ONTOP tar_obj1 tar_obj1 is interactable; The agent’s right hand holds an object. The object in the agent’s right hand is on top of tar_obj1.
LEFT_PLACE_INSIDE tar_obj1 tar_obj1 is interactable; The agent’s left hand holds an object; tar_obj1 is big enough and open if openable. The object in the agent’s left hand is inside tar_obj1.
RIGHT_PLACE_INSIDE tar_obj1 tar_obj1 is interactable; The agent’s right hand holds an object; tar_obj1 is big enough and open if openable. The object in the agent’s right hand is inside tar_obj1.
LEFT_PLACE_NEXTTO tar_obj1 tar_obj1 is interactable; The agent’s left hand holds an object. The object in the agent’s left hand is next to tar_obj1.
RIGHT_PLACE_NEXTTO tar_obj1 tar_obj1 is interactable; The agent’s right hand holds an object. The object in the agent’s right hand is next to tar_obj1.
LEFT_PLACE_UNDER tar_obj1 tar_obj1 is interactable; The agent’s left hand holds an object. The object in the agent’s left hand is under tar_obj1.
RIGHT_PLACE_UNDER tar_obj1 tar_obj1 is interactable; The agent’s right hand holds an object. The object in the agent’s right hand is under tar_obj1.
LEFT_TRANSFER_ CONTENTS_INSIDE tar_obj1 tar_obj1 is interactable; The agent’s left hand holds an object; tar_obj1 is big enough and open if openable. The contents inside the object in the agent’s left hand are in tar_obj1.
RIGHT_TRANSFER_ CONTENTS_INSIDE tar_obj1 tar_obj1 is interactable; The agent’s right hand holds an object; tar_obj1 is big enough and open if openable. The contents inside the object in the agent’s right hand are in tar_obj1.
LEFT_TRANSFER_ CONTENTS_ONTOP tar_obj1 tar_obj1 is interactable; The agent’s left hand holds an object. The contents inside the object in the agent’s left hand are on top of tar_obj1.
RIGHT_TRANSFER_ CONTENTS_ONTOP tar_obj1 tar_obj1 is interactable; The agent’s right hand holds an object. The contents inside the object in the agent’s right hand are on top of tar_obj1.
LEFT_PLACE_ NEXTTO_ONTOP tar_obj1, tar_obj2 tar_obj1 and tar_obj2 are interactable; The agent’s left hand holds an object. The object in the agent’s left hand is next to tar_obj1 and on top of tar_obj2.
RIGHT_PLACE_ NEXTTO_ONTOP tar_obj1, tar_obj2 tar_obj1 and tar_obj2 are interactable; The agent’s right hand holds an object. The object in the agent’s right hand is next to tar_obj1 and on top of tar_obj2.
TOGGLE_ON tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is toggled off and closed. tar_obj1 is toggled on.
TOGGLE_OFF tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is toggled on. tar_obj1 is toggled off.
CLOSE tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is open. tar_obj1 is closed.
OPEN tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is closed and not toggled on. tar_obj1 is open.
CLEAN tar_obj1 tar_obj1 is interactable; tar_obj1 is dusty or stained; the agent has a cleaning tool. tar_obj1 is not dusty or stained (according to the tool).
DRY tar_obj1 tar_obj1 is interactable; tar_obj1 is soaked. tar_obj1 is dry.
SLICE tar_obj1 tar_obj1 is interactable; tar_obj1 is not sliced; the agent has a knife. tar_obj1 is sliced.
SOAK tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is dry and in a toggled on sink or a pot. tar_obj1 is soaked.
FREEZE tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is unfrozen and in the fridge. tar_obj1 is frozen.
UNFREEZE tar_obj1 tar_obj1 is interactable; tar_obj1 is frozen. tar_obj1 is unfrozen.
COOK tar_obj1 tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is not cooked; tar_obj1 is on/in the pan. tar_obj1 is cooked.

N.1.2 BEHAVIOR State Space and Action Space

The implemented BEHAVIOR vocabulary contains fifteen object states and thirty actions. Unary state changes and binary spatial changes are separated. All state/action names and definitions are retained.

Table 28: BEHAVIOR State Space and Action Space.
State Space Action Space
Unary States Binary States State Actions Spatial Actions
Cooked Inside TOGGLE_ON LEFT_GRASP
Dusty OnFloor TOGGLE_OFF RIGHT_GRASP
Frozen OnTop CLOSE LEFT_RELEASE
Open Under OPEN RIGHT_RELEASE
Sliced NextTo CLEAN LEFT_PLACE_ONTOP
Soaked DRY RIGHT_PLACE_ONTOP
Stained SLICE LEFT_PLACE_NEXTTO
ToggledOn SOAK RIGHT_PLACE_NEXTTO
Slicer FREEZE LEFT_PLACE_INSIDE
CleaningTool UNFREEZE RIGHT_PLACE_INSIDE
COOK LEFT_PLACE_UNDER
RIGHT_PLACE_UNDER
LEFT_TRANSFER_CONTENTS_INSIDE
RIGHT_TRANSFER_CONTENTS_INSIDE
LEFT_TRANSFER_CONTENTS_ONTOP
RIGHT_TRANSFER_CONTENTS_ONTOP
LEFT_PLACE_NEXTTO_ONTOP
RIGHT_PLACE_NEXTTO_ONTOP
NAVIGATE_TO

N.2 VirtualHome Implementation Details

N.2.1 VirtualHome State Space and Action Space

A VirtualHome environment graph and MotionPlanner track state and categorize runtime errors for 42 actions. Executors are modified for consistent evaluation. The complete vocabulary table specifies the actual benchmark interface.

Table 29: VirtualHome State Space and Action Space.
State Space Action Space
Unary States Binary States State Actions Spatial Actions
Closed On OPEN TURN_TO
Open Inside SIT PUT_BACK
On Between STANDUP PUT_IN
Off Close SLEEP POUR
Sitting Facing WAKEUP PUT_ON
Dirty Holds_RH CLOSE FIND
Clean Holds_LH DRINK RUN
Lying GRAB WALK
Plugged_in LOOKAT POINT_AT
Plugged_out LOOKAT_SHORT TOUCH
LOOKAT_LONG WATCH
SWITCH_OFF MOVE
SWITCH_ON RELEASE
TYPE DROP
PUSH
PULL
SQUEEZE
WASH
RINSE
SCRUB
EAT
PLUG_IN
PLUG_OUT
CUT
READ
LIE

Appendix O Evaluation Settings of LLMs

O.1 Decoding Parameters

The paper reports temperature-zero decoding to standardize generation and reduce sampling variation. This is the stated evaluation setting, not a guarantee of determinism across changing hosted model implementations.

O.2 Evaluation Cost

The authors estimate 192.28 million tokens, 84,960 queries, and US$1,540.70 for the reported benchmark run, using contemporary API pricing including Together AI for open models. These are historical costs. The stated 180 runs should not be inferred directly from eighteen models times four modules times two simulators without additional setup details.

O.3 Model Cards

The full model-card table preserves exact evaluated model identifiers and versions; shorthand names elsewhere should be interpreted through this table.

Table 30: Model Cards for All Evaluated Large Language Models
Model Name Creator Complete Model ID Release Hosting
Claude-3 Haiku Anthropic claude-3-haiku-20240307 03/07/24 Anthropic
Claude-3 Sonnet Anthropic claude-3-sonnet-20240229 02/29/24 Anthropic
Claude-3 Opus Anthropic claude-3-opus-20240229 02/29/24 Anthropic
Claude-3.5 Sonnet Anthropic claude-3-5-sonnet-20240620 06/20/24 Anthropic
Cohere Command R Cohere command-r 03/11/24 Cohere
Cohere Command R+ Cohere command-r-plus 04/04/24 Cohere
Gemini 1.0 Pro Google gemini-pro 12/13/23 GCP Vertex
Gemini 1.5 Flash Google gemini-1.5-flash-preview-0514 05/14/24 GCP Vertex
Gemini 1.5 Pro Google gemini-1.5-pro-preview-0409 04/09/24 GCP Vertex
GPT-3.5-turbo OpenAI gpt-3.5-turbo-0125 01/25/24 OpenAI
GPT-4-turbo OpenAI gpt-4-turbo-2024-04-09 04/09/24 OpenAI
GPT-4o OpenAI gpt-4o-2024-05-13 05/13/24 OpenAI
Llama 3 8B Instruct Meta meta-llama-3-8b-instruct 04/18/24 TogetherAI
Llama 3 70B Instruct Meta meta-llama-3-70b-instruct 04/18/24 TogetherAI
Mistral Large MistralAI mistral-large-2402 02/26/24 MistralAI
Mixtral 8x22B MoE MistralAI mixtral-8x22b-instruct-v0.1 04/17/24 TogetherAI
o1-mini OpenAI o1-mini-2024-09-12 09/12/24 OpenAI
o1-preview OpenAI o1-preview-2024-09-12 09/12/24 OpenAI
Table 30: Model Cards for Various Models

Appendix P Extensive Related Work

P.1 LLMs for Embodied Planning

The extended survey categorizes prior embodied planning systems by whether language models interpret goals, decompose subgoals, sequence actions, or model transitions. Its complete mapping is retained.

Table 31: Categorization of Existing Embodied Agent Planning Works’ Usage of Large Language Models: Each “LLMs” refers to the usage of LLMs to perform an ability module. For example, Ada [8] uses LLMs for Action Sequencing and Transition Modeling, while LLM+P [7] uses LLMs for Goal Interpretation.
Existing Work Ref. Goal Action Subgoal Transition
Interpretation Sequencing Decomposition Modeling
SayCan [2] LLMs LLMs
Ada [8] LLMs LLMs
LLP+P [7] LLMs
AutoTAMP [18] LLMs LLMs
Code as Policies [6] LLMs LLMs LLMs
Voyager [35] LLMs LLMs
Demo2Code [13] LLMs LLMs LLMs
LM as ZeroShot Planner [3] LLMs LLMs
SayPlan [15] LLMs LLMs LLMs
Text2Motion [48] LLMs
LLMGROP [46] LLMs LLMs
REFLECT [66] LLMs LLMs
Generating Consistent PDDL Domains with LLMs [32] LLMs LLMs
PlanSeqLearn [64] LLMs
COWP [70] LLMs LLMs LLMs
HumanAssisted Continual Robot Learning [65] LLMs
DECKARD [34] LLMs LLMs
MOSAIC [63] LLMs
Interactive Task Planning with Language Models [14] LLMs LLMs
RoCo [62] LLMs
Cook2LTL [42] LLMs
InnerMonologue [21] LLMs
MLDT [20] LLMs
Learning to Reason over Scene Graphs [61] LLMs LLMs LLMs
GRID [19] LLMs LLMs
LLMplanner [33] LLMs LLMs
DELTA [17] LLMs
Look Before You Leap [60] LLMs LLMs
CAPE [36] LLMs LLMs
HERACLEs [43] LLMs
RoboTool [16] LLMs LLMs
PROMST [59] LLMs
LLM3 [31] LLMs LLMs
Ghost in the Minecraft [27] LLMs
PlanBench [98] LLMs LLMs
TaPA [30] LLMs LLMs LLMs
ChatGPT Robot Control [29] LLMs
LLM World Models for Planning [28] LLMs LLMs
DEPS [81] LLMs LLMs
Grounded Decoding [25] LLMs
ProgPrompt [37] LLMs LLMs
DROC [24] LLMs LLMs
LMPC [58] LLMs LLMs
GPTPDDL [47] LLMs
RAP [57] LLMs
LEAGUE++ [38] LLMs LLMs
CoPAL [23] LLMs LLMs
SayCanPay [22] LLMs LLMs
LLMGenPlan [56] LLMs

P.2 LTL Agent Interface

Temporal logic has a history in program verification and robot planning. This work adopts it as an expressive interface between language and symbolic decision modules. Using formal specifications does not by itself provide robot safety guarantees when model definitions or perception are incorrect.

P.3 Embodied Agent Benchmarks

The comparison discusses planning benchmarks, household simulators, dynamic-environment tasks, and embodied action understanding. EAI’s distinguishing emphasis is a common interface with module-level diagnostics and temporal goals. Its broader diagnostic scope complements, rather than replaces, perception/control benchmarks.

Appendix Q Maintenance Plan

Q.1 Dataset URLs, License, and Hosting Plan

The paper lists Hugging Face and GitHub hosting, author contact channels, and GitHub issues for corrections. It states CC-BY-4.0 for the new annotations. These are the authors’ release commitments; underlying simulator assets retain their own terms.

Dataset and metadata · Code, evaluators, and issue tracker · Project page.

Q.2 Long-term Preservation and DOI

The source provides a Dryad share URL as a preservation pointer, describing it as a persistent identifier. The printed URL is not itself a DOI-form identifier.

Preservation link printed in the paper.

Q.3 URL of Croissant Metadata Record

Structured dataset metadata are linked through the Hugging Face dataset page.

Q.4 Author Statement

The authors state responsibility for annotation rights and the CC-BY-4.0 release.

Q.5 URLs of Code and Re-productivity

The shared repository provides evaluation, annotation, prompt-generation, and inference instructions.

Q.5.1 Code for VirtualHome Evaluator and Computation Resources

VirtualHome evaluation and interactive goal-annotation tools are provided with Docker instructions. Reported runs use CPU systems: Ryzen 7 5800H with 16 GB for decomposition and a Xeon-based Linux system with 32 GB for the other modules.

Q.5.2 Code for BEHAVIOR Evaluator and Computation Resources

BEHAVIOR evaluation scripts cover all four modules. Reported experiments use a Ryzen 7 7800X3D Windows system with 32 GB RAM.

Q.5.3 Code for LLMs Implementations and Computation Resources

Batch inference integrates with HELM and optionally hosted Together AI APIs, avoiding a requirement to run each open model locally.

Appendix R Datasheets for Embodied Agent Interface ( EAI )

The datasheet records author-curated additions made in 2024 over existing simulators. VirtualHome contains 338 trajectories, 2,960 steps, and 801 goal conditions; BEHAVIOR contains 100 trajectories, 1,460 steps, and 673 goal conditions. It reports 33 annotated VirtualHome transition models with 99 preconditions/57 effects and thirty BEHAVIOR models with 84 preconditions/51 effects. The intended use is out-of-the-box model evaluation, with related task instances sharing goals or names and differing trajectories. Authors checked executions and curated irrelevant states. The text describes public JSON/evaluator distribution, issue-based corrections, expansion, and reviewed contributions, and states that the annotations contain no sensitive personal data. Several datasheet statements predate revisions: it mentions fifteen evaluated models where the final paper reports eighteen and describes visual evaluation as future work despite the added appendix. Its references to simulator licenses should be read alongside the separate CC-BY-4.0 annotation statement, rather than as one license for all assets.

Appendix S Impact, Limitations and Future Directions

S.1 Broader Impact

The proposed benefit is more informative diagnosis of language-driven robot plans, helping developers identify failures and choose which components need external support. Potential deployment and safety benefits are prospective; the benchmark itself demonstrates symbolic evaluation rather than reliable real-world autonomy.

S.2 Limitations

Relational graphs omit sensory ambiguity, continuous geometry, physical dynamics, and fine motor control. The main evaluations do not jointly test language grounding in perception and execution through actuators. These omissions are central boundaries on what the scores establish.

S.3 Potential Negative Social Impact

The authors discuss privacy, security, misuse, bias, labor disruption, and unequal impacts of future embodied agents. They call for safeguards and stakeholder participation; no empirical societal-impact study is reported.

S.4 Potential Future Directions

Proposed extensions include multimodal and low-level control interfaces, episodic/state memory, physical and geometric reasoning, navigation/exploration, forward/backward/counterfactual prediction, richer simulators, and fine-tuning on diagnostic feedback. These are research directions beyond the released benchmark’s demonstrated capabilities.

Concept demonstration

Open image ↗