Overview
At a glance
Embodied Agent Interface standardizes four language-model roles in household decision making: interpreting goals, decomposing them into subgoals, sequencing actions, and predicting action preconditions/effects. A shared object-centric temporal-logic representation connects these roles and supports diagnosis beyond final success. Eighteen contemporary models are evaluated in symbolic versions of BEHAVIOR and VirtualHome. The benchmark exposes missing prerequisites, redundant actions, relational errors, and mismatches between a planner’s internal model and the environment. Its primary scope is reasoning over supplied symbolic state, rather than end-to-end robot perception and control.
1 Introduction
Previous systems ask language models to produce different artifacts under different assumptions, so their success rates are difficult to compare. This work makes goal specifications, module boundaries, and error categories explicit. The resulting interface can evaluate one model component while holding the others fixed, or compose components into a pipeline. The central empirical finding is that coherent language and valid syntax are insufficient: models often omit necessary physical relationships, confuse intermediate states with final goals, or issue actions whose prerequisites are false. Model rankings vary across simulator and ability; aggregate scores should not replace the module-level results.
2 Embodied Agent Interface Based on LTL
The representation separates a finite object universe, Boolean relational state, parameterized actions, task goals, subgoals, and trajectories. Four typed input/output interfaces sit above this shared representation. The notation table is retained in full.
| Notation | Symbol |
Description | |
|
Environment Representations |
Object |
An object, which has relational features | |
| State |
A tuple of the universe of objects and relational features | ||
| Action |
A tuple of the action name and arguments | ||
| Operator |
An action schema: a tuple of the name and a list of parameters. Each can be instantiated into an action | ||
| Transition Model |
The deterministic transition function of the environment | ||
| Natural Language Goal |
A sentence in English | ||
| LTL Goal |
An LTL formula. Here, we only consider formulas containing a sequence of action items and a conjunction of propositions (for the final state): . | ||
| Action Trajectory |
A sequence of actions | ||
| Subgoal Trajectory |
A sequence of LTL subgoals connected by “then” | ||
| State-action Trajectory |
A sequence of state-action pairs. | ||
| Task |
A tuple of the initial state and the LTL/Natural Language goals | ||
|
Abilities |
Goal Interpretation |
Initial State & Natural Language Goal LTL Goal | |
| Subgoal Decomposition |
Initial State & Goal Subgoal Trajectory | ||
| Action Sequencing |
Initial State & Goal & Transition Model Action Trajectory | ||
| Transition Modeling |
Initial State & Goal & Operator Preconditions & Effects |
2.1 Representation for Objects, States and Actions
A state contains a fixed finite set of objects and Boolean features over object tuples. Unary features represent object properties; higher-arity features represent relations. Actions are named primitives with ordered object arguments. This abstracts continuous geometry, sensing uncertainty, and motor execution into simulator-recognized symbols.
2.2 Representation for Goals, Subgoals, Action Sequences, and State-Action Trajectories
Goals, subgoals, and ordered action descriptions use a finite-trajectory fragment of linear temporal logic. It permits state/action predicates, Boolean connectives, universal and existential quantifiers, exact-count quantification, and a then operator. A state formula holds if it is true at some trajectory step; then requires a prefix/suffix split with the first formula before the second. Globally and until are not implemented. This distinction matters: the general temporal semantics are not simply a test of the last state, although several benchmark metrics explicitly inspect final-state conditions.
2.3 Ability Module 1: Goal Interpretation 𝒢 : ⟨ s 0 , l g ⟩ → g
Goal interpretation maps an initial state and natural-language instruction into a symbolic goal. Predictions are restricted to an ordered action sequence followed by a conjunction of final-state predicates. Quantified reference goals are grounded into alternative simple goals, and the prediction receives the best set-matching F1 among valid alternatives. The ordered action sequence is treated as a single matching element. This evaluates formal translation, not execution of a plan.
2.4 Ability Module 2: Subgoal Decomposition Φ : ⟨ s 0 , g ⟩ → ϕ ¯
Subgoal decomposition outputs declarative intermediate conditions. A breadth-first planner searches for actions connecting those conditions, after which the same feasibility and goal metrics used for action sequencing apply. Because several action realizations can satisfy one decomposition, the benchmark reports the maximum score over considered realizations. This provides an optimistic assessment of whether the subgoal plan can be grounded by the supplied planner, not the success of a unique directly predicted action sequence.
2.5 Ability Module 3: Action Sequencing 𝒬 : ⟨ s 0 , g ⟩ , ℳ → a ¯
Action sequencing outputs executable primitives given the initial state, goal, and transition information. Simulation tests both whether actions execute and whether the resulting trajectory satisfies the task. Partial credit counts satisfied state conditions and required action subsequences, taking the best valid grounding of quantified goals. Execution can terminate at the first blocking error, so a fluent or partly correct plan need not complete the task.
2.6 Ability Module 4: Transition Modeling 𝒯 : ⟨ s 0 , g ⟩ , o → ⟨ pre , eff ⟩
Transition modeling asks the model to generate PDDL preconditions and effects for supplied operator names and arguments. Relevant operators are selected using the reference trajectory. Evaluation combines structural F1 against expert definitions with whether an external planner finds a plan under the predicted model. The latter considers state goals and omits action subgoals. A plan found under incorrect dynamics need not execute in the real simulator; this metric must be read alongside logical accuracy and error analysis.
3 Dataset Annotations and Benchmark Implementations
The authors annotate goals, task-relevant actions, and transition models for two household simulators: BEHAVIOR [4] and VirtualHome [5]. VirtualHome includes action goals such as touching an object when no persistent state change establishes completion. BEHAVIOR has quantified goals with many valid object assignments: the paper reports 6.7 grounded conditions and 4,164.4 goal options per task on average. A new symbolic EvalGibson layer provides 30 actions for BEHAVIOR, and VirtualHome is revised for evaluation. These benchmark implementations define the action semantics being tested.
| VirtualHome | BEHAVIOR | |
| #task name | 26 | 100 |
| #task instruction | 338 | 100 |
| #goal | 801 | 673 |
| - #state | 340 | 153 |
| - #relation | 299 | 520 |
| - #action | 162 | - |
| #trajectory | 338 | 100 |
| - #step | 2960 | 1460 |
| - avg. step | 8.76 | 14.6 |
| #transition model | 33 | 30 |
| - #precondition | 99 | 84 |
| - #effect | 57 | 51 |
4 Results
Across eighteen models, performance depends on both ability and simulator. The reported BEHAVIOR aggregate favors o1-preview at 74.9%, compared with 64.2% for the next result discussed by the authors; this aggregate combines different metrics and is not a single task-success rate. BEHAVIOR trajectories are longer and harder to execute, while VirtualHome instructions can make goal grounding more ambiguous. Relations are often harder than unary object states. Examples include placing food directly on a table instead of on a plate on the table, and omitting an implicit floor relation. These illustrate disagreement with the benchmark’s physically explicit target annotations, rather than a universal interpretation of every natural-language instruction.
| Model | Goal Interpretation | Action Sequencing | Subgoal Decomposition | Transition Modeling | Average Perf. | |||||||||||
| Task SR | Execution SR | Task SR | Execution SR | Planner SR | Module SR | |||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3 Haiku | 28.0 | 52.5 | 54.8 | 26.0 | 60.7 | 32.0 | 78.4 | 30.0 | 82.8 | 35.0 | 42.3 | 51.6 | 30.4 | 64.0 | 49.4 | 41.6 |
| Claude-3 Sonnet | 29.4 | 69.4 | 58.0 | 44.0 | 63.3 | 60.0 | 83.1 | 39.0 | 86.4 | 43.0 | 41.2 | 56.2 | 13.2 | 80.0 | 49.4 | 55.1 |
| Claude-3 Opus | 31.4 | 77.0 | 64.6 | 51.0 | 69.5 | 59.0 | 86.7 | 41.0 | 89.9 | 47.0 | 48.8 | 63.4 | 61.8 | 82.0 | 59.5 | 60.4 |
| Claude-3.5 Sonnet | 33.0 | 82.7 | 76.1 | 60.0 | 81.3 | 69.0 | 89.1 | 39.0 | 92.0 | 44.0 | 48.9 | 67.9 | 80.5 | 82.0 | 65.7 | 64.2 |
| Cohere Command R | 36.7 | 36.0 | 44.9 | 16.0 | 44.3 | 19.0 | 71.3 | 15.0 | 78.1 | 25.0 | 11.7 | 24.1 | 51.1 | 41.0 | 46.1 | 24.9 |
| Cohere Command R+ | 22.4 | 51.2 | 54.1 | 27.0 | 65.2 | 35.0 | 77.8 | 25.0 | 83.7 | 37.0 | 30.8 | 49.7 | 37.2 | 59.0 | 47.1 | 39.1 |
| Gemini 1.0 Pro | 23.8 | 60.0 | 45.6 | 27.0 | 56.7 | 32.0 | 70.4 | 24.0 | 84.6 | 33.0 | 41.8 | 45.8 | 11.8 | 16.0 | 41.7 | 35.5 |
| Gemini 1.5 Flash | 26.8 | 74.8 | 69.5 | 40.0 | 75.4 | 52.0 | 89.1 | 34.0 | 94.1 | 42.0 | 45.7 | 53.4 | 46.6 | 66.0 | 57.9 | 52.1 |
| Gemini 1.5 Pro | 36.2 | 79.6 | 76.7 | 42.0 | 83.6 | 54.0 | 87.0 | 31.0 | 91.1 | 37.0 | 34.1 | 45.8 | 91.9 | 39.0 | 65.7 | 48.8 |
| GPT-3.5-turbo | 22.7 | 50.4 | 24.9 | 16.0 | 40.7 | 20.0 | 69.2 | 24.0 | 81.4 | 36.0 | 30.0 | 42.1 | 0.7 | 41.0 | 33.0 | 33.0 |
| GPT-4-turbo | 33.2 | 77.2 | 60.0 | 38.0 | 65.2 | 45.0 | 85.5 | 38.0 | 94.1 | 47.0 | 42.9 | 44.2 | 56.1 | 46.0 | 57.1 | 49.6 |
| GPT-4o | 36.5 | 79.2 | 71.5 | 47.0 | 81.3 | 53.0 | 87.6 | 49.0 | 91.1 | 55.0 | 46.7 | 60.9 | 68.2 | 67.0 | 63.3 | 59.8 |
| Llama 3 8B Instruct | 22.6 | 28.3 | 21.3 | 10.0 | 23.6 | 16.0 | 48.8 | 22.0 | 58.0 | 29.0 | 12.9 | 35.0 | 28.7 | 29.0 | 28.4 | 23.1 |
| Llama 3 70B Instruct | 26.9 | 70.9 | 59.0 | 34.0 | 66.6 | 42.0 | 78.4 | 21.0 | 87.3 | 30.0 | 37.4 | 55.1 | 12.2 | 78.0 | 47.3 | 48.1 |
| Mistral Large | 26.8 | 74.3 | 78.4 | 33.0 | 84.6 | 50.0 | 84.3 | 31.0 | 92.0 | 38.0 | 36.1 | 49.5 | 31.1 | 77.0 | 55.8 | 50.4 |
| Mixtral 8x22B MoE | 26.6 | 54.7 | 63.3 | 30.0 | 67.9 | 40.0 | 80.5 | 28.0 | 90.2 | 33.0 | 42.0 | 52.4 | 37.5 | 55.0 | 52.5 | 41.6 |
| o1-mini | 31.2 | 76.4 | 71.5 | 56.0 | 76.4 | 65.0 | 79.3 | 31.0 | 84.6 | 39.0 | 41.5 | 56.4 | 69.0 | 77.0 | 59.3 | 57.5 |
| o1-preview | 42.7 | 81.6 | 65.2 | 81.0 | 72.5 | 91.0 | 89.4 | 60.0 | 93.2 | 62.0 | 48.0 | 69.5 | 72.4 | 89.0 | 64.4 | 74.9 |
| Model | State Goal | Relation Goal | Action Goal | Overall | ||||||||||||||||||||
| Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | |||||||||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3.5 Sonnet | 25.3 | 74.0 | 60.9 | 94.8 | 35.8 | 83.1 | 31.1 | 84.4 | 63.8 | 81.3 | 41.8 | 82.9 | 14.0 | - | 98.8 | - | 24.5 | - | 21.7 | 81.1 | 69.6 | 84.4 | 33.0 | 82.7 |
| Gemini 1.5 Pro | 47.2 | 94.0 | 47.5 | 92.8 | 47.3 | 93.4 | 42.0 | 74.4 | 7.2 | 76.7 | 12.4 | 75.6 | 24.1 | - | 81.4 | - | 37.2 | - | 33.6 | 78.8 | 39.3 | 80.4 | 36.2 | 79.6 |
| GPT-4o | 29.0 | 67.1 | 60.0 | 94.8 | 39.1 | 78.6 | 31.5 | 81.1 | 43.6 | 78.5 | 36.6 | 79.8 | 20.5 | - | 85.8 | - | 33.1 | - | 26.4 | 76.5 | 59.1 | 82.2 | 36.5 | 79.2 |
| Llama 3 70B | 23.9 | 69.5 | 61.2 | 95.4 | 34.3 | 80.4 | 22.6 | 70.0 | 37.5 | 73.3 | 28.2 | 71.6 | 11.2 | - | 88.8 | - | 19.8 | - | 17.5 | 64.7 | 58.0 | 78.3 | 26.9 | 70.9 |
| o1-mini | 26.3 | 63.8 | 58.6 | 90.8 | 36.3 | 74.9 | 30.4 | 77.3 | 39.9 | 76.5 | 34.5 | 76.9 | 13.5 | - | 56.8 | - | 21.8 | - | 22.4 | 73.3 | 51.3 | 79.8 | 31.2 | 76.4 |
| o1-preview | 28.2 | 66.8 | 60.3 | 94.8 | 38.5 | 78.4 | 44.9 | 82.9 | 62.4 | 82.7 | 52.2 | 82.8 | 26.0 | - | 81.5 | - | 39.5 | - | 31.8 | 78.1 | 65.4 | 85.4 | 42.7 | 81.6 |
| Model | Action Sequencing | Subgoal Decomposition | ||||||||||||||
| State Goal | Relation Goal | Action Goal | Total | State Goal | Relation Goal | Action Goal | Total | |||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3.5 Sonnet | 87.8 | 63.0 | 83.3 | 62.4 | 60.8 | - | 79.9 | 62.6 | 92.9 | 41.0 | 88.6 | 39.5 | 87.0 | - | 90.1 | 39.9 |
| Claude-3 Opus | 57.2 | 45.0 | 77.8 | 53.0 | 54.7 | - | 62.7 | 50.8 | 92.4 | 43.0 | 88.6 | 41.6 | 83.3 | - | 89.1 | 42.0 |
| Gemini 1.5 Pro | 85.6 | 41.0 | 76.7 | 43.2 | 62.2 | - | 77.2 | 42.6 | 91.2 | 31.0 | 72.5 | 37.1 | 89.5 | - | 83.9 | 35.4 |
| GPT-4o | 87.1 | 49.0 | 76.1 | 45.5 | 56.1 | - | 76.2 | 46.5 | 92.1 | 50.0 | 84.2 | 53.2 | 93.2 | - | 89.4 | 52.3 |
| Llama 3 70B | 61.5 | 31.0 | 68.3 | 45.5 | 42.6 | - | 58.9 | 41.5 | 93.2 | 25.0 | 63.4 | 27.7 | 82.7 | - | 80.0 | 27.0 |
| o1-mini | 88.5 | 64.0 | 72.2 | 66.9 | 57.4 | - | 76.1 | 66.1 | 89.7 | 28.0 | 68.8 | 38.0 | 81.5 | - | 80.3 | 35.3 |
| o1-preview | 80.9 | 89.5 | 65.0 | 84.4 | 46.6 | - | 67.8 | 85.8 | 91.8 | 56.5 | 88.3 | 69.4 | 92.6 | - | 90.6 | 65.9 |
4.1 Ability Module Analysis
Goal interpretation frequently includes temporary states in the final goal. Planning errors mostly arise during execution, especially overlooked prerequisites and redundant actions. Subgoal decomposition still requires detailed reasoning, despite expressing states rather than commands. Transition models often omit necessary preconditions or invent effects, allowing an external planner to succeed in an inaccurate model. The appendix studies sensitivity to individual operators, module composition, replanning, and prompt choices. Statements of numerical error prevalence differ between the introduction and Appendix A; the complete result tables are retained instead of treating one prose percentage as a universal rate.
| Model | Goal Evaluation | Trajectory Evaluation | ||||||||||||||||
| Task SR | Execution SR | Grammar Error () | Runtime Error () | |||||||||||||||
| Parsing | Hallucination | Predicate-Arg Num | Wrong Order | Missing Step | Affordance | Additional Step | ||||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Action Sequencing | ||||||||||||||||||
| Claude-3.5 Sonnet | 76.7 | 60.0 | 81.3 | 69.0 | 0.0 | 0.0 | 2.0 | 0.0 | 0.3 | 0.0 | 0.3 | 5.0 | 14.4 | 25.0 | 1.6 | 1.0 | 1.3 | 2.0 |
| Claude-3 Opus | 64.9 | 51.0 | 69.5 | 59.0 | 0.0 | 0.0 | 17.0 | 0.0 | 0.0 | 0.0 | 0.3 | 3.0 | 12.8 | 35.0 | 0.3 | 3.0 | 2.3 | 2.0 |
| Gemini 1.5 Pro | 76.7 | 42.0 | 83.6 | 54.0 | 0.0 | 0.0 | 1.3 | 0.0 | 0.7 | 0.0 | 0.3 | 6.0 | 14.1 | 39.0 | 0.0 | 1.0 | 3.0 | 2.0 |
| GPT-4o | 71.5 | 47.0 | 81.3 | 53.0 | 1.0 | 0.0 | 2.0 | 1.0 | 0.7 | 0.0 | 0.3 | 9.0 | 15.1 | 36.0 | 0.0 | 1.0 | 2.3 | 0.0 |
| Llama 3 70B | 59.0 | 34.0 | 66.6 | 42.0 | 0.0 | 0.0 | 14.1 | 2.0 | 8.2 | 0.0 | 2.0 | 15.0 | 9.2 | 38.0 | 0.0 | 3.0 | 6.2 | 6.0 |
| o1-mini | 71.5 | 56.0 | 76.4 | 65.0 | 4.9 | 0.0 | 2.0 | 3.0 | 0.0 | 0.0 | 1.0 | 7.0 | 17.7 | 17.0 | 0.3 | 6.0 | 2.6 | 5.0 |
| o1-preview | 65.2 | 81.0 | 72.5 | 91.0 | 6.6 | 0.0 | 11.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 12.1 | 6.0 | 0.3 | 2.0 | 2.0 | 3.0 |
| Subgoal Decomposition | ||||||||||||||||||
| Claude-3.5 Sonnet | 89.1 | 39.0 | 92.0 | 44.0 | 0.0 | 0.0 | 1.8 | 1.0 | 0.0 | 0.0 | 1.5 | 11.0 | 2.7 | 44.0 | 2.1 | 0.0 | 24.6 | 4.0 |
| Claude-3 Opus | 87.0 | 39.0 | 89.9 | 47.0 | 0.3 | 0.0 | 3.3 | 3.0 | 0.0 | 0.0 | 1.2 | 5.0 | 3.0 | 45.0 | 2.4 | 0.0 | 16.0 | 5.0 |
| Gemini 1.5 Pro | 87.0 | 31.0 | 91.1 | 37.0 | 0.0 | 1.0 | 1.5 | 0.0 | 1.8 | 1.0 | 0.0 | 3.0 | 5.6 | 59.0 | 0.0 | 0.0 | 16.0 | 2.0 |
| GPT-4o | 88.8 | 48.0 | 90.2 | 55.0 | 0.0 | 0.0 | 6.2 | 3.0 | 0.0 | 0.0 | 1.2 | 5.0 | 2.4 | 37.0 | 0.0 | 0.0 | 15.7 | 5.0 |
| Llama 3 70B | 78.4 | 20.0 | 87.3 | 30.0 | 0.0 | 1.0 | 2.4 | 5.0 | 0.9 | 1.0 | 2.4 | 8.0 | 5.3 | 51.0 | 1.8 | 4.0 | 20.4 | 4.0 |
| o1-mini | 79.3 | 31.0 | 84.6 | 39.0 | 0.0 | 0.0 | 1.5 | 3.0 | 0.6 | 3.0 | 0.3 | 7.0 | 8.9 | 46.0 | 4.1 | 2.0 | 21.9 | 1.0 |
| o1-preview | 89.4 | 57.0 | 93.2 | 62.0 | 0.0 | 2.0 | 1.5 | 3.0 | 0.0 | 0.0 | 0.3 | 5.0 | 2.7 | 25.0 | 2.4 | 3.0 | 12.1 | 7.0 |
| Model | Object States | Object Orientation | Object Affordance | Spatial Relations | Non-Spatial Relations | |||||||||||||||
| SR | SR | SR | SR | SR | ||||||||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3.5 Sonnet | 60.5 | 78.8 | 67.4 | 86.7 | 95.3 | - | 96.4 | - | 76.6 | - | 67.7 | - | 42.4 | 58.6 | 96.6 | 80.9 | 5.9 | 73.6 | 91.9 | 80.3 |
| Claude-3 Opus | 63.0 | 71.9 | 63.5 | 84.4 | 62.6 | - | 71.4 | - | 75.5 | - | 58.7 | - | 38.7 | 54.6 | 64.8 | 80.9 | 7.0 | 68.8 | 55.4 | 82.0 |
| Gemini 1.5 Pro | 18.8 | 55.9 | 94.4 | 35.6 | 90.9 | - | 89.3 | - | 77.7 | - | 95.8 | - | 38.7 | 35.9 | 89.0 | 40.4 | 7.8 | 52.8 | 83.8 | 39.3 |
| GPT-4o | 54.6 | 71.3 | 71.9 | 68.9 | 52.8 | - | 78.6 | - | 74.9 | - | 63.5 | - | 40.8 | 45.9 | 66.9 | 64.9 | 7.5 | 73.0 | 68.9 | 68.9 |
| Llama-3 70B | 32.5 | 66.3 | 10.1 | 68.9 | 56.6 | - | 3.6 | - | 57.0 | - | 6.6 | - | 27.0 | 47.2 | 15.2 | 77.7 | 3.0 | 58.9 | 18.9 | 85.2 |
| o1-mini | 59.0 | 41.3 | 63.5 | 77.8 | 56.3 | - | 82.1 | - | 58.5 | - | 59.3 | - | 32.5 | 53.1 | 75.9 | 77.7 | 4.5 | 67.5 | 71.6 | 75.4 |
| o1-preview | 58.5 | 78.3 | 69.1 | 86.7 | 78.4 | - | 100.0 | - | 77.5 | - | 67.1 | - | 38.8 | 56.3 | 76.6 | 89.4 | 11.8 | 83.5 | 78.4 | 90.2 |
5 Related Work
Earlier embodied systems use language models for one or several of the four roles, often coupled to planners, skill libraries, or feedback. This benchmark contributes a common representation and diagnostic protocol for comparing those roles. The retained survey table maps prior systems to their interfaces; the contribution is evaluation and composition support rather than a new universal robot controller.
| Goal Interpretation | Subgoal Decomposition | Action Sequencing | Transition Modeling |
| [2, 7, 46, 47, 21, 48, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 13, 14, 15, 49, 50, 51, 52, 53] | [2, 27, 34, 18, 33, 35, 54, 55] | [6, 8, 35, 56, 57, 58, 59, 16, 43, 60, 17, 19, 61, 20, 42, 62, 14, 63, 64, 65, 66, 3, 15, 45, 67, 68, 69] | [8, 28, 32, 36, 70, 37, 13, 38] |
6 Conclusions and Future Work
The interface makes symbolic decision-making failures easier to localize and suggests selective use of language models with external tools. The principal scope limit is that objects, states, actions, and goals are already expressible in a supplied symbolic vocabulary. Sensory grounding, actuation, memory, geometric reasoning, navigation, and uncertain world models require further integration. The later visual and perturbation experiments probe some of these gaps without removing them.
Acknowledgments and Disclosure of Funding
The authors disclose academic, US research-agency, Amazon, and Microsoft support.
Checklist
The checklist reports released code, annotations, evaluation settings, and asset information. Main model comparisons do not include repeated-run error bars because of inference cost. The authors state that existing public data contain no personally identifiable or offensive material and discuss limitations and negative impacts in Appendix S.
Appendix
Appendix A Summary of Empirical Findings
The empirical summary reiterates that models confuse intermediate and final goals, struggle with relational predicates, and generate infeasible action sequences. Reported BEHAVIOR action task/execution success is 81%/91% for o1-preview versus 47%/53% for GPT-4o; VirtualHome rankings differ, with Mistral Large and Gemini 1.5 Pro ahead of o1-preview on the cited action metrics. Learned subgoal sequences are not uniformly easier than actions. Operator consistency matters when predicted and reference transitions are mixed. Composition can preserve executability while degrading task achievement, and feedback can improve both. Some claims in this summary reflect different model subsets or conflict with main-text rankings; consult the complete tables for the relevant metric and simulator.
Appendix B Embodied Agent Interface Design
This appendix relates the diagnostic design, module interfaces, and symbolic decision process.
B.1 Motivation
The design separates goal-directed environmental interaction from general language generation. Each task package contains instructions, symbolic goals, a reference action trajectory, and transition models. The retained diagrams show how standardized goals, interfaces, and error metrics connect these annotations to simulator evaluation.
B.2 Input and Output Details
Inputs combine task instructions, module instructions/examples, and an object-state-relation description. Goal interpretation grounds names into object identifiers; decomposition proposes intermediate states; action sequencing realizes them; transition modeling supplies operator preconditions and effects. The cleaning example illustrates that one state transition may require several actions. The text’s description of transition modeling as a low-level controller refers here to symbolic transitions, not continuous motor control.
B.3 Grounding to Markov Decision Process
The Markov decision-process account uses objects, states, actions, a transition function, reward, and goal. Reward is illustrated as a binary goal-satisfaction indicator. The four modules correspond to interpreting the target, producing state or action trajectories, and modeling transitions. This is a useful symbolic decomposition; temporal goals require trajectory context, so the simple state-only reward expression is not by itself a complete realization of every temporal specification.
B.4 The Relationship between Ability Modules
Isolating a module with reference data or external tools diagnoses its contribution. Decomposition outputs declarative conditions; sequencing outputs imperative commands; transition models enable symbolic search over those commands. Pipeline experiments then measure how errors interact. Isolated scores should therefore not be read as end-to-end performance of one autonomous agent.
Appendix C LTL Representation and Implementation
C.1 Why LTL
Temporal logic gives a compact common language for final-state constraints, required actions, ordering, and alternatives. The decomposition pipeline parses model output into an expression tree, samples a concrete subgoal path, searches for actions, and tests execution and goal achievement. Representation unification does not make all module metrics equivalent.
C.2 Comparision with Traditional LTL Representation
The implemented language extends relational predicates with quantifiers and exact counts, and uses a custom then connective over finite trajectories. It shares recursive logical structure with LTL but is a particular supported fragment, not the full standard temporal language.
C.3 Syntax and Semantics of LTL Formulas
Atomic propositions describe a state relation or an action with object arguments. Boolean operators and quantifiers are evaluated over the finite object universe; exact-count quantification requires exactly the specified number of satisfying objects. A non-temporal formula holds at some state-action pair in the trajectory. Then holds when the trajectory can be divided into a prefix satisfying the first formula and a strictly later suffix satisfying the second. Parsing and dynamic programming evaluate nested expressions. The complete displayed syntax, semantics, and example tree remain available.
Equation
?start: stmt
primitive: VARNAME "(" [args] ")“ # primitive format looks like varname(param)
object_name: VARNAME # object_name can be an object name (eg. pants)
| VARNAMEWITHID # or object name with ID (eg. pants.1000)
args: object_name ("," object_name)* # definition of arguments
?stmt: then_stmt | primitive_stmt # a formula is a Boolean stmt
then_stmt: or_stmt ("then" or_stmt)* # connective priority order:
or_stmt: and_stmt ("or" and_stmt)* # then < or < and < not < forall = forn = exists
and_stmt: primitive_stmt ("and" primitive_stmt)*
primitive_stmt: "not" primitive_stmt -> not_stmt
| primitive
| "(" stmt ")"
| "forall" VARNAME "." "(" stmt ")" -> forall_stmt
| "forn" VARNAME "." "(" stmt ")" -> forn_stmt
| "exists" VARNAME "." "(" stmt ")" -> exists_stmt
VARNAME: /[a-zA-Z_]\w*/
VARNAMEWITHID: /[a-zA-Z_]\w*\.[0-9]+/
Equation
Appendix D Fine-Grained Metrics and Automatic Error Detection
The following protocols distinguish formal correctness, execution, and task achievement.
D.1 Goal Interpretation: State Goal, Relation Goal and Action Goal
Goal predictions are checked for format and invented symbols, then compared with reference unary state goals, binary relation goals, and action goals. Precision penalizes extra predicates, recall measures omitted requirements, and F1 combines them. Action goals are used for VirtualHome tasks without a persistent completion effect; BEHAVIOR does not include them.
D.2 Action Sequencing: Trajectory Error Detection for Missing Step, Additional Step, Wrong Temporal Order, Affordance Error
Action plans are checked against simulator vocabulary, state transitions, and task conditions.
Input Instruction.
The model receives task-relevant objects and initial states, the action/predicate vocabulary, and goal conditions rephrased in natural language. Relevance selection uses reference initial/final differences, so this input is already curated rather than a raw, unfiltered scene.
Trajectory Evaluation.
Grammar checks distinguish malformed output, invented action/object names, and wrong argument counts. Runtime categorization first tests object affordances, then whether the intended effect is already true. Unsatisfied prerequisites become missing-step errors if they were never true in the history, or wrong-order errors if they were true earlier. This deterministic hierarchy makes diagnosis reproducible, while its labels remain dependent on the simulator’s transition rules.
def check_runtime_errors(action, current_state, historical_states):
# Check affordance error
if not is_affordable(action, current_state):
return "Affordance Error"
# Check if action effect is redundant (Additional Step)
if is_effect_redundant(action, current_state):
return "Additional Step"
# Check if precondition for action is met
if not is_precondition_satisfied(action, current_state):
# Check if the effect has not been satisfied
if not is_precondition_ever_satisfied(action, historical_states):
# Check historical states for precondition satisfaction
if not precondition_satisfied_in_history(action, historical_states):
return "Missing Step"
else:
return "Wrong Order"
return "No Runtime Error"
Goal Satisfaction
Execution proceeds until a blocking action or the end of the sequence. The resulting state and executed actions are checked against node, relation, and action goals. Full task success requires all specified conditions; partial goal recall is reported separately. A partially satisfied task is not counted as complete.
D.3 Subgoal Decomposition: Converting Subgoal Trajectory to Action Trajectory with BFS Searching
Breadth-first search connects each generated subgoal to an action trajectory, carrying the reached state forward. Several refinements may be possible; evaluation uses the best considered realization. The paper notes that breadth-first search returns shortest paths in its discrete search space and can be expensive at scale. The main text describes a per-subgoal action restriction, while this appendix says no additional search-space control is needed for the benchmark’s small vocabulary; neither implies scalability to unconstrained robot action spaces.
Initialize an empty state-action sequence Set the current state to the initial state For each subgoal in the subgoal sequence : For each subgoal in the subgoal sequence : Perform BFS starting from to find a state that satisfies . Extract the path from to and append the corresponding state-action pairs to . Set to . Return the complete state-action sequence .
D.4 Transition Modeling: Evaluating with PDDL Planners
Transition-model evaluation supplies object/predicate information and operator signatures, then scores generated bodies. Recursive clause parsing and bipartite matching compare preconditions and effects with reference logical structure, producing precision, recall, and F1. A PDDL planner additionally tests whether the predicted domain admits a plan. Surface/structural matching can penalize alternative formulations, and a satisfiable predicted domain can still encode physically incorrect dynamics.
if expr1 and expr2 are literals:
return expr1 == expr2
if type(expr1) != type(expr2):
return 0
if isinstance(expr1, Not):
return match_expressions(expr1.expression, expr2.expression)
if isinstance(expr1, When):
if match_expressions(expr1.condition, expr2.condition)
and match_expressions(expr1.consequence, expr2.consequence):
return 1
return 0
if isinstance(expr1, (Exists, Forall)):
if quantifier are the same:
return match_expressions(expr1.body, expr2.body)
return 0
if isinstance(expr1, (And, Or)):
adj_matrix = np.zeros((len(expr1.args), len(expr2.args)), dtype=int)
for i in range(size1):
for j in range(size2):
adj_matrix[i, j] = match_expressions(sub1[i], sub2[j])
match_result = maximum_bipartite_matching(sparse_matrix, perm_type=’column’)
total_match = sum(adj_matrix[i, match_result[i]]
for i in range(size1) if match_result[i] != -1)
max_possible_match = min(size1, size2)
return total_match / max_possible_match if max_possible_match > 0 else 0
Equations
D.5 Average Performance
The overall score averages goal-interpretation F1, subgoal task success, action task success, and an average of transition-model F1 and planner success. This mixes complementary measures; it is a diagnostic aggregate, not a probability that a full robot pipeline succeeds.
Equation
Appendix E Full Results with 18 models
Complete results for all eighteen models and both simulators are preserved, alongside qualitative error cases. Scores and model names are those of the evaluated versions, not a current model ranking.
E.1 Goal Interpretation
Goal-interpretation results separate unary state, relation, and VirtualHome action goals, with micro-averaged overall precision, recall, and F1.
E.1.1 Result Analysis
The detailed analysis identifies o1-preview as the highest overall F1 model on VirtualHome and Claude-3.5 Sonnet on BEHAVIOR, with Gemini 1.5 Pro competitive and Llama-3 70B strongest among the discussed open models. Category leaders differ. This supersedes the appendix summary’s blanket claim that Gemini leads both simulators; the retained table provides the exact per-category values.
E.1.2 Case Studies
Cases distinguish malformed syntax and invented symbols from semantically missing or extra goal conditions. Strong proprietary models rarely fail basic formatting under these prompts, but still mistake temporary states for final requirements and omit spatial links such as food-on-plate. The authors suggest implicit reasoning may leak intermediate states into outputs; this is an interpretation of errors, not a demonstrated mechanism inside the models.
Top-performing Models: o1-preview and Claude 3.5 Sonnet achieves top overall goal interpretation performance (F1-score) in VirtualHome and BEHAVIOR simulators, respectively. Whereas Gemini 1.5 Pro achieves the highest accuracy in VirtualHome .
(2)Grammar Errors: Current SOTA proprietary LLMs generally make very little to no grammar errors in goal interpretation, whereas top open-source LLMs including Llama 3 70B Instruct tend to suffer more from format/parsing errors and object/state hallucination.
(3)Goal Prediction Errors: Both open-source and proprietary LLMs suffer significantly from two common types of goal prediction errors:
(a)Generating intermediate goals in place of final goals (when asked to directly output final goals), resulting in false positive goal predictions.
(b)Omitting conversationally uncommon spatial relationship goals that are essential for precise robotic control, resulting in missing goals (reporting bias).
| Model Name | Goal Interpretation | |||||||||||||||||||||||
| State | Spatial | Action | Overall | |||||||||||||||||||||
| Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | |||||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3 Haiku | 21.8 | 22.8 | 58.9 | 93.5 | 31.8 | 36.7 | 24.2 | 64.5 | 50.8 | 64.6 | 32.8 | 64.6 | 12.2 | - | 95.7 | - | 21.6 | - | 18.0 | 41.5 | 63.2 | 71.2 | 28.0 | 52.5 |
| Claude-3 Sonnet | 23.3 | 36.8 | 57.1 | 88.9 | 33.1 | 52.0 | 26.6 | 76.2 | 53.0 | 79.8 | 35.5 | 77.9 | 12.4 | - | 85.8 | - | 21.7 | - | 19.3 | 60.2 | 61.5 | 81.9 | 29.4 | 69.4 |
| Claude-3 Opus | 27.0 | 72.6 | 66.9 | 93.5 | 38.5 | 81.7 | 22.6 | 75.2 | 46.8 | 79.2 | 30.5 | 77.1 | 14.5 | - | 92.6 | - | 25.1 | - | 20.7 | 72.2 | 65.0 | 82.5 | 31.4 | 77.0 |
| Claude-3.5 Sonnet | 25.3 | 74.0 | 60.9 | 94.8 | 35.8 | 83.1 | 31.1 | 84.4 | 63.8 | 81.3 | 41.8 | 82.9 | 14.0 | - | 98.8 | - | 24.5 | - | 21.7 | 81.1 | 69.6 | 84.4 | 33.0 | 82.7 |
| Cohere Command R | 51.1 | 7.7 | 69.6 | 31.4 | 58.9 | 12.4 | 34.5 | 56.8 | 21.3 | 55.0 | 26.3 | 55.9 | 3.6 | - | 38.9 | - | 6.5 | - | 27.4 | 28.2 | 55.7 | 49.6 | 36.7 | 36.0 |
| Cohere Command R+ | 20.9 | 23.3 | 52.0 | 79.1 | 29.8 | 36.0 | 17.9 | 66.7 | 15.2 | 61.5 | 16.4 | 64.0 | 10.4 | - | 82.6 | - | 18.5 | - | 14.9 | 42.0 | 44.5 | 65.5 | 22.4 | 51.2 |
| Gemini 1.0 Pro | 25.3 | 27.4 | 57.9 | 81.1 | 34.9 | 41.0 | 17.0 | 75.2 | 20.6 | 70.4 | 18.6 | 72.7 | 9.9 | - | 68.7 | - | 17.2 | - | 16.2 | 51.0 | 45.2 | 72.8 | 23.8 | 60.0 |
| Gemini 1.5 Flash | 23.6 | 55.8 | 57.9 | 94.1 | 33.5 | 70.1 | 19.8 | 76.6 | 21.1 | 76.7 | 20.5 | 76.7 | 13.5 | - | 90.1 | - | 23.5 | - | 18.2 | 69.7 | 50.8 | 80.7 | 26.8 | 74.8 |
| Gemini 1.5 Pro | 47.2 | 94.0 | 47.5 | 92.8 | 47.3 | 93.4 | 42.0 | 74.4 | 7.2 | 76.7 | 12.4 | 75.6 | 24.1 | - | 81.4 | - | 37.2 | - | 33.6 | 78.8 | 39.3 | 80.4 | 36.2 | 79.6 |
| GPT-3.5-turbo | 22.4 | 52.0 | 50.0 | 66.7 | 30.9 | 58.5 | 8.5 | 51.5 | 18.8 | 46.9 | 11.7 | 49.1 | 15.2 | - | 60.5 | - | 24.4 | - | 15.7 | 49.5 | 40.5 | 51.4 | 22.7 | 50.4 |
| GPT-4-turbo | 28.6 | 70.4 | 58.5 | 86.9 | 38.4 | 77.8 | 24.7 | 77.5 | 32.9 | 76.4 | 28.2 | 76.9 | 19.0 | - | 82.1 | - | 30.9 | - | 24.0 | 75.6 | 53.8 | 78.8 | 33.2 | 77.2 |
| GPT-4o | 29.0 | 67.1 | 60.0 | 94.8 | 39.1 | 78.6 | 31.5 | 81.1 | 43.6 | 78.5 | 36.6 | 79.8 | 20.5 | - | 85.8 | - | 33.1 | - | 26.4 | 76.5 | 59.1 | 82.2 | 36.5 | 79.2 |
| Llama 3 8B Instruct | 21.7 | 17.3 | 54.4 | 80.4 | 31.0 | 28.4 | 14.0 | 51.4 | 7.4 | 20.8 | 9.7 | 29.6 | 11.1 | - | 79.4 | - | 19.4 | - | 15.5 | 24.1 | 41.9 | 34.3 | 22.6 | 28.3 |
| Llama 3 70B Instruct | 23.9 | 69.5 | 61.2 | 95.4 | 34.3 | 80.4 | 22.6 | 70.0 | 37.5 | 73.3 | 28.2 | 71.6 | 11.2 | - | 88.8 | - | 19.8 | - | 17.5 | 64.7 | 58.0 | 78.3 | 26.9 | 70.9 |
| Mistral Large | 23.6 | 63.5 | 59.1 | 92.2 | 32.8 | 75.2 | 23.7 | 75.1 | 40.3 | 76.2 | 29.8 | 75.6 | 11.2 | - | 84.0 | - | 19.7 | - | 17.5 | 69.6 | 57.1 | 79.8 | 26.8 | 74.3 |
| Mixtral 8x22B MoE | 23.6 | 22.9 | 56.9 | 83.7 | 33.4 | 36.0 | 22.2 | 70.7 | 36.3 | 67.7 | 27.5 | 69.2 | 11.2 | - | 94.8 | - | 20.0 | - | 17.4 | 44.4 | 56.2 | 71.3 | 26.6 | 54.7 |
| o1-mini | 26.3 | 63.8 | 58.6 | 90.8 | 36.3 | 74.9 | 30.4 | 77.3 | 39.9 | 76.5 | 34.5 | 76.9 | 13.5 | - | 56.8 | - | 21.8 | - | 22.4 | 73.3 | 51.3 | 79.8 | 31.2 | 76.4 |
| o1-preview | 28.2 | 66.8 | 60.3 | 94.8 | 38.5 | 78.4 | 44.9 | 82.9 | 62.4 | 82.7 | 52.2 | 82.8 | 26.0 | - | 81.5 | - | 39.5 | - | 31.8 | 78.1 | 65.4 | 85.4 | 42.7 | 81.6 |
E.2 Subgoal Decomposition
E.2.1 Result Analysis
The complete decomposition tables separate executable refinements, full task success, grammar/runtime failures, and partial goal satisfaction.
VirtualHome
VirtualHome subgoal task success reaches 89.4% for o1-preview and 89.1% for Gemini 1.5 Flash and Claude-3.5 Sonnet. Execution success can be higher, illustrating that feasible plans may miss goals. Strong models usually follow syntax but can invent predicates such as POUR. Redundant subgoals are common when the initial scene already satisfies them; weaker models also miss container-opening and standing prerequisites. State conditions are generally easier than relations or required actions.
BEHAVIOR
BEHAVIOR is substantially harder: o1-preview reaches 57% task success and 62% execution success in this comparison. Missing prerequisites dominate; even that model has a 25% missing-step rate. The simulator enforces tool possession and other conditions absent from analogous VirtualHome actions. Quantified requirements are scored as combined conditions, so one unmet instance can fail the whole quantified goal. Differences in these semantics and initial states contribute to the cross-simulator gap.
| Model | Goal Evaluation | Trajectory Evaluation | ||||||||||||||||
| Task SR | Execution SR | Grammar Error () | Runtime Error () | |||||||||||||||
| Parsing | Hallucination | Predicate-Arg Num | Wrong Order | Missing Step | Affordance | Additional Step | ||||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3 Haiku | 78.4 | 30.0 | 82.8 | 35.0 | 0.3 | 0.0 | 2.4 | 1.0 | 1.8 | 0.0 | 1.8 | 3.0 | 2.7 | 58.0 | 8.3 | 3.0 | 20.4 | 3.0 |
| Claude-3 Sonnet | 83.1 | 39.0 | 86.4 | 43.0 | 0.0 | 0.0 | 1.8 | 2.0 | 0.0 | 2.0 | 0.6 | 3.0 | 2.7 | 51.0 | 8.6 | 1.0 | 33.7 | 3.0 |
| Claude-3 Opus | 87.0 | 41.0 | 90.0 | 47.0 | 0.3 | 0.0 | 3.6 | 3.0 | 0.0 | 0.0 | 1.2 | 5.0 | 3.0 | 45.0 | 2.4 | 0.0 | 16.0 | 6.0 |
| Claude-3.5 Sonnet | 89.1 | 39.0 | 92.0 | 44.0 | 0.0 | 0.0 | 1.8 | 1.0 | 0.0 | 0.0 | 1.5 | 11.0 | 2.7 | 44.0 | 2.1 | 0.0 | 24.6 | 4.0 |
| Gemini 1.0 Pro | 70.4 | 24.0 | 84.6 | 33.0 | 0.6 | 2.0 | 3.3 | 4.0 | 2.4 | 0.0 | 1.2 | 3.0 | 2.7 | 51.0 | 5.3 | 7.0 | 10.4 | 3.0 |
| Gemini 1.5 Flash | 89.1 | 34.0 | 94.1 | 42.0 | 0.0 | 2.0 | 1.5 | 1.0 | 0.0 | 0.0 | 0.6 | 2.0 | 3.9 | 53.0 | 0.0 | 0.0 | 13.3 | 3.0 |
| Gemini 1.5 Pro | 87.0 | 31.0 | 91.1 | 37.0 | 0.0 | 1.0 | 1.5 | 0.0 | 1.8 | 1.0 | 0.0 | 3.0 | 5.6 | 59.0 | 0.0 | 0.0 | 16.0 | 2.0 |
| GPT-3.5-turbo | 69.2 | 24.0 | 81.4 | 36.0 | 1.5 | 2.0 | 0.0 | 3.0 | 0.6 | 0.0 | 1.5 | 4.0 | 11.8 | 51.0 | 3.3 | 4.0 | 20.4 | 3.0 |
| GPT-4-turbo | 85.5 | 38.0 | 94.1 | 47.0 | 0.0 | 0.0 | 1.8 | 3.0 | 0.0 | 0.0 | 1.5 | 9.0 | 2.4 | 40.0 | 0.3 | 1.0 | 22.2 | 6.0 |
| GPT-4o | 88.8 | 49.0 | 90.2 | 55.0 | 0.0 | 0.0 | 6.2 | 3.0 | 0.0 | 0.0 | 1.2 | 6.0 | 2.4 | 36.0 | 0.0 | 0.0 | 15.7 | 5.0 |
| Cohere Command R | 71.3 | 15.0 | 79.6 | 25.0 | 2.1 | 23.0 | 3.9 | 10.0 | 0.9 | 0.0 | 1.5 | 0.0 | 6.2 | 37.0 | 5.9 | 5.0 | 14.5 | 4.0 |
| Cohere Command R+ | 79.0 | 25.0 | 83.7 | 37.0 | 1.5 | 2.0 | 4.5 | 4.0 | 2.1 | 0.0 | 0.9 | 4.0 | 7.7 | 52.0 | 2.7 | 1.0 | 16.0 | 6.0 |
| Mistral Large | 84.3 | 31.0 | 92.0 | 38.0 | 0.3 | 1.0 | 1.8 | 3.0 | 0.3 | 0.0 | 2.1 | 4.0 | 3.3 | 52.0 | 0.3 | 2.0 | 11.0 | 1.0 |
| Mixtral 8x22B MoE | 80.5 | 28.0 | 90.2 | 33.0 | 0.3 | 0.0 | 2.4 | 4.0 | 0.0 | 0.0 | 3.0 | 2.0 | 3.9 | 59.0 | 0.3 | 2.0 | 11.2 | 0.0 |
| Llama 3 8B | 48.8 | 21.0 | 58.0 | 29.0 | 0.6 | 2.0 | 2.4 | 11.0 | 0.6 | 0.0 | 6.8 | 6.0 | 5.0 | 44.0 | 26.6 | 8.0 | 18.3 | 7.0 |
| Llama 3 70B | 78.4 | 20.0 | 87.3 | 30.0 | 0.0 | 1.0 | 2.4 | 5.0 | 0.9 | 1.0 | 2.4 | 8.0 | 5.3 | 51.0 | 1.8 | 4.0 | 20.4 | 4.0 |
| o1-mini | 79.3 | 31.0 | 84.6 | 39.0 | 0.0 | 0.0 | 1.5 | 3.0 | 0.6 | 3.0 | 0.3 | 7.0 | 8.9 | 46.0 | 4.1 | 2.0 | 21.9 | 1.0 |
| o1-preview | 89.4 | 57.0 | 93.2 | 62.0 | 0.0 | 2.0 | 1.5 | 3.0 | 0.0 | 0.0 | 0.3 | 5.0 | 2.7 | 25.0 | 2.4 | 3.0 | 12.1 | 7.0 |
| Model | Action Sequencing | Subgoal Decomposition | ||||||||||||||
| State Goal | Relation Goal | Action Goal | Total | State Goal | Relation Goal | Action Goal | Total | |||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3 Haiku | 71.2 | 27.0 | 53.9 | 38.7 | 41.9 | - | 58.9 | 35.5 | 89.4 | 26.0 | 82.2 | 34.8 | 71.6 | - | 83.1 | 32.4 |
| Claude-3 Sonnet | 80.2 | 41.0 | 68.3 | 59.8 | 38.5 | - | 66.5 | 54.6 | 89.1 | 37.0 | 89.3 | 49.8 | 83.3 | - | 88.0 | 46.3 |
| Claude-3 Opus | 57.2 | 45.0 | 77.8 | 53.0 | 54.7 | - | 62.7 | 50.8 | 92.4 | 43.0 | 88.6 | 41.6 | 83.3 | - | 89.1 | 42.0 |
| Claude-3.5 Sonnet | 87.8 | 63.0 | 83.3 | 62.4 | 60.8 | - | 79.9 | 62.6 | 92.9 | 41.0 | 88.6 | 39.5 | 87.0 | - | 90.1 | 39.9 |
| Gemini 1.0 Pro | 70.5 | 28.0 | 41.7 | 32.0 | 34.5 | - | 53.1 | 30.9 | 84.4 | 26.0 | 61.5 | 31.1 | 72.8 | - | 73.5 | 29.7 |
| Gemini 1.5 Flash | 86.0 | 34.0 | 73.3 | 50.0 | 47.3 | - | 72.8 | 45.6 | 93.5 | 44.0 | 88.3 | 36.0 | 92.0 | - | 91.3 | 38.2 |
| Gemini 1.5 Pro | 85.6 | 41.0 | 76.7 | 43.2 | 62.2 | - | 77.2 | 42.6 | 91.2 | 31.0 | 72.5 | 37.1 | 89.5 | - | 83.9 | 35.4 |
| GPT-3.5-turbo | 49.3 | 20.0 | 25.0 | 22.6 | 33.8 | - | 38.3 | 21.9 | 84.7 | 28.0 | 54.4 | 28.5 | 64.8 | - | 69.4 | 28.3 |
| GPT-4-turbo | 75.9 | 39.0 | 75.0 | 39.5 | 48.6 | - | 69.0 | 39.3 | 93.5 | 45.0 | 84.2 | 46.1 | 90.7 | - | 89.5 | 45.8 |
| GPT-4o | 87.1 | 49.0 | 76.1 | 45.5 | 56.1 | - | 76.2 | 46.5 | 92.1 | 50.0 | 84.2 | 53.2 | 93.2 | - | 89.4 | 52.3 |
| Cohere Command R | 46.8 | 20.0 | 38.9 | 25.9 | 48.6 | - | 44.9 | 24.3 | 85.3 | 20.0 | 67.4 | 21.4 | 60.5 | - | 73.6 | 21.0 |
| Cohere Command R+ | 68.7 | 28.0 | 54.4 | 32.0 | 50.7 | - | 60.1 | 30.9 | 89.4 | 34.0 | 66.8 | 29.6 | 75.9 | - | 78.3 | 30.8 |
| Mistral Large | 88.5 | 38.5 | 81.1 | 41.2 | 70.9 | - | 82.0 | 40.4 | 92.9 | 33.0 | 71.5 | 35.6 | 90.1 | - | 84.4 | 34.9 |
| Mixtral 8x22B MoE | 69.4 | 30.0 | 62.2 | 36.8 | 57.4 | - | 64.4 | 35.0 | 92.1 | 30.0 | 74.8 | 34.1 | 87.7 | - | 84.8 | 33.0 |
| Llama 3 8B | 34.2 | 16.0 | 16.7 | 23.7 | 11.5 | - | 23.4 | 21.6 | 68.8 | 21.0 | 54.7 | 23.6 | 50.0 | - | 59.8 | 22.9 |
| Llama 3 70B | 61.5 | 31.0 | 68.3 | 45.5 | 42.6 | - | 58.9 | 41.5 | 93.2 | 25.0 | 63.4 | 27.7 | 82.7 | - | 80.0 | 27.0 |
| o1-mini | 88.5 | 64.0 | 72.2 | 66.9 | 57.4 | - | 76.1 | 66.1 | 89.7 | 28.0 | 68.8 | 38.0 | 81.5 | - | 80.3 | 35.3 |
| o1-preview | 80.9 | 89.5 | 65.0 | 84.4 | 46.6 | - | 67.8 | 85.8 | 91.8 | 56.5 | 88.3 | 69.4 | 92.6 | - | 90.6 | 65.9 |
E.2.2 Case Studies
The preserved case figures show how formal, temporal, affordance, and goal errors differ.
Grammar Errors
Examples include duplicated punctuation, unmatched closing parentheses, incorrect predicate arity, and object identifiers absent from the scene. Such failures are detected before executing a subgoal realization.
Runtime Errors
Runtime cases include filling both hands before a plug action, dropping soap before cleaning, retrieving items from closed containers, trying to hold a fridge or slice a knife, and opening an already open container. The distinctions rely on both present state and historical prerequisite satisfaction.
Goal Satisfaction Errors
A plan may execute yet omit an intended light/window state, required placement, or DRINK action. Another example satisfies one adjacency relation and later destroys it while moving a shared object. Satisfying each subgoal once is therefore insufficient when final conditions must hold together.
Top-performing Models: o1-preview demonstrates superior performance in both VirtualHome and BEHAVIOR simulator compared with other SOTA LLMs, achieving the highest task success rates of 89.4% and 57.0%. Claude-3.5 Sonnet and Gemini 1.5 Flash also demonstrate high performance in VirtualHome with a success rate close to 90%.
(2)Grammar and Runtime Errors: State-of-the-art models generally avoid grammar errors but can hallucinate actions and objects. The most common runtime errors in VirtualHome are additional steps, whereas most of runtime errors in BEHAVIOR are missing steps.
(3)Goal Satisfaction: Stronger LLMs like o1-preview and GPT-4o show higher accuracy in action goal success rates in VirtualHome compared to weaker models like Llama 3 8B. However, achieving state and relation goals in BEHAVIOR is challenging due to more complex task representations and stricter precondition checks.
(4)Impact of Task Complexity In Different Simulators: Overall LLM performance is lower in BEHAVIOR compared with VirtualHome due to complex task representations involving quantifiers like forall and forpairs, which articulate complex temporal and spatial requirements.
E.3 Action Sequencing
Action-sequencing rankings differ between the relatively short VirtualHome plans and longer BEHAVIOR tasks. The text alternates Mistral Large and Mixtral Large naming and retains an outdated claim that the best BEHAVIOR model completes only half the tasks, whereas its updated results report o1-preview at 81%; the tables should govern quantitative comparisons. Strong models make fewer syntax errors but still omit tool/sink/container prerequisites, act in a destructive order, or repeat completed actions. State-versus-relation difficulty also depends on simulator semantics. Explanations involving overthinking, model training quality, or memory failures are hypotheses rather than controlled causal findings.
Larger models did not always outperform smaller ones, especially for shorter and simpler tasks.
2.o1-preview takes the lead. Other LLMs (including powerful ones like GPT-4-turbo, Gemini 1.5 Pro, and Claude-3 Opus) consistently outperformed others across all runtime error metrics.
3.Generally, larger model sizes and higher quality training data correlated with lower grammar error rates.
4.Commonsense reasoning about object properties, agent states, and preconditions for actions remains a significant challenge for current LLMs, particularly when given limited context.
5.Errors in goal satisfaction can be attributed to: 1) Missed goals: Failing to generate actions to achieve the goal. 2) Wrong actions: Incorrectly mapping goals to inappropriate actions.
| Model | Goal Evaluation | Trajectory Evaluation | ||||||||||||||||
| Task SR | Execution SR | Grammar Error () | Runtime Error () | |||||||||||||||
| Parsing | Hallucination | Predicate-Arg Num | Wrong Order | Missing Step | Affordance | Additional Step | ||||||||||||
| V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | V | B | |
| Claude-3 Haiku | 54.8 | 26.0 | 60.7 | 32.0 | 0.0 | 0.0 | 6.6 | 6.0 | 0.0 | 0.0 | 0.7 | 7.0 | 30.5 | 54.0 | 1.6 | 1.0 | 1.6 | 1.0 |
| Claude-3 Sonnet | 58.4 | 44.0 | 63.3 | 57.0 | 0.0 | 0.0 | 4.9 | 1.0 | 0.0 | 7.9 | 4.6 | 11.0 | 26.9 | 19.0 | 0.3 | 11.0 | 3.0 | 2.0 |
| Claude-3 Opus | 64.9 | 51.0 | 69.5 | 59.0 | 0.0 | 0.0 | 17.0 | 0.0 | 0.0 | 0.0 | 0.3 | 3.0 | 12.8 | 35.0 | 0.3 | 3.0 | 2.3 | 2.0 |
| Claude-3.5 Sonnet | 76.7 | 60.0 | 81.3 | 69.0 | 0.0 | 0.0 | 2.0 | 0.0 | 0.3 | 0.0 | 0.3 | 5.0 | 14.4 | 25.0 | 1.6 | 1.0 | 1.3 | 2.0 |
| Gemini 1.0 Pro | 45.6 | 27.0 | 56.7 | 32.0 | 0.0 | 7.0 | 3.6 | 3.0 | 2.6 | 6.0 | 2.0 | 13.0 | 33.4 | 35.0 | 1.6 | 4.0 | 8.2 | 4.0 |
| Gemini 1.5 Flash | 69.5 | 40.0 | 75.4 | 52.0 | 0.0 | 0.0 | 2.0 | 0.0 | 0.0 | 0.0 | 0.0 | 5.0 | 22.3 | 42.0 | 0.3 | 1.0 | 1.6 | 2.0 |
| Gemini 1.5 Pro | 76.7 | 42.0 | 83.6 | 54.0 | 0.0 | 0.0 | 1.3 | 0.0 | 0.7 | 0.0 | 0.3 | 6.0 | 14.1 | 39.0 | 0.0 | 1.0 | 3.0 | 2.0 |
| GPT-3.5-turbo | 24.9 | 16.0 | 40.7 | 20.0 | 10.2 | 4.0 | 1.0 | 7.0 | 2.0 | 23.0 | 0.0 | 1.0 | 41.6 | 36.0 | 4.6 | 8.0 | 3.0 | 1.3 |
| GPT-4-turbo | 60.0 | 38.0 | 65.2 | 45.0 | 0.0 | 0.0 | 2.0 | 0.0 | 1.6 | 0.0 | 0.0 | 7.0 | 30.8 | 47.0 | 0.3 | 1.0 | 1.3 | 0.0 |
| GPT-4o | 71.5 | 47.0 | 81.3 | 53.0 | 1.0 | 0.0 | 2.0 | 1.0 | 0.7 | 0.0 | 0.3 | 9.0 | 15.1 | 36.0 | 0.0 | 1.0 | 2.3 | 0.0 |
| Cohere Command R | 44.9 | 16.0 | 44.3 | 19.0 | 2.0 | 5.0 | 23.3 | 13.0 | 2.6 | 0.0 | 2.0 | 8.0 | 17.0 | 43.0 | 9.8 | 12.0 | 3.6 | 4.0 |
| Cohere Command R+ | 54.1 | 27.0 | 65.2 | 35.0 | 6.9 | 0.0 | 12.5 | 1.0 | 2.0 | 15.0 | 0.3 | 10.0 | 11.8 | 39.0 | 1.6 | 0.0 | 3.9 | 15.0 |
| Mistral Large | 78.4 | 33.0 | 84.6 | 50.0 | 0.0 | 0.0 | 2.0 | 0.0 | 0.3 | 0.0 | 0.0 | 8.0 | 12.5 | 35.0 | 0.7 | 6.0 | 3.9 | 7.0 |
| Mixtral 8x22B MoE | 63.3 | 30.0 | 67.9 | 40.0 | 0.0 | 3.0 | 13.1 | 6.0 | 1.0 | 0.0 | 1.3 | 10.0 | 14.1 | 32.0 | 2.6 | 9.0 | 5.6 | 2.0 |
| Llama 3 8B | 21.3 | 10.0 | 23.6 | 16.0 | 1.0 | 0.0 | 34.1 | 15.0 | 8.2 | 9.0 | 1.0 | 6.0 | 30.8 | 44.0 | 1.3 | 9.0 | 2.6 | 5.0 |
| Llama 3 70B | 59.0 | 34.0 | 66.6 | 42.0 | 0.0 | 0.0 | 14.1 | 2.0 | 8.2 | 0.0 | 2.0 | 15.0 | 9.2 | 38.0 | 0.0 | 3.0 | 6.2 | 6.0 |
| o1-mini | 71.5 | 56.0 | 76.4 | 65.0 | 4.9 | 0.0 | 2.0 | 3.0 | 0.0 | 0.0 | 1.0 | 7.0 | 17.7 | 17.0 | 0.3 | 6.0 | 2.6 | 5.0 |
| o1-preview | 65.2 | 81.0 | 72.5 | 91.0 | 6.6 | 0.0 | 11.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 12.1 | 6.0 | 0.3 | 2.0 | 2.0 | 3.0 |
E.4 Transition Modeling
Full transition-model results are retained for logical matching and planner success, including category breakdowns.
| Model | Object States | Object Orientation | Object Affordance | Spatial Relations | Non-Spatial Relations | ||||||||||
| Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | |
| Claude-3 Haiku | 76.0 | 40.1 | 52.5 | 19.0 | 34.4 | 24.4 | 67.8 | 73.9 | 70.7 | 37.7 | 38.7 | 38.2 | 2.0 | 1.5 | 1.7 |
| Claude-3 Opus | 87.4 | 49.2 | 63.0 | 46.3 | 96.9 | 62.6 | 76.8 | 74.3 | 75.5 | 37.6 | 39.9 | 38.7 | 10.4 | 5.2 | 7.0 |
| Claude-3 Sonnet | 76.6 | 37.4 | 50.3 | 48.1 | 78.1 | 59.5 | 60.7 | 74.3 | 66.8 | 32.3 | 39.9 | 35.7 | 6.2 | 4.1 | 4.9 |
| Claude-3.5 Sonnet | 86.1 | 46.7 | 60.5 | 93.9 | 96.9 | 95.3 | 77.7 | 75.5 | 76.6 | 45.3 | 39.8 | 42.4 | 7.1 | 5.1 | 5.9 |
| Cohere Command R | 18.0 | 6.8 | 9.9 | 38.7 | 90.6 | 54.2 | 40.2 | 23.0 | 29.2 | 12.6 | 6.7 | 8.8 | 3.3 | 0.9 | 1.4 |
| Cohere Command R+ | 44.9 | 19.0 | 26.3 | 34.6 | 68.8 | 45.9 | 51.0 | 62.1 | 56.0 | 30.1 | 34.8 | 32.4 | 7.6 | 3.1 | 4.4 |
| Gemini 1.0 Pro | 68.4 | 12.3 | 20.4 | 16.3 | 62.5 | 27.9 | 55.3 | 20.1 | 29.6 | 45.0 | 16.5 | 24.3 | 7.7 | 2.5 | 3.8 |
| Gemini 1.5 Flash | 82.3 | 37.6 | 51.6 | 2.0 | 3.1 | 2.5 | 54.4 | 74.7 | 62.9 | 47.4 | 42.9 | 45.0 | 16.3 | 5.2 | 7.9 |
| Gemini 1.5 Pro | 45.3 | 11.9 | 18.8 | 88.2 | 93.8 | 90.9 | 79.9 | 75.5 | 77.7 | 42.2 | 35.8 | 38.7 | 15.5 | 5.2 | 7.8 |
| GPT-3.5-turbo | 63.5 | 21.9 | 32.5 | 11.4 | 15.6 | 13.2 | 57.2 | 53.1 | 54.9 | 35.2 | 21.7 | 26.8 | 1.7 | 0.3 | 0.6 |
| GPT-4-turbo | 79.3 | 44.2 | 56.7 | 10.1 | 31.3 | 15.3 | 65.9 | 71.0 | 68.4 | 31.8 | 34.2 | 32.9 | 3.8 | 1.0 | 1.6 |
| GPT-4o | 80.2 | 41.5 | 54.6 | 48.0 | 59.4 | 52.8 | 76.2 | 73.7 | 74.9 | 40.8 | 40.7 | 40.8 | 14.8 | 5.1 | 7.5 |
| Llama 3 8b | 30.8 | 13.7 | 18.9 | 0.0 | 0.0 | 0.0 | 1.6 | 3.2 | 2.1 | 15.5 | 18.2 | 16.8 | 0.0 | 0.0 | 0.0 |
| Llama 3 70b | 63.5 | 21.9 | 32.5 | 49.0 | 66.3 | 56.6 | 65.0 | 50.0 | 57.0 | 27.0 | 27.0 | 27.0 | 5.0 | 2.0 | 3.0 |
| Mistral Large | 30.0 | 8.0 | 13.0 | 48.0 | 88.0 | 62.0 | 72.0 | 29.0 | 41.0 | 35.0 | 18.0 | 24.0 | 3.0 | 1.0 | 1.0 |
| Mixtral 8x22B MoE | 72.0 | 33.0 | 45.0 | 43.0 | 83.0 | 57.0 | 64.0 | 74.0 | 69.0 | 40.0 | 38.0 | 39.0 | 12.0 | 4.0 | 6.0 |
| o1-mini | 82.5 | 45.9 | 59.0 | 51.3 | 62.5 | 56.3 | 59.8 | 57.1 | 58.5 | 32.1 | 32.8 | 32.5 | 5.0 | 4.1 | 4.5 |
| o1-preview | 83.0 | 45.1 | 58.5 | 69.0 | 90.6 | 78.4 | 84.7 | 71.4 | 77.5 | 39.8 | 37.8 | 38.8 | 17.1 | 9.0 | 11.8 |
| Model | Object States | Spatial Relations | Non-Spatial Relations | ||||||
| Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1 | |
| Claude-3.5 Sonnet | 83.3 | 74.8 | 78.8 | 73.3 | 48.8 | 58.6 | 82.9 | 66.2 | 73.6 |
| Claude-3 Haiku | 64.1 | 55.2 | 59.3 | 54.7 | 37.4 | 44.4 | 63.3 | 51.4 | 56.7 |
| Claude-3 Opus | 74.6 | 69.4 | 71.9 | 70.4 | 44.6 | 54.6 | 68.5 | 69.1 | 68.8 |
| Claude-3 Sonnet | 66.2 | 68.7 | 67.5 | 62.8 | 39.8 | 48.7 | 68.8 | 52.0 | 59.2 |
| Cohere Command R | 59.7 | 43.9 | 50.6 | 29.1 | 11.6 | 16.6 | 27.2 | 15.3 | 19.6 |
| Cohere Command R+ | 58.0 | 58.4 | 58.2 | 54.2 | 33.6 | 41.5 | 53.0 | 56.6 | 54.7 |
| Gemini 1.0 Pro | 67.2 | 55.2 | 60.6 | 47.5 | 35.3 | 40.5 | 43.8 | 48.3 | 45.9 |
| Gemini 1.5 Flash | 73.9 | 57.2 | 64.5 | 54.5 | 40.7 | 46.6 | 60.7 | 53.8 | 57.0 |
| Gemini 1.5 Pro | 69.6 | 46.7 | 55.9 | 52.9 | 27.2 | 35.9 | 59.6 | 47.4 | 52.8 |
| GPT-3.5-turbo | 67.1 | 46.1 | 54.6 | 57.6 | 31.6 | 40.9 | 40.8 | 36.1 | 38.3 |
| GPT-4-turbo | 58.2 | 59.4 | 58.8 | 50.3 | 27.8 | 35.8 | 58.5 | 38.4 | 46.4 |
| GPT-4o | 73.1 | 69.6 | 71.3 | 63.9 | 35.8 | 45.9 | 84.7 | 64.2 | 73.0 |
| Llama 3 70b | 68.1 | 64.6 | 66.3 | 60.3 | 38.8 | 47.2 | 65.1 | 53.8 | 58.9 |
| Llama 3 8b | 40.3 | 32.4 | 35.9 | 29.6 | 22.7 | 25.7 | 48.9 | 43.9 | 46.2 |
| Mistral Large | 67.5 | 66.5 | 67.0 | 54.9 | 32.3 | 40.7 | 59.7 | 44.6 | 51.1 |
| Mixtral 8x22B MoE | 60.2 | 60.0 | 60.1 | 53.2 | 39.9 | 45.6 | 57.9 | 55.8 | 56.8 |
| o1-mini | 46.3 | 37.2 | 41.3 | 71.1 | 42.3 | 53.1 | 80.1 | 58.3 | 67.5 |
| o1-preview | 85.5 | 72.3 | 78.3 | 72.4 | 46.1 | 56.3 | 88.0 | 79.5 | 83.5 |
| Model | Object States | Object Orientation | Object Affordance | Spatial Relations | Non-Spatial Relations | |||||
| V | B | V | B | V | B | V | B | V | B | |
| Claude-3 Haiku | 13.5 | 68.9 | 3.6 | - | 19.8 | - | 46.9 | 62.8 | 73.0 | 62.3 |
| Claude-3 Opus | 63.5 | 84.4 | 71.4 | - | 58.7 | - | 64.8 | 80.9 | 55.4 | 82.0 |
| Claude-3 Sonnet | 11.2 | 80.0 | 3.6 | - | 10.8 | - | 20.0 | 79.8 | 13.5 | 80.3 |
| Claude-3.5 Sonnet | 67.4 | 86.7 | 96.4 | - | 67.8 | - | 96.6 | 80.8 | 91.9 | 80.3 |
| Cohere Command R | 44.6 | 48.9 | 82.1 | - | 40.1 | - | 62.6 | 38.3 | 58.3 | 39.3 |
| Cohere Command R+ | 36.5 | 77.8 | 46.4 | - | 35.3 | - | 40.7 | 57.4 | 31.1 | 47.5 |
| Gemini 1.0 Pro | 10.7 | 22.2 | 0.0 | - | 10.2 | - | 14.5 | 13.8 | 2.7 | 14.8 |
| Gemini 1.5 Flash | 34.8 | 55.6 | 7.1 | - | 46.7 | - | 61.4 | 68.1 | 60.8 | 70.5 |
| Gemini 1.5 Pro | 94.4 | 35.6 | 89.3 | - | 95.8 | - | 89.0 | 40.4 | 83.8 | 39.3 |
| GPT-3.5-turbo | 1.1 | 26.7 | 25.0 | - | 1.2 | - | 0.0 | 39.4 | 0.0 | 54.1 |
| GPT-4-turbo | 51.7 | 40.0 | 50.0 | - | 47.9 | - | 67.6 | 44.7 | 64.9 | 52.5 |
| GPT-4o | 71.9 | 68.9 | 78.6 | - | 63.5 | - | 66.9 | 64.9 | 68.9 | 68.9 |
| Llama 3 8b | 27.0 | 35.6 | 0.0 | - | 26.4 | - | 37.9 | 27.7 | 31.1 | 26.2 |
| Llama 3 70b | 10.1 | 68.9 | 3.6 | - | 6.6 | - | 15.2 | 77.7 | 18.9 | 85.2 |
| Mistral Large | 15.7 | 73.3 | 7.1 | - | 14.4 | - | 17.9 | 76.6 | 8.1 | 80.3 |
| Mixtral 8x22B MoE | 36.5 | 57.8 | 50.0 | - | 28.1 | - | 44.1 | 52.1 | 43.2 | 57.4 |
| o1-mini | 63.5 | 77.8 | 82.1 | - | 59.3 | - | 75.9 | 77.7 | 71.6 | 75.4 |
| o1-preview | 69.1 | 86.7 | 100.0 | - | 67.1 | - | 76.6 | 89.4 | 78.4 | 90.2 |
E.4.1 Logic Form Accuracy
Accuracy varies across object states, affordances, orientation, spatial relations, and non-spatial relations. Complex possession and container conditions are difficult to predict exactly. Claude-3.5 Sonnet performs strongly in several categories, while o1-preview is notably strong on BEHAVIOR non-spatial relations. Some narrative category-leader statements conflict with updated tables; they do not establish architectural or training causes for performance differences.
E.4.2 Planner Success Rate Analysis
Planner success measures whether the predicted operators collectively admit a solution. Simplified operators may score well despite missing real-world prerequisites, so high success can reflect internal consistency rather than faithful dynamics. The reported 100% o1-preview VirtualHome orientation result and 90.2% BEHAVIOR non-spatial result must be interpreted in this sense.
E.4.3 Error categorization
Missing predicates omit necessary conditions; extra predicates may overconstrain actions, put effects on the wrong object, or introduce unsupported properties. A detailed plug-in example shows that an entirely predicted domain can remain mutually consistent while mixing one predicted operator into reference definitions breaks planning. Thus operator replacement tests both individual correctness and compatibility with the surrounding domain. The three full planning examples preserve the exact differences.
(:action plug_in
:parameters (?char - character ?obj - object)
:precondition (or (and (next_to ?char ?obj) (has_plug ?obj)
(plugged_out ?obj)) (and (next_to ?char ?obj)
(has_switch ?obj) (plugged_out ?obj)))
:effect (and (plugged_in ?obj) (not (plugged_out ?obj)))
)
(:action walk_towards
:parameters (?char - character ?obj - object)
:precondition (and (not (sitting ?char)) (not (lying ?char))
(next_to ?char ?obj) (forall (?far_obj - object)
(when (not (obj_next_to ?far_obj ?obj))
(not (next_to ?char ?far_obj)))) (forall (?close_obj - object)
(when (obj_next_to ?close_obj ?obj)
(next_to ?char ?close_obj))))
:effect (and (next_to ?char ?obj))
)
(:action switch_on
:parameters (?char - character ?obj - object)
:precondition (and (has_switch ?obj) (off ?obj)
(plugged_in ?obj) (next_to ?char ?obj))
:effect (and (on ?obj) (not (off ?obj)))
)
(:action plug_in
:parameters (?char - character ?obj - object)
:precondition (and (has_plug ?obj) (plugged_out ?obj)
(next_to ?char ?obj))
:effect (and (plugged_in ?obj) (not (plugged_out ?obj)))
)
(:action walk_towards
:parameters (?char - character ?obj - object)
:precondition (and (inside ?char ?room) (inside ?obj ?room))
:effect (next_to ?char ?obj)
)
(:action switch_on
:parameters (?char - character ?obj - object)
:precondition (and (has_switch ?obj) (off ?obj)
(next_to ?char ?obj))
:effect (and (on ?obj) (not (off ?obj)))
)
(:action plug_in
:parameters (?char - character ?obj - object)
:precondition (and (has_plug ?obj) (plugged_out ?obj)
(next_to ?char ?obj))
:effect (and (plugged_in ?obj) (not (plugged_out ?obj)))
)
(:action walk_towards
:parameters (?char - character ?obj - object)
:precondition (and (not (sitting ?char)) (not (lying ?char))
(next_to ?char ?obj) (forall (?far_obj - object)
(when (not (obj_next_to ?far_obj ?obj))
(not (next_to ?char ?far_obj)))) (forall (?close_obj - object)
(when (obj_next_to ?close_obj ?obj)
(next_to ?char ?close_obj))))
:effect (and (next_to ?char ?obj))
)
(:action switch_on
:parameters (?char - character ?obj - object)
:precondition (and (has_switch ?obj) (off ?obj)
(plugged_in ?obj) (next_to ?char ?obj))
:effect (and (on ?obj) (not (off ?obj)))
)
Specialized Performance: Models such as o1-preview and Claude-3 Opus excel in specific categories like object states and object orientation, respectively, suggesting that targeted training or specialized architectures enhance LLM capabilities in understanding different types of tasks in transition modeling.
(2)Difficulty with Non-Spatial Relations: Across various models, non-spatial relations consistently pose a challenge, highlighting a gap in the ability of LLMs to grasp complex relational dynamics that are crucial for realistic scenario modeling.
(3)Consistency in Action Space: The effectiveness of planning relies heavily on the consistency of the predicted action space by LLMs; discrepancies between mixed predicted and ground truth actions lead to reduced planner success, emphasizing the need for coherent action definitions in LLM outputs.
E.5 Correlection with Action Length and Goal Complexity
For GPT-4o action sequencing in BEHAVIOR, success generally falls with more goals and longer reference plans. Tool-dependent actions such as soaking, cleaning, and slicing are difficult. The zero slicing success is based on only two cases, so it is not a precise estimate of general slicing competence. These are associations within this task set, not controlled evidence that one complexity factor alone causes failure.
Task complexity adversely affects goal success rates, including the number of total goals, state goals, relation goals, and ground-truth action sequence length.
2.Complexity of action preconditions adversely affects success rates, especially for actions requiring tool usage, e.g., Soak, Slice, and Clean.
3.Actions performed with the right hand have a slightly higher success rate than those with the left hand.
Appendix F Sensitivity Analysis
F.1 Motivation and Problem Formulation
Sensitivity analysis replaces one reference operator with its predicted counterpart and asks whether the reference planning problem remains solvable. This localizes disruptive definitions that an all-predicted, mutually consistent domain can conceal.
F.2 Implementation Details
For every task-relevant operator, the evaluator changes only that definition, runs a PDDL planner, and aggregates results by object/relational task category.
F.3 Result Analysis
VirtualHome plug_in and spatial movement definitions are particularly sensitive; plug_in must handle the benchmark’s switch-related cases as well as explicit plugs. BEHAVIOR slicing and some possession/placement definitions also disrupt planning. All ten sensitivity charts are retained. These tests assess symbolic domain compatibility and the selected task distribution; the narrative’s references to motor precision or deficiencies in training data are not directly tested by this experiment.
Action-Specific Sensitivities: The analysis reveals significant variations in the success rates for specific actions across tasks, with actions like "plug_in" and "walk_towards" consistently showing low success due to complex preconditions and spatial requirements. This underscores the need for LLMs to better understand and generate actions with multi-faceted constraints.
(2)Challenges with Complex Interactions: Complex interactions involving detailed object manipulation, such as "slice_carvingknife" and "place_inside," present notable challenges, highlighting deficiencies in the model’s training on precise movements and tool use. Enhancing data representation for such scenarios in training sets is crucial for improving performance.
(3)Robustness in Basic Tasks vs. Nuanced Deficiencies: While the model performs well in straightforward tasks, it struggles with nuanced actions that require fine motor skills or advanced spatial awareness. This contrast points to the necessity of incorporating more complex and varied interaction scenarios in training to refine the model’s capabilities in handling real-world complexities.
(4)Training and Fine-Tuning Needs: The consistent issues across different types of tasks suggest that the current training regimens may not fully capture the diversity of real-world interactions, especially in spatial and object-oriented tasks. A targeted approach to fine-tuning the model with specialized datasets that focus on underperforming actions could lead to significant improvements in model robustness and applicability in practical settings.
Appendix G Pipeline-Based vs Modularized
G.1 Motivation and Problem Formulation
Two-stage pipelines test the more realistic input condition in which the system starts with a natural-language instruction rather than a reference symbolic goal.
G.2 Implementation Details
In BEHAVIOR, predicted goal interpretation feeds either action sequencing or subgoal decomposition. These are compared with the corresponding isolated modules supplied with reference goals.
G.3 Result Analysis
Executability can remain similar while full goal success falls, because a downstream module can produce a feasible plan for the wrong interpreted goal. For example, the appendix summary reports GPT-4o decomposition task success falling from 48% to 38% under composition. This supports separate measurement of goal correctness and action feasibility rather than assuming one implies the other.
Similar Trajectory Execution Rates: Both modularized and pipeline-based methods have similar trajectory executable rates for pipelines and .
(2)Error Accumulation in Pipelines: Pipeline-based methods suffer from error accumulation, particularly due to the composition of two modules, which impacts their performance negatively.
(3)Grammar Error Trends: State-of-the-art (SOTA) LLMs generally avoid grammar errors for both pipeline-based and modularized methods, unlike less advanced models such as Gemini 1.0 Pro, GPT-3.5-turbo, and some open-sourced LLMs.
(4)Runtime Errors Persist: All LLMs, regardless of their level of advancement, are prone to runtime errors, missing necessary steps in their generation process.
| Model | Goal Evaluation | Trajectory Evaluation | ||||||||||||||||
| Task SR | Execution SR | Grammar Error () | Runtime Error () | |||||||||||||||
| Parsing | Hallucination | Predicate-Arg Num | Wrong Order | Missing Step | Affordance | Additional Step | ||||||||||||
| M | P | M | P | M | P | M | P | M | P | M | P | M | P | M | P | M | P | |
| Goal Interpretation + Action Sequencing | ||||||||||||||||||
| Claude-3 Haiku | 26.0 | 21.0 | 32.0 | 29.0 | 0.0 | 0.0 | 6.0 | 6.0 | 0.0 | 0.0 | 7.0 | 6.0 | 54.0 | 52.0 | 1.0 | 7.0 | 1.0 | 17.0 |
| Claude-3 Sonnet | 44.0 | 41.0 | 57.0 | 53.0 | 0.0 | 0.0 | 1.0 | 3.0 | 0.0 | 0.0 | 11.0 | 14.0 | 19.0 | 21.0 | 11.0 | 9.0 | 2.0 | 12.0 |
| Claude-3 Opus | 51.0 | 46.0 | 59.0 | 54.0 | 0.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 3.0 | 6.0 | 35.0 | 35.0 | 3.0 | 3.0 | 2.0 | 4.0 |
| Gemini 1.0 Pro | 27.0 | 26.0 | 32.0 | 35.0 | 7.0 | 5.0 | 3.0 | 3.0 | 6.0 | 6.0 | 13.0 | 14.0 | 35.0 | 38.0 | 4.0 | 2.0 | 4.0 | 11.0 |
| Gemini 1.5 Flash | 40.0 | 35.0 | 52.0 | 49.0 | 0.0 | 0.0 | 0.0 | 2.0 | 0.0 | 0.0 | 5.0 | 10.0 | 42.0 | 41.0 | 1.0 | 0.0 | 2.0 | 7.0 |
| Gemini 1.5 Pro | 42.0 | 37.0 | 54.0 | 55.0 | 0.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 6.0 | 7.0 | 39.0 | 35.0 | 1.0 | 1.0 | 2.0 | 0.0 |
| GPT-3.5-turbo | 16.0 | 14.0 | 20.0 | 32.0 | 4.0 | 1.0 | 7.0 | 3.0 | 23.0 | 15.0 | 1.0 | 5.0 | 36.0 | 39.0 | 8.0 | 6.0 | 1.0 | 3.0 |
| GPT-4-turbo | 38.0 | 32.0 | 45.0 | 47.0 | 0.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 7.0 | 9.0 | 47.0 | 41.0 | 1.0 | 1.0 | 0.0 | 0.0 |
| GPT-4o | 47.0 | 42.0 | 53.0 | 55.0 | 0.0 | 0.0 | 1.0 | 3.0 | 0.0 | 0.0 | 9.0 | 6.0 | 36.0 | 35.0 | 1.0 | 1.0 | 0.0 | 4.0 |
| Cohere Command R | 16.0 | 5.0 | 19.0 | 9.0 | 5.0 | 3.0 | 13.0 | 38.0 | 0.0 | 1.0 | 8.0 | 8.0 | 43.0 | 31.0 | 12.0 | 12.0 | 4.0 | 8.0 |
| Cohere Command R+ | 27.0 | 15.0 | 35.0 | 29.0 | 0.0 | 0.0 | 1.0 | 8.0 | 15.0 | 14.0 | 10.0 | 30.0 | 39.0 | 31.0 | 0.0 | 2.0 | 15.0 | 22.0 |
| Mistral Large | 33.0 | 31.0 | 50.0 | 38.0 | 0.0 | 0.0 | 0.0 | 3.0 | 0.0 | 0.0 | 8.0 | 14.0 | 35.0 | 37.0 | 6.0 | 8.0 | 7.0 | 5.0 |
| Mixtral 8x22B MoE | 30.0 | 26.0 | 40.0 | 36.0 | 3.0 | 3.0 | 6.0 | 13.0 | 0.0 | 0.0 | 10.0 | 14.0 | 32.0 | 21.0 | 9.0 | 13.0 | 2.0 | 15.0 |
| Llama 3 8B | 10.0 | 0.0 | 16.0 | 5.0 | 0.0 | 2.0 | 15.0 | 25.0 | 9.0 | 6.0 | 6.0 | 11.0 | 44.0 | 34.0 | 9.0 | 17.0 | 5.0 | 14.0 |
| Llama 3 70B | 34.0 | 26.0 | 42.0 | 40.0 | 0.0 | 1.0 | 2.0 | 3.0 | 0.0 | 0.0 | 15.0 | 18.0 | 38.0 | 35.0 | 3.0 | 5.0 | 6.0 | 9.0 |
| Goal Interpretation + Subgoal Decomposition | ||||||||||||||||||
| Claude-3 Haiku | 29.0 | 21.0 | 35.0 | 40.0 | 0.0 | 0.0 | 1.0 | 5.0 | 0.0 | 0.0 | 2.0 | 2.0 | 59.0 | 46.0 | 3.0 | 7.0 | 3.0 | 16.0 |
| Claude-3 Sonnet | 38.0 | 31.0 | 43.0 | 45.0 | 0.0 | 0.0 | 2.0 | 3.0 | 0.0 | 0.0 | 3.0 | 2.0 | 51.0 | 47.0 | 1.0 | 3.0 | 3.0 | 18.0 |
| Claude-3 Opus | 39.0 | 35.0 | 47.0 | 45.0 | 0.0 | 0.0 | 3.0 | 8.0 | 0.0 | 0.0 | 5.0 | 4.0 | 45.0 | 42.0 | 0.0 | 1.0 | 5.0 | 7.0 |
| Gemini 1.0 Pro | 23.0 | 14.0 | 33.0 | 30.0 | 2.0 | 0.0 | 4.0 | 10.0 | 0.0 | 1.0 | 3.0 | 1.0 | 51.0 | 45.0 | 7.0 | 13.0 | 3.0 | 17.0 |
| Gemini 1.5 Flash | 34.0 | 32.0 | 42.0 | 44.0 | 2.0 | 1.0 | 1.0 | 3.0 | 0.0 | 0.0 | 2.0 | 2.0 | 53.0 | 48.0 | 0.0 | 2.0 | 3.0 | 7.0 |
| Gemini 1.5 Pro | 31.0 | 26.0 | 37.0 | 38.0 | 0.0 | 1.0 | 1.0 | 3.0 | 0.0 | 0.0 | 3.0 | 2.0 | 59.0 | 56.0 | 0.0 | 0.0 | 2.0 | 1.0 |
| GPT-3.5-turbo | 24.0 | 14.0 | 36.0 | 27.0 | 2.0 | 0.0 | 3.0 | 12.0 | 0.0 | 22.0 | 3.0 | 1.0 | 52.0 | 32.0 | 4.0 | 6.0 | 3.0 | 5.0 |
| GPT-4-turbo | 37.0 | 37.0 | 47.0 | 49.0 | 0.0 | 0.0 | 3.0 | 4.0 | 0.0 | 0.0 | 9.0 | 8.0 | 40.0 | 37.0 | 1.0 | 2.0 | 6.0 | 6.0 |
| GPT-4o | 48.0 | 38.0 | 55.0 | 52.0 | 0.0 | 0.0 | 3.0 | 4.0 | 0.0 | 0.0 | 5.0 | 6.0 | 37.0 | 35.0 | 0.0 | 3.0 | 5.0 | 9.0 |
| Cohere Command R | 15.0 | 8.0 | 25.0 | 15.0 | 21.0 | 13.0 | 11.0 | 32.0 | 0.0 | 1.0 | 0.0 | 1.0 | 38.0 | 32.0 | 4.0 | 6.0 | 4.0 | 12.0 |
| Cohere Command R+ | 24.0 | 17.0 | 37.0 | 31.0 | 2.0 | 6.0 | 4.0 | 10.0 | 0.0 | 2.0 | 5.0 | 7.0 | 51.0 | 40.0 | 1.0 | 4.0 | 6.0 | 14.0 |
| Mistral Large | 30.0 | 22.0 | 38.0 | 29.0 | 1.0 | 1.0 | 3.0 | 12.0 | 0.0 | 1.0 | 4.0 | 5.0 | 52.0 | 50.0 | 2.0 | 2.0 | 1.0 | 5.0 |
| Mixtral 8x22B MoE | 27.0 | 22.0 | 33.0 | 29.0 | 0.0 | 0.0 | 4.0 | 9.0 | 0.0 | 2.0 | 2.0 | 2.0 | 59.0 | 45.0 | 2.0 | 13.0 | 0.0 | 17.0 |
| Llama 3 8B | 21.0 | 3.0 | 29.0 | 14.0 | 2.0 | 7.0 | 11.0 | 29.0 | 0.0 | 2.0 | 6.0 | 3.0 | 44.0 | 30.0 | 8.0 | 15.0 | 7.0 | 7.0 |
| Llama 3 70B | 20.0 | 19.0 | 30.0 | 31.0 | 1.0 | 1.0 | 5.0 | 22.0 | 1.0 | 1.0 | 8.0 | 7.0 | 51.0 | 35.0 | 4.0 | 3.0 | 4.0 | 7.0 |
Appendix H Replanning and Feedback
H.1 Motivation and Problem Formulation
Replanning tests whether a model can use explicit simulator feedback after a failed initial plan.
H.2 Implementation Details
GPT-4o receives its previous plan, categorized execution errors or unmet goal conditions, and the accumulated retry history. At most three replans are allowed. This supplies detailed symbolic feedback, including which requirements remain unsatisfied.
H.3 Result Analysis
In BEHAVIOR action sequencing, the reported task/execution success improves from 47%/53% to 59%/63%. Redundant actions can become more frequent as the model edits prior plans. The gain includes additional attempts and feedback, so it is not directly comparable to a single-call budget.
Enhanced Adaptability through Replanning: Incorporating the ability for models to replan based on feedback significantly improves their performance, demonstrating over a 10% increase in success rates. This shows that LLMs can effectively learn from their errors and adjust their action sequences to better align with the required task outcomes, enhancing overall adaptability. Replanning also helps to reduce the repetition of similar errors, showcasing the model’s ability to evolve its strategy over successive iterations.
(2)Risk of Over-Correction: While replanning generally leads to improvements in task execution, it can sometimes result in the over-generation of actions, as indicated by an increased rate of additional step errors. This suggests that while the model is adept at adjusting to feedback, it may also benefit from more precise guidance to avoid introducing unnecessary actions in its revised plans.
H.4 Replanning with Stochastic Actions
A further BEHAVIOR experiment injects stochastic action failures and allows the same three retries. Replanning helps across the tested failure probabilities, with a generally larger gap as failures increase. The complete table preserves the tested probabilities and scores; this is simulated action unreliability rather than measured hardware failure.
Appendix I Prompt and Analysis
I.1 Prompt of Goal Interpretation
The full goal-interpretation template is preserved, including the expected structured goal output and supplied environment vocabulary.
# Behavior Goal Interpretation Prompt Template
SYSTEM_PROMPT = "For this task, please only output a parsable json string inside brackets. Please start your answer with { and end your answer with }. Don't include any notes or explanations with the output json string."
USER_PROMPT = "You are a helpful assistant for goal interpretation in an embodied environment. You should only output in json format. Your task is to understand natural language goals for a household robot, reason about the object states and relationships, and turn natural language goals into symbolic goal states in the designated format. The goals include: unary goals describing one object's own unary states, and binary goals describing object-object binary relationships. The input will be the goal's name, the goal's description, relevant objects as well as their possible unary states, and all initial unary and binary states. The output should be the symbolic version of the goal states.
Relevant objects in the scene indicates those objects involved in the action execution initially. It will include the object name, and the object's all possible unary states (In goal conditions, each state can be set to true or not true). It follows the format: object name including object id, possible unary states: ...(all possible unary states). Your proposed unary object states should be within the following set: {'Cooked', 'Open', 'Frozen', 'Dusty', 'Stained', 'Sliced', 'Soaked', 'Toggled_On'}
Relevant objects in the scene are:
<object_in_scene>
All initial states in the scene are:
<all_initial_states>
Symbolic goals format:
Node goal states should be a set indicating the desired final goal states of single objects. Each goal in the list should be a list with two elements: the first element is the state name, which comes from the set {'Cooked', 'Open', 'Frozen', 'Dusty', 'Stained', 'Sliced', 'Soaked', 'Toggled_On'}; the second element is the object name, which comes from the list of relevant objects in the scene provided above.
Edge goal states should be a set indicating the desired binary relationships between two objects. Each goal state in the set is a list of three elements: the first element is the state name, which comes from the set {'NextTo', 'Inside', 'OnFloor', 'Touching', 'Under'}, the second and third elements are the object names, with relationship as indicated by the first element.
Here is an example of desired node and edge goal states:
<in_context_example>
Task Name and Goal Instructions:
<instructions_str>
Now using json format, output just the symbolic version of the goal states without any explanation. Output a single json object string, whose keys are 'node goals' and 'edge goals', and values are your output of symbolic node goals and symbolic edge goals, respectively. That is, your output should be of the format: {'node goals': SYMBOLIC_NODE_GOALS, 'edge goals': SYMBOLIC_EDGE_GOALS}. Also, please strictly follow the aforementioned symbolic goal format."
I.2 Prompt of Subgoal Decomposition
The full decomposition template is preserved, including its declarative subgoal syntax and ordering requirements.
# Behavior Subgoal Decomposition Prompt Template
SYSTEM_PROMPT = '''# Background Introduction
You are determining the complete state transitions of a household task solving by a robot. The goal is to list all intermediate states (which is called subgoals as a whole) to achieve the final goal states from the initial states. The output consists of a list of boolean expression, which is a combination of the state predicates. Note that your output boolean expression list is in temporal order, therefore, it must be consistent and logical. In short, your task is to output the subgoal plan in the required format.
# Data Vocabulary Introduction
Below we introduce the detailed data vocabulary and their format that you can use to generate the subgoal plan.
## Available States
The state is represented as a first-order predicate, which is a tuple of a predicate name and its arguments. Its formal definition looks like this "<PredicateName>(Params)", where <PredicateName> is the state name and each param should be ended with an id. An example is "inside(coin.1, jar.1)". Below is a list of available states and their descriptions.
<available_behavior_states>
## Available Connectives
The connectives are used to satisfy the complex conditions. They are used to combine the state predicates.
<available_behavior_connectives>
# Rules You Must Follow
- The initial states are the states that are given at the beginning of the task.
- Your output must be a list of boolean expressions that are in temporal order.
- You must follow the data format and the available states and connectives defined above.
- The output must be consistent, logical and as detailed as possible. View your output as a complete state transition from the initial states to the final goal states.
- Please note that the robot can only hold one object in one hand. Also, the robot needs to have at least one hand free to perform any action other than put, place, take, hold.
- Use holds_rh and holds_lh in your plan if necessary. For example, stained(shoe) cannot directly change to not stained(shoe), but needs intermediate states like [soaked(rag), holds_rh(rag), not stained(shoe)] or [holds_rh(detergent), not stained(shoe)].
- Your output follows the temporal order line by line. If you think there is no temporal order requirement for certain states, you can use connective "and" to combine them. If you think some states are equivalent, you can use connective "or" to combine them.
- Please use provided relevant objects well to help you achieve the final goal states. Note that inside(obj1, agent) is an invalid state, therefore you cannot output it in your plan.
- Do not output redundant states. A redundant state means a state that is either not necessary or has been satisfied before without broken.
- Your must strictly follow the json format like this {"output": [<your subgoal plan>]}, where <your subgoal plan> is a list of boolean expressions presented in the temporal order.
- Start your output with "{" and end with "}". For each line of the output, DO NOT INCLUDE IRRELEVANT INFORMATION (like number of line, explanation, etc.).
# Example: Task is bottling_fruit
Below we provide an example for your better understanding.
<in_context_example>'''
USER_PROMPT = '''Now, it is time for you to generate the subgoal plan for the following task.
# Target Task: <task_name>
## Relevant objects in this scene
<relevant_objects>
## Initial States
<initial_states>
## Goal States
<goal_states>
## Output: Based on initial states in this task, achieve final goal states logically and reasonably. It does not matter which state should be satisfied first, as long as all goal states can be satisfied at the end. Make sure your output follows the json format, and do not include irrelevant information, do not include any explanation. Output concrete states and do not use quantifiers like "forall" or "exists".'''
I.3 Prompt of Action Sequencing
The full action-sequencing template is preserved, including action names, argument conventions, and structured output instructions.
The reproduced prompt contains several left/right-hand inconsistencies in action descriptions. These are preserved as published and are relevant when interpreting instruction-following failures.
SYSTEM_PROMPT = """For this task, please only output a parsable json string inside brackets. Please start your answer with [ and end your answer with ]. Don't include any notes or explanations with the output json string."""
USER_PROMPT="""
Problem:
You are designing instructions for a household robot.
The goal is to guide the robot to modify its environment from an initial state to a desired final state.
The input will be the initial environment state, the target environment state, the objects you can interact with in the environment.
The output should be a list of action commands so that after the robot executes the action commands sequentially, the environment will change from the initial state to the target state.
Data format: After # is the explanation.
Format of the states:
The environment state is a list starts with a uniary predicate or a binary prediate, followed by one or two obejcts.
You will be provided with multiple environment states as the initial state and the target state.
For example:
['inside', 'strawberry_0', 'fridge_97'] #strawberry_0 is inside fridge_97
['not', 'sliced', 'peach_0'] #peach_0 is not sliced
['ontop', 'jar_1', 'countertop_84'] #jar_1 is on top of countertop_84
Format of the action commands:
Action commands is a dictionary with the following format:
{{
'action': 'action_name',
'object': 'target_obj_name',
}}
or
{{
'action': 'action_name',
'object': 'target_obj_name1,target_obj_name2',
}}
The action_name must be one of the following:
LEFT_GRASP # the robot grasps the object with its left hand, to execute the action, the robot's left hand must be empty, e.g. {{'action': 'LEFT_GRASP', 'object': 'apple_0'}}.
RIGHT_GRASP # the robot grasps the object with its right hand, to execute the action, the robot's right hand must be empty, e.g. {{'action': 'RIGHT_GRASP', 'object': 'apple_0'}}.
LEFT_PLACE_ONTOP # the robot places the object in its left hand on top of the target object and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_ONTOP', 'object': 'table_1'}}.
RIGHT_PLACE_ONTOP # the robot places the object in its right hand on top of the target object and release the object in its left hand, e.g. {{'action': 'RIGHT_PLACE_ONTOP', 'object': 'table_1'}}.
LEFT_PLACE_INSIDE # the robot places the object in its left hand inside the target object and release the object in its left hand, to execute the action, the robot's left hand must hold an object, and the target object can't be closed e.g. {{'action': 'LEFT_PLACE_INSIDE', 'object': 'fridge_1'}}.
RIGHT_PLACE_INSIDE # the robot places the object in its right hand inside the target object and release the object in its left hand, to execute the action, the robot's right hand must hold an object, and the target object can't be closed, e.g. {{'action': 'RIGHT_PLACE_INSIDE', 'object': 'fridge_1'}}.
RIGHT_RELEASE # the robot directly releases the object in its right hand, to execute the action, the robot's left hand must hold an object, e.g. {{'action': 'RIGHT_RELEASE', 'object': 'apple_0'}}.
LEFT_RELEASE # the robot directly releases the object in its left hand, to execute the action, the robot's right hand must hold an object, e.g. {{'action': 'LEFT_RELEASE', 'object': 'apple_0'}}.
OPEN # the robot opens the target object, to execute the action, the target object should be openable and closed, also, toggle off the target object first if want to open it, e.g. {{'action': 'OPEN', 'object': 'fridge_1'}}.
CLOSE # the robot closes the target object, to execute the action, the target object should be openable and open, e.g. {{'action': 'CLOSE', 'object': 'fridge_1'}}.
COOK # the robot cooks the target object, to execute the action, the target object should be put in a pan, e.g. {{'action': 'COOK', 'object': 'apple_0'}}.
CLEAN # the robot cleans the target object, to execute the action, the robot should have a cleaning tool such as rag, the cleaning tool should be soaked if possible, or the target object should be put into a toggled on cleaner like a sink or a dishwasher, e.g. {{'action': 'CLEAN', 'object': 'window_0'}}.
FREEZE # the robot freezes the target object e.g. {{'action': 'FREEZE', 'object': 'apple_0'}}.
UNFREEZE # the robot unfreezes the target object, e.g. {{'action': 'UNFREEZE', 'object': 'apple_0'}}.
SLICE # the robot slices the target object, to execute the action, the robot should have a knife in hand, e.g. {{'action': 'SLICE', 'object': 'apple_0'}}.
SOAK # the robot soaks the target object, to execute the action, the target object must be put in a toggled on sink, e.g. {{'action': 'SOAK', 'object': 'rag_0'}}.
DRY # the robot dries the target object, e.g. {{'action': 'DRY', 'object': 'rag_0'}}.
TOGGLE_ON # the robot toggles on the target object, to execute the action, the target object must be closed if the target object is openable and open e.g. {{'action': 'TOGGLE_ON', 'object': 'light_0'}}.
TOGGLE_OFF # the robot toggles off the target object, e.g. {{'action': 'TOGGLE_OFF', 'object': 'light_0'}}.
LEFT_PLACE_NEXTTO # the robot places the object in its left hand next to the target object and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_NEXTTO', 'object': 'table_1'}}.
RIGHT_PLACE_NEXTTO # the robot places the object in its right hand next to the target object and release the object in its right hand, e.g. {{'action': 'RIGHT_PLACE_NEXTTO', 'object': 'table_1'}}.
LEFT_TRANSFER_CONTENTS_INSIDE # the robot transfers the contents in the object in its left hand inside the target object, e.g. {{'action': 'LEFT_TRANSFER_CONTENTS_INSIDE', 'object': 'bow_1'}}.
RIGHT_TRANSFER_CONTENTS_INSIDE # the robot transfers the contents in the object in its right hand inside the target object, e.g. {{'action': 'RIGHT_TRANSFER_CONTENTS_INSIDE', 'object': 'bow_1'}}.
LEFT_TRANSFER_CONTENTS_ONTOP # the robot transfers the contents in the object in its left hand on top of the target object, e.g. {{'action': 'LEFT_TRANSFER_CONTENTS_ONTOP', 'object': 'table_1'}}.
RIGHT_TRANSFER_CONTENTS_ONTOP # the robot transfers the contents in the object in its right hand on top of the target object, e.g. {{'action': 'RIGHT_TRANSFER_CONTENTS_ONTOP', 'object': 'table_1'}}.
LEFT_PLACE_NEXTTO_ONTOP # the robot places the object in its left hand next to target object 1 and on top of the target object 2 and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_NEXTTO_ONTOP', 'object': 'window_0, table_1'}}.
RIGHT_PLACE_NEXTTO_ONTOP # the robot places the object in its right hand next to object 1 and on top of the target object 2 and release the object in its right hand, e.g. {{'action': 'RIGHT_PLACE_NEXTTO_ONTOP', 'object': 'window_0, table_1'}}.
LEFT_PLACE_UNDER # the robot places the object in its left hand under the target object and release the object in its left hand, e.g. {{'action': 'LEFT_PLACE_UNDER', 'object': 'table_1'}}.
RIGHT_PLACE_UNDER # the robot places the object in its right hand under the target object and release the object in its right hand, e.g. {{'action': 'RIGHT_PLACE_UNDER', 'object': 'table_1'}}.
Format of the interactable objects:
Interactable object will contain multiple lines, each line is a dictionary with the following format:
{{
'name': 'object_name',
'category': 'object_category'
}}
object_name is the name of the object, which you must use in the action command, object_category is the category of the object, which provides a hint for you in interpreting initial and goal condtions.
Please pay specail attention:
1. The robot can only hold one object in each hand.
2. Action name must be one of the above action names, and the object name must be one of the object names listed in the interactable objects.
3. All PLACE actions will release the object in the robot's hand, you don't need to explicitly RELEASE the object after the PLACE action.
4. For LEFT_PLACE_NEXTTO_ONTOP and RIGHT_PLACE_NEXTTO_ONTOP, the action command are in the format of {{'action': 'action_name', 'object': 'obj_name1, obj_name2'}}
5. If you want to perform an action to an target object, you must make sure the target object is not inside a closed object.
6. For actions like OPEN, CLOSE, SLICE, COOK, CLEAN, SOAK, DRY, FREEZE, UNFREEZE, TOGGLE_ON, TOGGLE_OFF, at least one of the robot's hands must be empty, and the target object must have the corresponding property like they're openable, toggleable, etc.
7. For PLACE actions and RELEASE actions, the robot must hold an object in the corresponding hand.
8. Before slicing an object, the robot can only interact with the object (e.g. peach_0), after slicing the object, the robot can only interact with the sliced object (e.g. peach_0_part_0).
Examples: after# is the explanation.
Example 1:
Input:
initial environment state:
['stained', 'sink_7']
['stained', 'bathtub_4']
['not', 'soaked', 'rag_0']
['onfloor', 'rag_0', 'room_floor_bathroom_0']
['inside', 'rag_0', 'cabinet_1']
['not', 'open', 'cabinet_1']
target environment state:
['not', 'stained', 'bathtub_4']
['not', 'stained', 'sink_7']
['and', 'soaked', 'rag_0', 'inside', 'rag_0', 'bucket_0']
interactable objects:
{{'name': 'sink_7', 'category': 'sink.n.01'}}
{{'name': 'bathtub_4', 'category': 'bathtub.n.01'}}
{{'name': 'bucket_0', 'category': 'bucket.n.01'}}
{{'name': 'rag_0', 'category': 'rag.n.01'}}
{{'name': 'cabinet_1', 'category': 'cabinet.n.01'}}
Please output the list of action commands (in the given format) so that after the robot executes the action commands sequentially, the current environment state will change to target environment state. Usually, the robot needs to execute multiple action commands consecutively to achieve final state. Please output multiple action commands rather than just one. Only output the list of action commands with nothing else.
Output:
[
{{
'action': 'OPEN',
'object': 'cabinet_1'
}}, # you want to get the rag_0 from cabinet_1, should open it first
{{
'action': 'RIGHT_GRASP',
'object': 'rag_0'
}}, # you want to clean the sink_7 and bathtub_4, you found them stained, so you need to soak the rag_0 first
{{
'action': 'RIGHT_PLACE_INSIDE',
'object': 'sink_7'
}}, # to soak the rag_0, you need to place it inside the sink_7
{{
'action': 'TOGGLE_ON',
'object': 'sink_7'
}}, # to soak the rag_0, you need to toggle on the sink_7
{{
'action': 'SOAK',
'object': 'rag_0'
}}, # now you can soak the rag_0
{{
'action': 'TOGGLE_OFF',
'object': 'sink_7'
}}, # after soaking the rag_0, you need to toggle off the sink_7
{{
'action': 'LEFT_GRASP',
'object': 'rag_0'
}}, # now you can grasp soaked rag_0 to clean stain
{{
'action': 'CLEAN',
'object': 'sink_7'
}}, # now you clean the sink_7
{{
'action': 'CLEAN',
'object': 'bathtub_4'
}}, # now you clean the bathtub_4
{{
'action': 'LEFT_PLACE_INSIDE',
'object': 'bucket_0'
}} # after cleaning the sink_7, you need to place the rag_0 inside the bucket_0
]
Your task:
Input:
initial environment state:
{init_state}
target environment state:
{target_state}
interactable objects:
{obj_list}
Please output the list of action commands (in the given format) so that after the robot executes the action commands sequentially, the current environment state will change to target environment state. Usually, the robot needs to execute multiple action commands consecutively to achieve final state. Please output multiple action commands rather than just one. Only output the list of action commands with nothing else.
Output:
"""
I.4 Prompt of Transition Modeling
The full transition-model template is preserved, including operator signatures and the requested precondition/effect definitions.
# Virtualhome Transition Modeling Prompt Template
SYSTEM_PROMPT = "You are a software engineer who will be writing action definitions for a household robot in the PDDL planning language given the problem file and predicates in domain file. For this task, please only output a parsable json string inside brackets. Please start your answer with { and end your answer with }. Don't include any notes or explanations with the output json string."
USER_PROMPT = "The following is predicates defined in this domain file. Pay attention to the types for each predicate.
(define (domain virtualhome)
(:requirements :typing)
;; types in virtualhome domain
(:types
object character ; Define 'object' and 'character' as types
)
;; Predicates defined on this domain. Note the types for each predicate.
(:predicates
(closed ?obj - object) ; obj is closed
(open ?obj - object) ; obj is open
(on ?obj - object) ; obj is turned on, or it is activated
(off ?obj - object) ; obj is turned off, or it is deactivated
(plugged_in ?obj - object) ; obj is plugged in
(plugged_out ?obj - object) ; obj is unplugged
(sitting ?char - character) ; char is sitting, and this represents a state of a character
(lying ?char - character) ; char is lying
(clean ?obj - object) ; obj is clean
(dirty ?obj - object) ; obj is dirty
(obj_ontop ?obj1 ?obj2 - object) ; obj1 is on top of obj2
(ontop ?char - character ?obj - object) ; char is on obj
(on_char ?obj - object ?char - character) ; obj is on char
(inside_room ?obj ?room - object) ; obj is inside room
(obj_inside ?obj1 ?obj2 - object) ; obj1 is inside obj2
(inside ?char - character ?obj - object) ; char is inside obj
(obj_next_to ?obj1 ?obj2 - object) ; obj1 is close to or next to obj2
(next_to ?char - character ?obj - object) ; char is close to or next to obj
(between ?obj1 ?obj2 ?obj3 - object) ; obj1 is between obj2 and obj3
(facing ?char - character ?obj - object) ; char is facing obj
(holds_rh ?char - character ?obj - object) ; char is holding obj with right hand
(holds_lh ?char - character ?obj - object) ; char is holding obj with left hand
(grabbable ?obj - object) ; obj can be grabbed
(cuttable ?obj - object) ; obj can be cut
(can_open ?obj - object) ; obj can be opened
(readable ?obj - object) ; obj can be read
(has_paper ?obj - object) ; obj has paper
(movable ?obj - object) ; obj is movable
(pourable ?obj - object) ; obj can be poured from
(cream ?obj - object) ; obj is cream
(has_switch ?obj - object) ; obj has a switch
(lookable ?obj - object) ; obj can be looked at
(has_plug ?obj - object) ; obj has a plug
(drinkable ?obj - object) ; obj is drinkable
(body_part ?obj - object) ; obj is a body part
(recipient ?obj - object) ; obj is a recipient
(containers ?obj - object) ; obj is a container
(cover_object ?obj - object) ; obj is a cover object
(surfaces ?obj - object) ; obj has surfaces
(sittable ?obj - object) ; obj can be sat on
(lieable ?obj - object) ; obj can be lied on
(person ?obj - object) ; obj is a person
(hangable ?obj - object) ; obj can be hanged
(clothes ?obj - object) ; obj is clothes
(eatable ?obj - object) ; obj is eatable
)
;; Actions to be predicted
)
Objective: Given the problem file of pddl, which defines objects in the task (:objects), initial conditions (:init) and goal conditions (:goal), write the body of PDDL actions (:precondition and :effect) given specific action names and parameters, so that after executing the actions in some order, the goal conditions can be reached from initial conditions.
Each PDDL action definition consists of four main components: action name, parameters, precondition, and effect. Here is the general format to follow:
(:action [action name]
:parameters ([action parameters])
:precondition ([action precondition])
:effect ([action effect])
)
The :parameters is the list of variables on which the action operates. It lists variable names and variable types.
The :precondition is a first-order logic sentence specifying preconditions for an action. The precondition consists of predicates and 4 possible logical operators: or, and, not, exists!
1. The precondition should be structured in Disjunctive Normal Form (DNF), meaning an OR of ANDs.
2. The not operator should only be used within these conjunctions. For example, (or (and (predicate1 ?x) (predicate2 ?y)) (and (predicate3 ?x)))
3. Exists operator is followed by two parts, variable and body. It follows the format: exists (?x - variable type) (predicate1 ?x), which means there exists an object ?x of certain variable type, that predicate1 ?x satisfies.
The :effect lists the changes which the action imposes on the current state. The precondition consists of predicates and 6 possible logical operators: or, and, not, exists, when, forall.
1. The effects should generally be several effects connected by AND operators.
2. For each effect, if it is a conditional effect, use WHEN to check the conditions. The semantics of (when [condition] [effect]) are as follows: If [condition] is true before the action, then [effect] occurs afterwards.
3. If it is not a conditional effect, use predicates directly.
4. The NOT operator is used to negate a predicate, signifying that the condition will not hold after the action is executed.
5. Forall operator is followed by two parts, variable and body. It follows the format: forall (?x - variable type) (predicate1 ?x), which means there for all objects ?x of certain variable type, that predicate1 ?x satisfies.
6. An example of effect is (and (when (predicate1 ?x) (not (predicate2 ?y))) (predicate3 ?x))
Formally, the preconditions and effects are all clauses <Clause>.
<Clause> := (predicate ?x)
<Clause> := (and <Clause1> <Clause2> ...)
<Clause> := (or <Clause1> <Clause2> ...)
<Clause> := (not <Clause>)
<Clause> := (when <Clause1> <Clause2>)
<Clause> := (exists (?x - object type) <Clause>)
<Clause> := (forall (?x - object type) <Clause>)
In any case, the occurrence of a predicate should agree with its declaration in terms of number and types of arguments defined in DOMAIN FILE at the beginning.
Here is an example of the input problem file and unfinished action. Observe carefully how to think step by step to write the action body of hang_up_clothes:
Input:
Problem file:
(define (problem hang-clothes-problem)
(:domain household)
(:objects
character - character
shirt - object
hanger - object
) ; This section declares the instances needed for the problem: character is an instance of a character; shirt is an instance of an object classified as clothes; hanger is an object that is suitable for hanging clothes.
(:init
(clothes shirt)
(hangable hanger)
(holds_rh alice shirt)
(next_to alice hanger)
) ; This section declares the initial conditions. (clothes shirt) and (hangable hanger) tells the properties of objects; (holds_rh alice shirt) indicates that Alice is holding the shirt in her right hand; (next_to alice hanger) means Alice is next to the hanger, ready to hang the shirt.
(:goal
(and
(ontop shirt hanger)
)
) ; This section declares the goal. (ontop shirt hanger) is the goal, where the shirt should end up hanging on the hanger.
)
Action to be finished:
(:action hang_up_clothes
:parameters (?char - character ?clothes - object ?hang_obj - object)
:precondition ()
:effect ()
)
Example output:
Given the objects in the problem file, and what typically needs to be true to perform an action like hanging up clothes: 1. clothes must indeed be a type of clothing. 2. hang_obj should be something on which clothes can be hung (hangable). 3. char should be holding the clothes, either in the right or left hand. 4. char needs to be next to the hanging object to hang the clothes. Besides, we need to write preconditions in Disjunctive Normal Form.
These insights guide us to write:
:precondition (or
(and
(clothes ?clothes) ; the object must be a piece of clothing
(hangable ?hang_obj) ; the target must be an object suitable for hanging clothes
(holds_rh ?char ?clothes) ; character is holding clothes in the right hand
(next_to ?char ?hang_obj) ; character is next to the hanging object
)
(and
(clothes ?clothes) ; the object must be a piece of clothing
(hangable ?hang_obj) ; the target must be an object suitable for hanging clothes
(holds_lh ?char ?clothes) ; character is holding clothes in the left hand
(next_to ?char ?hang_obj) ; character is next to the hanging object
)
)
Effects describe how the world state changes due to the action. After hanging up clothes, you'd expect: 1. char is no longer holding the clothes. 2. clothes is now on the hang_obj.
These expectations convert into effects:
:effect (and
(when (holds_rh ?char ?clothes)(not (holds_rh ?char ?clothes))) ; if clothes are held in the right hand, they are no longer held
(when (holds_lh ?char ?clothes)(not (holds_lh ?char ?clothes))) ; if clothes are held in the left hand, they are no longer held
(ontop ?clothes ?hang_obj) ; clothes are now hanging on the object
)
Combining these parts, the complete hang_up_clothes action becomes:
(:action hang_up_clothes
:parameters (?char - character ?clothes - object ?hang_obj - object)
:precondition (or
(and
(clothes ?clothes)
(hangable ?hang_obj)
(holds_rh ?char ?clothes)
(next_to ?char ?hang_obj)
)
(and
(clothes ?clothes)
(hangable ?hang_obj)
(holds_lh ?char ?clothes)
(next_to ?char ?hang_obj)
)
)
:effect (and
(when (holds_rh ?char ?clothes)(not (holds_rh ?char ?clothes)))
(when (holds_lh ?char ?clothes)(not (holds_lh ?char ?clothes)))
(ontop ?clothes ?hang_obj)
)
)
Above is a good example of given predicates in domain file, problem file, action names and parameters, how to reason step by step and write the action body in PDDL. Pay attention to the usage of different connectives and their underlying logic.
Here are some other commonly used actions and their PDDL definition:
(:action put_to
:parameters (?char - character ?obj - object ?dest - object)
:precondition (or
(and
(hold_lh ?obj) ; The character should hold either with left hand or right hand
(next_to ?char ?dest) ; The character should be close to destination
)
(and
(hold_rh ?obj) ; The character should hold either with left hand or right hand
(next_to ?char ?dest) ; The character should be close to destination
)
)
:effect (obj_ontop ?obj ?dest) ; The object is now on the destination
)
This case illustrates the use of OR to include all possible preconditions of an action.
(:action pick_and_place
:parameters (?char - character ?obj - object ?dest - object)
:precondition (and
(grabbable ?obj) ; The object must be grabbable
(next_to ?char ?obj) ; The character must be next to the object
(not (obj_ontop ?obj ?dest)) ; Ensure the object is not already on the destination
)
:effect (and
(obj_ontop ?obj ?dest) ; The object is now on the destination
(next_to ?char ?dest) ; The character is now next to the destination
)
)
This case illustrates a plain case with only AND operator.
(:action bow
:parameters (?char - character ?target - character)
:precondition (and
(next_to ?char ?target) ; The character must be next to the target to perform the bow
)
:effect ()
)
This case illustrates the action can have no effect (or no precondition.)
hint:
1. Don't enforce the use of WHEN everywhere.
2. You MUST only use predicates and object types exactly as they appear in the domain file at the beginning.
3. Use and only use the arguments provided in :parameters for each action. Don't propose additional arguments, unless you are using exists or forall.
4. It is possible that action has no precondition or effect.
5. The KEY of the task is to ensure after executing your proposed actions in some order, the intial state (:init) in problem file can reach the goals (:goal)!!! Pay attention to the initial state and final goals in problem file.
6. Preconditions and effects are <Clause> defined above. When there is only one predicate, do not use logic connectives.
For actions to be finished, write their preconditions and effects, and return in standard PDDL format:
(:action [action name]
:parameters ([action parameters])
:precondition ([action precondition])
:effect ([action effect])
)
Concatenate all actions PDDL string into a single string. Output in json format where key is 'output' and value is your output string: {'output': YOUR OUTPUT STRING}
Input:
<problem_file>
<action_handlers>
Output:"
I.5 Prompt of Environment Representation
The complete environment-representation example shows how objects, initial states, and relations are supplied as text.
# Behavior Environment Prompt Template
AVAILABLE_STATES = '''The state is represented as a first-order predicate, which is a tuple of a predicate name and its arguments. Its formal definition looks like this "<PredicateName>(Params)", where <PredicateName> is the state name and each param should be ended with an id. An example is "inside(coin.1, jar.1)". Below is a list of available states and their descriptions.
| State Name | Arguments | Description |
| --- | --- | --- |
| inside | (obj1.id, obj2.id) | obj1 is inside obj2. If we have state inside(A, B), and you want to take A out of B while B is openable and stayed at "not open" state, please open B first. Also, inside(obj1, agent) is invalid.|
| ontop | (obj1.id, obj2.id) | obj1 is on top of obj2 |
| nextto | (obj1.id, obj2.id) | obj1 is next to obj2 |
| under | (obj1.id, obj2.id) | obj1 is under obj2 |
| onfloor | (obj1.id, floor2.id) | obj1 is on the floor2 |
| touching | (obj1.id, obj2.id) | obj1 is touching or next to obj2 |
| cooked | (obj1.id) | obj1 is cooked |
| burnt | (obj1.id) | obj1 is burnt |
| dusty | (obj1.id) | obj1 is dusty. If want to change dusty(obj1.id) to "not dusty(obj1.id)", there are two ways to do it, depending on task conditions. Here, all objects other than obj1 are types but not instances: 1. [inside(obj1.id, dishwasher or sink), toggledon(dishwasher or sink)] 2. holding other cleanning tool |
| frozen | (obj1.id) | obj1 is frozen |
| open | (obj1.id) | obj1 is open |
| sliced | (obj1.id) | obj1 is sliced. If want to change "not sliced(obj1.id)" to "sliced(obj1.id)", one must have a slicer. |
| soaked | (obj1.id) | obj1 is soaked |
| stained | (obj1.id) | obj1 is stained. If want to change stained(obj1.id) to "not stained(obj1.id)", there are three ways to do it, depending on task conditions. Here, all objects other than obj1 are types but not instances: 1. [inside(obj1.id, sink), toggledon(sink)] 2. [soaked(cleaner)] 3. holding detergent. |
| toggledon | (obj1.id) | obj1 is toggled on |
| holds_rh | (obj1.id) | obj1 is in the right hand of the robot |
| holds_lh | (obj1.id) | obj1 is in the left hand of the robot |'''
AVAILABLE_CONNECTIVES = '''The connectives are used to satisfy the complex conditions. They are used to combine the state predicates.
| Connective Name | Arguments | Description |
| --- | --- | --- |
| and | exp1 and exp2 | evaluates to true if both exp1 and exp2 are true |
| or | exp1 or exp2 | evaluates to true if either exp1 or exp2 is true |
| not | not exp | evaluates to true if exp is false |
| forall | forall(x, exp) | evaluates to true if exp is true for all x |
| exists | exists(x, exp) | evaluates to true if exp is true for at least one x |
| forpairs | forpairs(x, y, exp) | evaluates to true if exp is true for all pairs of x and y. For example, forpairs(watch, basket, inside(watch, basket)) means that for each watch and basket, the watch is inside the basket. |
| forn | forn(n, x, exp) | evaluates to true if exp is true for exactly n times for x. For example, forn(2, jar_n_01, (not open(jar_n_01)) means that there are exactly two jars that are not open. |
| fornpairs | fornpairs(n, x, y, exp) | evaluates to true if exp is true for exactly n times for pairs of x and y. For example, fornpairs(2, watch, basket, inside(watch, basket)) means that there are exactly two watches inside the basket. |'''
EXAMPLE_RELEVANT_OBJECTS = '''## Relevant objects in this scene
{'name': 'strawberry.0', 'category': 'strawberry_n_01'}
{'name': 'fridge.97', 'category': 'electric_refrigerator_n_01'}
{'name': 'peach.0', 'category': 'peach_n_03'}
{'name': 'countertop.84', 'category': 'countertop_n_01'}
{'name': 'jar.0', 'category': 'jar_n_01'}
{'name': 'jar.1', 'category': 'jar_n_01'}
{'name': 'carving_knife.0', 'category': 'carving_knife_n_01'}
{'name': 'bottom_cabinet_no_top.80', 'category': 'cabinet_n_01'}
{'name': 'room_floor_kitchen.0', 'category': 'floor_n_01'}'''
INITIAL_STATES = '''inside(strawberry.0, fridge.97)
inside(peach.0, fridge.97)
not sliced(strawberry.0)
not sliced(peach.0)
ontop(jar.0, countertop.84)
ontop(jar.1, countertop.84)
ontop(carving_knife.0, countertop.84)
onfloor(agent_n_01.1, room_floor_kitchen.0)'''
GOAL_STATES = '''exists(jar_n_01, (inside(strawberry.0, jar_n_01) and (not inside(peach.0, jar_n_01))))
exists(jar_n_01, (inside(peach.0, jar_n_01) and (not inside(strawberry.0, jar_n_01))))
forall(jar_n_01, (not open(jar_n_01)))
sliced(strawberry.0)
sliced(peach.0)'''
I.6 Prompt Analysis and Learned Lessons
Prompts favor structured JSON and Python-list outputs over unconstrained prose, because extra explanations and malformed strings can make plans unparsable. A common prompt is used across models. The authors’ prompting recommendations arise from qualitative development experiments, rather than a broad independently held-out prompt-optimization study.
System Prompts are preferred over user prompts for specifying output structure and overall output style as it is more strongly enforced.
•Stop Sequences are provided to prevent repetitive output and end-of-output explanations. This can improve format following and save token usage.
•Shorter Strings are encouraged to reduce grammar errors. For example, in the goal interpretation, we ask models to generate [“cond1”, “and”, “cond2”, “or”, “cond3”] instead of “cond1 and cond2 or cond3".
•Start / End Tokens are specified to discourage models from generating additional explanations and self-repetition of answers / prompt instructions.
I.7 Further Consideration about Prompt Variability
A limited prompt-variation study evaluates Claude-3 Opus, GPT-4o, and Gemini 1.5 Pro on VirtualHome goal interpretation. Improvements often transfer between models, supporting a shared prompt in that setting. The source’s statement that no model exceeds a 3.8% format-error rate is not a blanket result across every module: its BEHAVIOR decomposition discussion reports 23% parsing errors for Cohere Command R. Prompt robustness is therefore task- and model-dependent.
Appendix J Human Performance Comparison
The human comparison reports better human goal interpretation and action execution, while GPT-4o scores higher on the displayed decomposition and transition-planning measures. A perfect planner success score still refers to solvability under predicted operators. The table and brief description do not establish broad human-versus-model superiority, especially without detailed participant, training, and uncertainty information; the source also alternates GPT-4 and GPT-4o naming.
Appendix K Further Discussion on Visual Information in Our Benchmark
K.1 Integration of Visual Inputs in Long-Horizon Decision Making
Supplementary experiments examine how adding visual inputs changes the symbolic-planning setting.
K.1.1 Challenges with Existing VLMs
Complex multi-room scenes can contain more information than a single image or present visual model reliably captures. The authors distinguish this perception challenge from the decision-making abilities isolated by the main benchmark.
K.1.2 Comparative Experiments with LLMs and VLMs
The controlled comparison uses Llama 3 and a LLaVA model based on Llama 3 in BEHAVIOR.
Experiment Settings
Three inputs are compared: a scene graph for the language model, images alone for the vision-language model, and images plus a scene graph for the vision-language model.
Results and Analysis
Image-only LLaVA performs worse than scene-graph-conditioned Llama 3. Adding the graph helps LLaVA but does not close the reported gap. This demonstrates a limitation of these tested configurations; it does not establish that visual input is inherently unhelpful or that all vision-language models share the same ranking. End-to-end scores also combine perception and planning errors.
K.1.3 Why LLMs Benchmarking is Useful
Structured language-model evaluation remains relevant to modular systems that combine perception, planning, and control.
Experiment Settings
Two further settings add broader context to the scene-graph language model and to the image-plus-graph vision-language model.
Result Analysis
Additional context raises the reported Llama goal F1 from 31.5 to 31.9 and action success from 11.1 to 13.9; LLaVA changes from 25.8 to 25.9 and 11.0 to 12.3, respectively. These small improvements suggest some transfer of input-design choices, with no reported uncertainty establishing statistical significance.
K.1.4 Implications for Future Research
The authors advocate better visual/text integration and evaluations that separate perception from planning. Transfer of prompt and representation improvements remains a hypothesis to test across broader model families.
K.2 Impact of Perception and State Estimation Errors
A further experiment supplies language planners with scene graphs generated from images, focusing on subgoal decomposition.
K.2.1 Experiment Setup (Exp.5)
Different vision-language models produce BEHAVIOR scene graphs, which then become planner inputs. This directly exposes downstream decisions to perception errors.
K.2.2 Results Analysis
The retained table and examples distinguish extra or missing state predicates, incorrect spatial relations, and invented objects. Higher predicate error rates accompany lower decomposition success; Claude-3.5 Sonnet reaches the reported best 75% in this experiment. This supplementary score uses a different input/evaluation setting from the main decomposition table and should not be interchanged with it.
| VLM | Object Error Rate (%) | Predicate Error Rate (%) | Steps Generated Num | Success Rate (%) |
| Claude-3.5-Sonnet | 0.0 | 8.0 | 13 | 75.0 |
| ChatGPT-4 | 0.0 | 20.0 | 8 | 50.0 |
| LLaVA-Vicuna-13B | 0.0 | 83.0 | 6 | 12.5 |
K.3 Assumptions on Scene Graphs in Our Benchmark
Scene graphs are a practical abstraction for modular robot reasoning. In the main benchmark they are supplied perception results, so benchmark success does not establish that those graphs can be recovered reliably from real-world images.
Appendix L Dataset Statistics and Analysis
L.1 Dataset Structure
Each task instance combines a name, natural-language instruction, symbolic/LTL goal, symbolic action trajectory, and relevant transition models. BEHAVIOR also includes demonstration-video references. VirtualHome goals are annotated to exclude incidental final-state changes, making evaluation less dependent on one reference execution. Complete JSON examples for both simulators are preserved.
# Below is an example of VirtualHome
"1057_1": {
"task_name": "Watch TV",
"natural_language_description": "Go to the living room, sit on the couch, find the
remote, switch on the TV and watch",
"vh_goal": {
"actions": [
"LOOKAT|WATCH"
],
"goal": [{
"id": 410,
"class_name": "television",
"state": "ON"
}, {
"id": 410,
"class_name": "television",
"state": "PLUGGED_IN"
}, {
"from_id": 65,
"relation_type": "FACING",
"to_id": 410
}]
},
"tl_goal": "(exists x0. ((LOOKAT(x0) or WATCH(x0))) then (ON(television.410) and
PLUGGED_IN(television.410) and FACING(character.65, television.410)))",
"action_trajectory": [
"[WALK] <home_office> (319)",
"[WALK] <couch> (352)",
"[FIND] <couch> (352)",
"[SIT] <couch> (352)",
"[FIND] <remote_control> (1000)",
"[FIND] <television> (410)",
"[SWITCHON] <television> (410)",
"[TURNTO] <television> (410)",
"[WATCH] <television> (410)"
],
"transition_model": <pddl_definition>
}
# Below is an example of BEHAVIOR
"cleaning_high_chair_0_Wainscott_0_int_0_2021-06-05_18-03-15": {
"task_name": "cleaning_high_chair",
"natural_description": "Clean the high chair.",
"raw_bddl_goal": "(define (problem cleaning_high_chair_0)\n (:domain igibson)\n\n
(:objects\n highchair.n.01_1 - highchair.n.01\n
piece_of_cloth.n.01_1 - piece_of_cloth.n.01\n
cabinet.n.01_1 - cabinet.n.01\n
...",
"tl_goal": "not dusty(highchair.0)",
"action_trajectory": [
{
"action": "OPEN",
"object": "bottom_cabinet_no_top_80"
},
{
"action": "RIGHT_GRASP",
"object": "paper_towel_0"
},
{
"action": "LEFT_GRASP",
"object": "highchair_0"
},
{
"action": "CLEAN",
"object": "highchair_0"
}
],
"transition_model": <pddl_definition>,
"demo": <demo_link>
}
L.2 Data Statistics and Distribution
The statistics describe the selected benchmark subset, not every task available in the underlying simulators.
VirtualHome
VirtualHome contains 801 annotated goal conditions: 340 state, 299 relation, and 162 action goals. Reference trajectories average 8.76 actions and span 2–54; about one third exceed ten actions. Counts by action, object, and action–object pair remain available in the distribution plots.
BEHAVIOR
BEHAVIOR reference trajectories average 14.6 actions. The average 3.7 goals counts top-level BDDL expressions, which can each contain quantified requirements. It differs from the average number of grounded atomic goals.
L.3 Goal Complexity Analysis
Quantified goals can be satisfied by many object assignments. The benchmark distinguishes one valid assignment’s atomic conditions from the number of alternative valid assignments. BEHAVIOR averages 6.7 grounded conditions and 4,164.4 goal options per task; distributions, rather than these averages alone, show their variability.
L.4 Task List
BEHAVIOR selection uses tasks with human demonstrations. VirtualHome selection uses executable tasks in a chosen scene. The full task lists and action lengths are retained. This supports diverse household-task evaluation within a constrained selection, rather than a representative sample of all robot environments or proof of cross-domain deployment.
| Task Name | Task Number | Task Name | Task Number |
| Wash hands | 22 | Pet cat | 27 |
| Drink | 28 | Work | 35 |
| Turn on light | 34 | Pick up phone | 28 |
| Go to toilet | 11 | Get some water | 1 |
| Wash teeth | 11 | Read book | 28 |
| Make coffee | 10 | Listen to music | 22 |
| Wash dishes by hand | 11 | Relax on sofa | 35 |
| Wash clothes | 21 | Wash dishes with dishwasher | 27 |
| Browse internet | 31 | Put groceries in Fridge | 23 |
| Set up table | 8 | Write an email | 21 |
| Take shower | 2 | Watch TV | 32 |
| Cook some food | 7 | Change TV channel | 29 |
| Go to sleep | 9 | ||
| Brush teeth | 5 |
| Task Name | Length | Task Name | Length |
|---|---|---|---|
| assembling_gift_baskets | 32 | brushing_lint_off_clothing | 13 |
| boxing_books_up_for_storage | 16 | collecting_aluminum_cans | 12 |
| mopping_floors | 15 | preserving_food | 20 |
| re-shelving_library_books | 8 | polishing_silver | 6 |
| packing_boxes_for_household_move_or_trip | 20 | cleaning_freezer | 15 |
| installing_a_modem | 3 | cleaning_up_after_a_meal | 31 |
| putting_away_Christmas_decorations | 17 | setting_up_candles | 12 |
| cleaning_shoes | 10 | cleaning_sneakers | 30 |
| locking_every_window | 4 | washing_cars_or_other_vehicles | 8 |
| washing_dishes | 11 | putting_away_Halloween_decorations | 15 |
| collect_misplaced_items | 11 | cleaning_high_chair | 4 |
| washing_floor | 8 | sorting_mail | 12 |
| cleaning_bathrooms | 12 | cleaning_kitchen_cupboard | 18 |
| preparing_salad | 34 | putting_leftovers_away | 18 |
| preparing_a_shower_for_child | 8 | cleaning_stove | 14 |
| putting_dishes_away_after_cleaning | 17 | packing_child_s_bag | 13 |
| cleaning_windows | 13 | bottling_fruit | 21 |
| thawing_frozen_food | 20 | setting_mousetraps | 8 |
| washing_pots_and_pans | 24 | laying_wood_floors | 8 |
| putting_away_toys | 16 | cleaning_garage | 19 |
| watering_houseplants | 14 | installing_alarms | 6 |
| organizing_boxes_in_garage | 12 | putting_up_Christmas_decorations_inside | 18 |
| cleaning_bathtub | 8 | clearing_the_table_after_dinner | 14 |
| loading_the_dishwasher | 13 | filling_an_Easter_basket | 26 |
| cleaning_the_pool | 8 | opening_packages | 6 |
| packing_bags_or_suitcase | 15 | cleaning_up_refrigerator | 23 |
| making_tea | 13 | picking_up_trash | 10 |
| polishing_furniture | 4 | organizing_school_stuff | 15 |
| cleaning_cupboards | 24 | installing_a_printer | 3 |
| packing_food_for_work | 12 | storing_the_groceries | 23 |
L.5 Task Categorization
Transition predicates are grouped into object affordance, object state, orientation, spatial relation, and non-spatial relation categories. Inverse-document-frequency weights emphasize rare predicates; each program is assigned its two highest-scoring categories. Counts can therefore overlap across categories. The complete scoring equations and category-count table are preserved.
Equations
| VirtualHome | BEHAVIOR | |
| Object States | 178 | 45 |
| Object Orientation | 167 | - |
| Object Affordance | 28 | - |
| Spatial Relations | 145 | 94 |
| Non-spatial Relations | 74 | 61 |
Appendix M Annotation Details
M.1 Simulator Comparison and Selection
The authors choose VirtualHome and BEHAVIOR to emphasize long plans and complex household goals, while discussing AI2-THOR as an alternative. This survey mixes platform-wide and benchmark-specific action/scene counts and contains inconsistent descriptions of BEHAVIOR physics. The concrete benchmark vocabulary and evaluator behavior in Appendix N are the relevant implementation specification.
M.2 BEHAVIOR
BEHAVIOR annotations add action sequences and natural-language descriptions to the existing task definitions and demonstrations.
M.2.1 Annotating Action Sequence based on VR Human Demonstrations
Object-state tracking and segmentation of VR demonstrations provide candidate changes, which authors manually map to symbolic action sequences for the 100 tasks.
M.2.2 Complicated Goals with Quantifiers
Quantified BDDL conditions are grounded into atomic alternatives. Different pairings can satisfy the same task, so reference evaluation must permit these alternatives rather than insist on one arbitrary assignment.
M.2.3 Annotating Transition Models
For transition-planner evaluation, a solvable BDDL goal option is sampled and a bounded subset of its grounded conditions is selected to keep planning tractable. Relevant objects and initial conditions form the PDDL problem. An existing BEHAVIOR domain is modified in names, predicates, preconditions, and effects. Thus these planner problems may cover fewer conditions than the full original BDDL task.
M.3 VirtualHome
VirtualHome extends RobotHow with manually curated symbolic goals for 338 instructions. The source describes 30 task categories here but 26 in its statistics discussion; the retained lists expose the actual scope. Annotation separates task-essential outcomes from incidental states in demonstrations.
M.3.1 Wildcard Representations: Deriving Common Task-Related Final States
Common outcomes across instructions are expressed with wildcards and object properties, allowing interchangeable items such as different kinds of clothing or books.
M.3.2 Extracting and Annotating Abstract Goals
Annotators derive state, relation, and action goals and add missing conditions. Action goals capture required behaviors such as touching a cat even when the simulator records no lasting effect.
M.3.3 Grounding to Concrete Goals from Abstract Goals
Abstract wildcard goals are grounded into actual object identifiers in the selected scene. Multiple grounded targets can remain valid, preserving alternative task solutions.
M.3.4 Annotating Transition Models
PDDL action definitions are derived from VirtualHome’s implementation, including quantified and conditional effects. Roughly ten body-action operators have no environmental effect and can be irrelevant to state-goal planning; they are nevertheless included in logical-accuracy evaluation. This further explains why planner success alone does not cover every action requirement.
M.4 Quality Verification
Annotation verification combines simulator execution checks with human ratings of action accuracy, coverage, and preference.
M.4.1 Annotation Quality Evaluation
The complete quality table reports means and weighted mean-squared disagreement for these three attributes.
| Attribute | Mean Score | Weighted MSE |
| Action Accuracy | 3.73 | 0.4062 |
| Action Coverage | 4.07 | 1.8438 |
| Human Preference | 4.27 | 0.8125 |
Explanation of Metrics:
Ratings average above 3.7 out of 5. Per-task squared deviations from the mean are averaged with weights reflecting evaluation frequency. This measures rater agreement and perceived annotation quality; it is not an exhaustive proof that every symbolic definition is correct. The full rating distributions are preserved.
Appendix N Simulator Implementation Details
N.1 BEHAVIOR Implementation Details
BEHAVIOR has both an iGibson/PyBullet rendering/physics implementation and a graph-based symbolic implementation, sharing actions and states.
N.1.1 Building the Symbolic Simulator and Transition Model Implementation
The full action table defines arguments, prerequisites, and effects. Interactability excludes enclosed objects and ordinarily requires reachability; automatic NAVIGATE_TO removes the reachability burden from the language model. Consequently, benchmark planning does not solve explicit navigation or continuous reachability.
| Action Name | Arguments | Preconditions | Post Effects |
|---|---|---|---|
| NAVIGATE_TO | tar_obj1 | tar_obj1 is interactable. | The agent is next to tar_obj1. |
| LEFT_GRASP | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is small enough. | tar_obj1 is in the agent’s left hand. |
| RIGHT_GRASP | tar_obj1 | tar_obj1 is interactable; The agent’s right hand is empty; tar_obj1 is small enough. | tar_obj1 is in the agent’s right hand. |
| LEFT_RELEASE | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds tar_obj1. | tar_obj1 is released to the floor. |
| RIGHT_RELEASE | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds tar_obj1. | tar_obj1 is released to the floor. |
| LEFT_PLACE_ONTOP | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds an object. | The object in the agent’s left hand is on top of tar_obj1. |
| RIGHT_PLACE_ONTOP | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds an object. | The object in the agent’s right hand is on top of tar_obj1. |
| LEFT_PLACE_INSIDE | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds an object; tar_obj1 is big enough and open if openable. | The object in the agent’s left hand is inside tar_obj1. |
| RIGHT_PLACE_INSIDE | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds an object; tar_obj1 is big enough and open if openable. | The object in the agent’s right hand is inside tar_obj1. |
| LEFT_PLACE_NEXTTO | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds an object. | The object in the agent’s left hand is next to tar_obj1. |
| RIGHT_PLACE_NEXTTO | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds an object. | The object in the agent’s right hand is next to tar_obj1. |
| LEFT_PLACE_UNDER | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds an object. | The object in the agent’s left hand is under tar_obj1. |
| RIGHT_PLACE_UNDER | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds an object. | The object in the agent’s right hand is under tar_obj1. |
| LEFT_TRANSFER_ CONTENTS_INSIDE | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds an object; tar_obj1 is big enough and open if openable. | The contents inside the object in the agent’s left hand are in tar_obj1. |
| RIGHT_TRANSFER_ CONTENTS_INSIDE | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds an object; tar_obj1 is big enough and open if openable. | The contents inside the object in the agent’s right hand are in tar_obj1. |
| LEFT_TRANSFER_ CONTENTS_ONTOP | tar_obj1 | tar_obj1 is interactable; The agent’s left hand holds an object. | The contents inside the object in the agent’s left hand are on top of tar_obj1. |
| RIGHT_TRANSFER_ CONTENTS_ONTOP | tar_obj1 | tar_obj1 is interactable; The agent’s right hand holds an object. | The contents inside the object in the agent’s right hand are on top of tar_obj1. |
| LEFT_PLACE_ NEXTTO_ONTOP | tar_obj1, tar_obj2 | tar_obj1 and tar_obj2 are interactable; The agent’s left hand holds an object. | The object in the agent’s left hand is next to tar_obj1 and on top of tar_obj2. |
| RIGHT_PLACE_ NEXTTO_ONTOP | tar_obj1, tar_obj2 | tar_obj1 and tar_obj2 are interactable; The agent’s right hand holds an object. | The object in the agent’s right hand is next to tar_obj1 and on top of tar_obj2. |
| TOGGLE_ON | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is toggled off and closed. | tar_obj1 is toggled on. |
| TOGGLE_OFF | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is toggled on. | tar_obj1 is toggled off. |
| CLOSE | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is open. | tar_obj1 is closed. |
| OPEN | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is closed and not toggled on. | tar_obj1 is open. |
| CLEAN | tar_obj1 | tar_obj1 is interactable; tar_obj1 is dusty or stained; the agent has a cleaning tool. | tar_obj1 is not dusty or stained (according to the tool). |
| DRY | tar_obj1 | tar_obj1 is interactable; tar_obj1 is soaked. | tar_obj1 is dry. |
| SLICE | tar_obj1 | tar_obj1 is interactable; tar_obj1 is not sliced; the agent has a knife. | tar_obj1 is sliced. |
| SOAK | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is dry and in a toggled on sink or a pot. | tar_obj1 is soaked. |
| FREEZE | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is unfrozen and in the fridge. | tar_obj1 is frozen. |
| UNFREEZE | tar_obj1 | tar_obj1 is interactable; tar_obj1 is frozen. | tar_obj1 is unfrozen. |
| COOK | tar_obj1 | tar_obj1 is interactable; One of the agent’s hands is empty; tar_obj1 is not cooked; tar_obj1 is on/in the pan. | tar_obj1 is cooked. |
N.1.2 BEHAVIOR State Space and Action Space
The implemented BEHAVIOR vocabulary contains fifteen object states and thirty actions. Unary state changes and binary spatial changes are separated. All state/action names and definitions are retained.
| State Space | Action Space | ||
| Unary States | Binary States | State Actions | Spatial Actions |
| Cooked | Inside | TOGGLE_ON | LEFT_GRASP |
| Dusty | OnFloor | TOGGLE_OFF | RIGHT_GRASP |
| Frozen | OnTop | CLOSE | LEFT_RELEASE |
| Open | Under | OPEN | RIGHT_RELEASE |
| Sliced | NextTo | CLEAN | LEFT_PLACE_ONTOP |
| Soaked | DRY | RIGHT_PLACE_ONTOP | |
| Stained | SLICE | LEFT_PLACE_NEXTTO | |
| ToggledOn | SOAK | RIGHT_PLACE_NEXTTO | |
| Slicer | FREEZE | LEFT_PLACE_INSIDE | |
| CleaningTool | UNFREEZE | RIGHT_PLACE_INSIDE | |
| COOK | LEFT_PLACE_UNDER | ||
| RIGHT_PLACE_UNDER | |||
| LEFT_TRANSFER_CONTENTS_INSIDE | |||
| RIGHT_TRANSFER_CONTENTS_INSIDE | |||
| LEFT_TRANSFER_CONTENTS_ONTOP | |||
| RIGHT_TRANSFER_CONTENTS_ONTOP | |||
| LEFT_PLACE_NEXTTO_ONTOP | |||
| RIGHT_PLACE_NEXTTO_ONTOP | |||
| NAVIGATE_TO | |||
N.2 VirtualHome Implementation Details
N.2.1 VirtualHome State Space and Action Space
A VirtualHome environment graph and MotionPlanner track state and categorize runtime errors for 42 actions. Executors are modified for consistent evaluation. The complete vocabulary table specifies the actual benchmark interface.
| State Space | Action Space | ||
| Unary States | Binary States | State Actions | Spatial Actions |
| Closed | On | OPEN | TURN_TO |
| Open | Inside | SIT | PUT_BACK |
| On | Between | STANDUP | PUT_IN |
| Off | Close | SLEEP | POUR |
| Sitting | Facing | WAKEUP | PUT_ON |
| Dirty | Holds_RH | CLOSE | FIND |
| Clean | Holds_LH | DRINK | RUN |
| Lying | GRAB | WALK | |
| Plugged_in | LOOKAT | POINT_AT | |
| Plugged_out | LOOKAT_SHORT | TOUCH | |
| LOOKAT_LONG | WATCH | ||
| SWITCH_OFF | MOVE | ||
| SWITCH_ON | RELEASE | ||
| TYPE | DROP | ||
| PUSH | |||
| PULL | |||
| SQUEEZE | |||
| WASH | |||
| RINSE | |||
| SCRUB | |||
| EAT | |||
| PLUG_IN | |||
| PLUG_OUT | |||
| CUT | |||
| READ | |||
| LIE | |||
Appendix O Evaluation Settings of LLMs
O.1 Decoding Parameters
The paper reports temperature-zero decoding to standardize generation and reduce sampling variation. This is the stated evaluation setting, not a guarantee of determinism across changing hosted model implementations.
O.2 Evaluation Cost
The authors estimate 192.28 million tokens, 84,960 queries, and US$1,540.70 for the reported benchmark run, using contemporary API pricing including Together AI for open models. These are historical costs. The stated 180 runs should not be inferred directly from eighteen models times four modules times two simulators without additional setup details.
O.3 Model Cards
The full model-card table preserves exact evaluated model identifiers and versions; shorthand names elsewhere should be interpreted through this table.
| Model Name | Creator | Complete Model ID | Release | Hosting |
|---|---|---|---|---|
| Claude-3 Haiku | Anthropic | claude-3-haiku-20240307 | 03/07/24 | Anthropic |
| Claude-3 Sonnet | Anthropic | claude-3-sonnet-20240229 | 02/29/24 | Anthropic |
| Claude-3 Opus | Anthropic | claude-3-opus-20240229 | 02/29/24 | Anthropic |
| Claude-3.5 Sonnet | Anthropic | claude-3-5-sonnet-20240620 | 06/20/24 | Anthropic |
| Cohere Command R | Cohere | command-r | 03/11/24 | Cohere |
| Cohere Command R+ | Cohere | command-r-plus | 04/04/24 | Cohere |
| Gemini 1.0 Pro | gemini-pro | 12/13/23 | GCP Vertex | |
| Gemini 1.5 Flash | gemini-1.5-flash-preview-0514 | 05/14/24 | GCP Vertex | |
| Gemini 1.5 Pro | gemini-1.5-pro-preview-0409 | 04/09/24 | GCP Vertex | |
| GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-0125 | 01/25/24 | OpenAI |
| GPT-4-turbo | OpenAI | gpt-4-turbo-2024-04-09 | 04/09/24 | OpenAI |
| GPT-4o | OpenAI | gpt-4o-2024-05-13 | 05/13/24 | OpenAI |
| Llama 3 8B Instruct | Meta | meta-llama-3-8b-instruct | 04/18/24 | TogetherAI |
| Llama 3 70B Instruct | Meta | meta-llama-3-70b-instruct | 04/18/24 | TogetherAI |
| Mistral Large | MistralAI | mistral-large-2402 | 02/26/24 | MistralAI |
| Mixtral 8x22B MoE | MistralAI | mixtral-8x22b-instruct-v0.1 | 04/17/24 | TogetherAI |
| o1-mini | OpenAI | o1-mini-2024-09-12 | 09/12/24 | OpenAI |
| o1-preview | OpenAI | o1-preview-2024-09-12 | 09/12/24 | OpenAI |
Appendix P Extensive Related Work
P.1 LLMs for Embodied Planning
The extended survey categorizes prior embodied planning systems by whether language models interpret goals, decompose subgoals, sequence actions, or model transitions. Its complete mapping is retained.
| Existing Work | Ref. | Goal | Action | Subgoal | Transition |
|---|---|---|---|---|---|
| Interpretation | Sequencing | Decomposition | Modeling | ||
| SayCan | [2] | LLMs | LLMs | ||
| Ada | [8] | LLMs | LLMs | ||
| LLP+P | [7] | LLMs | |||
| AutoTAMP | [18] | LLMs | LLMs | ||
| Code as Policies | [6] | LLMs | LLMs | LLMs | |
| Voyager | [35] | LLMs | LLMs | ||
| Demo2Code | [13] | LLMs | LLMs | LLMs | |
| LM as ZeroShot Planner | [3] | LLMs | LLMs | ||
| SayPlan | [15] | LLMs | LLMs | LLMs | |
| Text2Motion | [48] | LLMs | |||
| LLMGROP | [46] | LLMs | LLMs | ||
| REFLECT | [66] | LLMs | LLMs | ||
| Generating Consistent PDDL Domains with LLMs | [32] | LLMs | LLMs | ||
| PlanSeqLearn | [64] | LLMs | |||
| COWP | [70] | LLMs | LLMs | LLMs | |
| HumanAssisted Continual Robot Learning | [65] | LLMs | |||
| DECKARD | [34] | LLMs | LLMs | ||
| MOSAIC | [63] | LLMs | |||
| Interactive Task Planning with Language Models | [14] | LLMs | LLMs | ||
| RoCo | [62] | LLMs | |||
| Cook2LTL | [42] | LLMs | |||
| InnerMonologue | [21] | LLMs | |||
| MLDT | [20] | LLMs | |||
| Learning to Reason over Scene Graphs | [61] | LLMs | LLMs | LLMs | |
| GRID | [19] | LLMs | LLMs | ||
| LLMplanner | [33] | LLMs | LLMs | ||
| DELTA | [17] | LLMs | |||
| Look Before You Leap | [60] | LLMs | LLMs | ||
| CAPE | [36] | LLMs | LLMs | ||
| HERACLEs | [43] | LLMs | |||
| RoboTool | [16] | LLMs | LLMs | ||
| PROMST | [59] | LLMs | |||
| LLM3 | [31] | LLMs | LLMs | ||
| Ghost in the Minecraft | [27] | LLMs | |||
| PlanBench | [98] | LLMs | LLMs | ||
| TaPA | [30] | LLMs | LLMs | LLMs | |
| ChatGPT Robot Control | [29] | LLMs | |||
| LLM World Models for Planning | [28] | LLMs | LLMs | ||
| DEPS | [81] | LLMs | LLMs | ||
| Grounded Decoding | [25] | LLMs | |||
| ProgPrompt | [37] | LLMs | LLMs | ||
| DROC | [24] | LLMs | LLMs | ||
| LMPC | [58] | LLMs | LLMs | ||
| GPTPDDL | [47] | LLMs | |||
| RAP | [57] | LLMs | |||
| LEAGUE++ | [38] | LLMs | LLMs | ||
| CoPAL | [23] | LLMs | LLMs | ||
| SayCanPay | [22] | LLMs | LLMs | ||
| LLMGenPlan | [56] | LLMs |
P.2 LTL Agent Interface
Temporal logic has a history in program verification and robot planning. This work adopts it as an expressive interface between language and symbolic decision modules. Using formal specifications does not by itself provide robot safety guarantees when model definitions or perception are incorrect.
P.3 Embodied Agent Benchmarks
The comparison discusses planning benchmarks, household simulators, dynamic-environment tasks, and embodied action understanding. EAI’s distinguishing emphasis is a common interface with module-level diagnostics and temporal goals. Its broader diagnostic scope complements, rather than replaces, perception/control benchmarks.
Appendix Q Maintenance Plan
Q.1 Dataset URLs, License, and Hosting Plan
The paper lists Hugging Face and GitHub hosting, author contact channels, and GitHub issues for corrections. It states CC-BY-4.0 for the new annotations. These are the authors’ release commitments; underlying simulator assets retain their own terms.
Dataset and metadata · Code, evaluators, and issue tracker · Project page.
Q.2 Long-term Preservation and DOI
The source provides a Dryad share URL as a preservation pointer, describing it as a persistent identifier. The printed URL is not itself a DOI-form identifier.
Q.3 URL of Croissant Metadata Record
Structured dataset metadata are linked through the Hugging Face dataset page.
Q.4 Author Statement
The authors state responsibility for annotation rights and the CC-BY-4.0 release.
Q.5 URLs of Code and Re-productivity
The shared repository provides evaluation, annotation, prompt-generation, and inference instructions.
Q.5.1 Code for VirtualHome Evaluator and Computation Resources
VirtualHome evaluation and interactive goal-annotation tools are provided with Docker instructions. Reported runs use CPU systems: Ryzen 7 5800H with 16 GB for decomposition and a Xeon-based Linux system with 32 GB for the other modules.
Q.5.2 Code for BEHAVIOR Evaluator and Computation Resources
BEHAVIOR evaluation scripts cover all four modules. Reported experiments use a Ryzen 7 7800X3D Windows system with 32 GB RAM.
Q.5.3 Code for LLMs Implementations and Computation Resources
Batch inference integrates with HELM and optionally hosted Together AI APIs, avoiding a requirement to run each open model locally.
Appendix R Datasheets for Embodied Agent Interface ( EAI )
The datasheet records author-curated additions made in 2024 over existing simulators. VirtualHome contains 338 trajectories, 2,960 steps, and 801 goal conditions; BEHAVIOR contains 100 trajectories, 1,460 steps, and 673 goal conditions. It reports 33 annotated VirtualHome transition models with 99 preconditions/57 effects and thirty BEHAVIOR models with 84 preconditions/51 effects. The intended use is out-of-the-box model evaluation, with related task instances sharing goals or names and differing trajectories. Authors checked executions and curated irrelevant states. The text describes public JSON/evaluator distribution, issue-based corrections, expansion, and reviewed contributions, and states that the annotations contain no sensitive personal data. Several datasheet statements predate revisions: it mentions fifteen evaluated models where the final paper reports eighteen and describes visual evaluation as future work despite the added appendix. Its references to simulator licenses should be read alongside the separate CC-BY-4.0 annotation statement, rather than as one license for all assets.
Appendix S Impact, Limitations and Future Directions
S.1 Broader Impact
The proposed benefit is more informative diagnosis of language-driven robot plans, helping developers identify failures and choose which components need external support. Potential deployment and safety benefits are prospective; the benchmark itself demonstrates symbolic evaluation rather than reliable real-world autonomy.
S.2 Limitations
Relational graphs omit sensory ambiguity, continuous geometry, physical dynamics, and fine motor control. The main evaluations do not jointly test language grounding in perception and execution through actuators. These omissions are central boundaries on what the scores establish.
S.3 Potential Negative Social Impact
The authors discuss privacy, security, misuse, bias, labor disruption, and unequal impacts of future embodied agents. They call for safeguards and stakeholder participation; no empirical societal-impact study is reported.
S.4 Potential Future Directions
Proposed extensions include multimodal and low-level control interfaces, episodic/state memory, physical and geometric reasoning, navigation/exploration, forward/backward/counterfactual prediction, richer simulators, and fine-tuning on diagnostic feedback. These are research directions beyond the released benchmark’s demonstrated capabilities.