Literature
Display
← All papers

Infer Human's Intentions Before Following Natural Language Instructions

Yanming Wan · Yue Wu · Yiping Wang · Jiayuan Mao · Natasha Jaques

· AAAI 2025 ↗

AI agents11 min text · CondensedOriginal paper ↗
Contents

Overview

The authors are affiliated with the University of Washington and MIT CSAIL. This edition follows the complete arXiv v2 source, including its implementation details and prompt appendix.

At a glance

FISER resolves ambiguous requests by predicting the human-assigned robot task before selecting actions. An optional earlier stage predicts the human subgoal that motivates the request. Small supervised Transformers implement these stages using symbolic world information, a human action history, and language.

On fully observable, text-based HandMeThat tasks, the best staged model averages 64.5% success across four difficulty levels. It exceeds the evaluated GPT-4 Turbo prompts, but this is an in-domain benchmark comparison with intermediate supervision and known applicable actions, not a demonstration of general human-intention understanding or physical robot assistance.

1 Introduction

“Could you pass that from the sofa?” becomes informative when the listener sees that the speaker has been boxing books. The observed actions suggest a goal; the request identifies a useful contribution to it. FISER makes this inference explicit instead of treating language and history only as inputs to an action predictor.

The framework separates social reasoning about the requested task from embodied planning. It also tests whether predicting a higher-level human subgoal helps. Supervised Transformer variants and structured GPT-4 Turbo prompts provide complementary implementations; the central result is that intermediate task inference improves performance in the tested setting.

Refer to caption
Figure 1: An example scenario where the human’s natural language instruction (“Could you pass that from the sofa?”) is inherently ambiguous. Standard language grounding and planning methods fail to resolve ambiguity. We propose FISER, which explicitly reasons about human’s internal intentions as intermediate steps. The robot disamiguates the instruction into a concrete robot-understandable task in the social reasoning phase (Phase 1), and then accomplishes the grounded planning in the embodied reasoning phase (Phase 2). We further propose an optional enhancement to Phase 1 by explicitly recognize the human’s overall plan first, and then infer what the human wants the robot to do.

2 Related Work

The method connects grounded instruction following, collaborative communication, goal recognition, and intermediate-step reasoning.

2.1 Grounded language learning

Systems such as SayCan use language-model knowledge to select feasible robot skills. FISER focuses on requests that omit information about the speaker’s goal, making the human’s preceding actions useful for deciding what the request means.

2.2 Collaborative communication

HandMeThat combines explicit language with implicit information from human actions. Unlike dialogue tasks with a disembodied instruction giver, its speaker is already acting in the shared environment. Earlier logic-guided HMT work supplies hand-designed symbolic structure; FISER instead learns supervised intermediate predictions.

2.3 Goal recognition

Goal recognition often interprets behavior as approximately rational action toward a hidden goal. Watch-and-Help similarly uses observed activity, but without a natural-language request. Here the target is the subgoal relevant to helping, rather than complete recovery of the human’s entire plan.

2.4 Reasoning with intermediate steps

The design follows work on intermediate representations and chain-of-thought prompting: expose a useful subproblem before predicting the final answer. The empirical question is whether human-task inference is such a useful intermediate step for collaborative instruction following.

3 FISER: Follow Instructions with Social and Embodied Reasoning

A factorized human–robot decision process distinguishes observable behavior and language from hidden human goals and the robot task assigned through speech.

3.1 Problem Formulation

The state records objects and agents, their types, locations, and attributes. Human and robot actions have deterministic transitions. In each episode the human acts first, stops once, and issues one request. The robot receives the preceding trajectory, current state, and utterance, then selects an action sequence to maximize discounted reward. There is no repeated dialogue or clarification loop in this formulation.

Equations⟨𝒮,𝒜h,r,𝒯,𝒰,Rr,γ,T⟩\langle{\mathcal{S}},{\mathcal{A}}^{h,r},{\mathcal{T}},{\mathcal{U}},R^{r},\gamma,T\rangle∑t=t′T[γt−t′​Rr​(st+1)]\sum_{t=t^{\prime}}^{T}[\gamma^{t-t^{\prime}}R^{r}(s_{t+1})]
Refer to caption
Figure 2: Graph for our problem formulation and proposed method. White nodes are observable variables, while grey nodes are unobservable. The robot is given the trajectory τt′\tau_{t^{\prime}}, a final state st′s_{t^{\prime}}, and an utterance uu. We propose to explicitly model human’s intentions by modeling the human’s overall plan Gh∈𝒢hG^{h}\in\mathcal{G}^{h} as a set of predicates pkp_{k}. We further assume that human selects a subgoal p∗p^{*} that needs help, and then specifies a robot’s task GrG^{r}, which is the underlying intention of human when saying uu.

3.2 Modeling the Human’s Intentions

The human has a fixed hidden goal and is assumed to act rationally toward it. A goal can be represented as predicates, such as putting all books into a box. The human chooses an unfinished subgoal and requests a concrete robot contribution that advances it.

The relevant distinctions are the overall goal, the selected subgoal, the robot’s assigned task, and the utterance expressing that task. The same words can correspond to different tasks when the observed history or goal changes. The model uses this structure without recovering an exact complete human plan.

Equations⟨∃y,box(y),∀x,book(x)⇒in(x,y)⟩\langle\exists y,\text{box}(y),\forall x,\text{book}(x)\Rightarrow\text{in}(x,y)\rangleGh={p1,p2,⋯,pl}G^{h}=\{p_{1},p_{2},\cdots,p_{l}\}p∗∈{pk,…,pl}p^{*}\in\{p_{k},\ldots,p_{l}\}Rh​(s′∣{p∗})>0R^{h}(s^{\prime}\mid\{p^{*}\})>0

3.3 Step-wise Reasoning over Human Intentions

Task recognition and grounded planning are separate stages; optional plan recognition supplies a compact subgoal between the observed trajectory and task prediction.

Social Reasoning: Robot’s Task Recognition

Task recognition maps the state, human history, and utterance to an explicit robot goal, such as the identity of the object to hand over. This replaces an ambiguous request with a concrete prediction that can be supervised and evaluated.

EquationTR​(st′,τt′,u)→Gr.\text{TR}(s_{t^{\prime}},\tau_{t^{\prime}},u)\rightarrow G^{r}.

Social Reasoning: Human’s Plan Recognition.

Despite the name “plan recognition,” the implemented stage predicts only the relevant human subgoal. It uses the state, trajectory, and utterance, then replaces the trajectory with that subgoal for task recognition. This limits the inference problem to the part of the plan needed for assistance.

EquationsPR​(st′,τt′,u)→p∗\text{PR}(s_{t^{\prime}},\tau_{t^{\prime}},u)\rightarrow p^{*}TR​(st′,p∗,u)→Gr.\text{TR}(s_{t^{\prime}},p^{*},u)\rightarrow G^{r}.

Embodied Reasoning: Grounded Planning.

Grounded planning conditions on the predicted robot task and current context to choose executable actions. Its inputs can be symbolic, so either a planner or a neural policy could implement it; this study uses a learned model. When plan recognition is enabled, its predicted subgoal replaces the human-trajectory input, as specified in the source footnote.

EquationGP​(st′,τt′,Gr)→{atr}t≥t′\text{GP}(s_{t^{\prime}},\tau_{t^{\prime}},G^{r})\rightarrow\{a_{t}^{r}\}_{t\geq t^{\prime}}

4 Transformer-based Model Implementation

The implementation processes four input streams with Transformer encoders and predicts symbolic tasks and action strings. Figure 3 retains the complete architecture.

Original figure
Figure 3: The Transformer-based model has four parts of inputs, which are passed separately into different Transformer Encoder Layers, and interact with each other through a Modality Interaction module after each Transformer layer. The first 2​N2N layers form the social reasoning phase and the last NN layers form the embodied reasoning phase. The embeddings at Layer 2​N2N are used for recognizing robot’s task, and the last layer embeddings are used for predicting actions.

4.1 Inputs

World state, human trajectory, instruction, and the robot’s past outputs are encoded separately to manage long household descriptions.

World state description.

An object token combines category, attributes, and spatial relationships; human and robot are special objects. The representation uses structured object information rather than perception from camera images.

Human’s trajectory.

The human’s action sequence is rendered as text, with object-name embeddings replaced by the corresponding object tokens so language refers to the same world entities.

Language instruction.

The instruction is tokenized and embedded directly.

Model’s past outputs.

Past robot actions and environment feedback are concatenated with separator tokens. This history records what the robot has already attempted and how the environment changed.

4.2 Model Architecture Overview

Nine encoder layers use N = 3. Optional human-subgoal prediction occurs after layer three; the predicted subgoal replaces the trajectory stream. Robot-task prediction occurs after layer six; its output replaces the instruction stream. The final three layers produce actions.

End-to-end training retains intermediate auxiliary losses while learning final actions jointly. Multi-staged training separates social reasoning from the action module. The latter learns with ground-truth intermediate inputs but receives predicted inputs at evaluation, creating a possible train–test mismatch.

4.3 Encoder Layers

Each layer has four Transformer branches followed by a module that exchanges information among their summary tokens.

Transformer Layers.

World-state tokens omit positional encodings because object order is arbitrary. The other streams retain their sequence structure.

Modality Interaction.

Following GreaseLM, each stream reserves an interaction token. A shared multilayer perceptron mixes these four tokens, and the updated tokens carry information back into their respective streams.

4.4 Prediction Layers

Intermediate predictions and actions use cross-entropy supervision. Human subgoals describe object types; robot tasks identify concrete objects.

Social Reasoning: Human’s Plan Recognition.

A human subgoal is simplified to four symbols: quantifier, subject type, relation, and object type, for example “for all, book, inside, box.” The preserved likelihood factorization predicts quantifier and relation first, then conditions subject and object on them. This fixed vocabulary is a domain assumption.

Equation
log⁡Pr⁡[Q,S,V,O]≈\displaystyle\log\Pr[\text{Q},\text{S},\text{V},\text{O}]\approxlog⁡Pr⁡[Q]+log⁡Pr⁡[V]\displaystyle\log\Pr[\text{Q}]+\log\Pr[\text{V}]
+\displaystyle+logPr[S|Q,V]+logPr[O|Q,V],\displaystyle\log\Pr[\text{S}|\text{Q},\text{V}]+\log\Pr[\text{O}|\text{Q},\text{V}],

Social Reasoning: Robot’s Task Recognition.

Task recognition selects a particular object in the current world. Predicting “book” as a human-goal category is therefore distinct from selecting the specific book the robot should manipulate.

Embodied Reasoning: Action Prediction.

Each action is a type with up to two arguments. Argument predictions condition on the action type. The model scores triples and selects the highest-scoring currently applicable one; access to the valid-action set is assumed, so the task does not require learning all physical preconditions from scratch.

Equation
logPr[Action,\displaystyle\log\Pr[\text{Action},Arg1,Arg2]≈log⁡Pr⁡[Action]\displaystyle\text{Arg1},\text{Arg2}]\approx\log\Pr[\text{Action}]
+\displaystyle+log⁡Pr⁡[Arg1|Action]+log⁡Pr⁡[Arg2|Action].\displaystyle\log\Pr[\text{Arg1}|\text{Action}]+\log\Pr[\text{Arg2}|\text{Action}].

5 Experiments

Experiments test whether explicit task recognition helps, whether optional human-subgoal recognition adds value, and how these supervised models compare with the paper’s GPT-4 Turbo prompting variants.

5.1 HandMeThat Environment

HandMeThat version 2 represents a household in text. Level 1 supplies an unambiguous task; Level 2 requires inferring the human goal; Level 3 additionally requires pragmatic interpretation of why particular wording was chosen; Level 4 remains ambiguous even with the available evidence and therefore rewards useful priors.

Evaluation uses fully observable states and one attempt of four or five actions, preventing brute-force object enumeration. Success measures whether the intended robot goal is achieved. This differs from the original benchmark metric, which also penalizes action count, and does not test vision, physical manipulation, or interactive clarification.

5.2 Model Details

The comparisons include published benchmark baselines, newly trained Transformer variants, and increasingly informative prompts for GPT-4 Turbo.

Baseline models.

Human, heuristic, and sequence-to-sequence results come from the HMT benchmark. The heuristic has privileged symbolic state and utterance-logic access. The comparison with Cao et al. uses a selected version-1 subset with “bring me” requests and goals also present in version 2; it is not a matched full-dataset reimplementation.

Transformer-based model.

The vanilla Transformer predicts actions directly; FISER adds task prediction; FISER+PR also predicts a human subgoal. Both joint and staged training are tested, with intermediate QSVO and object accuracies reported. Models train from scratch: approximately 5.1 million parameters jointly, or 4.7 million across the two staged modules.

Prompted GPT-4.

GPT-4 Turbo receives plain task inputs, then domain knowledge and an example through prompt engineering, then explicit object inference and optional higher-level goal inference. Appendix B retains the complete prompt components. A separate diagnostic removes irrelevant objects using ground-truth goals, an oracle aid unavailable to an ordinary deployed agent.

6 Results

Multi-staged FISER achieves 89.0%, 74.0%, 52.3%, and 42.7% success across Levels 1–4, averaging 64.5%. Tables retain all baseline, staged/joint, intermediate-accuracy, and version-1-subset results. Transformer estimates report mean and standard error across three runs; these are not standard deviations.

Model Baseline Models GPT-4 Turbo Transformer-based Models On HMT Version 1
Human Heur. Seq2Seq Vanilla +PE +FISER +PR Vanilla +FISER +PR Cao et al. FISER
Level 1 100.0 100.0 30.4 72.0 82.0 80.0 77.0 77.7±\pm1.6 89.0±\pm1.5 72.0±\pm1.5 27.7±\pm0.3 89.7±\pm0.4
Level 2 80.0 64.0 28.8 16.0 25.0 36.0 34.0 55.3±\pm0.4 74.0±\pm0.3 74.0±\pm0.9 24.8±\pm0.4 63.0±\pm0.3
Level 3 40.0 39.0 12.8 5.0 13.0 18.0 17.0 36.3±\pm1.0 52.3±\pm2.3 52.3±\pm1.0 21.0±\pm0.1 28.3±\pm2.5
Level 4 30.0 29.0 14.8 9.0 9.0 17.0 20.0 38.3±\pm1.4 42.7±\pm0.2 51.0±\pm1.2 21.7±\pm0.2 40.7±\pm1.9
Table 1: Success rate (%) of models over HMT in the fully observable setting. The results for Transformer-based models are the mean and standard error values over three runs. *Cao et al. (2024) is evaluating version 1 of HandMeThat dataset, and thus we provide the results of Transformer+FISER model over a subset of version 1 for a fair comparison. Overall, FISER improves the performance across all levels compared to the vanilla Transformer. While applying FISER and PR to GPT-4 improves its performance, overall GPT-4 cannot perform well on these tasks even with very careful prompting, achieving less than half the success rate of our model for ambiguous instructions in levels 2-4.
Model Level 1 Level 2 Level 3 Level 4
End-to-End 77.7±\pm1.6 (N/A, N/A) 55.3±\pm0.4 (N/A, N/A) 36.3±\pm1.0 (N/A, N/A) 38.3±\pm1.4 (N/A, N/A)
End-to-End+FISER 74.3±\pm0.2 (N/A, 87.8) 73.7±\pm0.5 (N/A, 80.9) 47.3±\pm2.1 (N/A, 73.1) 41.3±\pm1.9 (N/A, 64.7)
End-to-End+FISER+PR 61.0±\pm1.5 (73.2, 81.3) 66.7±\pm1.1 (64.9, 81.0) 47.0±\pm1.2 (59.0, 73.4) 42.0±\pm0.3 (55.0, 66.5)
Multi-Staged+FISER 89.0±\pm1.5 (N/A, 93.4) 74.0±\pm0.3 (N/A, 82.3) 52.3±\pm2.3 (N/A, 76.5) 42.7±\pm0.2 (N/A, 68.1)
Multi-Staged+FISER+PR 72.0±\pm1.5 (74.4, 82.4) 74.0±\pm0.9 (71.9, 82.3) 52.3±\pm1.0 (67.1, 75.6) 51.0±\pm1.2 (65.3, 71.2)
Table 2: Comparison between End-to-End and Multi-staged training of Transformer-based models over HMT in the fully observable setting. All models are evaluated by the success rate (%). The prediction accuracy (%) of QSVO and Obj (intermediate outputs) are presented in parentheses. The results are the mean values over three runs, and the standard error values for success rates are provided. Overall, training in a multi-staged manner works better than end-to-end in our tasks, implying that fully separating social from embodied reasoning provides the best performance.

H1: Explicitly modeling human intentions works better than directly predicting actions.

The intermediate-task hypothesis is supported overall, with important exceptions when an additional inference stage is unnecessary.

(a) Separating the social and embodied reasoning steps by explicitly recognizing the robot’s task is beneficial.

Staged FISER exceeds the vanilla Transformer at every level and its jointly trained counterpart at every level. This is consistent with separating task interpretation from planning, though it does not independently establish a causal explanation in terms of gradient interference.

GPT-4 FISER improves ambiguous Levels 2–4 over prompt engineering alone, but Level 1 falls from 82% to 80%. The joint Transformer similarly falls from 77.7% to 74.3% at Level 1. Thus intermediate reasoning does not improve every configuration on every difficulty level.

(b) Explicitly recognizing the human’s plan further helps with the social reasoning stage.

Adding plan recognition raises staged Level-4 success from 42.7% to 51.0%, leaves Levels 2 and 3 unchanged at the reported precision, and lowers Level 1 from 89.0% to 72.0%. Extra goal inference is most useful when instructions are intrinsically ambiguous; unnecessary intermediate predictions can hurt simple tasks.

H2: Pre-trained LLMs, despite having access to common-sense knowledge, do not adequately perform the complex social and embodied reasoning in this task. Incorporation of domain-specific knowledge through CoT can help.

The best prompted GPT-4 Turbo variants average roughly 38% across the four levels, versus 64.5% for staged FISER: about a 70% relative gain, not a 70-point absolute gain. These results concern this model/version, prompt set, supervision regime, and benchmark.

The Level-2 failure study uses 100 examples. Errors include locating/planning failures, returning objects already handled, and choosing the wrong intended object. Categories can overlap. The trained model reduces planning and redundant-action errors, while object-intention mistakes remain.

Oracle filtering improves GPT-4 markedly, but retaining only relevant objects yields approximately 80–86% success in Figure 5. The source prose says “less than 85%,” which is not consistent with every plotted point. Figure 5’s horizontal axis is the percentage of irrelevant objects remaining, not removed. This diagnostic implicates both information selection and planning; it supplies privileged goal information and is not a deployable filtering method.

Refer to caption
Figure 4: Failure case analysis for state-of-the-art LLMs with CoT prompts following FISER framework versus Transformer-based models trained from scratch with the FISER framework over 100 data points on Level 2.
Refer to caption
Figure 5: Success rates of GPT-4 Turbo under different settings given that different percentages of irrelevant objects are filtered out from the environment.

7 Conclusion

FISER supports a practical decomposition: infer the specific requested contribution before planning it. Optional higher-level inference helps the most ambiguous cases, but is not uniformly beneficial. Evidence remains limited to a fully observable symbolic household benchmark, supervised intermediate labels, and the tested GPT-4 Turbo baselines; real sensing, richer human behavior, and generalization beyond the task vocabulary remain unestablished.

Appendix A Model Implementation Details

The appendix specifies baseline differences, training hyperparameters, model sizes, and compute resources.

A.1 Baseline Models

Baseline results need to be interpreted with their original information access and dataset restrictions.

Baselines in HandMeThat

The published heuristic grounds the utterance’s underlying logical formula and prefers object categories recently manipulated by the human. It sees full object states and is designed for the fully observable setting. Published human and neural baselines are reused rather than all being retrained here.

Cao et al. (2024)

The version-1 comparison retains “bring me” instructions with goals included in version 2. FISER trains on 25 goal templates and is evaluated on new instances of those goals. The authors contrast their supervised subgoal labels and symbolic states with Cao et al.’s hand-crafted goal structures and rule-based text extraction; these protocol differences limit a strictly controlled method comparison.

A.2 Transformer-based Models

The implementation uses the architecture described in Section 4. The introduction links a code repository, while this appendix still describes release as future work.

Hyperparameters

Exact parameter counts are 5,057,945 for the joint model, 3,020,483 for social reasoning, and 1,701,750 for embodied reasoning. Joint/action models train for twenty epochs; social reasoning trains for forty. Batch size 32 is selected from 32, 64, and 128. Adam uses learning rate 10⁻⁴, betas (0.9, 0.98), epsilon 10⁻⁹, and seeds 0, 1, and 2.

Compute Resources

Each model uses one NVIDIA RTX A6000. Joint and action-model epochs take approximately eight hours, making twenty epochs roughly a week; social-model epochs take under an hour. Small parameter counts therefore should not be read as negligible total training cost.

A.3 Prompted GPT-4

GPT-4 Turbo is queried through the paid OpenAI API. The appendix does not identify an exact dated model snapshot, so the results should not be generalized to subsequent model versions.

Appendix B GPT-4 Prompts

All eleven original prompt boxes are preserved below, including their line structure, example world, twenty-five goal descriptions, output-format constraints, and blue/red highlighting. Successive variants reuse components omitted from later source listings; “Data” marks insertion points.

B.1 GPT-4

The base prompt supplies the world, human trajectory, and instruction, and requests an action-only final block. The first box defines action templates; the second inserts the current example.

You are a robot. Every time you will receive a description of the world and an instruction given by the human in the following templates:
[World Description]:
[Human Trajectory]:
[Human Instruction]:
You are expected to generate a sequence of actions that are consistent with the human’s instruction and the world description.
Please remember that each object has a number ID.
!!!!!THE LAST PART OF THE RESPONSE should be a sequence of actions starting with the string ”Actions:” (!!Only actions! No numbering for actions, no description, no extra words)!!!!!
The available action templates contain:
move to XXX
pick up XXX
pick up XXX from XXX
put XXX into XXX
put XXX onto XXX
take XXX from XXX
give XXX to human
open XXX
A typical answer format is as follows, where the ”open” step is optional.
Actions:
move to XXX
open XXX
pick up XXX
move to human
give XXX to human
======================== Now please begin to generate the output for given scenarios ========================
[World Description]: Data
[Human Trajectory]: Data
[Human Instruction]: Data
You are expected to generate a sequence of actions that are consistent with the human’s instruction and the world description.
Please remember that each object has a number ID.
!!!!!THE LAST PART OF THE RESPONSE should be a sequence of actions starting with the string ”Actions:” (!!Only actions! No numbering for actions, no description, no extra words)!!!!!
======================== Let’s start! ========================

B.2 GPT-4+PE

Prompt engineering adds opening-object rules, the complete twenty-five-goal domain, a detailed household example, and its action answer. Task instructions and current-example formatting reuse the base prompt. The example preserves the source’s “give dusty document 2 to human” wording.

NOTE!!: Remember to open something like refrigerator if it is closed before pick up something.
There are some openable objects: vessel, bag, box, package, cabinet, microwave, oven, dishwasher, refrigerator
Take an action like open refrigerator, open cabinet, open box, open bag, open vessel, open package, open microwave, open oven, open dishwasher
========================================================================================
To let you answer better, we give you the description of the goal space for the world as follows. The human has one of the following goals:
Goal 0: boxing books up for storage
Goal description: Put all the paper products of a certain kind into a box.
Goal 1: bring in wood
Goal description: Put all the building materials of a certain kind on top of the floor.
Goal 2: clearing the table after dinner
Goal description: Put all of one certain kind of cutlery inside some bucket, put all of another kind of cutlery into another bucket, and put all the flavorer of a certain kind into a bucket.
Goal 3: collect misplaced items
Goal description: Put all of a certain footwear, a certain decoration, and a certain paper product on top of the table.
Goal 4: collect aluminum cans
Goal description: Put all drinks into the ashcan.
Goal 5: installing alarms
Goal description: Put an electrical device on the table, on the countertop, and on the sofa.
Goal 6: laying tile floors
Goal description: Put all of a certain building material on top of the floor.
Goal 7: loading the dishwasher
Goal description: Put all the tablewares of two certain kinds and all of a certain vessel into the sink.
Goal 8: moving boxes to storage
Goal description: Put all the boxes on top of the floor.
Goal 9: oraganizing boxes in garage
Goal description: Put all of a certain plaything into a box, put all of a certain cutlery into another box, put all of a certain cleasing thing into a third box, and put all the boxes on top of the floor.
Goal 10: organize file cabinet
Goal description: Put all the writing tools of a certain kind on top of the table, and put all the paper products of a certain kind into the cabinet.
Goal 11: pick up trash
Goal description: Put all of a certain paper product and all of a certain drink into the ashcan.
Goal 12: put away Christmas decorations
Goal description: Put all the decorations of three certain kinds into the cabinet.
Goal 13: put away Halloween decorations
Goal description: Put all the vegetable of a certain kind and all the illumination tools of a certain kind into the cabinet, and put all of a certain vessels on top of the table.
Goal 14: put away toys
Goal description: Put all the playthings of a certain kind into a closed box.
Goal 15: put dishes away after cleaning
Goal description: Put all the tablewares of a certain kind into the cabinet.
Goal 16: put leftovers away
Goal description: Put all of a certain prepared food and all of a certain flavorer into the refrigerator.
Goal 17: put up Christmas decorations inside
Goal description: Put all of a certain illumination tool and all the decorations of two certain kinds on top of the table, and put all the decorations of another certain kind on top of the sofa.
Goal 18: re-shelve library books
Goal description: Put all of a certain paper product on top of the shelf.
Goal 19: serve hors d’oeuvres
Goal description: Put all of a certain baked food, a certain vegetable, a certain prepared food, and the trays on top of the table.
Goal 20: sort books
Goal description: Put all the paper products of two certain kinds on top of the shelf.
Goal 21: store food
Goal description: Put all of a certain prepared food, a certain snacks, and two certain kinds of flavorers into the cabinet.
Goal 22: store the groceries
Goal description: Put all of a certain fruit, a certain protein, and two certain kinds of vegetables into the refrigerator.
Goal 23: thaw frozen food
Goal description: Put all of a certain vegetable, a certain fruit, and a certain protein into the sink.
Goal 24: throw away leftovers
Goal description: Put all the snacks of a certain kind into the ashcan.
=========================================================================
The following is an example of prompt inputs and ground truth outputs. You should generate output in a similar way:
[World Description]:
Welcome to the world!
In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
Now you are standing on the floor. You are at the floor.
You see floor. On floor you can see green highchair 0 red dusty seat 0 blue dusty seat 1 blue chair 0 red dusty chair 1 green dusty chair 2 dusty bucket 0 bucket 1 dusty bucket 2 green large closed package 2 small dusty ashcan 0 small dusty ashcan 1 large ashcan 2 xmas_stocking 0 dusty xmas_tree 0 xmas_tree 1 xmas_tree 2 shoe 0 dusty sock 0 sandal 2. In bucket 1 you can see dishtowel 1 detergent 0.
You see countertop. On countertop you can see green closed bottle 0 red closed kettle 1 red closed dusty kettle 2 small dusty plate 0 red closed dusty briefcase 1 large dusty tray 2 xmas_stocking 2 strawberry 1 strawberry 2 carrot 1 beer 1 fish 0 fish 1 parsley 1 bread 2 cookie 0 saw 0 saw 1 hammer 0 carving_knife 0 toggled-off facsimile 0 calculator 1 mouse 0 earphone 0 pencil 1 highlighter 1 fork 1 spoon 0.
You see dusty stained sofa. On sofa you can see green closed duffel_bag 0 green basket 0 green basket 1 red large open dusty box 0 red large closed dusty box 1 blue small closed dusty package 0 ribbon 1 dusty bow 1 document 1 dusty document 2 hardback 1 hat 0 cube 0. In box 0 you can see document 0. In package 0 you can see dusty ribbon 0 bow 0 bracelet 2 hat 2.
You see dusty bed. On bed you can see blue basket 2 bracelet 0 dusty jewelery 0 hardback 0 dusty apparel 0 apparel 1 ball 0.
You see toggled-off stove. On stove you can see nothing.
You see dusty table. On table you can see green closed bottle 1 green closed sack 0 blue closed briefcase 0 blue closed dusty briefcase 2 small tray 1 blue large closed dusty box 2 xmas_stocking 1 strawberry 0 banana 2 pop 0 vegetable_oil 0 vegetable_oil 1 sugar 1 cracker 0 cracker 1 bread 0 bread 1 cookie 2 salad 0 soup 0 pasta 2 carving_knife 1 scraper 0 scraper 1 screwdriver 1 toggled-off printer 0 toggled-off printer 1 calculator 0 mouse 2 pencil 2 highlighter 0 lamp 0 lamp 1. On tray 1 you can see sandwich 1 soup 2.
You see stained shelf. On shelf you can see blue small closed dusty package 1 saw 2 hammer 1 screwdriver 0 toggled-off scanner 0 mouse 1 earphone 1 lamp 2.
You see dusty stained toilet. On toilet you can see toothpaste 1 toothpaste 2.
You see open dusty cabinet. In cabinet you can see blue closed dusty kettle 0 small dusty bowl 0 large bowl 1 pan 2 pop 1 beer 2 juice 1 parsley 0 vegetable_oil 2 hammer 2 scraper 2 toggled-off modem 0 toggled-off modem 1 pencil 0 dishtowel 0 rag 1 soap 1 toothpaste 0 detergent 2 bracelet 1 painting 0 dusty painting 1 painting 2 book 0 book 1 notebook 0 dusty sandal 0 sandal 1 dusty hat 1 cube 1.
You see stained bathtub. In bathtub you can see makeup 1 rag 0 vacuum 0 broom 0 broom 1 shampoo 2.
You see closed toggled-off stained microwave. In microwave you can see cooked salad 1 cooked soup 1.
You see closed toggled-off dusty oven. In oven you can see cooked cracker 2 cooked cake 0.
You see closed toggled-off dishwasher. In dishwasher you can see dusty pan 0 knife 0 spoon 1.
You see closed toggled-on dusty refrigerator. In refrigerator you can see large dusty mug 0 small bowl 2 large tray 0 frozen banana 0 frozen banana 1 frozen carrot 0 frozen carrot 2 frozen beer 0 frozen juice 0 frozen fish 2 frozen sugar 0 frozen tea_bag 0 frozen tea_bag 1 frozen cookie 1 frozen cake 2 frozen sandwich 0 frozen sandwich 2 frozen salad 2 frozen pasta 0 fork 0 knife 1. On tray 0 you can see cake 1 pasta 1.
You see toggled-off stained sink. In sink you can see dusty pan 1 makeup 0 makeup 2 rag 2 hand_towel 0 vacuum 1 scrubbrush 0.
You see stained pool. In pool you can see soap 0 detergent 1 shampoo 0 shampoo 1 dusty tile 0 tile 1.
Human is currently holding nothing. Now it is your turn to help human to achieve the goal!
[Human Trajectory]:
The human agent has taken a list of actions towards a goal, which includes:
Human moves to the cabinet.
Human opens the cabinet.
Human picks up the document 0 at the cabinet.
Human moves to the sofa.
Human opens the box 0 at the sofa.
Human puts the document 0 into the box 0.
Human stops and says,
[Human Instruction]:
’Please give me the dusty one.’
——————————–
Actions:
move to sofa
pick up document 2
move to human
give dusty document 2 to human

B.3 GPT-4+PE+FISER

FISER asks which object the human intends before requesting actions, with a worked explanation for the dusty document. Blue passages mark added instructions. Domain knowledge and the example world reuse the preceding variant.

You are a robot. Every time you will receive a description of the world and an instruction given by the human in the following templates:
[World Description]:
[Human Trajectory]:
[Human Instruction]:
You are expected to generate a sequence of actions that are consistent with the human’s instruction and the world description.
Please remember that each object has a number ID.
Remember in the final output, the type of object should be specific!! you CAN NOT say something like e.g., note !!!!!
Please first think step-by-step!
1. Which object do you think the human is asking for?
2. What are your actions?
Answer the above questions one by one NO MATTER YOU KNOW THE GOAL SPACE OR NOT!
!!!!!THE LAST PART OF THE RESPONSE should be a sequence of actions starting with the string ”Actions:” (!!Only actions! No numbering for actions, no description, no extra words)!!!!!
The available action templates contain:
move to XXX
pick up XXX
pick up XXX from XXX
put XXX into XXX
put XXX onto XXX
take XXX from XXX
give XXX to human
open XXX
A typical answer format is as follows, where the ”open” step is optional.
Actions:
move to XXX
open XXX
pick up XXX
move to human
give XXX to human
[Questions]:
1. Which object do you think the human is asking for?
The human has taken some actions related to the document, and is probably asking for another document to put into the box. The human also requires that the object should be dusty. Note that there is a dusty document 2 on the sofa, so the human is probably asking for document 2.
2. What are your actions?
I should navigate to the sofa and pick up the dusty document 2. Then I should give it to the human.
Actions:
move to sofa
pick up document 2
move to human
give document 2 to human
======================== Now please begin to generate the output for given scenarios ========================
[World Description]: Data
[Human Trajectory]: Data
[Human Instruction]: Data
You are expected to generate a sequence of actions that are consistent with the human’s instruction and the world description.
Please remember that each object has a number ID.
Remember in the final output, the type of object should be specific!! you CAN NOT say something like e.g., note !!!!!
Please first think step-by-step!
1. Which object do you think the human is asking for?
2. What are your actions?
Answer the above questions one by one NO MATTER YOU KNOW THE GOAL SPACE OR NOT!
!!!!!THE LAST PART OF THE RESPONSE should be a sequence of actions starting with the string ”Actions:” (!!Only actions! No numbering for actions, no description, no extra words)!!!!!
======================== Let’s start! ========================

B.4 GPT-4+PE+FISER+PR

Plan recognition additionally asks what the human is doing, including a goal name and identifier when supplied, before object inference and action generation. The complete worked example and current-example template are retained; the source’s wording and formatting are unchanged.

You are a robot. Every time you will receive a description of the world and an instruction given by the human in the following templates:
[World Description]:
[Human Trajectory]:
[Human Instruction]:
You are expected to generate a sequence of actions that are consistent with the human’s instruction and the world description.
Please remember that each object has a number ID.
Remember in the final output, the type of object should be specific!! you CAN NOT say something like e.g., note !!!!!
Please first think step-by-step!
1. What is human doing? (Please find the most possible goal name and goal id if you are given the goal space, otherwise you should infer by yourself, but you STILL need output!)
2. Which object do you think the human is asking for?
3. What are your actions?
Answer the above questions one by one NO MATTER YOU KNOW THE GOAL SPACE OR NOT!
!!!!!THE LAST PART OF THE RESPONSE should be a sequence of actions starting with the string ”Actions:” (!!Only actions! No numbering for actions, no description, no extra words)!!!!!
The available action templates contain:
move to XXX
pick up XXX
pick up XXX from XXX
put XXX into XXX
put XXX onto XXX
take XXX from XXX
give XXX to human
open XXX
A typical answer format is as follows, where the ”open” step is optional.
Actions:
move to XXX
open XXX
pick up XXX
move to human
give XXX to human
[Questions]:
1. What is human doing?
The human has taken some actions related to the document. By putting document 0 into box 0 on the sofa, the human is probably organizing the documents. It aligns with Goal 0: boxing books up for storage.
2. Which object do you think the human is asking for?
Given that the overall goal can be organizing the documents, the human is probably asking for another document to put into the box. The human also requires that the object should be dusty. Note that there is a dusty document 2 on the sofa, so the human is probably asking for document 2.
3. What are your actions?
I should navigate to the sofa and pick up the dusty document 2. Then I should give it to the human.
Actions:
move to sofa
pick up document 2
move to human
give document 2 to human
======================== Now please begin to generate the output for given scenarios ========================
[World Description]: Data
[Human Trajectory]: Data
[Human Instruction]: Data
You are expected to generate a sequence of actions that are consistent with the human’s instruction and the world description.
Please remember that each object has a number ID.
Remember in the final output, the type of object should be specific!! you CAN NOT say something like e.g., note !!!!!
Please first think step-by-step!
1. What is human doing? (Please find the most possible goal name and goal id if you are given the goal space, otherwise you should infer by yourself, but you STILL need output!)
2. Which object do you think the human is asking for?
3. What are your actions?
Answer the above questions one by one NO MATTER YOU KNOW THE GOAL SPACE OR NOT!
!!!!!THE LAST PART OF THE RESPONSE should be a sequence of actions starting with the string ”Actions:” (!!Only actions! No numbering for actions, no description, no extra words)!!!!!
======================== Let’s start! ========================
Concept demonstration

Open image ↗