Literature
Display
← All papers

HandMeThat: Human-Robot Communication in Physical and Social Environments

Yanming Wan · Jiayuan Mao · Joshua B. Tenenbaum

· NeurIPS 2022 Datasets and Benchmarks Track ↗

AI agents10 min text · CondensedOriginal paper ↗
Contents

Overview

The supplement specifies the object ontology, action schemas, goal templates, language and command templates, costs, generation parameters, dataset statistics, baseline details, and a possible clarification interface. It includes the original benchmark and later Version 2 additions. These exact definitions are essential because both the simulated human’s behavior and the test labels come from the generator.

At a glance

HandMeThat tests whether an agent can infer an intended request from household state, a simulated human’s previous actions, and an ambiguous instruction. The evaluated interface is textual and symbolic, so it isolates grounding, goal inference, pragmatic reasoning, and planning from visual perception and physical control. The original release has ten thousand episodes; the revised version adds more data with narrower tasks. Baseline failures demonstrate the difficulty of these generated settings, not an evaluation of unrestricted human–robot conversation.

1 Introduction

An utterance such as asking for an object can be underspecified because its speaker assumes shared knowledge of a larger task. HandMeThat makes that larger goal latent: the robot observes a human-like action prefix, then acts to satisfy the requested subgoal. Large object sets and compositional household goals create ambiguity, while symbolic simulation permits automatic generation and scoring. The demonstrated system uses generated language and plans rather than recorded everyday dialogues or physical robot trials.

Refer to caption
Figure 1: An example HandMeThat task, rendered in images. A robot agent observes a sequence of actions performed by the human (step 1-3), and receives a quest (step 4). The robot needs to interpret the natural language quest based on physical and social context, and select the relevant object from the environment: the knife on the table in this case. Currently, the HandMeThat benchmark is released as a text-only environment.

2 Related Work

The benchmark combines household task structure from BEHAVIOR with goal recognition and pragmatic language interpretation. It differs from direct instruction following by requiring inference about an assisting subgoal, and from pure goal recognition by providing an utterance that may be ambiguous. The paper’s comparison describes contemporary benchmarks and its chosen task scope; using human-annotated goal templates does not itself validate the distribution of generated conversations.

Object
Interaction
Goal Space
(#Templates)
Common-Sense
Goal Prior
Instruction
Interpretation
Social
Reasoning
Pragmatic
Inference
Collaborative
Task Completion
BEHAVIOR [7] ✓ 100 (100) ✓ ✗ ✗ ✗ ✗
ALFRED [6] ✓ 7,000 (7) ✗ ✓ ✗ ✗ ✗
Watch-and-Help [8] ✓ 5 (5) ✗ ✗ ✓ ✗ ✓
CerealBar [9] ✓ 1202* ✗ ✓ ✓ ✗ ✓
ALFworld [10] ✓ 7,000 (7) ✗ ✓ ✗ ✗ ✗
DialFRED [11] ✓ 36,912 (25)* ✗ ✓ ✗ ✗ ✓
TEACh [12] ✓ 7,000 (7) ✗ ✓ ✓ ✗ ✓
Two Body [13] ✓ 1 (1) ✗ ✗ ✗ ✗ ✓
TextWorld [14] ✓ — ✗ ✓ ✗ ✗ ✗
LIGHT [15] ✗ 10,777* ✗ ✓ ✓ ✗ ✗
SCONE [16, 17] ✗ 13,951* ✗ ✓ ✗ ✓ ✗
HandMeThat (ours) ✓ >300k (69) ✓ ✓ ✓ ✓ ✓
Table 1: Comparison between HandMeThat and other related benchmarks. *: indicates the number of instructions or dialogues in that dataset, in contrast to explicit goals/tasks. The numbers in the parenthesis in the third column represents the number of goal templates.

3 The HandMeThat Benchmark

A deterministic symbolic world represents object attributes and spatial relations with Boolean predicates. Human and robot actions have explicit preconditions and effects. Each episode samples a state and hidden human goal, generates a plan prefix, chooses a useful assisting subgoal, and generates an utterance for it. The robot receives observations and must satisfy that subgoal within an action budget. Scoring depends on success and action costs, while the human’s complete goal remains unrevealed.

Original figure
Figure 2: (a) A pipeline for HandMeThat task formulation. Stage 1: Human takes TT steps from initial state s0s_{0} towards a goal GG, giving a trajectory Ω\Omega. At state sTs_{T}, she generates a subgoal mm for robot and utters it as uu. Stage 2: Robot perceives and acts in the world, following the human’s instruction. Evaluation: When the robot stops, we check if human’s quest has been satisfied, and count robot’s action costs to give a final score. (b) A pipeline for HandMeThat data generation. We first sample initial state s0s_{0}, human’s goal GG, and solve a plan for human to execute. At a randomly sampled step TT, the human stops and generates a subgoal, including both the internal mm and utterance uu.

3.1 Intial States, Goals, and Human Trajectories

Scenes sample object instances, attributes, and placements, typically yielding over two hundred entities. Household goals instantiate first-order templates derived from BEHAVIOR-100. The generator plans toward a goal and truncates that trajectory to simulate a request for help. Its rational-planning assumption supplies a useful signal for inverse reasoning, but this is a model of human behavior; expert demonstrations elsewhere use greedy best-first search and are not automatically guaranteed globally optimal.

Original figure
Figure 3: Example of our generated scene, goal, and quest. (a) The sampled initial state. (b) The sampled internal goal from BEHAVIOR-100 templates and the plan. (c) The generated quest in mind based on human’s utility and a corresponding utterance generated using the Rational Speech Acts (RSA) model [41].

3.2 Subgoals and Utterances

Requests are bring-me, move-to, or change-state subgoals. A lifted subgoal describes acceptable objects by properties rather than unique identifiers. Its utility is the expected reduction in the human’s remaining goal cost, averaged over matching grounded subgoals. A Boltzmann distribution trades that utility against description complexity.

The Rational Speech Acts model alternates a listener interpreting meanings and a speaker selecting informative, economical utterances. Utterances omit some specifiers from the intended meaning, then templates turn them into English. Crucially, the generator’s listener prior is conditioned on the true human goal. A tested robot does not receive that goal, so some generated tasks demand information unavailable from its observations. The three original displays retain the utility, subgoal distribution, and recursive speaker–listener definitions.

Equations
UG​(sT,m)=VG∗​(sT)−1|A⁡(m)|​∑mg∈A⁡(m)[VG∗​(γsubgoal​(sT,mg))]\displaystyle U_{G}(s_{T},m)=V^{*}_{G}(s_{T})-\frac{1}{|A(m)|}\sum_{\textit{mg}\in A(m)}\left[V^{*}_{G}\left(\gamma_{\text{subgoal}}(s_{T},\textit{mg})\right)\right]
P⁡(m|sT,G)∝exp⁡[β1​UG​(sT,m)−β2​c​(m)],P(m|s_{T},G)\propto\exp[\beta_{1}U_{G}(s_{T},m)-\beta_{2}\textit{c}(m)],
PL0​(m|u)∝l⁡(u,m)​P​(m);\displaystyle P_{L_{0}}(m|u)\propto l(u,m)P(m);
PSk​(u|m)∝exp⁡[α​USk​(u,m)];\displaystyle P_{S_{k}}(u|m)\propto\exp[\alpha U_{S_{k}}(u,m)];\quad PLk​(m|u)∝PSk​(u|m)​P​(m).\displaystyle P_{L_{k}}(m|u)\propto P_{S_{k}}(u|m)P(m).

3.3 Hardness Levels

Level 1 needs literal grounding: utterance and intended meaning identify the same acceptable subgoals. Level 2 also needs the human goal to eliminate unhelpful groundings. Level 3 requires pragmatic inference under the supplied listener model. Level 4 remains ambiguous even under that model and needs more information. These are generator-based categories, not guarantees of monotonic empirical difficulty; Level 3 can also be unresolvable when the true goal cannot be inferred from the observed action prefix.

3.4 Text-Based Interactive Interface

The Gym-style interface returns text about human actions, the request, and the environment. Commands move, inspect, or manipulate objects. Success yields a reward of one hundred, with a forty-action horizon and action costs. Full observability supplies all object states; partial observability restricts inspection to the current location and requires opening containers. The 250-word vocabulary and templated observations deliberately remove open-ended language and perception difficulties.

3.5 Challenges in HandMeThat Tasks

Successful assistance can require using actions and language jointly, selecting only the relevant subgoal, and exploring to gather missing state information. The paper suggests learning human preferences and integrating probabilistic goal recognition with planning. Those are future directions: the released symbolic distributions provide priors, but do not demonstrate adaptation to an individual human’s unmodeled habits.

4 Experiments

The experiments compare random and privileged heuristic agents with behavior-cloned sequence generation and Q-learning. They report success, total score, and steps among successful episodes. Because information access and stopping rules differ, these are diagnostic baselines rather than a controlled comparison of learning algorithms alone.

4.1 Model Details

The heuristic receives full symbolic state, the logical utterance, and a planner; it selects a single grounding by favoring object categories used in the human’s action history. Seq2Seq learns action strings from planner-generated demonstrations, with optional oracle goal or subgoal inputs. DRRN ranks valid actions by learned Q values, either through online exploration or offline demonstrations. Oracle variants test the value of missing information and are not ordinary inference-time baselines.

Model Partially Observable
Level 1 Level 2 Level 3 Level 4
Human (76%, 5.1) (52%, 4.8) (20%, 5.8) (8%, 5.0)
Random -40.0 (0.0%, N/A) -39.5 (0.4%, 16.0) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Seq2Seq -5.4 (25.5%, 4.2) -25.8 (10.4%, 4.1) -34.7 (3.9%, 4.1) -32.8 (5.3%, 4.1)
   +goal -11.4 (21.0%, 4.1) -28.8 (8.2%, 4.1) -30.0 (7.4%, 4.2) -32.5 (5.5%, 4.0)
   +subgoal -10.1 (22.0%, 4.1) -18.7 (15.7%, 4.1) -24.5 (11.4%, 4.1) -21.9 (13.4%, 4.1)
DRRN -40.0 (0.0%, N/A) -40.0 (0.0%, N/A). -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
   +offline -40.0 (0.0%, N/A) -40.0 (0.0%, N/A). -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Table 2: Experiment results in the partially observable setting. Each model is evaluated on 4 hardness levels with 3 metrics: 1) the average score, 2) the success rate, and 3) the average number of moves in successful episodes.
Model Fully Observable
Level 1 Level 2 Level 3 Level 4
Human (92%, 4.4) (80%, 4.8) (36%, 4.3) (16%, 4.5)
Heuristic 88.9 (94.9%, 4.2) -1.9 (28.1%, 4.4) -23.7 (12.0%, 4.3) -15.9 (17.8%, 4.4)
Random -40.0 (0.0%, N/A) -39.8 (0.1%, 30.0) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Seq2Seq -9.5 (22.4%, 4.0) -30.7 (6.8%, 4.3) -34.6 (4.0%, 4.3) -32.5 (5.5%, 4.1)
   +goal -15.7 (17.9%, 4.3) -25.8 (10.4%, 4.3) -30.5 (7.0%, 4.4) -29.0 (8.0%, 4.0)
   +subgoal -4.9 (25.9%, 4.2) -22.0 (13.2%, 4.2) -21.0 (14.0%, 4.0) -19.3 (15.3%, 4.2)
DRRN -39.8 (0.1%, 5.0) -39.8 (0.1%, 10.0) -39.8 (0.1%, 9.0) -40.0 (0.0%, N/A)
   +offline -40.0 (0.0%, N/A) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Table 3: Experiment results in the fully observable setting. Each model is evaluated on 4 hardness levels with 3 metrics: 1) the average score, 2) the success rate, and 3) the average number of moves in successful episodes.

4.2 Results

In Version 1, Seq2Seq success falls from 22.4% to 4.0% between Levels 1 and 3 under full observability; partial-observation rates are 25.5% and 3.9%. The heuristic reaches 94.9% on unambiguous Level 1 but only 12.0% on Level 3, despite its extra state, logic, and planning access. DRRN is at or near zero. Short successful trajectories mainly reflect frequent pick-and-place templates, and an average over successes does not describe failed episodes.

Human evaluation uses twenty-five episodes per level and observability setting with one chosen object and no clarification. Full-observation success is 92%, 80%, 36%, and 16% across the four levels. Oracle subgoal information often helps learned agents, but goal information does not improve every cell. The authors identify both intended Level 4 ambiguity and unintended Level 3 ambiguity caused by the generator’s true-goal prior. Thus low performance mixes reasoning difficulty, sparse exploration rewards, representational limits, and missing information.

5 HandMeThat Dataset Version 2

Version 2 addresses sparse data and some ambiguities by narrowing the task family, increasing examples, and modifying where human trajectories are interrupted. It changes the task distribution as well as dataset size, so improvements cannot be attributed solely to more training data.

5.1 Revision Details

Version 2 contains 116,146 episodes over twenty-five goal templates, with bring-me requests and pick-and-place goals. Heuristics choose interruption points intended to reveal the relevant subgoal from prior human actions, and a logical subgoal label supplies optional supervision. Hardness counts are 4,008, 78,488, 28,343, and 5,307, creating a strongly uneven distribution. The stated total is about 11.6 times the original ten thousand; the source’s “forty times” claim better matches its approximate per-template increase than the total dataset ratio.

Level 1 Level 2 Level 3 Level 4
Train 3620 70583 25488 4804
Validation 207 3937 1433 247
Test 181 3968 1422 256
Table 4: The distribution of 116,146 instances in HandMeThat (Version 2).

5.2 Experiment Results

Seq2Seq improves on the revised distribution: full-observation success is 30.4%, 28.8%, 12.8%, and 14.8%; partial-observation success is 30.4%, 31.2%, 16.4%, and 12.0%. Oracle subgoals substantially help some settings, reaching 69.1% on fully observed Level 1, while the privileged heuristic reaches 100%, 64%, 39.2%, and 29.2%. Online and offline DRRN remain at zero in the reported Version 2 tables.

Uneven goal frequencies affect difficulty: Level 2 includes many goals that Seq2Seq handles relatively well. Human and heuristic agents can stop early, whereas failed learned agents use their budget, making raw scores sensitive to stopping behavior. The revisions reduce some ambiguities but retain intrinsically ambiguous Level 4 cases; they do not establish general conversational competence.

Model Partially Observable
Level 1 Level 2 Level 3 Level 4
Human (100%, 7.8) (80%, 7.5) (40%, 8.4) (30%, 8.0)
Random -40.0 (0.0%, N/A) -39.5 (0.4%, 16.0) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Seq2Seq 1.3 (30.4%, 4.0) 2.4 (31.2%, 4.0) -17.7 (16.4%, 4.1) -23.7 (12.0%, 4.0)
   +goal 0.6 (29.8%, 4.0) 3.0 (31.6%, 4.0) -9.6 (22.4%, 4.1) -9.6 (22.4%, 4.1)
   +subgoal 0.6 (29.8%, 4.0) 14.4 (40.0%, 4.0) -1.9 (28.0%, 4.1) -1.4 (28.4%, 4.0)
DRRN -40.0 (0.0%, N/A) -40.0 (0.0%, N/A). -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
   +offline -40.0 (0.0%, N/A) -40.0 (0.0%, N/A). -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Table 5: Experiment results for version 2 dataset in the partially observable setting. Each model is evaluated on 4 hardness levels with 3 metrics: 1) the average score, 2) the success rate, and 3) the average number of moves in successful episodes. This table repeats the result of Table 2 on the version 2 dataset.
Model Fully Observable
Level 1 Level 2 Level 3 Level 4
Human (100%, 4.1) (80%, 4.2) (40%, 5.2) (30%, 4.5)
Heuristic 95.8 (100.0%, 4.2) 59.5 (64.0%, 4.1) 34.5 (39.2%, 4.1) 24.4 (29.2%, 4.1)
Random -40.0 (0.0%, N/A) -39.8 (0.1%, 30.0) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Seq2Seq 1.3 (30.4%, 4.0) -0.8 (28.8%, 4.0) -22.6 (12.8%, 4.0) -19.9 (14.8%, 4.0)
   +goal -4.7 (26.0%, 4.0) 7.3 (34.8%, 4.1) -6.9 (24.4%, 4.1) -6.9 (24.4%, 4.1)
   +subgoal 53.8 (69.1%, 4.2) 51.3 (67.2%, 4.1) 1.3 (30.4%, 4.1) 3.0 (31.6%, 4.1)
DRRN -40.0 (0.0%, N/A) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
   +offline -40.0 (0.0%, N/A) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A) -40.0 (0.0%, N/A)
Table 6: Experiment results for Version 2 dataset in the fully observable setting. Each model is evaluated on 4 hardness levels with 3 metrics: 1) the average score, 2) the success rate, and 3) the average number of moves in successful episodes. This table repeats the result of Table 3 on the version 2 dataset.

6 Conclusion

HandMeThat makes physical state, action history, and speaker intent jointly relevant to instruction following. Its textual, templated, deterministic setup omits visual perception, gestures, much of human variability, and sustained dialogue. Visual interfaces, multi-round clarification, and joint information gathering are proposed extensions. The benchmark is useful for studying these controlled reasoning dependencies while its generation assumptions constrain conclusions about real assistance.

Appendix A Task Design

The appendix makes the symbolic generator concrete: object categories and properties, action constraints, initial-state sampling, goal templates, request templates, costs, parameters, and observation strings. All original tables are retained so these assumptions remain inspectable rather than being replaced by a high-level benchmark description.

A.1 Object Space

The ontology has 155 categories grouped into locations, receptacles, food, tools, and things. Size and color are static; eight Boolean properties describe changeable state. Only direct inside/on-top relations are stored, avoiding redundant updates when containers move. Sampling creates each fixed location, up to three objects per movable category, admissible placements, and specified state distributions. Tables 7–11 retain the hierarchy, meta-properties, action semantics, constraints, and placement rules; the complete preconditions and effects reside in the released code.

Class Subclass Category
location (has-ontop)
floor, countertop, sofa, bed,
stove, table, shelf, toilet
(has-inside)
cabinet, bathtub, microwave, oven,
dishwasher, refrigerator, sink, pool
receptacle furniture highchair, chair, seat
vessel bottle, jar, kettle, caldron
tableware bowl, mug, plate, dish, cup
utensil saucepan, pan, casserole
bag duffel bag, sack, backpack, briefcase
bucket bucket
tray tray
basket basket
box box
package package
ashcan ashcan
Xmas stocking Xmas stocking
Xmas tree Xmas tree
food fruit
apple, banana, melon, grape, lemon, orange, peach,
strawberry, raspberry, date, olive, chestnut
vegetable
carrot, radish, tomato, broccoli,
mushroom, onion, lettuce, pumpkin
drink pop, beer, juice, water, milk
protein beef, chicken, pork, fish, egg
flavorer catsup, sauce, parsley, tea bag, suger, vegetable oil
baked food cracker, bread, cookie, cake
snack chip, hamburger, sandwich, candy
prepared food oatmeal, sushi, salad, soup, pasta
tool metal tool carving knife, hammer, screwdriver, scraper, saw
electric equipment printer, scanner, facsimile, modem
electrical device calculator, headset, earphone, mouse, alarm
toiletry toothbrush, perfume, makeup
writing tool highlighter, marker, pen, pencil
piece of cloth dishtower, hand towel, rag
cleaning tool scrub brush, broom, vacuum
cleansing soap, shampoo, detergent, toothpaste
cutlery fork, spoon, knife
illumination tool lamp, candle
thing decoration necklace, bracelet, jewelry, bow, wreath, ribbon
paper product
hardback, notebook, book, newspaper,
painting, pad, document
footwear gym shoe, sandal, shoe, sock
headwear hat, sunglass
clothing shirt, sweater, underwear, apparel
building materials tile, plywood
plaything cube, ball
Table 7: Object hierarchy designed for HandMeThat based on BEHAVIOR-100.
Meta-Properties Categories
has-inside
cabinet, bathtub, microwave, oven, dishwasher, refrigerator, sink, pool,
vessel, tableware, utensil, bag, basket, box, package, ashcan, bucket, Xmas stocking
has-ontop
floor, countertop, sofa, bed, stove, table, shelf, toilet
furniture, tray, Xmas tree
has-size tableware tray box package ashcan
has-color furniture vessel bag basket box package
openable cabinet, microwave, oven, dishwasher, refrigerator, vessel, bag, box, package
toggleable microwave, oven dishwasher, refrigerator, stove, sink, electric equipment
cookable food
freezable food
sliceable fruit, vegetable, protein
dustyable location, receptacle, thing
stainable location
soakable piece of cloth, clothing
Table 8: All meta-properties in HandMeThat and their corresponding categories. “has-inside” and “has-ontop” indicate whether the objects in this category can hold other objects in the corresponding relation; “has-size” and “has-color” indicate whether the objects in this category will be assigned with size or color; the remaining properties (X-able) indicate whether objects in this category can be manipulated through corresponding verb. The category list contains class/subclass names: all categories derived from the listed class/subclass are associated with the corresponding meta-property.
Action Name and Parameters Semantics
Action: move
Args: ?a - agent ?from - location ?to - location
Agent [?a] moves from [?from] to [?to].
Action: pick-up-at-loc
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] picks up [?obj] at [?loc].
Action: pick-up-from-rec-at-loc
Args: ?a - agent ?obj - object ?rec - receptacle ?loc - location
Agent [?a] picks up [?obj] from [?rec] at [?loc].
Action: put-inside-loc
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] puts [?obj] into [?loc].
Action: put-ontop-loc
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] puts [?obj] onto [?loc].
Action: put-inside-rec-at-loc
Args: ?a - agent ?obj - object ?rec - receptacle ?loc - location
Agent [?a] puts [?obj] into [?rec] at [?loc].
Action: put-ontop-loc-at-loc
Args: ?a - agent ?obj - object ?rec - receptacle ?loc - location
Agent [?a] puts [?obj] onto [?rec] at [?loc].
Action: open-loc
Args: ?a - agent ?loc - location
Agent [?a] opens [?loc].
Action: close-loc
Args: ?a - agent ?loc - location
Agent [?a] closes [?loc].
Action: open-rec-at-loc
Args: ?a - agent ?rec - receptacle ?loc - location
Agent [?a] opens [?rec] at [?loc].
Action: close-rec-at-loc
Args: ?a - agent ?rec - receptacle ?loc - location
Agent [?a] closes [?rec] at [?loc].
Action: toggle-on-loc
Args: ?a - agent ?loc - location
Agent [?a] toggles on [?loc].
Action: toggle-off-loc
Args: ?a - agent ?loc - location
Agent [?a] toggles off [?loc].
Action: toggle-on-obj-at-loc
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] toggles on [?obj] at [?loc].
Action: toggle-off-obj-at-loc
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] toggles off [?obj] at [?loc].
Action: heat-obj
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] heats up [?obj] at [?loc].
Action: cool-obj
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] cools down [?obj] at [?loc].
Action: soak-obj
Args: ?a - agent ?obj - object ?loc - location
Agent [?a] makes [?obj] soaked at [?loc].
Action: slice-obj
Args: ?a - agent ?obj - object ?tool - tool ?loc - location
Agent [?a] slices up [?obj] with [?tool] at [?loc].
Action: clean-obj-at-loc
Args: ?a - agent ?obj - object ?tool - tool ?loc - location
Agent [?a] cleans up [?obj] with [?tool] at [?loc].
Action: clean-loc
Args: ?a - agent ?tool - tool ?loc - location
Agent [?a] cleans up [?loc] with [?tool].
Action: bring-to-human
Args: ?r - robot ?obj - object ?h - human
Robot [?r] brings [?obj] to human [?h].
Action: take-from-human
Args: ?r - robot ?obj - object ?h - human
Robot [?r] takes [?obj] from human [?h].
Table 9: All actions in HandMeThat. If the action schema has an argument of type “agent,” the action can be executed by either the human or the robot.
Action Name Constraints
pick-up
put-inside
The receptacle/location must be open if it’s openable.
heat The agent can only heat things at microwave, oven and stove.
cool The agent can only cool things at refrigerator.
soak The agent can only soak things at sink.
slice The agent can only slice things with knife.
clean
The agent can only clean things with suitable cleaning tool
within rag, dishtowel, hand towel, scrub brush, vacuum, broom.
Table 10: Action constraints in HandMeThat.
Category Valid Positions
furniture floor
vessel countertop table cabinet
tableware countertop table cabinet dishwasher refrigerator sink
utensil countertop table cabinet dishwasher refrigerator sink
bag floor countertop table sofa bed
bucket floor countertop table
tray countertop table cabinet refrigerator
basket floor countertop table shelf cabinet sofa bed
box floor countertop table shelf cabinet sofa bed
package floor countertop table shelf cabinet sofa bed
ashcan floor
xmas tree floor
xmas stocking floor countertop table shelf cabinet sofa bed
fruit table countertop refrigerator utensil
vegetable table countertop refrigerator stove utensil
drink table countertop refrigerator cabinet bag
protein table countertop refrigerator stove utensil
flavorer table countertop refrigerator cabinet bag
baked food table countertop refrigerator oven tray
snack table countertop refrigerator microwave tray
prepared food table countertop refrigerator microwave tray
metal tool countertop table cabinet shelf furniture
electric equipment countertop table cabinet shelf furniture
electrical device countertop table cabinet shelf furniture
toiletry cabinet toilet bathtub sink pool bag
writing tool countertop table cabinet shelf bag
piece of cloth cabinet toilet bathtub sink pool bucket
cleaning tool cabinet toilet bathtub sink pool bucket
cleansing cabinet toilet bathtub sink pool bucket
cutlery countertop table cabinet dishwasher refrigerator utensil
illumination tool countertop table sofa bed shelf
decoration cabinet sofa bed package
paper product cabinet sofa bed package
footwear cabinet floor
headwear cabinet sofa bed package
clothing cabinet sofa bed package
building materials pool
plaything cabinet sofa bed package
Table 11: Valid positions for each category when sampling the initial state.

A.2 Goal Space

The original goal space uses sixty-nine first-order templates after removing duplicate or incompatible BEHAVIOR tasks. Category substitution creates many concrete goals, but the logical vocabulary and action model remain supplied. Tables 12–16 preserve every template. Version 2 selects twenty-five goal names and restricts the manipulation setting, trading task breadth for more samples and easier control of ambiguity.

Goal Name First-Order Logic Formula
assembling gift baskets ∀y,∃x,?[illumination tool]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[illumination tool]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[snack]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[snack]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[baked food]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[baked food]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[decoration]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[decoration]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
bottling fruit ∃y1,y2,jar​(y1),jar​(y2),…\exists y_{1},y_{2},\text{jar}(y_{1}),\text{jar}(y_{2}),\ldots
   ∀x,?[fruit]1​(x)⇒in​(x,y1)\forall x,\text{?[fruit]}_{1}(x)\Rightarrow\text{in}(x,y_{1})
   ∀x,?[fruit]2​(x)⇒in​(x,y2)\forall x,\text{?[fruit]}_{2}(x)\Rightarrow\text{in}(x,y_{2})
boxing books up for storage ∃y,box​(y),∀x,?[paper product]​(x)⇒in​(x,y)\exists y,\text{box}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{in}(x,y)
bringing in wood ∃y,floor​(y),∀x,?[building material]​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{?[building material]}(x)\Rightarrow\text{on}(x,y)
brushing lint off clothing ∃y,bed​(y),∀x,?[clothing]​(x)⇒on​(x,y)∧¬dusty​(x)\exists y,\text{bed}(y),\forall x,\text{?[clothing]}(x)\Rightarrow\text{on}(x,y)\land\neg\text{dusty}(x)
chopping vegetables ∃y,?[tableware]​(y),∀x,?[fruit]1​(x)⇒in​(x,y)∧sliced​(x)\exists y,\text{?[tableware]}(y),\forall x,\text{?[fruit]}_{1}(x)\Rightarrow\text{in}(x,y)\land\text{sliced}(x)
∃y,?[tableware]​(y),∀x,?[fruit]2​(x)⇒in​(x,y)∧sliced​(x)\exists y,\text{?[tableware]}(y),\forall x,\text{?[fruit]}_{2}(x)\Rightarrow\text{in}(x,y)\land\text{sliced}(x)
∃y,?[tableware]​(y),∀x,?[vegetable]1​(x)⇒in​(x,y)∧sliced​(x)\exists y,\text{?[tableware]}(y),\forall x,\text{?[vegetable]}_{1}(x)\Rightarrow\text{in}(x,y)\land\text{sliced}(x)
∃y,?[tableware]​(y),∀x,?[vegetable]2​(x)⇒in​(x,y)∧sliced​(x)\exists y,\text{?[tableware]}(y),\forall x,\text{?[vegetable]}_{2}(x)\Rightarrow\text{in}(x,y)\land\text{sliced}(x)
cleaning bathrooms ∀x,toilet​(x)⇒¬stained​(x)\forall x,\text{toilet}(x)\Rightarrow\neg\text{stained}(x)
∀x,bathtub​(x)⇒¬stained​(x)\forall x,\text{bathtub}(x)\Rightarrow\neg\text{stained}(x)
∀x,sink​(x)⇒¬stained​(x)\forall x,\text{sink}(x)\Rightarrow\neg\text{stained}(x)
∀x,floor​(x)⇒¬stained​(x)\forall x,\text{floor}(x)\Rightarrow\neg\text{stained}(x)
∃y,bucket​(y),∃x,rag​(x)∧in​(x,y)∧soaked​(x)\exists y,\text{bucket}(y),\exists x,\text{rag}(x)\land\text{in}(x,y)\land\text{soaked}(x)
cleaning bedrooms ∃y,cabinet​(y),∀x,?[clothing]​(x)⇒in​(x,y)∧¬dusty​(y)\exists y,\text{cabinet}(y),\forall x,\text{?[clothing]}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(y)
∃y,cabinet​(y),∀x,?[decoration]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[decoration]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[toiletry]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[toiletry]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[paper product]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{in}(x,y)
cleaning closet ∃y,cabinet​(y),∀x,?[decoration]​(x)⇒in​(x,y)∧¬dusty​(y)\exists y,\text{cabinet}(y),\forall x,\text{?[decoration]}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(y)
∃y,shelf​(y),∀x,?[headwear]​(x)⇒on​(x,y)∧¬dusty​(y)\exists y,\text{shelf}(y),\forall x,\text{?[headwear]}(x)\Rightarrow\text{on}(x,y)\land\neg\text{dusty}(y)
∃y,floor​(y),∀x,?[footwear]​(x)⇒on​(x,y)∧¬dusty​(y)\exists y,\text{floor}(y),\forall x,\text{?[footwear]}(x)\Rightarrow\text{on}(x,y)\land\neg\text{dusty}(y)
cleaning floors ∀x,floor​(x)⇒¬stained​(x)∧¬dusty​(x)\forall x,\text{floor}(x)\Rightarrow\neg\text{stained}(x)\land\neg\text{dusty}(x)
cleaning garage ∀x,floor​(x)⇒¬dusty​(x)\forall x,\text{floor}(x)\Rightarrow\neg\text{dusty}(x)
∀x,cabinet​(x)⇒¬stained​(x)\forall x,\text{cabinet}(x)\Rightarrow\neg\text{stained}(x)
∀x,cabinet​(x)⇒¬dusty​(x)\forall x,\text{cabinet}(x)\Rightarrow\neg\text{dusty}(x)
∃y,ashcan​(y),∀x,?[paper product]​(x)⇒in​(x,y)\exists y,\text{ashcan}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{in}(x,y)
∃y,table​(y),∀x,?[vessel]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[vessel]}(x)\Rightarrow\text{on}(x,y)
cleaning high chair ∀x,?[furniture]​(x)⇒¬dusty​(x)\forall x,\text{?[furniture]}(x)\Rightarrow\neg\text{dusty}(x)
cleaning kitchen cupboard ∀x,cabinet​(x)⇒¬dusty​(x)\forall x,\text{cabinet}(x)\Rightarrow\neg\text{dusty}(x)
∃y,cabinet​(y),∀x,?[tableware]1​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[tableware]}_{1}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[tableware]2​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[tableware]}_{2}(x)\Rightarrow\text{in}(x,y)
cleaning microwave oven ∀x,microwave​(x)⇒¬stained​(x)∧¬dusty​(x)\forall x,\text{microwave}(x)\Rightarrow\neg\text{stained}(x)\land\neg\text{dusty}(x)
cleaning out drawers ∃y,sink​(y),∀x,?[piece of cloth]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[piece of cloth]}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,?[tableware]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[tableware]}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,?[cutlery]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[cutlery]}(x)\Rightarrow\text{in}(x,y)
cleaning oven ∀x,oven​(x)⇒¬stained​(x)\forall x,\text{oven}(x)\Rightarrow\neg\text{stained}(x)
∃x,rag​(x)∧soaked​(x)\exists x,\text{rag}(x)\land\text{soaked}(x)
cleaning shoes ∃y,floor​(y),∀x,?[footwear]​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{?[footwear]}(x)\Rightarrow\text{on}(x,y)
∃y,floor​(y),∀x,?[footwear]​(x)⇒on​(x,y)∧¬dusty​(x)\exists y,\text{floor}(y),\forall x,\text{?[footwear]}(x)\Rightarrow\text{on}(x,y)\land\neg\text{dusty}(x)
cleaning stove ∀x,stove​(x)⇒¬stained​(x)∧¬dusty​(x)\forall x,\text{stove}(x)\Rightarrow\neg\text{stained}(x)\land\neg\text{dusty}(x)
∀x,rag​(x)⇒soaked​(x)\forall x,\text{rag}(x)\Rightarrow\text{soaked}(x)
∀x,dishtowel​(x)⇒soaked​(x)\forall x,\text{dishtowel}(x)\Rightarrow\text{soaked}(x)
cleaning table after clearing ∀x,table​(x)⇒¬stained​(x)\forall x,\text{table}(x)\Rightarrow\neg\text{stained}(x)
cleaning the pool ∀x,pool​(x)⇒¬stained​(x)\forall x,\text{pool}(x)\Rightarrow\neg\text{stained}(x)
∃y,shelf​(y),∀x,scrub brush​(x)⇒on​(x,y)\exists y,\text{shelf}(y),\forall x,\text{scrub brush}(x)\Rightarrow\text{on}(x,y)
∃y,floor​(y),∀x,cleansing​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{cleansing}(x)\Rightarrow\text{on}(x,y)
Table 12: The goal names are collected from original BEHAVIOR-100 tasks, and the formulas are the rewritten version for original goal predicates.
Goal Name First-Order Logic Formula
cleaning toilet ∀x,toilet​(x)⇒¬stained​(x)\forall x,\text{toilet}(x)\Rightarrow\neg\text{stained}(x)
∃y,floor​(y),∀x,scrub brush​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{scrub brush}(x)\Rightarrow\text{on}(x,y)
∃y,floor​(y),∀x,cleansing​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{cleansing}(x)\Rightarrow\text{on}(x,y)
cleaning up after a meal ∀x,table​(x)⇒¬stained​(x)\forall x,\text{table}(x)\Rightarrow\neg\text{stained}(x)
∀x,floor​(x)⇒¬stained​(x)\forall x,\text{floor}(x)\Rightarrow\neg\text{stained}(x)
∀x,?[furniture]​(x)⇒¬dusty​(x)\forall x,\text{?[furniture]}(x)\Rightarrow\neg\text{dusty}(x)
∀x,?[tableware]1​(x)⇒¬dusty​(x)\forall x,\text{?[tableware]}_{1}(x)\Rightarrow\neg\text{dusty}(x)
∀x,?[tableware]2​(x)⇒¬dusty​(x)\forall x,\text{?[tableware]}_{2}(x)\Rightarrow\neg\text{dusty}(x)
∀x,?[tableware]3​(x)⇒¬dusty​(x)\forall x,\text{?[tableware]}_{3}(x)\Rightarrow\neg\text{dusty}(x)
∃y,?[bag]​(y),∀x,?[snack]​(x)⇒in​(x,y)\exists y,\text{?[bag]}(y),\forall x,\text{?[snack]}(x)\Rightarrow\text{in}(x,y)
cleaning up refrigerator ∃y,sink​(y),∀x,rag​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{rag}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,cleansing​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{cleansing}(x)\Rightarrow\text{in}(x,y)
∃y,refrigerator​(y),∀x,tray​(x)⇒in​(x,y)∧¬dusty​(x)∧¬stained​(y)\exists y,\text{refrigerator}(y),\forall x,\text{tray}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)\land\neg\text{stained}(y)
∃y,sink​(y),∀x,?[tableware]​(x)⇒in​(x,y)∧¬dusty​(x)\exists y,\text{sink}(y),\forall x,\text{?[tableware]}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)
cleaning up the kitchen only ∃y,countertop​(y),∀x,?[vessel]​(x)⇒on​(x,y)\exists y,\text{countertop}(y),\forall x,\text{?[vessel]}(x)\Rightarrow\text{on}(x,y)
∃y,sink​(y),∀x,?[cleansing]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[cleansing]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[flavorer]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[flavorer]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[tableware]​(x)⇒in​(x,y)∧¬dusty​(x)∧¬dusty​(y)\exists y,\text{cabinet}(y),\forall x,\text{?[tableware]}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)\land\neg\text{dusty}(y)
∃y,sink​(y),∀x,rag​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{rag}(x)\Rightarrow\text{in}(x,y)
∃y,refrigerator​(y),∀x,?[utensil]​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[utensil]}(x)\Rightarrow\text{in}(x,y)
∃y,refrigerator​(y),∀x,?[fruit]​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[fruit]}(x)\Rightarrow\text{in}(x,y)
clearing the table after dinner ∃y1,bucket​(y1),∀x,?[cutlery]1​(x)⇒in​(x,y1)\exists y_{1},\text{bucket}(y_{1}),\forall x,\text{?[cutlery]}_{1}(x)\Rightarrow\text{in}(x,y_{1})
∃y2,bucket​(y2),∀x,?[cutlery]2​(x)⇒in​(x,y2)\exists y_{2},\text{bucket}(y_{2}),\forall x,\text{?[cutlery]}_{2}(x)\Rightarrow\text{in}(x,y_{2})
∃y,bucket​(y),∀x,?[flavorer]​(x)⇒in​(x,y)\exists y,\text{bucket}(y),\forall x,\text{?[flavorer]}(x)\Rightarrow\text{in}(x,y)
collect misplaced items ∃y,table​(y),∀x,?[footwear]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[footwear]}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[decoration]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[decoration]}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[paper product]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{on}(x,y)
collecting aluminum cans ∃y,ashcan​(y),∀x,?[drink]​(x)⇒in​(x,y)\exists y,\text{ashcan}(y),\forall x,\text{?[drink]}(x)\Rightarrow\text{in}(x,y)
filling a Christmas stocking ∀y,∃x,?[plaything]​(x)∧(Xmas stocking​(y)⇒in​(x,y))\forall y,\exists x,\text{?[plaything]}(x)\land(\text{Xmas stocking}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[snack]​(x)∧(Xmas stocking​(y)⇒in​(x,y))\forall y,\exists x,\text{?[snack]}(x)\land(\text{Xmas stocking}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[writing tool]​(x)∧(Xmas stocking​(y)⇒in​(x,y))\forall y,\exists x,\text{?[writing tool]}(x)\land(\text{Xmas stocking}(y)\Rightarrow\text{in}(x,y))
filling a Easter basket ∃y,countertop​(y),∀x,?[basket]​(x)⇒on​(x,y)\exists y,\text{countertop}(y),\forall x,\text{?[basket]}(x)\Rightarrow\text{on}(x,y)
∀y,∃x,?[protein]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[protein]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[snack]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[snack]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[paper product]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[paper product]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∀y,∃x,?[plaything]​(x)∧(basket​(y)⇒in​(x,y))\forall y,\exists x,\text{?[plaything]}(x)\land(\text{basket}(y)\Rightarrow\text{in}(x,y))
∃y,basket​(y),∀x,?[decoration]​(x)⇒in​(x,y)\exists y,\text{basket}(y),\forall x,\text{?[decoration]}(x)\Rightarrow\text{in}(x,y)
installing a fax machine ∃y,table(y),∀x,?[electric equipment](x)⇒on(x,y))∧toggled(x)\exists y,\text{table}(y),\forall x,\text{?[electric equipment]}(x)\Rightarrow\text{on}(x,y))\land\text{toggled}(x)
installing alarms ∀y,∃x,?[electrical device]​(x)∧(table​(y)⇒on​(x,y))\forall y,\exists x,\text{?[electrical device]}(x)\land(\text{table}(y)\Rightarrow\text{on}(x,y))
∀y,∃x,?[electrical device]​(x)∧(countertop​(y)⇒on​(x,y))\forall y,\exists x,\text{?[electrical device]}(x)\land(\text{countertop}(y)\Rightarrow\text{on}(x,y))
∀y,∃x,?[electrical device]​(x)∧(sofa​(y)⇒on​(x,y))\forall y,\exists x,\text{?[electrical device]}(x)\land(\text{sofa}(y)\Rightarrow\text{on}(x,y))
laying tile floors ∃y,floor​(y),∀x,?[building material]​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{?[building material]}(x)\Rightarrow\text{on}(x,y)
loading the dishwasher ∃y,sink​(y),∀x,?[tableware]1​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[tableware]}_{1}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,?[tableware]2​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[tableware]}_{2}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,?[vessel]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[vessel]}(x)\Rightarrow\text{in}(x,y)
moving boxes to storage ∃y,floor​(y),∀x,box​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{box}(x)\Rightarrow\text{on}(x,y)
opening packages ∀x,package​(x)⇒open​(x)\forall x,\text{package}(x)\Rightarrow\text{open}(x)
organizing boxes in garage ∃y,floor​(y),∀x,box​(x)⇒on​(x,y)\exists y,\text{floor}(y),\forall x,\text{box}(x)\Rightarrow\text{on}(x,y)
∃y1,y2,y3,box​(y1),box​(y2),box​(y3),…\exists y_{1},y_{2},y_{3},\text{box}(y_{1}),\text{box}(y_{2}),\text{box}(y_{3}),\ldots
   ∀x,?[plaything]​(x)⇒in​(x,y1)\forall x,\text{?[plaything]}(x)\Rightarrow\text{in}(x,y_{1})
   ∀x,?[cutlery]​(x)⇒in​(x,y2)\forall x,\text{?[cutlery]}(x)\Rightarrow\text{in}(x,y_{2})
   ∀x,?[cleansing]​(x)⇒in​(x,y3)\forall x,\text{?[cleansing]}(x)\Rightarrow\text{in}(x,y_{3})
Table 13: (Cont’d.) The goal names are collected from original BEHAVIOR-100 tasks, and the formulas are the rewritten version for original goal predicates.
Goal Name First-Order Logic Formula
organizing file cabinet ∃y,table​(y),∀x,?[writing tool]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[writing tool]}(x)\Rightarrow\text{on}(x,y)
∃y,cabinet​(y),∀x,?[paper product]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{in}(x,y)
organizing school stuff ∃y,backpack​(y),…\exists y,\text{backpack}(y),\ldots
   ∃x,?[paper product]​(x)∧in​(x,y)\exists x,\text{?[paper product]}(x)\land\text{in}(x,y)
   ∃x,?[writing tool]1​(x)∧in​(x,y)\exists x,\text{?[writing tool]}_{1}(x)\land\text{in}(x,y)
   ∃x,?[writing tool]2​(x)∧in​(x,y)\exists x,\text{?[writing tool]}_{2}(x)\land\text{in}(x,y)
   ∃x,?[electrical device]​(x)∧in​(x,y)\exists x,\text{?[electrical device]}(x)\land\text{in}(x,y)
packing adult’s bag ∃y,backpack​(y),…\exists y,\text{backpack}(y),\ldots
   ∀x,?[decoration]​(x)⇒in​(x,y)\forall x,\text{?[decoration]}(x)\Rightarrow\text{in}(x,y)
   ∃x,?[toiletry]1​(x)∧in​(x,y)\exists x,\text{?[toiletry]}_{1}(x)\land\text{in}(x,y)
   ∃x,?[toiletry]2​(x)∧in​(x,y)\exists x,\text{?[toiletry]}_{2}(x)\land\text{in}(x,y)
   ∃x,?[electrical device]​(x)∧in​(x,y)\exists x,\text{?[electrical device]}(x)\land\text{in}(x,y)
packing bags or suitcase ∃y,briefcase​(y),…\exists y,\text{briefcase}(y),\ldots
   ∀x,?[clothing]​(x)⇒in​(x,y)\forall x,\text{?[clothing]}(x)\Rightarrow\text{in}(x,y)
   ∃x,?[toiletry]​(x)∧in​(x,y)\exists x,\text{?[toiletry]}(x)\land\text{in}(x,y)
   ∃x,?[cleansing]1​(x)∧in​(x,y)\exists x,\text{?[cleansing]}_{1}(x)\land\text{in}(x,y)
   ∃x,?[cleansing]2​(x)∧in​(x,y)\exists x,\text{?[cleansing]}_{2}(x)\land\text{in}(x,y)
   ∃x,?[paper product]​(x)∧in​(x,y)\exists x,\text{?[paper product]}(x)\land\text{in}(x,y)
packing boxes for household move or trip ∃y1,y2,box​(y1),box​(y2),…\exists y_{1},y_{2},\text{box}(y_{1}),\text{box}(y_{2}),\ldots
   ∀x,?[cutlery]​(x)⇒in​(x,y1)\forall x,\text{?[cutlery]}(x)\Rightarrow\text{in}(x,y_{1})
   ∃x,?[piece of cloth]​(x)∧in​(x,y1)\exists x,\text{?[piece of cloth]}(x)\land\text{in}(x,y_{1})
   ∀x,book​(x)⇒in​(x,y2)\forall x,\text{book}(x)\Rightarrow\text{in}(x,y_{2})
   ∃x,?[clothing]​(x)∧in​(x,y2)\exists x,\text{?[clothing]}(x)\land\text{in}(x,y_{2})
packing child’s bag ∃y,backpack​(y),…\exists y,\text{backpack}(y),\ldots
   ∃x,?[headwear]​(x)∧in​(x,y)\exists x,\text{?[headwear]}(x)\land\text{in}(x,y)
   ∃x,?[decoration]​(x)∧in​(x,y)\exists x,\text{?[decoration]}(x)\land\text{in}(x,y)
   ∃x,?fruit]​(x)∧in​(x,y)\exists x,\text{?fruit]}(x)\land\text{in}(x,y)
   ∃x,?[paper product]​(x)∧in​(x,y)\exists x,\text{?[paper product]}(x)\land\text{in}(x,y)
   ∃x,?[electrical device]​(x)∧in​(x,y)\exists x,\text{?[electrical device]}(x)\land\text{in}(x,y)
packing box for work ∃y,box​(y),…\exists y,\text{box}(y),\ldots
   ∃x,?[fruit]​(x)∧in​(x,y)\exists x,\text{?[fruit]}(x)\land\text{in}(x,y)
   ∃x,?[drink]​(x)∧in​(x,y)\exists x,\text{?[drink]}(x)\land\text{in}(x,y)
   ∃x,?[snack]1​(x)∧in​(x,y)\exists x,\text{?[snack]}_{1}(x)\land\text{in}(x,y)
   ∃x,?[snack]2​(x)∧in​(x,y)\exists x,\text{?[snack]}_{2}(x)\land\text{in}(x,y)
packing lunches ∃y1,y2,box​(y1),box​(y2),…\exists y_{1},y_{2},\text{box}(y_{1}),\text{box}(y_{2}),\ldots
   ∃x,?[snack]1​(x)∧in​(x,y1)\exists x,\text{?[snack]}_{1}(x)\land\text{in}(x,y_{1})
   ∃x,?[snack]1​(x)∧in​(x,y2)\exists x,\text{?[snack]}_{1}(x)\land\text{in}(x,y_{2})
   ∃x,?[baked food]1​(x)∧in​(x,y1)\exists x,\text{?[baked food]}_{1}(x)\land\text{in}(x,y_{1})
   ∃x,?[baked food]1​(x)∧in​(x,y2)\exists x,\text{?[baked food]}_{1}(x)\land\text{in}(x,y_{2})
   ∃x,?[prepared food]​(x)∧in​(x,y1)\exists x,\text{?[prepared food]}(x)\land\text{in}(x,y_{1})
   ∃x,?[snack]2​(x)∧in​(x,y2)\exists x,\text{?[snack]}_{2}(x)\land\text{in}(x,y_{2})
   ∃x,?[drink]1​(x)∧in​(x,y1)\exists x,\text{?[drink]}_{1}(x)\land\text{in}(x,y_{1})
   ∃x,?[drink]2​(x)∧in​(x,y2)\exists x,\text{?[drink]}_{2}(x)\land\text{in}(x,y_{2})
   ∃x,?[fruit]1​(x)∧in​(x,y1)\exists x,\text{?[fruit]}_{1}(x)\land\text{in}(x,y_{1})
   ∃x,?[fruit]2​(x)∧in​(x,y2)\exists x,\text{?[fruit]}_{2}(x)\land\text{in}(x,y_{2})
packing picnics ∃y1,y2,y3,box​(y1),box​(y2),box​(y3),…\exists y_{1},y_{2},y_{3},\text{box}(y_{1}),\text{box}(y_{2}),\text{box}(y_{3}),\ldots
   ∃x,?[snack]1​(x)∧in​(x,y1)\exists x,\text{?[snack]}_{1}(x)\land\text{in}(x,y_{1})
   ∃x,?[snack]2​(x)∧in​(x,y1)\exists x,\text{?[snack]}_{2}(x)\land\text{in}(x,y_{1})
   ∃x,?[fruit]1​(x)∧in​(x,y2)\exists x,\text{?[fruit]}_{1}(x)\land\text{in}(x,y_{2})
   ∃x,?[fruit]2​(x)∧in​(x,y2)\exists x,\text{?[fruit]}_{2}(x)\land\text{in}(x,y_{2})
   ∃x,?[fruit]3​(x)∧in​(x,y2)\exists x,\text{?[fruit]}_{3}(x)\land\text{in}(x,y_{2})
   ∃x,?[drink]1​(x)∧in​(x,y3)\exists x,\text{?[drink]}_{1}(x)\land\text{in}(x,y_{3})
   ∃x,?[drink]2​(x)∧in​(x,y3)\exists x,\text{?[drink]}_{2}(x)\land\text{in}(x,y_{3})
   ∃x,?[drink]3​(x)∧in​(x,y3)\exists x,\text{?[drink]}_{3}(x)\land\text{in}(x,y_{3})
picking up take out food ∃y,box​(y),∀x,?[prepared food]​(x)⇒in​(x,y)\exists y,\text{box}(y),\forall x,\text{?[prepared food]}(x)\Rightarrow\text{in}(x,y)
∃y,box​(y),∀x,?[snack]​(x)⇒in​(x,y)\exists y,\text{box}(y),\forall x,\text{?[snack]}(x)\Rightarrow\text{in}(x,y)
∃y,floor​(y),∀x,?[box]​(x)⇒on​(x,y)∧open​(x)\exists y,\text{floor}(y),\forall x,\text{?[box]}(x)\Rightarrow\text{on}(x,y)\land\text{open}(x)
Table 14: (Cont’d.) The goal names are collected from original BEHAVIOR-100 tasks, and the formulas are the rewritten version for original goal predicates.
Goal Name First-Order Logic Formula
picking up trash ∃y,ashcan​(y),∀x,?[paper product]​(x)⇒in​(x,y)\exists y,\text{ashcan}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{in}(x,y)
∃y,ashcan​(y),∀x,?[drink]​(x)⇒in​(x,y)\exists y,\text{ashcan}(y),\forall x,\text{?[drink]}(x)\Rightarrow\text{in}(x,y)
polishing furniture ∀x,table​(x)⇒¬dusty​(x)\forall x,\text{table}(x)\Rightarrow\neg\text{dusty}(x)
∀x,shelf​(x)⇒¬dusty​(x)\forall x,\text{shelf}(x)\Rightarrow\neg\text{dusty}(x)
polishing shoes ∃y,sink​(y),∀x,rag​(x)⇒in​(x,y)∧soaked​(x)\exists y,\text{sink}(y),\forall x,\text{rag}(x)\Rightarrow\text{in}(x,y)\land\text{soaked}(x)
∀x,?[footwear]​(x)⇒¬dusty​(x)\forall x,\text{?[footwear]}(x)\Rightarrow\neg\text{dusty}(x)
polishing silver ∃y,cabinet​(y),∀x,?[cutlery]​(x)⇒in​(x,y)∧¬dusty​(x)\exists y,\text{cabinet}(y),\forall x,\text{?[cutlery]}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)
∃y,cabinet​(y),∀x,rag​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{rag}(x)\Rightarrow\text{in}(x,y)
preparing food ∃y1,y2,plate​(y1),plate​(y2),…\exists y_{1},y_{2},\text{plate}(y_{1}),\text{plate}(y_{2}),\ldots
   ∃x,?[vegetable]1​(x)⇒in​(x,y1)\exists x,\text{?[vegetable]}_{1}(x)\Rightarrow\text{in}(x,y_{1})
   ∃x,?[vegetable]1​(x)⇒in​(x,y2)\exists x,\text{?[vegetable]}_{1}(x)\Rightarrow\text{in}(x,y_{2})
   ∃x,?[vegetable]2​(x)⇒in​(x,y1)\exists x,\text{?[vegetable]}_{2}(x)\Rightarrow\text{in}(x,y_{1})
   ∃x,?[vegetable]2​(x)⇒in​(x,y2)\exists x,\text{?[vegetable]}_{2}(x)\Rightarrow\text{in}(x,y_{2})
   ∃x,?[vegetable]3​(x)⇒in​(x,y1)∧sliced​(x)\exists x,\text{?[vegetable]}_{3}(x)\Rightarrow\text{in}(x,y_{1})\land\text{sliced}(x)
   ∃x,?[vegetable]3​(x)⇒in​(x,y2)∧sliced​(x)\exists x,\text{?[vegetable]}_{3}(x)\Rightarrow\text{in}(x,y_{2})\land\text{sliced}(x)
   ∃x,?[fruit]1​(x)⇒in​(x,y1)∧sliced​(x)\exists x,\text{?[fruit]}_{1}(x)\Rightarrow\text{in}(x,y_{1})\land\text{sliced}(x)
   ∃x,?[fruit]1​(x)⇒in​(x,y2)∧sliced​(x)\exists x,\text{?[fruit]}_{1}(x)\Rightarrow\text{in}(x,y_{2})\land\text{sliced}(x)
preserving food ∃y,?[vessel]​(y),∀x,?[fruit]​(x)⇒in​(x,y)∧sliced​(x)∧cooked​(x)\exists y,\text{?[vessel]}(y),\forall x,\text{?[fruit]}(x)\Rightarrow\text{in}(x,y)\land\text{sliced}(x)\land\text{cooked}(x)
∃y,refrigerator​(y),∀x,?[protein]​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[protein]}(x)\Rightarrow\text{in}(x,y)
putting away Christmas decorations ∃y,cabinet​(y),∀x,?[decoration]1​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[decoration]}_{1}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[decoration]2​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[decoration]}_{2}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[decoration]3​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[decoration]}_{3}(x)\Rightarrow\text{in}(x,y)
putting away Halloween decorations ∃y,cabinet​(y),∀x,?[vegetable]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[vegetable]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[illumination tool]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[illumination tool]}(x)\Rightarrow\text{in}(x,y)
∃y,table​(y),∀x,?[vessel]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[vessel]}(x)\Rightarrow\text{on}(x,y)
putting away toys ∃y,box​(y),∀x,?[plaything]​(x)⇒in​(x,y)∧¬open​(y)\exists y,\text{box}(y),\forall x,\text{?[plaything]}(x)\Rightarrow\text{in}(x,y)\land\neg\text{open}(y)
putting dishes away after cleaning ∃y,cabinet​(y),∀x,?[tableware]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[tableware]}(x)\Rightarrow\text{in}(x,y)
putting up Christmas decorations inside ∃y,table​(y),∀x,?[illumination tool]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[illumination tool]}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[decoration]1​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[decoration]}_{1}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[decoration]2​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[decoration]}_{2}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[decoration]3​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[decoration]}_{3}(x)\Rightarrow\text{on}(x,y)
re-shelving library books ∃y,shelf​(y),∀x,?[paper product]​(x)⇒on​(x,y)\exists y,\text{shelf}(y),\forall x,\text{?[paper product]}(x)\Rightarrow\text{on}(x,y)
serving a meal ∃y1,y2,y3,table​(y1),dish​(y2),dish​(y3),…\exists y_{1},y_{2},y_{3},\text{table}(y_{1}),\text{dish}(y_{2}),\text{dish}(y_{3}),\ldots
   ∀x,dish​(x)∧on​(x,y1)\forall x,\text{dish}(x)\land\text{on}(x,y_{1})
   ∀x,bowl​(x)∧on​(x,y1)\forall x,\text{bowl}(x)\land\text{on}(x,y_{1})
   ∀x,?[cutlery]1​(x)∧on​(x,y1)\forall x,\text{?[cutlery]}_{1}(x)\land\text{on}(x,y_{1})
   ∀x,?[cutlery]2​(x)∧on​(x,y1)\forall x,\text{?[cutlery]}_{2}(x)\land\text{on}(x,y_{1})
   ∀x,?[drink]​(x)∧on​(x,y1)\forall x,\text{?[drink]}(x)\land\text{on}(x,y_{1})
   ∃x,?[protein]​(x)∧in​(x,y2)\exists x,\text{?[protein]}(x)\land\text{in}(x,y_{2})
   ∃x,?[protein]​(x)∧in​(x,y2)\exists x,\text{?[protein]}(x)\land\text{in}(x,y_{2})
   ∃x,?[prepared food]​(x)∧in​(x,y2)\exists x,\text{?[prepared food]}(x)\land\text{in}(x,y_{2})
   ∃x,?[prepared food]​(x)∧in​(x,y3)\exists x,\text{?[prepared food]}(x)\land\text{in}(x,y_{3})
   ∃x,?[baked food]​(x)∧in​(x,y2)\exists x,\text{?[baked food]}(x)\land\text{in}(x,y_{2})
   ∃x,?[baked food]​(x)∧in​(x,y3)\exists x,\text{?[baked food]}(x)\land\text{in}(x,y_{3})
serving hors d’oeuvres ∃y,table​(y),∀x,tray​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{tray}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[baked food]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[baked food]}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[vegetable]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[vegetable]}(x)\Rightarrow\text{on}(x,y)
∃y,table​(y),∀x,?[prepared food]​(x)⇒on​(x,y)\exists y,\text{table}(y),\forall x,\text{?[prepared food]}(x)\Rightarrow\text{on}(x,y)
setting up candles ∀y,∃x,?[illumination tool]​(x)∧(?[furniture]​(y)⇒on​(x,y))\forall y,\exists x,\text{?[illumination tool]}(x)\land(\text{?[furniture]}(y)\Rightarrow\text{on}(x,y))
sorting books ∃y,shelf​(y),∀x,?[paper product]1​(x)⇒on​(x,y)\exists y,\text{shelf}(y),\forall x,\text{?[paper product]}_{1}(x)\Rightarrow\text{on}(x,y)
∃y,shelf​(y),∀x,?[paper product]2​(x)⇒on​(x,y)\exists y,\text{shelf}(y),\forall x,\text{?[paper product]}_{2}(x)\Rightarrow\text{on}(x,y)
Table 15: (Cont’d.) The goal names are collected from original BEHAVIOR-100 tasks, and the formulas are the rewritten version for original goal predicates.
Goal Name First-Order Logic Formula
storing food ∃y,cabinet​(y),∀x,?[prepared food]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[prepared food]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[snack]​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[snack]}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[flavorer]1​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[flavorer]}_{1}(x)\Rightarrow\text{in}(x,y)
∃y,cabinet​(y),∀x,?[flavorer]2​(x)⇒in​(x,y)\exists y,\text{cabinet}(y),\forall x,\text{?[flavorer]}_{2}(x)\Rightarrow\text{in}(x,y)
storing the groceries ∃y,refrigerator​(y),∀x,?[fruit]​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[fruit]}(x)\Rightarrow\text{in}(x,y)
∃y,refrigerator​(y),∀x,?[protein]​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[protein]}(x)\Rightarrow\text{in}(x,y)
∃y,refrigerator​(y),∀x,?[vegetable]1​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[vegetable]}_{1}(x)\Rightarrow\text{in}(x,y)
∃y,refrigerator​(y),∀x,?[vegetable]2​(x)⇒in​(x,y)\exists y,\text{refrigerator}(y),\forall x,\text{?[vegetable]}_{2}(x)\Rightarrow\text{in}(x,y)
thawing frozen food ∃y,sink​(y),∀x,?[fruit]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[fruit]}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,?[protein]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[protein]}(x)\Rightarrow\text{in}(x,y)
∃y,sink​(y),∀x,?[vegetable]​(x)⇒in​(x,y)\exists y,\text{sink}(y),\forall x,\text{?[vegetable]}(x)\Rightarrow\text{in}(x,y)
throwing away leftovers ∃y,ashcan​(y),∀x,?[snack]​(x)⇒in​(x,y)\exists y,\text{ashcan}(y),\forall x,\text{?[snack]}(x)\Rightarrow\text{in}(x,y)
washing dishes ∀x,?[tableware]1​(x)⇒¬dusty​(x)\forall x,\text{?[tableware]}_{1}(x)\Rightarrow\neg\text{dusty}(x)
∀x,?[tableware]2​(x)⇒¬dusty​(x)\forall x,\text{?[tableware]}_{2}(x)\Rightarrow\neg\text{dusty}(x)
∀x,?[tableware]3​(x)⇒¬dusty​(x)\forall x,\text{?[tableware]}_{3}(x)\Rightarrow\neg\text{dusty}(x)
washing pots and pans ∃y,cabinet​(y),∀x,?[utensil​(x)⇒in​(x,y)∧¬dusty​(x)\exists y,\text{cabinet}(y),\forall x,\text{?[utensil}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)
∃y,cabinet​(y),∀x,?[vessel]1​(x)⇒in​(x,y)∧¬dusty​(x)\exists y,\text{cabinet}(y),\forall x,\text{?[vessel]}_{1}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)
∃y,cabinet​(y),∀x,?[vessel]2​(x)⇒in​(x,y)∧¬dusty​(x)\exists y,\text{cabinet}(y),\forall x,\text{?[vessel]}_{2}(x)\Rightarrow\text{in}(x,y)\land\neg\text{dusty}(x)
Table 16: (Cont’d.) The goal names are collected from original BEHAVIOR-100 tasks, and the formulas are the rewritten version for original goal predicates.

A.3 Quests

Request templates combine selected static attributes, changeable states, categories, and locations. If the category is omitted, a pronoun replaces it. This provides controlled referential underspecification, while the linguistic forms remain generated from a limited grammar rather than collected natural conversations.

Utterance Natural Language Instruction
bring-me(None) Bring/Hand/Give me that.
bring-me(specifiers) Bring/Hand/Give me [Description(specifiers)].
move-to(None, None) Put it over there.
move-to(specifiers, None) Put [Description(specifiers)] over there.
move-to(None, position) Put/Move it to the [position].
move-to(specifiers, position) Put/Move [Description(specifiers)] to the [position].
change-state(None, verb, prep) [Verb] it [prep].
change-state(specifiers, verb, prep) [Verb] [prep] [Description(specifiers)].
Table 17: Translation from utterance to natural language instruction. “Description(specifiers)” transfers a set of specifiers to a natural language description. The brackets should be replaced by corresponding values, and the words in parentheses should be removed if that attribute doesn’t appear in the specifiers. Moreover, we randomly append “please,” “can you” or nothing to the beginning of an instruction.

A.4 Cost Functions

Human actions cost one. Descriptions charge for specifiers: category levels cost one, two, or three, while attributes, relations, and move-to destinations add unit costs. Robot actions also cost one except examine and inventory, which are free. These costs shape both synthetic requests and evaluation; they encode design choices rather than measured human effort.

Quest Cost
Bring me that. 0
Bring me the food. 1
Bring me the apple on the table. 3+1
Put it over there. 0
Put the large receptacle over there. 1+1
Move the large, red box to the sofa. 1+1+2+1
Slice it up. 0
Slice up the one on the table. 1
Slice up the unfrozen apple on the table. 1+3+1
Table 18: Examples for quest costs. For easier understanding, we represent the quests in natural language. Note that when we are generating data, we separately process three types of quests, so we only need to compare the costs within a same quest type.

A.5 Hyperparameters

Meaning selection uses inverse temperatures 3 and 1.5; the speech model uses temperature 2, description-cost weight 1, and ten recursion steps. The authors choose these values to diversify description lengths, not to fit measured human communication. They report that ten recursion steps approximates the limiting behavior adequately for their generator, without supplying a broad calibration study.

A.6 Textual Interface

Commands refer to concrete object identifiers and return templated observations or invalid-command messages. The initial message includes human history, the request, and the available scene view. Table 19 retains every action and observation template. Shortening observation strings reduces neural input cost but also makes this a specialized text interface.

Action Observation
(Welcome Message) Welcome to the world!
In the room there is a countertop, a sofa, …, a pool.
Now you are standing on the floor.
(Previous Trajectory) The human agent has taken a list of actions towards a goal,
which includes:
Human moves to the [location].
Human opens [receptacle] at the [location].
Human picks up [object] at the [location]…
(Human’s utterance) Human stops and says, “[Natural Language Instruction].”
Now it is your turn to help human to achieve the goal!
Examine/Look You are at the [location]. You see the [state] [location].
In/On [location] you can see [state] [object] [ID], …
In/On [receptacle] you can see [state] [object] [ID], …
  (First the [location] and then the [receptacle] here.)
Inventory You are holding nothing / [state] [object] [ID].
Recall your task: [Previous Trajectory], [Human’s Utterance].
Move to [location] You move from [location] to [location].
Pick up [movable] You pick up the [movable] from [location].
Pick up [movable] from [receptacle] You pick up the [movable] from the [receptacle] at the [location].
put [movable] into/onto [location] You put the [movable] into/onto the [location].
put [movable] into/onto [receptacle] You put the [movable] into/onto the [receptacle] at the [location].
take [movable] from human You take the [movable] from [location].
give [movable] to human You give the [movable] to [location].
open/close [receptacle/location] You open/close the [receptacle/location].
toggle on/off [toggleable] You toggle the [toggleable] on/off.
heat [cookable] You heat the [cookable] up with the [location].
cool [freezable] You cool the [freezable] down with the [location].
soak [soakable] You make the [soakable] soaked with the [location].
slice [sliceable] with [tool] You slice up the [sliceable] with the [tool].
clean [cleanable] with [tool] You clean up the [cleanable] with the [tool].
Table 19: This table presents the natural language templates for actions and corresponding observations in textual interface. Note that the initial observation is not related to player’s command, including welcome message and task description.

Appendix B Dataset Statistics

The reported statistics distinguish the balanced original dataset from the larger, uneven Version 2 release. Scene complexity, observation length, and expert-trajectory length help explain why a text-only interface can still present a substantial search problem.

Scene.

Original scenes average roughly 230 objects from 110 categories. These are per-scene samples from the larger ontology, not a fixed set of 230 categories.

Original figure
Figure 4: Number of categories and objects in the scene.

Human actions.

Human action prefixes average fifteen steps; the original full plans average twenty-five. The prefix is therefore partial evidence of the hidden goal rather than a completed demonstration of the requested robot behavior.

Original figure
Figure 5: Lengths of human’s previous trajectory and original full plan.

Demonstration.

Demonstrations use greedy best-first search with the FF heuristic and mostly take four or five steps. Their generator has full state information, so a partially observing agent may require additional search beyond those expert trajectories. This information mismatch qualifies imitation-learning comparisons.

Original figure
Figure 6: Lengths of trajectories given by expert demonstrations.

Textual interface.

The vocabulary contains 250 words. Initial observations average about 860 tokens with full state and 140 with partial state. More information also increases input-processing demands, so full observability does not guarantee better performance for a fixed-capacity model.

Original figure
Figure 7: Number of words in the initial observation in both fully- and partially-observable setting.

Data split.

Version 1 balances four hardness levels at 2,500 episodes each and splits into 8,000 training, 1,000 validation, and 1,000 test episodes. Bring-me and move-to dominate, with only 535 change-state episodes. Version 2 splits into 104,495 training, 5,824 validation, and 5,827 test episodes and has unequal level frequencies. The appendix’s unresolved table placeholder is a source omission; Table 4 supplies the actual level-by-split counts.

Level 1 Level 2 Level 3 Level 4
Train bring-me 1351 711 728 965
move-to 558 1061 1167 1037
change-state 78 242 80 22
Validation bring-me 174 89 88 123
move-to 73 117 167 111
change-state 9 31 12 6
Test bring-me 175 100 84 112
move-to 69 122 166 117
change-state 13 27 8 7
Table 20: The 10,000 HandMeThat episodes classified in 3 ways.

Appendix C Experimental Details

The supplement gives network inputs, training variants, uncertainty bars, heuristic access, offline-learning failures, and an unsuccessful reward-shaping attempt. These details identify why the baseline results should not be reduced to a single claim that reinforcement learning cannot solve the task.

C.1 Model Details

Seq2Seq separately represents the human history/request and the sequence of observations and prior commands, then predicts the next command. DRRN encodes candidate actions and textual state components to estimate Q values. Its offline variant trains only on successful expert transitions. The source calls its recurrent encoders “GNU,” without a clear expansion; that apparent typo is not silently converted into a verified architecture specification.

C.2 Results.

Tables 21–22 retain three-run values for the original dataset. The prose calls the uncertainty standard deviation, while the captions call it standard error; the source does not resolve this inconsistency. The heuristic’s one-guess restriction limits repeated attempts but does not remove its privileged state, logical request, or planner.

Offline DRRN overvalues actions near the end of successful demonstrations and can repeat them in inappropriate states. Conservative offline learning is suggested as a possible remedy, not evaluated here. Rewarding correct prefixes of demonstrations still yields zero success in the reported shaping experiment: some first actions improve without completing tasks.

Model Fully Observable
Level 1 Level 2 Level 3 Level 4
Human
N/A
9292
4.44.4
N/A
8080
4.84.8
N/A
3636
4.34.3
N/A
1616
4.54.5
Random
−40.0-40.0
0.00.0
N/A
−39.8-39.8
0.10.1
30.030.0
−40.0-40.0
0.00.0
N/A
−40.0-40.0
0.00.0
N/A
Heuristic (1-trial)
88.988.9
94.994.9
4.24.2
−1.9-1.9
28.128.1
4.44.4
−23.7-23.7
12.012.0
4.34.3
−15.9-15.9
17.817.8
4.44.4
Seq2Seq
−9.49±0.47-9.49\pm 0.47
22.44±0.3422.44\pm 0.34
4.03±0.014.03\pm 0.01
−30.74±0.63-30.74\pm 0.63
6.83±0.466.83\pm 0.46
4.29±0.044.29\pm 0.04
−34.56±0.17-34.56\pm 0.17
4.01±0.134.01\pm 0.13
4.29±0.014.29\pm 0.01
−32.51±0.00-32.51\pm 0.00
5.51±0.005.51\pm 0.00
4.13±0.024.13\pm 0.02
Seq2Seq+goal
−15.71±0.87-15.71\pm 0.87
17.90±0.6417.90\pm 0.64
4.30±0.054.30\pm 0.05
−25.83±0.00-25.83\pm 0.00
10.44±0.0010.44\pm 0.00
4.31±0.034.31\pm 0.03
−30.54±0.42-30.54\pm 0.42
6.98±0.316.98\pm 0.31
4.38±0.034.38\pm 0.03
−29.07±0.47-29.07\pm 0.47
8.05±0.348.05\pm 0.34
4.04±0.034.04\pm 0.03
Seq2Seq+subgoal
−4.86±0.22-4.86\pm 0.22
25.88±0.1625.88\pm 0.16
4.20±0.004.20\pm 0.00
−22.01±0.00-22.01\pm 0.00
13.25±0.0013.25\pm 0.00
4.20±0.014.20\pm 0.01
−21.03±0.00-21.03\pm 0.00
13.95±0.0013.95\pm 0.00
4.03±0.004.03\pm 0.00
−19.29±0.00-19.29\pm 0.00
15.25±0.0015.25\pm 0.00
4.22±0.004.22\pm 0.00
DRRN
−39.82±0.18-39.82\pm 0.18
0.13±0.130.13\pm 0.13
5.00±0.005.00\pm 0.00
−39.83±0.17-39.83\pm 0.17
0.13±0.130.13\pm 0.13
10.00±0.0010.00\pm 0.00
−39.83±0.17-39.83\pm 0.17
0.13±0.130.13\pm 0.13
9.00±0.009.00\pm 0.00
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
offline-DRRN
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
Table 21: Experiment results in the fully observable setting. Each model is evaluated on 4 hardness levels with 3 metrics, presented as three values in each cell: 1) the average score, 2) the success rate (%), and 3) the average number of moves in successful episodes. The results for learning models are the mean and standard error values over three runs.
Model Partially Observable
Level 1 Level 2 Level 3 Level 4
Human
N/A
7676
5.15.1
N/A
5252
4.84.8
N/A
2020
5.85.8
N/A
88
5.05.0
Random
−40.0-40.0
0.00.0
N/A
−39.8-39.8
0.10.1
16.016.0
−40.0-40.0
0.00.0
N/A
−40.0-40.0
0.00.0
N/A
Seq2Seq
−5.39±0.21-5.39\pm 0.21
25.49±0.1625.49\pm 0.16
4.19±0.014.19\pm 0.01
−25.81±0.45-25.81\pm 0.45
10.44±0.8910.44\pm 0.89
4.31±0.024.31\pm 0.02
−34.73±0.43-34.73\pm 0.43
3.88±0.313.88\pm 0.31
4.05±0.044.05\pm 0.04
−32.80±0.71-32.80\pm 0.71
5.30±0.525.30\pm 0.52
4.08±0.014.08\pm 0.01
Seq2Seq+goal
−11.45±0.00-11.45\pm 0.00
21.01±0.0021.01\pm 0.00
4.13±0.024.13\pm 0.02
−28.80±0.67-28.80\pm 0.67
8.24±0.498.24\pm 0.49
4.05±0.004.05\pm 0.00
−30.01±0.00-30.01\pm 0.00
7.36±0.007.36\pm 0.00
4.32±0.044.32\pm 0.04
−32.51±0.71-32.51\pm 0.71
5.51±0.525.51\pm 0.52
4.04±0.034.04\pm 0.03
Seq2Seq+subgoal
−10.12±0.65-10.12\pm 0.65
21.99±0.4821.99\pm 0.48
4.09±0.004.09\pm 0.00
−18.72±0.45-18.72\pm 0.45
15.66±0.3315.66\pm 0.33
4.12±0.024.12\pm 0.02
−24.46±0.22-24.46\pm 0.22
11.44±0.1611.44\pm 0.16
4.13±0.004.13\pm 0.00
−21.87±0.24-21.87\pm 0.24
13.35±0.1813.35\pm 0.18
4.11±0.014.11\pm 0.01
DRRN
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
offline-DRRN
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
−40.00±0.00-40.00\pm 0.00
0.00±0.000.00\pm 0.00
N/A
Table 22: Experiment results in the partially observable setting. Each model is evaluated on 4 hardness levels with 3 metrics, presented as three values in each cell: 1) the average score, 2) the success rate (%), and 3) the average number of moves in successful episodes. The results for learning models are the mean and standard error values over three runs.

Appendix D Extension: Human-Robot Interaction

The proposed clarification interface asks for object type, color, size, state, location, destination, or a fuller description. Answers derive from the hidden meaning and may indicate that an attribute is unrestricted. Charging for questions would create a tradeoff with environmental exploration. Table 23 preserves the allowed questions and answers; this extension was not enabled in the principal learning or human comparisons.

Valid Questions Example for Answers
Which type do you mean? I mean the apple.
I just want a fruit.
I just want a food.
Which color do you like? The red one.
Which size do you like? The large one.
Where is the object you want? On top of the table.
Where do you want to place it? Put it on the countertop.
Do you want a dusty/cooked/frozen/ Yes, I mean the dusty one.
sliced/toggled/soaked/open one? No, I mean the no dusty one.
Can you say it clearly? I mean the sliced apple on the table.
(No specified answer.) Not any specific type/color/…
Either is fine.
Table 23: All the valid questions in our textual interface and the corresponding answers.
Concept demonstration

Open image ↗