Overview
The supplement specifies the object ontology, action schemas, goal templates, language and command templates, costs, generation parameters, dataset statistics, baseline details, and a possible clarification interface. It includes the original benchmark and later Version 2 additions. These exact definitions are essential because both the simulated human’s behavior and the test labels come from the generator.
At a glance
HandMeThat tests whether an agent can infer an intended request from household state, a simulated human’s previous actions, and an ambiguous instruction. The evaluated interface is textual and symbolic, so it isolates grounding, goal inference, pragmatic reasoning, and planning from visual perception and physical control. The original release has ten thousand episodes; the revised version adds more data with narrower tasks. Baseline failures demonstrate the difficulty of these generated settings, not an evaluation of unrestricted human–robot conversation.
1 Introduction
An utterance such as asking for an object can be underspecified because its speaker assumes shared knowledge of a larger task. HandMeThat makes that larger goal latent: the robot observes a human-like action prefix, then acts to satisfy the requested subgoal. Large object sets and compositional household goals create ambiguity, while symbolic simulation permits automatic generation and scoring. The demonstrated system uses generated language and plans rather than recorded everyday dialogues or physical robot trials.
2 Related Work
The benchmark combines household task structure from BEHAVIOR with goal recognition and pragmatic language interpretation. It differs from direct instruction following by requiring inference about an assisting subgoal, and from pure goal recognition by providing an utterance that may be ambiguous. The paper’s comparison describes contemporary benchmarks and its chosen task scope; using human-annotated goal templates does not itself validate the distribution of generated conversations.
|
|
|
|
|
|
| |||||||||||||||
| BEHAVIOR [7] | ✓ | 100 (100) | ✓ | ✗ | ✗ | ✗ | ✗ | ||||||||||||||
| ALFRED [6] | ✓ | 7,000 (7) | ✗ | ✓ | ✗ | ✗ | ✗ | ||||||||||||||
| Watch-and-Help [8] | ✓ | 5 (5) | ✗ | ✗ | ✓ | ✗ | ✓ | ||||||||||||||
| CerealBar [9] | ✓ | 1202* | ✗ | ✓ | ✓ | ✗ | ✓ | ||||||||||||||
| ALFworld [10] | ✓ | 7,000 (7) | ✗ | ✓ | ✗ | ✗ | ✗ | ||||||||||||||
| DialFRED [11] | ✓ | 36,912 (25)* | ✗ | ✓ | ✗ | ✗ | ✓ | ||||||||||||||
| TEACh [12] | ✓ | 7,000 (7) | ✗ | ✓ | ✓ | ✗ | ✓ | ||||||||||||||
| Two Body [13] | ✓ | 1 (1) | ✗ | ✗ | ✗ | ✗ | ✓ | ||||||||||||||
| TextWorld [14] | ✓ | — | ✗ | ✓ | ✗ | ✗ | ✗ | ||||||||||||||
| LIGHT [15] | ✗ | 10,777* | ✗ | ✓ | ✓ | ✗ | ✗ | ||||||||||||||
| SCONE [16, 17] | ✗ | 13,951* | ✗ | ✓ | ✗ | ✓ | ✗ | ||||||||||||||
| HandMeThat (ours) | ✓ | >300k (69) | ✓ | ✓ | ✓ | ✓ | ✓ |
3 The HandMeThat Benchmark
A deterministic symbolic world represents object attributes and spatial relations with Boolean predicates. Human and robot actions have explicit preconditions and effects. Each episode samples a state and hidden human goal, generates a plan prefix, chooses a useful assisting subgoal, and generates an utterance for it. The robot receives observations and must satisfy that subgoal within an action budget. Scoring depends on success and action costs, while the human’s complete goal remains unrevealed.
3.1 Intial States, Goals, and Human Trajectories
Scenes sample object instances, attributes, and placements, typically yielding over two hundred entities. Household goals instantiate first-order templates derived from BEHAVIOR-100. The generator plans toward a goal and truncates that trajectory to simulate a request for help. Its rational-planning assumption supplies a useful signal for inverse reasoning, but this is a model of human behavior; expert demonstrations elsewhere use greedy best-first search and are not automatically guaranteed globally optimal.
3.2 Subgoals and Utterances
Requests are bring-me, move-to, or change-state subgoals. A lifted subgoal describes acceptable objects by properties rather than unique identifiers. Its utility is the expected reduction in the human’s remaining goal cost, averaged over matching grounded subgoals. A Boltzmann distribution trades that utility against description complexity.
The Rational Speech Acts model alternates a listener interpreting meanings and a speaker selecting informative, economical utterances. Utterances omit some specifiers from the intended meaning, then templates turn them into English. Crucially, the generator’s listener prior is conditioned on the true human goal. A tested robot does not receive that goal, so some generated tasks demand information unavailable from its observations. The three original displays retain the utility, subgoal distribution, and recursive speaker–listener definitions.
Equations
3.3 Hardness Levels
Level 1 needs literal grounding: utterance and intended meaning identify the same acceptable subgoals. Level 2 also needs the human goal to eliminate unhelpful groundings. Level 3 requires pragmatic inference under the supplied listener model. Level 4 remains ambiguous even under that model and needs more information. These are generator-based categories, not guarantees of monotonic empirical difficulty; Level 3 can also be unresolvable when the true goal cannot be inferred from the observed action prefix.
3.4 Text-Based Interactive Interface
The Gym-style interface returns text about human actions, the request, and the environment. Commands move, inspect, or manipulate objects. Success yields a reward of one hundred, with a forty-action horizon and action costs. Full observability supplies all object states; partial observability restricts inspection to the current location and requires opening containers. The 250-word vocabulary and templated observations deliberately remove open-ended language and perception difficulties.
3.5 Challenges in HandMeThat Tasks
Successful assistance can require using actions and language jointly, selecting only the relevant subgoal, and exploring to gather missing state information. The paper suggests learning human preferences and integrating probabilistic goal recognition with planning. Those are future directions: the released symbolic distributions provide priors, but do not demonstrate adaptation to an individual human’s unmodeled habits.
4 Experiments
The experiments compare random and privileged heuristic agents with behavior-cloned sequence generation and Q-learning. They report success, total score, and steps among successful episodes. Because information access and stopping rules differ, these are diagnostic baselines rather than a controlled comparison of learning algorithms alone.
4.1 Model Details
The heuristic receives full symbolic state, the logical utterance, and a planner; it selects a single grounding by favoring object categories used in the human’s action history. Seq2Seq learns action strings from planner-generated demonstrations, with optional oracle goal or subgoal inputs. DRRN ranks valid actions by learned Q values, either through online exploration or offline demonstrations. Oracle variants test the value of missing information and are not ordinary inference-time baselines.
| Model | Partially Observable | |||
| Level 1 | Level 2 | Level 3 | Level 4 | |
| Human | (76%, 5.1) | (52%, 4.8) | (20%, 5.8) | (8%, 5.0) |
| Random | -40.0 (0.0%, N/A) | -39.5 (0.4%, 16.0) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| Seq2Seq | -5.4 (25.5%, 4.2) | -25.8 (10.4%, 4.1) | -34.7 (3.9%, 4.1) | -32.8 (5.3%, 4.1) |
| +goal | -11.4 (21.0%, 4.1) | -28.8 (8.2%, 4.1) | -30.0 (7.4%, 4.2) | -32.5 (5.5%, 4.0) |
| +subgoal | -10.1 (22.0%, 4.1) | -18.7 (15.7%, 4.1) | -24.5 (11.4%, 4.1) | -21.9 (13.4%, 4.1) |
| DRRN | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A). | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| +offline | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A). | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| Model | Fully Observable | |||
| Level 1 | Level 2 | Level 3 | Level 4 | |
| Human | (92%, 4.4) | (80%, 4.8) | (36%, 4.3) | (16%, 4.5) |
| Heuristic | 88.9 (94.9%, 4.2) | -1.9 (28.1%, 4.4) | -23.7 (12.0%, 4.3) | -15.9 (17.8%, 4.4) |
| Random | -40.0 (0.0%, N/A) | -39.8 (0.1%, 30.0) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| Seq2Seq | -9.5 (22.4%, 4.0) | -30.7 (6.8%, 4.3) | -34.6 (4.0%, 4.3) | -32.5 (5.5%, 4.1) |
| +goal | -15.7 (17.9%, 4.3) | -25.8 (10.4%, 4.3) | -30.5 (7.0%, 4.4) | -29.0 (8.0%, 4.0) |
| +subgoal | -4.9 (25.9%, 4.2) | -22.0 (13.2%, 4.2) | -21.0 (14.0%, 4.0) | -19.3 (15.3%, 4.2) |
| DRRN | -39.8 (0.1%, 5.0) | -39.8 (0.1%, 10.0) | -39.8 (0.1%, 9.0) | -40.0 (0.0%, N/A) |
| +offline | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
4.2 Results
In Version 1, Seq2Seq success falls from 22.4% to 4.0% between Levels 1 and 3 under full observability; partial-observation rates are 25.5% and 3.9%. The heuristic reaches 94.9% on unambiguous Level 1 but only 12.0% on Level 3, despite its extra state, logic, and planning access. DRRN is at or near zero. Short successful trajectories mainly reflect frequent pick-and-place templates, and an average over successes does not describe failed episodes.
Human evaluation uses twenty-five episodes per level and observability setting with one chosen object and no clarification. Full-observation success is 92%, 80%, 36%, and 16% across the four levels. Oracle subgoal information often helps learned agents, but goal information does not improve every cell. The authors identify both intended Level 4 ambiguity and unintended Level 3 ambiguity caused by the generator’s true-goal prior. Thus low performance mixes reasoning difficulty, sparse exploration rewards, representational limits, and missing information.
5 HandMeThat Dataset Version 2
Version 2 addresses sparse data and some ambiguities by narrowing the task family, increasing examples, and modifying where human trajectories are interrupted. It changes the task distribution as well as dataset size, so improvements cannot be attributed solely to more training data.
5.1 Revision Details
Version 2 contains 116,146 episodes over twenty-five goal templates, with bring-me requests and pick-and-place goals. Heuristics choose interruption points intended to reveal the relevant subgoal from prior human actions, and a logical subgoal label supplies optional supervision. Hardness counts are 4,008, 78,488, 28,343, and 5,307, creating a strongly uneven distribution. The stated total is about 11.6 times the original ten thousand; the source’s “forty times” claim better matches its approximate per-template increase than the total dataset ratio.
| Level 1 | Level 2 | Level 3 | Level 4 | |
| Train | 3620 | 70583 | 25488 | 4804 |
| Validation | 207 | 3937 | 1433 | 247 |
| Test | 181 | 3968 | 1422 | 256 |
5.2 Experiment Results
Seq2Seq improves on the revised distribution: full-observation success is 30.4%, 28.8%, 12.8%, and 14.8%; partial-observation success is 30.4%, 31.2%, 16.4%, and 12.0%. Oracle subgoals substantially help some settings, reaching 69.1% on fully observed Level 1, while the privileged heuristic reaches 100%, 64%, 39.2%, and 29.2%. Online and offline DRRN remain at zero in the reported Version 2 tables.
Uneven goal frequencies affect difficulty: Level 2 includes many goals that Seq2Seq handles relatively well. Human and heuristic agents can stop early, whereas failed learned agents use their budget, making raw scores sensitive to stopping behavior. The revisions reduce some ambiguities but retain intrinsically ambiguous Level 4 cases; they do not establish general conversational competence.
| Model | Partially Observable | |||
| Level 1 | Level 2 | Level 3 | Level 4 | |
| Human | (100%, 7.8) | (80%, 7.5) | (40%, 8.4) | (30%, 8.0) |
| Random | -40.0 (0.0%, N/A) | -39.5 (0.4%, 16.0) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| Seq2Seq | 1.3 (30.4%, 4.0) | 2.4 (31.2%, 4.0) | -17.7 (16.4%, 4.1) | -23.7 (12.0%, 4.0) |
| +goal | 0.6 (29.8%, 4.0) | 3.0 (31.6%, 4.0) | -9.6 (22.4%, 4.1) | -9.6 (22.4%, 4.1) |
| +subgoal | 0.6 (29.8%, 4.0) | 14.4 (40.0%, 4.0) | -1.9 (28.0%, 4.1) | -1.4 (28.4%, 4.0) |
| DRRN | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A). | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| +offline | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A). | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| Model | Fully Observable | |||
| Level 1 | Level 2 | Level 3 | Level 4 | |
| Human | (100%, 4.1) | (80%, 4.2) | (40%, 5.2) | (30%, 4.5) |
| Heuristic | 95.8 (100.0%, 4.2) | 59.5 (64.0%, 4.1) | 34.5 (39.2%, 4.1) | 24.4 (29.2%, 4.1) |
| Random | -40.0 (0.0%, N/A) | -39.8 (0.1%, 30.0) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| Seq2Seq | 1.3 (30.4%, 4.0) | -0.8 (28.8%, 4.0) | -22.6 (12.8%, 4.0) | -19.9 (14.8%, 4.0) |
| +goal | -4.7 (26.0%, 4.0) | 7.3 (34.8%, 4.1) | -6.9 (24.4%, 4.1) | -6.9 (24.4%, 4.1) |
| +subgoal | 53.8 (69.1%, 4.2) | 51.3 (67.2%, 4.1) | 1.3 (30.4%, 4.1) | 3.0 (31.6%, 4.1) |
| DRRN | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
| +offline | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) | -40.0 (0.0%, N/A) |
6 Conclusion
HandMeThat makes physical state, action history, and speaker intent jointly relevant to instruction following. Its textual, templated, deterministic setup omits visual perception, gestures, much of human variability, and sustained dialogue. Visual interfaces, multi-round clarification, and joint information gathering are proposed extensions. The benchmark is useful for studying these controlled reasoning dependencies while its generation assumptions constrain conclusions about real assistance.
Appendix A Task Design
The appendix makes the symbolic generator concrete: object categories and properties, action constraints, initial-state sampling, goal templates, request templates, costs, parameters, and observation strings. All original tables are retained so these assumptions remain inspectable rather than being replaced by a high-level benchmark description.
A.1 Object Space
The ontology has 155 categories grouped into locations, receptacles, food, tools, and things. Size and color are static; eight Boolean properties describe changeable state. Only direct inside/on-top relations are stored, avoiding redundant updates when containers move. Sampling creates each fixed location, up to three objects per movable category, admissible placements, and specified state distributions. Tables 7–11 retain the hierarchy, meta-properties, action semantics, constraints, and placement rules; the complete preconditions and effects reside in the released code.
| Class | Subclass | Category | ||
| location | (has-ontop) |
| ||
| (has-inside) |
| |||
| receptacle | furniture | highchair, chair, seat | ||
| vessel | bottle, jar, kettle, caldron | |||
| tableware | bowl, mug, plate, dish, cup | |||
| utensil | saucepan, pan, casserole | |||
| bag | duffel bag, sack, backpack, briefcase | |||
| bucket | bucket | |||
| tray | tray | |||
| basket | basket | |||
| box | box | |||
| package | package | |||
| ashcan | ashcan | |||
| Xmas stocking | Xmas stocking | |||
| Xmas tree | Xmas tree | |||
| food | fruit |
| ||
| vegetable |
| |||
| drink | pop, beer, juice, water, milk | |||
| protein | beef, chicken, pork, fish, egg | |||
| flavorer | catsup, sauce, parsley, tea bag, suger, vegetable oil | |||
| baked food | cracker, bread, cookie, cake | |||
| snack | chip, hamburger, sandwich, candy | |||
| prepared food | oatmeal, sushi, salad, soup, pasta | |||
| tool | metal tool | carving knife, hammer, screwdriver, scraper, saw | ||
| electric equipment | printer, scanner, facsimile, modem | |||
| electrical device | calculator, headset, earphone, mouse, alarm | |||
| toiletry | toothbrush, perfume, makeup | |||
| writing tool | highlighter, marker, pen, pencil | |||
| piece of cloth | dishtower, hand towel, rag | |||
| cleaning tool | scrub brush, broom, vacuum | |||
| cleansing | soap, shampoo, detergent, toothpaste | |||
| cutlery | fork, spoon, knife | |||
| illumination tool | lamp, candle | |||
| thing | decoration | necklace, bracelet, jewelry, bow, wreath, ribbon | ||
| paper product |
| |||
| footwear | gym shoe, sandal, shoe, sock | |||
| headwear | hat, sunglass | |||
| clothing | shirt, sweater, underwear, apparel | |||
| building materials | tile, plywood | |||
| plaything | cube, ball |
| Meta-Properties | Categories | ||
| has-inside |
| ||
| has-ontop |
| ||
| has-size | tableware tray box package ashcan | ||
| has-color | furniture vessel bag basket box package | ||
| openable | cabinet, microwave, oven, dishwasher, refrigerator, vessel, bag, box, package | ||
| toggleable | microwave, oven dishwasher, refrigerator, stove, sink, electric equipment | ||
| cookable | food | ||
| freezable | food | ||
| sliceable | fruit, vegetable, protein | ||
| dustyable | location, receptacle, thing | ||
| stainable | location | ||
| soakable | piece of cloth, clothing |
| Action Name and Parameters | Semantics | ||
|
Agent [?a] moves from [?from] to [?to]. | ||
|
Agent [?a] picks up [?obj] at [?loc]. | ||
|
Agent [?a] picks up [?obj] from [?rec] at [?loc]. | ||
|
Agent [?a] puts [?obj] into [?loc]. | ||
|
Agent [?a] puts [?obj] onto [?loc]. | ||
|
Agent [?a] puts [?obj] into [?rec] at [?loc]. | ||
|
Agent [?a] puts [?obj] onto [?rec] at [?loc]. | ||
|
Agent [?a] opens [?loc]. | ||
|
Agent [?a] closes [?loc]. | ||
|
Agent [?a] opens [?rec] at [?loc]. | ||
|
Agent [?a] closes [?rec] at [?loc]. | ||
|
Agent [?a] toggles on [?loc]. | ||
|
Agent [?a] toggles off [?loc]. | ||
|
Agent [?a] toggles on [?obj] at [?loc]. | ||
|
Agent [?a] toggles off [?obj] at [?loc]. | ||
|
Agent [?a] heats up [?obj] at [?loc]. | ||
|
Agent [?a] cools down [?obj] at [?loc]. | ||
|
Agent [?a] makes [?obj] soaked at [?loc]. | ||
|
Agent [?a] slices up [?obj] with [?tool] at [?loc]. | ||
|
Agent [?a] cleans up [?obj] with [?tool] at [?loc]. | ||
|
Agent [?a] cleans up [?loc] with [?tool]. | ||
|
Robot [?r] brings [?obj] to human [?h]. | ||
|
Robot [?r] takes [?obj] from human [?h]. |
| Action Name | Constraints | ||
|
The receptacle/location must be open if it’s openable. | ||
| heat | The agent can only heat things at microwave, oven and stove. | ||
| cool | The agent can only cool things at refrigerator. | ||
| soak | The agent can only soak things at sink. | ||
| slice | The agent can only slice things with knife. | ||
| clean |
|
| Category | Valid Positions |
| furniture | floor |
| vessel | countertop table cabinet |
| tableware | countertop table cabinet dishwasher refrigerator sink |
| utensil | countertop table cabinet dishwasher refrigerator sink |
| bag | floor countertop table sofa bed |
| bucket | floor countertop table |
| tray | countertop table cabinet refrigerator |
| basket | floor countertop table shelf cabinet sofa bed |
| box | floor countertop table shelf cabinet sofa bed |
| package | floor countertop table shelf cabinet sofa bed |
| ashcan | floor |
| xmas tree | floor |
| xmas stocking | floor countertop table shelf cabinet sofa bed |
| fruit | table countertop refrigerator utensil |
| vegetable | table countertop refrigerator stove utensil |
| drink | table countertop refrigerator cabinet bag |
| protein | table countertop refrigerator stove utensil |
| flavorer | table countertop refrigerator cabinet bag |
| baked food | table countertop refrigerator oven tray |
| snack | table countertop refrigerator microwave tray |
| prepared food | table countertop refrigerator microwave tray |
| metal tool | countertop table cabinet shelf furniture |
| electric equipment | countertop table cabinet shelf furniture |
| electrical device | countertop table cabinet shelf furniture |
| toiletry | cabinet toilet bathtub sink pool bag |
| writing tool | countertop table cabinet shelf bag |
| piece of cloth | cabinet toilet bathtub sink pool bucket |
| cleaning tool | cabinet toilet bathtub sink pool bucket |
| cleansing | cabinet toilet bathtub sink pool bucket |
| cutlery | countertop table cabinet dishwasher refrigerator utensil |
| illumination tool | countertop table sofa bed shelf |
| decoration | cabinet sofa bed package |
| paper product | cabinet sofa bed package |
| footwear | cabinet floor |
| headwear | cabinet sofa bed package |
| clothing | cabinet sofa bed package |
| building materials | pool |
| plaything | cabinet sofa bed package |
A.2 Goal Space
The original goal space uses sixty-nine first-order templates after removing duplicate or incompatible BEHAVIOR tasks. Category substitution creates many concrete goals, but the logical vocabulary and action model remain supplied. Tables 12–16 preserve every template. Version 2 selects twenty-five goal names and restricts the manipulation setting, trading task breadth for more samples and easier control of ambiguity.
| Goal Name | First-Order Logic Formula |
| assembling gift baskets | |
| bottling fruit | |
| boxing books up for storage | |
| bringing in wood | |
| brushing lint off clothing | |
| chopping vegetables | |
| cleaning bathrooms | |
| cleaning bedrooms | |
| cleaning closet | |
| cleaning floors | |
| cleaning garage | |
| cleaning high chair | |
| cleaning kitchen cupboard | |
| cleaning microwave oven | |
| cleaning out drawers | |
| cleaning oven | |
| cleaning shoes | |
| cleaning stove | |
| cleaning table after clearing | |
| cleaning the pool | |
| Goal Name | First-Order Logic Formula |
| cleaning toilet | |
| cleaning up after a meal | |
| cleaning up refrigerator | |
| cleaning up the kitchen only | |
| clearing the table after dinner | |
| collect misplaced items | |
| collecting aluminum cans | |
| filling a Christmas stocking | |
| filling a Easter basket | |
| installing a fax machine | |
| installing alarms | |
| laying tile floors | |
| loading the dishwasher | |
| moving boxes to storage | |
| opening packages | |
| organizing boxes in garage | |
| Goal Name | First-Order Logic Formula |
| organizing file cabinet | |
| organizing school stuff | |
| packing adult’s bag | |
| packing bags or suitcase | |
| packing boxes for household move or trip | |
| packing child’s bag | |
| packing box for work | |
| packing lunches | |
| packing picnics | |
| picking up take out food | |
| Goal Name | First-Order Logic Formula |
| picking up trash | |
| polishing furniture | |
| polishing shoes | |
| polishing silver | |
| preparing food | |
| preserving food | |
| putting away Christmas decorations | |
| putting away Halloween decorations | |
| putting away toys | |
| putting dishes away after cleaning | |
| putting up Christmas decorations inside | |
| re-shelving library books | |
| serving a meal | |
| serving hors d’oeuvres | |
| setting up candles | |
| sorting books | |
| Goal Name | First-Order Logic Formula |
| storing food | |
| storing the groceries | |
| thawing frozen food | |
| throwing away leftovers | |
| washing dishes | |
| washing pots and pans | |
A.3 Quests
Request templates combine selected static attributes, changeable states, categories, and locations. If the category is omitted, a pronoun replaces it. This provides controlled referential underspecification, while the linguistic forms remain generated from a limited grammar rather than collected natural conversations.
| Utterance | Natural Language Instruction |
| bring-me(None) | Bring/Hand/Give me that. |
| bring-me(specifiers) | Bring/Hand/Give me [Description(specifiers)]. |
| move-to(None, None) | Put it over there. |
| move-to(specifiers, None) | Put [Description(specifiers)] over there. |
| move-to(None, position) | Put/Move it to the [position]. |
| move-to(specifiers, position) | Put/Move [Description(specifiers)] to the [position]. |
| change-state(None, verb, prep) | [Verb] it [prep]. |
| change-state(specifiers, verb, prep) | [Verb] [prep] [Description(specifiers)]. |
A.4 Cost Functions
Human actions cost one. Descriptions charge for specifiers: category levels cost one, two, or three, while attributes, relations, and move-to destinations add unit costs. Robot actions also cost one except examine and inventory, which are free. These costs shape both synthetic requests and evaluation; they encode design choices rather than measured human effort.
| Quest | Cost |
| Bring me that. | 0 |
| Bring me the food. | 1 |
| Bring me the apple on the table. | 3+1 |
| Put it over there. | 0 |
| Put the large receptacle over there. | 1+1 |
| Move the large, red box to the sofa. | 1+1+2+1 |
| Slice it up. | 0 |
| Slice up the one on the table. | 1 |
| Slice up the unfrozen apple on the table. | 1+3+1 |
A.5 Hyperparameters
Meaning selection uses inverse temperatures 3 and 1.5; the speech model uses temperature 2, description-cost weight 1, and ten recursion steps. The authors choose these values to diversify description lengths, not to fit measured human communication. They report that ten recursion steps approximates the limiting behavior adequately for their generator, without supplying a broad calibration study.
A.6 Textual Interface
Commands refer to concrete object identifiers and return templated observations or invalid-command messages. The initial message includes human history, the request, and the available scene view. Table 19 retains every action and observation template. Shortening observation strings reduces neural input cost but also makes this a specialized text interface.
| Action | Observation |
| (Welcome Message) | Welcome to the world! |
| In the room there is a countertop, a sofa, …, a pool. | |
| Now you are standing on the floor. | |
| (Previous Trajectory) | The human agent has taken a list of actions towards a goal, |
| which includes: | |
| Human moves to the [location]. | |
| Human opens [receptacle] at the [location]. | |
| Human picks up [object] at the [location]… | |
| (Human’s utterance) | Human stops and says, “[Natural Language Instruction].” |
| Now it is your turn to help human to achieve the goal! | |
| Examine/Look | You are at the [location]. You see the [state] [location]. |
| In/On [location] you can see [state] [object] [ID], … | |
| In/On [receptacle] you can see [state] [object] [ID], … | |
| (First the [location] and then the [receptacle] here.) | |
| Inventory | You are holding nothing / [state] [object] [ID]. |
| Recall your task: [Previous Trajectory], [Human’s Utterance]. | |
| Move to [location] | You move from [location] to [location]. |
| Pick up [movable] | You pick up the [movable] from [location]. |
| Pick up [movable] from [receptacle] | You pick up the [movable] from the [receptacle] at the [location]. |
| put [movable] into/onto [location] | You put the [movable] into/onto the [location]. |
| put [movable] into/onto [receptacle] | You put the [movable] into/onto the [receptacle] at the [location]. |
| take [movable] from human | You take the [movable] from [location]. |
| give [movable] to human | You give the [movable] to [location]. |
| open/close [receptacle/location] | You open/close the [receptacle/location]. |
| toggle on/off [toggleable] | You toggle the [toggleable] on/off. |
| heat [cookable] | You heat the [cookable] up with the [location]. |
| cool [freezable] | You cool the [freezable] down with the [location]. |
| soak [soakable] | You make the [soakable] soaked with the [location]. |
| slice [sliceable] with [tool] | You slice up the [sliceable] with the [tool]. |
| clean [cleanable] with [tool] | You clean up the [cleanable] with the [tool]. |
Appendix B Dataset Statistics
The reported statistics distinguish the balanced original dataset from the larger, uneven Version 2 release. Scene complexity, observation length, and expert-trajectory length help explain why a text-only interface can still present a substantial search problem.
Scene.
Original scenes average roughly 230 objects from 110 categories. These are per-scene samples from the larger ontology, not a fixed set of 230 categories.
Human actions.
Human action prefixes average fifteen steps; the original full plans average twenty-five. The prefix is therefore partial evidence of the hidden goal rather than a completed demonstration of the requested robot behavior.
Demonstration.
Demonstrations use greedy best-first search with the FF heuristic and mostly take four or five steps. Their generator has full state information, so a partially observing agent may require additional search beyond those expert trajectories. This information mismatch qualifies imitation-learning comparisons.
Textual interface.
The vocabulary contains 250 words. Initial observations average about 860 tokens with full state and 140 with partial state. More information also increases input-processing demands, so full observability does not guarantee better performance for a fixed-capacity model.
Data split.
Version 1 balances four hardness levels at 2,500 episodes each and splits into 8,000 training, 1,000 validation, and 1,000 test episodes. Bring-me and move-to dominate, with only 535 change-state episodes. Version 2 splits into 104,495 training, 5,824 validation, and 5,827 test episodes and has unequal level frequencies. The appendix’s unresolved table placeholder is a source omission; Table 4 supplies the actual level-by-split counts.
| Level 1 | Level 2 | Level 3 | Level 4 | ||
| Train | bring-me | 1351 | 711 | 728 | 965 |
| move-to | 558 | 1061 | 1167 | 1037 | |
| change-state | 78 | 242 | 80 | 22 | |
| Validation | bring-me | 174 | 89 | 88 | 123 |
| move-to | 73 | 117 | 167 | 111 | |
| change-state | 9 | 31 | 12 | 6 | |
| Test | bring-me | 175 | 100 | 84 | 112 |
| move-to | 69 | 122 | 166 | 117 | |
| change-state | 13 | 27 | 8 | 7 |
Appendix C Experimental Details
The supplement gives network inputs, training variants, uncertainty bars, heuristic access, offline-learning failures, and an unsuccessful reward-shaping attempt. These details identify why the baseline results should not be reduced to a single claim that reinforcement learning cannot solve the task.
C.1 Model Details
Seq2Seq separately represents the human history/request and the sequence of observations and prior commands, then predicts the next command. DRRN encodes candidate actions and textual state components to estimate Q values. Its offline variant trains only on successful expert transitions. The source calls its recurrent encoders “GNU,” without a clear expansion; that apparent typo is not silently converted into a verified architecture specification.
C.2 Results.
Tables 21–22 retain three-run values for the original dataset. The prose calls the uncertainty standard deviation, while the captions call it standard error; the source does not resolve this inconsistency. The heuristic’s one-guess restriction limits repeated attempts but does not remove its privileged state, logical request, or planner.
Offline DRRN overvalues actions near the end of successful demonstrations and can repeat them in inappropriate states. Conservative offline learning is suggested as a possible remedy, not evaluated here. Rewarding correct prefixes of demonstrations still yields zero success in the reported shaping experiment: some first actions improve without completing tasks.
| Model | Fully Observable | |||||||||||||||
| Level 1 | Level 2 | Level 3 | Level 4 | |||||||||||||
| Human |
|
|
|
| ||||||||||||
| Random |
|
|
|
| ||||||||||||
| Heuristic (1-trial) |
|
|
|
| ||||||||||||
| Seq2Seq |
|
|
|
| ||||||||||||
| Seq2Seq+goal |
|
|
|
| ||||||||||||
| Seq2Seq+subgoal |
|
|
|
| ||||||||||||
| DRRN |
|
|
|
| ||||||||||||
| offline-DRRN |
|
|
|
| ||||||||||||
| Model | Partially Observable | |||||||||||||||
| Level 1 | Level 2 | Level 3 | Level 4 | |||||||||||||
| Human |
|
|
|
| ||||||||||||
| Random |
|
|
|
| ||||||||||||
| Seq2Seq |
|
|
|
| ||||||||||||
| Seq2Seq+goal |
|
|
|
| ||||||||||||
| Seq2Seq+subgoal |
|
|
|
| ||||||||||||
| DRRN |
|
|
|
| ||||||||||||
| offline-DRRN |
|
|
|
| ||||||||||||
Appendix D Extension: Human-Robot Interaction
The proposed clarification interface asks for object type, color, size, state, location, destination, or a fuller description. Answers derive from the hidden meaning and may indicate that an attribute is unrestricted. Charging for questions would create a tradeoff with environmental exploration. Table 23 preserves the allowed questions and answers; this extension was not enabled in the principal learning or human comparisons.
| Valid Questions | Example for Answers |
| Which type do you mean? | I mean the apple. |
| I just want a fruit. | |
| I just want a food. | |
| Which color do you like? | The red one. |
| Which size do you like? | The large one. |
| Where is the object you want? | On top of the table. |
| Where do you want to place it? | Put it on the countertop. |
| Do you want a dusty/cooked/frozen/ | Yes, I mean the dusty one. |
| sliced/toggled/soaked/open one? | No, I mean the no dusty one. |
| Can you say it clearly? | I mean the sliced apple on the table. |
| (No specified answer.) | Not any specific type/color/… |
| Either is fine. |