Literature
Display
← All papers

Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation

Albert Yu · Chengshu Li · Luca Macesanu · Arnav Balaji · Ruchira Ray · Raymond Mooney · Roberto Martín-Martín

· ICRA 2026 ↗

Robotics11 min text · CondensedOriginal paper ↗
Contents

At a glance

MICoBot combines a language-model meta-planner, a constrained task allocator and an action executor. Human and robot can propose, accept or decline task assignments in speech; the allocator uses simulation-trained robot costs and a dialog-derived estimate of human willingness.

In a paired study with eighteen participants across three tasks, it completes 14/18 trials versus 5/18 for a pure language-model baseline, and fourteen participants prefer it. The system uses fixed, sequential task plans and researcher approval of physical actions. Project videos and materials accompany the paper.

I Introduction

A useful collaborator should recognize its own manipulation limits, request assistance, respond to human proposals and negotiate an appropriate division of work. MICoBot connects these conversational decisions to estimates of agent capability rather than asking a language model to assign all steps directly.

Its three levels generate planning code, optimize remaining assignments and execute a physical or verbal primitive. The reported 77.8% versus 27.8% success is a 50-percentage-point increase, not a 50% relative improvement.

Complete original Figure 1.
Fig. 1: We present MICoBot, a system for human-robot collaboration where both agents can initiate and carry out physical and verbal actions. MICoBot uses both the robot’s capability and the likelihood of human helping (inferred from previous dialog history) to determine whether the robot is better suited than the human to perform the skill. If it is, it attempts the skill itself. If not, it negotiates for human help.
Complete original Figure 2.
Fig. 2: MICoBot supports both robot-initiated (top row) and human-initiated (bottom row) task-directed speech2speech dialog, where both agents discuss who is best suited to perform steps in a long-horizon task. These are real dialog and physical interactions from our user studies (see our website1).

II Related Work

Mixed-initiative dialog allows either participant to pose substantive questions or proposals. Prior language-driven robotics often uses a leader/follower or clarification-only arrangement; other collaboration systems permit physical initiative without freeform negotiation.

Task-allocation work commonly assumes known competence and availability. MICoBot instead combines a supplied task structure with estimated willingness and explicit dialog constraints. Its contribution integrates these components in a physical study; broad claims that it is the first such system should be read in the scope of the surveyed comparison.

III Problem Setting: Task Collaboration with Mixed-Initiative Dialog

The robot observes RGB-D images, proprioception and optional human speech. Both agents have parameterized physical primitives and abstract verbal intents that generate open-vocabulary utterances. Humans are assumed able to execute every task step successfully once they agree; robots have limited skills.

A fixed ordered plan is supplied by instructions or another planner. At each decision, assign every remaining step to human or robot while considering expressed preferences. The formulation does not jointly learn the task plan, rearrange arbitrary steps or support simultaneous joint manipulation. The MDP description abstracts away uncertainty in human internal state.

Complete original Figure 3.
Fig. 3: Proposed MDP for Mixed-Initiative Collaboration.

IV MICoBot: Mixed-Initiative Collaborative Robot

Three modules connect dialog to execution: a meta-planner writes adaptive code, an iterative planner evaluates allocations, and an executor produces manipulation trajectories or a spoken response.

Complete original Figure 4.
Fig. 4: MICoBot consists of 3 decision-making modules: a meta-planner that produces a collaborative strategy expressed through adaptive planning code, a planner that executes the code and optimizes our objective (Eq. 1) to decide the next primitive action, and an action executor that outputs the low-level pose trajectory or verbal utterance to say to the human.

IV-A Collaborative Task Allocation as Optimization.

A skill receives reward −1 at each low-level timestep until success or timeout, with undiscounted return. Its Q-value is therefore negative expected duration, including the timeout penalty on failed attempts. A larger value indicates a faster or more reliable skill under that model.

Equation 1 scales the human’s negative cost by α/pH, where α values human effort relative to robot effort and pH estimates willingness to help. Dialog-derived constraints restrict assignments. The objective is a timeout-weighted duration surrogate; it does not separately guarantee maximal task-success probability or an optimal human-effort frontier. The printed division also requires care when pH approaches zero.

Equation 1
maxgt,…,gT\displaystyle\max_{g_{t},...,g_{T}} ∑tT−1(𝟙gt=H⋅αpH,t+𝟙gt=R)​Qgt​(st,at),\displaystyle\sum_{t}^{T-1}\left(\mathds{1}_{g_{t}=H}\cdot\frac{\alpha}{p_{H,t}}+\mathds{1}_{g_{t}=R}\right)Q_{g_{t}}(s_{t},a_{t}), (1)
s.t.\displaystyle{s.t.} C1,…,Cn​ are satisfied\displaystyle C_{1},\ldots,C_{n}\text{ are satisfied}

IV-B MICoBot Framework

The GPT-4o meta-planner uses symbolic state, task plan, dialog and about fifteen in-context examples to generate allocation and action-selection code. The iterative planner enumerates feasible assignments, evaluating the cost objective with learned robot and estimated human Q-functions. If constraints are infeasible, it relaxes the most recent one and explains the limitation.

Robot costs come from supervised simulation rollouts; human costs combine an LLM duration estimate with travel distance. Another LLM estimates willingness from dialog. These are approximate models, so exhaustive allocation is optimal only for the supplied states, costs and constraints.

Physical execution uses ROS navigation, Grounding DINO, RGB-D backprojection and inverse kinematics. Verbal execution uses ten examples to turn selected intents into task-grounded utterances. A plan hierarchy groups adjacent primitives into understandable requests and expands them only when negotiation requires finer detail.

Pour Package in Bowl
n=6n=6
Assemble Toy Car
n=6n=6
Pack Gift Box
n=6n=6
Average
n=18n=18
MICoBot LLM MICoBot LLM MICoBot LLM MICoBot (ours) LLM
Entire Task Success Rate (%, ↑\uparrow) 𝟏𝟎𝟎\mathbf{100} 8383 𝟔𝟕\mathbf{67} 00 𝟔𝟕\mathbf{67} 00 77.8±15.7\mathbf{77.8\pm 15.7} 27.8±39.327.8\pm 39.3
% of task steps completed (↑\uparrow) 𝟏𝟎𝟎\mathbf{100} 9393 𝟗𝟒\mathbf{94} 3131 𝟖𝟖\mathbf{88} 5050 93.8±5.1\mathbf{93.8\pm 5.1} 58.2±26.058.2\pm 26.0
% of steps performed by Human 2727 2929 6060 55 3535 2121 40.5±14.240.5\pm 14.2 18.2±9.718.2\pm 9.7
% Users Preferring … (↑\uparrow) 𝟔𝟕\mathbf{67} 3333 𝟏𝟎𝟎\mathbf{100} 00 𝟔𝟕\mathbf{67} 3333 77.8\mathbf{77.8} 22.222.2
Communicative ability (↑,/5\uparrow,/5) 3.73.7 3.8\mathbf{3.8} 4.3\mathbf{4.3} 1.31.3 2.8\mathbf{2.8} 2.32.3 3.6±1.0\mathbf{3.6\pm 1.0} 2.5±1.42.5\pm 1.4
Awareness of its Limitations (↑,/5\uparrow,/5) 3.3\mathbf{3.3} 3.3\mathbf{3.3} 3.7\mathbf{3.7} 1.21.2 4.2\mathbf{4.2} 2.52.5 3.7±1.4\mathbf{3.7\pm 1.4} 2.3±1.62.3\pm 1.6
Overall Satisfaction working w/ Robot (↑,/5\uparrow,/5) 3.8\mathbf{3.8} 3.73.7 3.5\mathbf{3.5} 1.51.5 3.5\mathbf{3.5} 2.52.5 3.6±0.8\mathbf{3.6\pm 0.8} 2.6±1.42.6\pm 1.4
TABLE I: Comparison between MICoBot (ours) and the LLM baseline across three real-world tasks on both objective (top 3 rows) and subjective (bottom 4 rows) metrics. Ratings out of 5 are on the Likert scale. Through more effective task allocation and communication, MICoBot achieves much higher task success rates and overall user satisfaction.

V Experimental Evaluation

Eighteen participants each perform one task with MICoBot and the pure-LLM baseline in randomized order: six participants per package-pouring, toy-car-assembly or gift-packing task. Each trial lasts about 20–30 minutes. Trials stop after an irrecoverable primitive failure, 4T decisions, repeated infeasible robot assignments or two refusals of an indispensable human step.

The baseline receives symbolic state, dialog, task plan, effort tradeoff and available skills. The random effort-controlled comparison is computed separately with oracle assumptions, not another matched human trial. Simulation adds random/RL baselines and ablations removing willingness estimation, plan hierarchy or either agent’s dialog initiative.

MICoBot reaches 77.8% entire-task success versus 27.8%, with 93.8% versus 58.2% steps completed in Table I. It uses more human help: 40.5% versus 18.2% of steps. The comparison supports better completion, not an absolute reduction in human effort.

Fourteen of eighteen participants prefer MICoBot, and reported rating tests favor it. It makes 2.9 help requests and 2.4 initiative shifts per trial versus 0.9 and 1.1. Acceptance rises from 55% initially to 86% after negotiation, versus 70%→75% for the baseline. These observational dialog statistics do not alone isolate a causal effect of negotiation.

Complete original Figure 5.
Fig. 5: In both real-world user studies (top) and simulation trials with a simulated human (bottom), our method (red) demonstrates the best tradeoff in achieving task success (y-axis) for a given amount of human effort (x-axis) than baselines (blue) and our method’s ablations (pink).
Complete original Figure 6.
Fig. 6: Our method substantially outperforms the pure LLM baseline in user ratings averaged over all n=18n=18 participants.

VI Conclusion

The system demonstrates that explicit capability-aware allocation can complement language-model dialog in a small physical collaboration study. It handles both agents’ proposals more effectively than the tested pure-language-model allocator, while relying on fixed plans, modeled competence and human assistance.

VII Limitations and Future Work

Agents act sequentially; the robot cannot hold an object while a person simultaneously manipulates it. The plan ordering is predetermined, and the person is assumed to understand its high-level structure. Time-based effort and text-derived willingness are crude proxies for workload and preferences.

Suggested extensions include spatial/temporal feedback, shared information under partial observability, replanning and richer affect cues. Those capabilities are not established by the current experiments. Researcher action approval and small task/participant coverage also limit claims of unattended household deployment.

Refer to caption
Refer to caption
Refer to caption
Fig. 7: Real-world tasks from left to right: pouring package into bowl, assembling toy car, and packing gift box.

Appendix -A Real-world Task Descriptions

The appendix specifies all physical tasks and their hierarchical plans, retaining the exact skill calls below.

Task Plans

Package pouring has five primitives; using scissors to open the package requires the human. Toy-car assembly has eight, with four drill/assembly steps impossible for the robot and drill-bit retrieval unreliable. Gift packing has eight, with lid placement, ribbon wrapping and tape cutting impossible and two other steps unreliable.

These tasks deliberately vary dependence on human assistance. Descriptive step numbers and the displayed hierarchical ordering differ in places; the exact plan trees are retained rather than silently renumbered.

Hierarchical Plan Trees for Each Task

The three original plan trees retain all 21 low-level skill calls and their grouping into high-level requests. The person is initially told only the high-level plan; the robot introduces finer divisions when needed.

Task 1: Pour Package into Bowl (5 low-level steps)
1. Bring bowl and package to coffee table.
  (a) pickplace(bowl, coffee_table)
  (b) pickplace(package, coffee_table)
2. Open package.
  (a) pickplace(scissors, coffee_table)
  (b) pick_open_place(scissors, package, coffee_table)
3. Pour package into bowl.
  (a) pick_pour_place(package, bowl, coffee_table)
Task 2: Assemble Toy Car (8 low-level steps)
1. Bring parts to coffee table.
  (a) pickplace(parts_tray, coffee_table)
  (b) pickplace(wheels, coffee_table)
2. Assemble wheels.
  (a) pickplace(drill, coffee_table)
  (b) put_on(wheels, car, drill)
3. Switch drill bit.
  (a) pickplace(hex_drill_bit, coffee_table)
  (b) switch(hex_drill_bit, drill)
4. Assemble rest of car.
  (a) put_on(window, car, drill)
  (b) put_on(seats, car, drill)
Task 3: Pack Gift Box (8 low-level steps)
1. Assemble box.
  (a) fold(box_flap)
2. Put in gift.
  (a) pickplace(gift_tissue_paper, box)
  (b) pickplace(toy_car, box)
3. Seal the box.
  (a) cover(box_lid, box)
  (b) pickplace(ribbons, coffee_table)
  (c) wrap(ribbons, box)
4. Decorate the box.
  (a) cut_put(tape, scissors, box)
  (b) pickplace(gift_bow, box_lid)

Appendix -B Dialog Excerpts from our User Studies

Two original exchanges illustrate successful negotiation and a failure to incorporate feedback. Speaker labels omitted from the HTML conversion are restored from the PDF.

Appendix -B 1 Convincing a Reluctant User to Perform a Step

A participant initially expects the robot to open a package. The robot explains that it can bring scissors but cannot use them, and the pair agrees on that split. The full exchange below shows how clarification and an explicit capability limit produce cooperation.

Robot: “Could you please open the package using the scissors on the coffee table? Thank you so much!”

Human: “There are no scissors on the coffee table.”

Robot: “Let’s collaborate to open the package! I can place the scissors onto the coffee table, and you can then use them to open the package. Thank you for your help!”

Human: “I think you should bring the scissors, and then you should use the scissors to open the package.”

Robot: “I’m happy to place the scissors on the coffee table for you. However, I’m not able to open the package with the scissors and place everything on the table. Let me know if there’s anything I can do to assist.”

Human: “Ok, if you bring the scissors, then I will open the package with the scissors.”

Robot: “Great! I’ll happily place the scissors on the coffee table. Thank you for taking care of opening the package with the scissors!”

Robot: [brings the scissors]

Human: “Thanks for bringing the scissors!”

Robot: “Could you please open the package with scissors and place it on the coffee table? Thank you!”

Human: “Ok, I will do that now!”

Human: [opens the package]

Appendix -B 2 Failure Case

A participant asks the robot to hold a toy car while they flatten tissue paper. The system fails to act on the requested motion-level correction, then confuses it with later lid/ribbon steps. The preserved exchange exposes limits of fixed-plan interpretation and simultaneous collaboration.

Human: “Please pack the gift.”

Robot: [Puts tissue paper into the box.]

Human: “It would be nice if the parchment was laying flatter.”

Human: “The compartment is kind of squished underneath the car. If you can hold the car for me, I can fix the parchment quickly.”

Robot: “I’m sorry, but I can’t cover the box with the lid. Could you help with that part? Thank you so much! Could you kindly help with sealing the box? Specifically, could you cover the box with the lid? Thank you so much for your assistance!”

Human: “Yeah, I can, but can we fix the parchment first? If you can pick up the toy car, I can fix the parchment.”

Robot: “I’m delighted to hear you’ll help! Unfortunately, I won’t be able to wrap the ribbons around the box.”

Human: “Okay, that’s fine. We can get to that later. Let me put the lid to the box on and then we can talk about the next steps.”

Appendix -C MICoBot Implementation Details

Implementation details explain how approximate capability, human-effort and symbolic-transition models support allocation.

Robot Q-function Q R training in OmniGibson

A coarse OmniGibson environment approximates furniture/object locations and the TIAGo skills. Roughly one hundred samples per skill-parameter action record symbolic state, action and time to success, substituting a fixed timeout for failures.

A two-layer network with hidden size 32 predicts negative duration using L2 regression and Adam. Symbolic inputs reduce visual transfer demands but do not establish calibration across arbitrary real-world objects or scenes.

Refer to caption
Refer to caption
Task 1: Pick package from shelf (left) and place on coffee table (right).
Refer to caption
Refer to caption
Task 3: Pick toy car from coffee table (left) and place into gift box (right).
Fig. 8: Frames from primitive rollouts in OmniGibson for task 1 (left two images) and task 3 (right two images). Left and right images within each task are frames near the beginning and end, respectively, of each skill. The square image at the top right of each frame represents the robot’s camera view observation.

Human Q-function Q H Estimation

Human cost adds an LLM estimate of stationary task duration to straight-line travel distance divided by 1.4 m/s. The study assumes the person starts on the couch; toy-car descriptions inform its task-time estimates.

These estimates omit individual skill variation, fatigue, interruption cost and actual motion paths. Perfect execution after acceptance is a modeling and study-protocol assumption.

Forward Dynamics Model

Future symbolic states are updated by hand-coded primitive effects, such as moving a bowl to the coffee table. This lets the allocator evaluate later skill costs without learning continuous dynamics. The simplification depends on accurate symbolic state and the supplied primitive semantics.

Appendix -D Additional Real-world Baselines and Success Rate / Human Effort Efficiency Metrics

Two computed oracle comparisons separate physical-skill failures from allocation: an LLM allocator with oracle skills/help scores 33.3%, while assigning the robot every feasible step with real skills scores 44.0%. Both are below MICoBot’s 77.8%, but neither is a globally optimal achievable benchmark.

Average estimated human effort is 94.3 seconds for MICoBot versus 23.3 for the LLM baseline. Success percentage per second is 0.83 versus 1.19, so MICoBot does not dominate this aggregate efficiency ratio. Its advantage emphasizes completion of harder tasks rather than minimizing help unconditionally.

TABLE II: Additional Real-world Baselines and Human Effort Efficiency
Real-World Task Metric LLM baseline MICoBot (B1) (B2)
Pour Package in Bowl Success Rate (%, ↑\uparrow) 83 100 83 41
Steps Completed (%, ↑\uparrow) 93.3 100 93.3 64.5
Human Effort (seconds) 36.7 48 46.7 32.7
Success Rate (%) / Human Effort (s) (↑\uparrow) 2.3 2.1 1.8 1.3
Assemble Toy Car Success Rate (%, ↑\uparrow) 0 67 0 11
Steps Completed (%, ↑\uparrow) 29 94 54.2 43.2
Human Effort (seconds) 20.8 197.3 56.7 55.0
Success Rate (%) / Human Effort (s) (↑\uparrow) 0 0.3 0 0.2
Pack Gift Box Success Rate (%, ↑\uparrow) 0 67 17 80
Steps Completed (%, ↑\uparrow) 50 88 62.5 97.5
Human Effort (seconds) 12.5 37.5 25 45.0
Success Rate (%) / Human Effort (s) (↑\uparrow) 0 1.8 0.68 1.8
Average Success Rate (%, ↑\uparrow) 27.8 77.8 33.3 44.0
Steps Completed (%, ↑\uparrow) 58.2 93.8 70 68.4
Human Effort (seconds) 23.3 94.3 42.8 44.2
Success Rate (%) / Human Effort (s) (↑\uparrow) 1.19 0.83 0.78 1.00

Appendix -E Statistical Testing

Table III reports Fisher’s exact p=0.006709 for success and Wilcoxon signed-rank p=0.002031 for completed steps; subjective-rating p-values range from 0.006502 to 0.023540. These are the authors’ reported tests, with no multiple-testing adjustment described.

Some Table III means/deviations and rating labels do not match Table I; the baseline completed-step mean is 57.5% here versus 58.2% there. Both original tables are preserved. Paired study design and these reporting inconsistencies should be considered when interpreting the numerical summaries.

TABLE III: Statistical Testing on Results Shown in Table I
Metric MICoBot (ours) LLM baseline Statistical Test Test statistic pp-val.
Overall User Satisfaction (↑\uparrow, /5) 3.61±0.953.61\pm 0.95 2.5±1.382.5\pm 1.38 Wilcoxon Signed-Rank W=17.0W=17.0 0.0235400.023540
Communicative Ability (↑\uparrow, /5) 3.72±1.413.72\pm 1.41 2.33±1.562.33\pm 1.56 W=8.5W=8.5 0.0092330.009233
Asked for Suitable Amt. of Help (↑\uparrow, /5) 3.61±0.833.61\pm 0.83 2.56±1.422.56\pm 1.42 W=15.0W=15.0 0.0086360.008636
Awareness of Its Limitations (↑\uparrow, /5) 3.94±0.853.94\pm 0.85 2.61±1.532.61\pm 1.53 W=16.0W=16.0 0.0065020.006502
Success Rate (%, ↑\uparrow) 77.8±15.777.8\pm 15.7 27.8±39.327.8\pm 39.3 Fisher’s Exact – 0.0067090.006709
Steps Completed (%, ↑\uparrow) 93.8±15.293.8\pm 15.2 57.5±30.657.5\pm 30.6 Wilcoxon Signed-Rank W=0.0W=0.0 0.0020310.002031

Appendix -F Mixed-Initiative Dialog: Real-world Metrics

The dialog audit records more robot initiations and more shifts of initiative with MICoBot. Human acceptance improves over negotiation rounds, while the baseline rarely asks for help.

Table IV gives the baseline 1.1 initiative shifts; the accompanying appendix prose says 0.9. The edition uses the table value and preserves both original quantitative material and its context.

TABLE IV: Mixed-Initiative Dialog Metrics
Metric (avg over each trial)
R = Robot, H = human
MICoBot (ours) LLM baseline
# R-Helpreqs 2.9±1.42.9\pm 1.4 0.9±1.10.9\pm 1.1
Initial H acceptance rate 55%±35%55\%\pm 35\% 70%±46%70\%\pm 46\%
H acceptance rate after R-negotiation 86%±27%86\%\pm 27\% 75%±40%75\%\pm 40\%
R-init dialogs 3.6±1.83.6\pm 1.8 0.9±1.10.9\pm 1.1
H-init dialogs 2.9±2.42.9\pm 2.4 2.3±2.92.3\pm 2.9
Initiative Shifts 2.4±2.12.4\pm 2.1 1.1±1.61.1\pm 1.6

Appendix -G Detailed Simulation Results

Controlled gridworld experiments vary synthetic humans’ willingness and language sentiment, then compare baseline and ablated systems.

Appendix -G 1 Setup

Eight conditions cross positive/negative dialog mood with nominal help probabilities 0, 0.3, 0.7 and 1.0. Each method receives ten trials in each condition. These LLM-controlled agents are a synthetic testbed, not additional human participants.

Appendix -G 2 Simulation Experiments

On simulated package pouring, MICoBot averages 63.75% success versus 45% for the LLM baseline, 47.5% random allocation and 37.5% RL. Robot-only initiative reaches 57.5%; human-only initiative falls to 5%.

Removing willingness estimation changes success only to 62.5%, while removing plan hierarchy yields 47.5%. The data give stronger support to hierarchy and robot initiative than to a large independent willingness-estimation gain. Results vary by condition and do not show universal per-condition dominance.

TABLE V: Simulation Task 1 Performance across different p~H,t\tilde{p}_{H,t} Values and Language Sentiments.
Human Parameters (Mood, p~H,t\tilde{p}_{H,t})
Method Metric Positive Mood Negative Mood Avg. (%)
0.0 0.3 0.7 1.0 0.0 0.3 0.7 1.0
Ours Success Rate 3/10 6/10 9/10 10/10 1/10 4/10 9/10 9/10 63.75
Num Plan Steps Completed 3.6/5 4.2/5 4.8/5 5.0/5 3.2/5 3.8/5 4.8/5 4.5/5 84.5
Prop. Plan Steps done by Human 0.1667 0.2381 0.3125 0.4 0.03125 0.1579 0.354 0.377 25.47
LLM Baseline Success Rate 2/10 2/10 4/10 7/10 3/10 6/10 6/10 6/10 45
Num Plan Steps Completed 3.4/5 3.4/5 3.7/5 4.4/5 3.6/5 4.2/5 4.0/5 4.2/5 77.25
Prop. Plan Steps done by Human 0.0588 0.05882 0.2162 0.1591 0.1111 0.1428 0.175 0.166 13.6
Random Agent Success Rate 2/10 5/10 6/10 7/10 2/10 3/10 6/10 7/10 47.5
Num Plan Steps Completed 3.4/5 3.5/5 4.0/5 4.4/5 3.4/5 2.8/5 4.0/5 4.4/5 74.75
Prop. Plan Steps done by Human 0.1176 0.4286 0.525 0.7045 0.1176 0.2143 0.525 0.7045 41.71
RL Success Rate 0/10 1/10 4/10 10/10 0/10 1/10 4/10 10/10 37.5
Num Plan Steps Completed 2.4/5 2.3/5 3.4/5 5.0/5 2.4/5 2.3/5 3.4/5 5.0/5 65.5
Prop. Plan Steps done by Human 0.125 0.1739 0.4412 0.54 0.125 0.1739 0.4412 0.54 32.0
Only R Init Success Rate 0/10 3/10 9/10 10/10 0/10 5/10 9/10 10/10 57.5
Num Plan Steps Completed 3.0/5 3.6/5 4.8/5 5.0/5 3.0/5 4.0/5 4.8/5 5.0/5 83
Prop. Plan Steps done by Human 0.0 0.1111 0.3542 0.4 0.0 0.225 0.354 0.4 23.05
Only H Init Success Rate 0/10 0/10 0/10 0/10 2/10 0/10 0/10 2/10 5.0
Num Plan Steps Completed 3.0/5 3.0/5 3.0/5 3.0/5 3.2/5 3.0/5 3.0/5 3.3/5 61.25
Prop. Plan Steps done by Human 0.0 0.0 0.0 0.0/3.0 0.1875 0.0 0.0 0.1212 3.86
Ours w/o p_help Success Rate 3/10 5/10 9/10 10/10 2/10 3/10 9/10 9/10 62.5
Num Plan Steps Completed 3.6/5 4.0/5 4.8/5 5.0/5 3.4/5 3.4/5 4.8/5 4.7/5 84.25
Prop. Plan Steps done by Human 0.1667 0.3 0.3333 0.38 0.1176 0.2059 0.3125 0.4468 28.29
Ours w/o Plan Hier. Success Rate 2/10 4/10 7/10 10/10 0/10 3/10 4/10 8/10 47.5
Num Plan Steps Completed 3.4/5 3.8/5 4.0/5 5.0/5 3.0/5 3.4/5 3.6/5 4.2/5 76
Prop. Plan Steps done by Human 0.0588 0.1316 0.25 0.24 0.0667 0.1176 0.1944 0.2381 16.22

Appendix -H User Study Details

The exact participant instructions and success criteria are retained to make the physical evaluation protocol inspectable.

Appendix -H 1 User Instructions

Participants were briefed on high-level steps and could work on unrelated tasks, accept/refuse help or propose splits. They were instructed to complete an accepted step successfully. Robot and person stayed on opposite sides of the table and acted sequentially.

A researcher reviewed every proposed physical action and could block unsafe motions. This gatekeeping is part of the reported experiment. The complete fourteen-item script below also records audio checks, initial object placement and voluntary withdrawal.

1. Thank you so much for coming for our user study! We wanted to remind you to review the RIS before proceeding, and that you may voluntarily opt-out of the study at any time.

2. You are working with the robot to perform the task of assembling the toy car. You must use the hexagonal drill bit to screw in the wheels, and the phillips drill bit to screw in the seat and the window. [Demonstrate these steps to the human]. You and the robot operate on a shared understanding of the plan. [Read the 4 high-level steps of the plan tree for this task. Do NOT discuss the low-level steps of the plan tree.]

3. Our goal is to simulate a home robot setting, where the human (you) is relatively busy with your own tasks, and once in a while you provide physical assistance and talk to the robot. So you are free to do work during each trial.

4. Once the robot asks you to do a step, and you accept, you must finish that step successfully.

5. We will perform 2 trials, each of a different method.

6. Both you and the robot can do a subset of the steps in the plan. You will communicate with the robot to determine who does what steps.

7. These are the objects you will work with during the task. I will move them now to their initial positions where they will start at the beginning of each trial. [Move objects to initial positions.]

8. For safety, I will gate-keep each of the robot’s physical actions. In other words, the actions are generated by the robot itself, but they will be displayed on the laptop screen with a confirmation message, and I can either allow that physical action to be executed by the robot, or block the action from being executed if it brings the robot to an unsafe location.

9. The robot will stay on the TV side of the coffee table, while you will sit on the couch and stay on the couch side of the coffee table.

10. You are free to get up off the couch if you want to volunteer to perform steps that involve going to the sink or shelf, but you can only go when the robot is stationary and waiting on the other side of the coffee table. Steps are done in sequential order; our system doesn’t support parallelization (agents working simultaneously).

11. You will be communicating to the robot through this headset. We will perform a mic-check now to make sure it can pick up your voice. [Do mic check.]

12. Now, this is what the robot will sound like when it talkes to you. [Play audio sample of the robot.] Try responding to it, and I will see if it can hear you.

13. The systems today can handle different kinds of dialog. (1) refusal/acceptance, (2) task allocation, such as (“Could you pour the package in the plate later?” Or: “I can pour the package onto the plate later.”), (3) silence—you don’t need to respond to the robot every time, and (4) a proposal to split up adjacent steps, such as “Please bring me the drill so that I can put on the wheels.” You may engage in any of these types of dialog, and the robot may also engage in them when communicating to you.

14. Do you have any questions before we start? I will let you know when each trial begins and ends. Sometimes trials may end prematurely.

Appendix -H 2 Success Rate

A primitive succeeds only if its goal state holds after execution; knocking a just-placed object off during arm retraction counts as failure. Completed-step fraction divides successful low-level steps by plan length, and whole-task success requires every step.

Appendix -I Real-world Failure Analysis

MICoBot fails four of eighteen trials: none on package pouring, two on car assembly and two on gift packing. Listed incidents include rejected indispensable help, inability to satisfy an out-of-plan request, grasp/release failures and a meta-planner parsing error.

The baseline fails thirteen trials, often assigning itself infeasible actions, alongside perception and manipulation errors. Incident counts include human-rectified failures and can exceed failed-trial counts; the Sankey diagrams and original breakdowns should not be treated as mutually exclusive one-error-per-trial totals.

Complete original Figure 9.
Fig. 9: MICoBot mainly fails from an inability to solicit human help. Metaplanner and primitive errors were also encountered.
Complete original Figure 10.
Fig. 10: The LLM baseline failed mainly by allocating itself steps it could not perform, due to the LLM’s lack of knowledge of its own affordances.

Appendix -J Fault Recovery

If generated code cannot execute, the meta-planner receives up to two more attempts. If those fail, the system drops the latest human utterance for two further retries. This fallback can restore execution at the cost of ignoring the most recent user preference.

Appendix -K RL Baseline Details

The hierarchical RL baseline chooses the agent, then a low-level robot policy selects physical moves or asks for help. Rewards arrive only after all five steps finish in order.

Training with uniformly varying human willingness failed to obtain nonzero returns, so the reported baseline was trained only with an always-helpful person. Its poor low-help evaluation partly reflects that training mismatch, not a general impossibility of learning collaborative allocation with RL.

Appendix -L Additional Experimental Investigations

A selected physical trace shows willingness estimates falling to 0.05 after refusals and rising toward 0.8 after cooperation. Positive remarks also affect the estimate; this qualitative trace does not calibrate the probability of the next response.

The controlled allocation example uses α=0.3, whereas physical trials use α=10. Its initial all-human allocation changes after a refusal. Table VI assigns steps 3–4 to the human at t=2, although the prose calls them steps 2–3; the source indexing inconsistency is preserved. Lower pH makes the negative human cost more negative in Equation 1, despite prose describing an increase.

Complete original Figure 11.
Fig. 11: From a real-world user study: MICoBot’s pH,tp_{H,t} estimation (blue line) reacts in real time to the human’s rejections (red), acceptances (green), and encouraging remarks. All dialog is shown as yy-labels. Green text denotes robot actions/dialog, and blue text denotes human dialog. The timestep tt increases from top to bottom on the yy-axis.
TABLE VI: Computed Best Task Allocation (and Agent Q-values) During a Sim Trial on Task 1.
Env. Timestep Step 1 Step 2 Step 3 Step 4 Step 5
t=0t=0 H (-9.6) H (-7.2) H (-13.2) H (-2.4) H (-2.4)
t=2t=2 R (-13.0) R (-9.0) H (-13.2) H (-4.8) R (-1.0)
t=6t=6 – R (-12.0) H (-13.2) H (-4.8) R (-1.0)
t=9t=9 – – H (-13.2) H (-4.8) R (-1.0)
t=16t=16 – – – – R (-3.0)

Appendix -M Meta-planner and p H Estimator Accuracy

On six selected user studies, 53 of 59 meta-planner programs match manual annotations (89.8%). Across 33 willingness estimates, mean absolute error is 0.11 and 76% lie within 0.15 of the comparison value.

That comparison value is the fraction of help requests accepted so far, not independently measured future willingness. The small selected sample does not establish broad next-turn probability calibration.

Appendix -N Further Connections to Prior Work

The framework combines prior ideas from physical/verbal action spaces, language-conditioned control and hierarchical planning.

Appendix -N 1 Agents with Both Physical and Verbal Actions

RL, imitation learning and multiagent communication have combined physical and verbal actions, mostly in simplified settings. MICoBot’s distinction is grounding freeform language in a supplied physical task hierarchy, dialog preferences and execution constraints.

Appendix -N 2 Natural Language and Robotics

The components draw on language-conditioned policies, LLM task planners, code-based policies, hierarchical control, vision-language representations and language-derived rewards. Mini-Behavior supplies the simulation substrate; combining these components does not remove their modeling assumptions.

Acknowledgments

The authors thank study participants, RobIn and Machine Learning Lab colleagues, debugging assistance and reviewers. Support includes DARPA TIAMAT HR0011-24-9-0428, Amazon, Emerson Electric and a UT Austin dissertation fellowship.

Concept demonstration

Open image ↗