At a glance
BUMBLE combines a vision-language model, object segmentation and depth, eight parameterized robot skills, and two forms of memory. It chooses a subtask and skill, grounds the skill’s parameters in the current scene, executes, and uses feedback for the next decision.
Across seven task/building combinations with ten trials each, Table I reports 47.1% success, versus 35.7% for an adapted COME baseline. Tasks involve as many as twelve skills and about fifteen minutes of execution, excluding VLM query time. Human-provided maps, landmarks, skill implementations, and precollected failure examples are part of the system; it does not learn a building-wide controller from scratch.
I Introduction
Building-wide mobile manipulation combines semantic choices with navigation and contact: find an appropriate drink, use an elevator, open a door, move an obstruction, and select the right object among distractors. A useful system needs both high-level reasoning and precisely grounded motor parameters.
BUMBLE uses the VLM as a decision interface over an existing skill library. Short-term memory records the current rollout; long-term memory supplies earlier failed predictions and explanations. The paper tests object/layout variation across three university buildings and demonstrates recovery by changing the next skill after failures.
II Related Work
Task-and-motion planning combines symbolic reasoning with physical feasibility but depends on supplied abstractions. Language-model planners provide semantic knowledge, while visual grounding helps connect instructions to objects. BUMBLE extends VLM mobile-manipulation approaches such as COME with a broader skill set and memory.
Its interactive-navigation skills move obstacles, open doors, and operate elevators. This adds semantic choices—such as avoiding a delicate object—to geometric route and contact decisions. The VLM selects actions; conventional mapping, planning, and controllers remain responsible for executing them.
III BUMBLE: VLM-Based Building-Wide MoMa
The architecture links open-world object perception, a fixed but extensible repertoire of skills, execution history, and a VLM that makes two successive predictions at each step.
III-A Perception System, Skill Library, and Memory
Perception supplies object geometry, skills turn discrete choices into motion, and memory supplies context for recovery and avoiding repeated mistakes.
Open-world Perception System
Grounded-SAM segments interactable objects. Depth back-projection provides their point clouds and distances to the robot. This supplies reachability and manipulation information that a VLM reading only RGB would not reliably recover. Pixel-level segmentation is a representation choice, not a guarantee against depth or mask errors.
Skill Library
Eight skills span landmark navigation, approaching objects, small base adjustments, picking up, pushing floor objects, opening doors, calling an elevator, and using it. Most parameters identify scene objects or discrete directions. GoToLandmark instead selects a human-collected image associated with a known pose on a floor map.
The VLM selects the skill and its parameters; the skill implementation maps them to base or arm commands. The restored appendix gives each original prompt and the constraints of its controller, including left-arm picking, push-only doors, and force-sensed elevator-button presses.
Memory
Short-term memory records each scene, subtask, skill, parameter choice, and detected outcome within the current trial. This supports responses to failed grasps, blocked paths, or invalid depth readings.
Long-term memory consists of human-flagged earlier prediction errors, their contexts, and VLM explanations. These examples are gathered before evaluation and frozen during testing. The reported benefit is therefore use of stored lessons, not autonomous online learning across the evaluation trials.
III-B VLM-based Decision Making
The first VLM query selects a subtask and skill; the second selects grounded parameters for that skill. Execution updates the short-term history before the next decision.
Subtask prediction and skill selection
The query combines the instruction, skill descriptions, current RGB image, location, and both memories. Set-of-Mark labels candidate objects in the image, while text provides measured distances. The VLM uses intermediate reasoning to choose an appropriate next action.
This avoids asking the VLM to interpret raw point clouds directly. It replaces explicit task-level search with iterative decisions, but retains lower-level motion planning and offers no completeness or optimality guarantee.
Skill parameter estimation
A second prompt combines the selected subtask, previous parameter mistakes, and visual markers for candidate choices. Base moves use directional markers; object skills use skill-specific segmentation queries, such as buttons for elevator use. The chosen parameter instantiates the controller and its outcome enters memory.
IV Experimental Evaluation
The main evaluation covers marker retrieval and diet-soda retrieval in three buildings, plus chair rearrangement in one: seventy trials total. Each task has three natural-language specifications. Initial floors/rooms, obstacles, object brands and positions, and clutter vary; clutter ranges from five to twenty-five distractors. The reported ninety-plus hours cover the broader evaluation effort.
Inner Monologue and COME receive BUMBLE’s skill library and short-term memory so their action repertoire is comparable. Inner Monologue uses text descriptions instead of RGB; COME uses vision but lacks the stored long-term failures. GPT-4o is the main backbone. An additional manually annotated dataset of about 120 images evaluates parameter selection for three skill types, rather than full closed-loop execution.
Table I gives BUMBLE 47.1% versus COME 35.7%, an 11.4-percentage-point difference; the prose’s 12.1% margin is inconsistent with those averages. BUMBLE ties COME for soda retrieval in Building 1 rather than strictly exceeding it in every cell. Inner Monologue is evaluated only in Building 1 after poor initial performance, so its 6.7% average has a narrower scope.
Offline parameter accuracy is 80.2% for BUMBLE, 72.6% for COME, and 61.7% for Inner Monologue. Removing intermediate reasoning reduces the average to 60.7%; replacing visual markers with textual object descriptions reduces it to 49.2%. Gains are not uniform: both vision methods tie on the two Pickup rows, and the no-marker ablation reaches 81.8% on pushing versus 81.0% for BUMBLE.
The backbone study reports 44.5/47.0/39.6/64.5% across Claude Haiku 3, Sonnet 3, Opus 3, and Sonnet 3.5; Gemini Flash/Pro 1.5 give 46.3/65.3%, and GPT-4o-mini/GPT-4o give 51.8/80.2%. Stronger models can help, but the Claude sequence is not monotonic. These are offline parameter-selection results, not building-wide success rates.
The failure chart attributes many errors to VLM choices, including wrong elevator buttons, unreachable adjustments, and wrong objects in clutter. It shows 32 successes and 38 failures, while the table’s per-condition counts imply 33 successes out of 70. Both originals are retained; the discrepancy prevents treating the failure chart as a perfectly reconciled breakdown of Table I. Repositioning after invalid depth readings illustrates recovery, not guaranteed fault tolerance.
Ten participants rate ten trials per method on two unseen tasks, choosing among five outcome categories. The prose reports means of 3.7 versus 2.6, but Figure 7’s displayed category percentages imply approximately 3.42 versus 2.31. The chart shows 22 points fewer ratings in the most severe failure category and 12 points fewer recoverable-failure ratings; these should not be relabeled a general 22% satisfaction increase. Ratings assess annotated rollout images, not a broad deployment study.
V Conclusion
BUMBLE shows how a VLM can coordinate diverse existing skills across rooms and floors, with stored failures improving some decisions. Reliability remains limited: fewer than half of the main task trials succeed, and clutter, geometry, sensors, and contact execution all matter.
Long-term context can grow expensive; retrieval or compression is future work. Human-collected landmarks remain necessary, and nonlinguistic skill interfaces need additional machinery. Greedy next-step decisions can select a merely adequate nearby object, motivating more explicit planning across multiple future steps.
VI Acknowledgments
The authors acknowledge robot infrastructure support, laboratory feedback, and funding from NSF, ONR, and DARPA TIAMAT.
VII Appendix
The appendix retains the full offline parameter table and restores method/evaluation details omitted by the source HTML conversion.
| Skill | BUMBLE | COME | Inner | BUMBLE | BUMBLE |
|---|---|---|---|---|---|
| Parameter | Monologue | w/o CoT [71] | w/o SoM [70] | ||
| Pickup [Obj.] (5-10 distractors) | |||||
| Pickup [Obj.] (20-25 distractors) | |||||
| PushObjectOnGround [Obj., Dir.] | |||||
| CallElevator [Button Location] | |||||
| Average |
VII-A Method details
Implementation details specify segmentation checkpoints, the eight skill interfaces, motion-planning software, and how failure examples are collected.
Perception System
The perception stack uses GroundingDINO checkpoint groundingdino_swinb_cogcoor for boxes and SAM-HQ checkpoint sam_hq_vit_b for masks, with code adapted from ORION.
Skill Library
Each skill exposes a semantically named interface and scene-grounded parameters. The following eight subsections retain the complete source prompts and summarize their mappings to robot commands.
goto_landmark
Select a marked landmark image on the desired floor. Its predefined 2D map pose becomes a navigation goal; global/local planners use the occupancy map and front lidar. Landmarks can include the elevator, but this skill navigates within a floor.
skill_name: goto_landmark
arguments: Selected landmark image from the
environment from various options.
description: Navigates to the landmark in the
environment. For instance, bedroom, kitchen,
tool shop, etc.move_base
Choose forward, backward, left, or right for a 30 cm base adjustment. Candidate resulting positions are shown on the image and transformed into map-frame targets. The skill is intended for final positioning, not inter-room travel.
skill_name: move_base
arguments: direction
description: Moves the robot base either forward,
backward, left, or right by 0.3 meters, w.r.t.
the camera view. This skill should only be used
to adjust the base of the robot in the final
few meters of navigation and not to traverse
between rooms.pick_up_object
Select a segmented object within arm reach. Its geometry defines approach and grasp poses for the left gripper; inverse kinematics and the joint controller execute the motion. This skill does not move the base or lift heavy furniture.
skill_name: pick_up_object
arguments: object_of_interest
description: pick_up_object skill moves its arms to
pick up object_of_interest. The pick_up_object
skill can only pick up objects within arm
reach and does not control the robot base. The
robot cannot pick up heavy objects like chairs,
tables, etc.push_object_on_ground
Choose an object and forward/left/right push direction in separate VLM queries. Predicted post-push positions are marked visually. Heuristics choose a base pose, then a predefined arm trajectory executes the push.
skill_name: push_object_on_ground
arguments: object, direction
description: Pushes an object on the ground one of
the directions. The robot can push objects that
are 2-3m away from the robot. The skill
decides which object and direction to push.
Example use cases: pushing obstacle to clear
the pathway for the robot to go forward,
pushing objects for rearrangement, etc.open_door
Choose the left or right arm for a push-door. The robot aligns, sets a predefined arm configuration, and drives forward using the forearm as contact. This is not a general latch-handling or pull-door skill.
skill_name: open_door
arguments: door_side
description: Opens the door by pushing on a certain
side of the door (left or right). The robot
moves forward with its door_side arm out to
push open the door.call_elevator
Select a segmented up/down button. The right wrist’s force-torque sensor detects a press threshold, after which the robot enters the elevator at a predefined map pose. The subtask supplies both current and destination floors.
skill_name: call_elevator
arguments: button position depending whether you
want to go to a floor above or below.
description: Equips the robot with calling the
elevator capability. The robot will push the
button selected in the argument to call the
elevator in the current floor. The subtask must
indicate the current floor number and the
destination floor number to go to. Example: ’Go
to the second floor from first floor.’use_elevator
Select the destination-floor button. After a force-threshold press, the robot exits and relocalizes on that floor’s 2D map. The prompt continues across PDF columns; it is restored here as a single complete block.
skill_name: use_elevator
arguments: button position of the floor to go to.
description: Equips the robot with using elevator
capabilities. The robot will push the button
selected in the argument. This skill is used to
change the floor of the robot after calling
the elevator. The subtask must indicate the
desired floor number to go to.ROS packages
The TIAGo implementation uses ROS gmapping for occupancy maps, AMCL for localization, and move_base with an A* global planner and a timed-elastic-band local planner. Arm motion uses the configured trajectory controller. Infrastructure is adapted from TeleMoMa.
Memory construction
Detected failures include NaN depth, infeasible inverse kinematics, and blocked collision-free paths. For long-term memory, the authors collect five annotated instances per skill, compare model predictions with ground truth, and retain only errors. This yields one to three examples for MoveBase, UseElevator, PushObjOnGround parameters and skill selection. Memory remains fixed during evaluation.
Task specifications and VLM checkpoints
The following source task wordings and dated backbone checkpoints are restored for reproducibility. These specify the tested conditions rather than current model recommendations.
Retrieve diet soda can: 1) “Could you grab me a drink that is low in calories?” 2) “Any chance you can find me a sugar-free soda?” 3) “I want something fizzy to drink, but I am on diet. Can you help me with that?” Retrieve colored marker: 1) “I want to color the sky in my drawing. Can you get me a marker?” 2) “I want to color grass in my drawing. Can you get me a marker?” 3) “I need to color some hearts. Can you get a marker for that?” Rearrange chairs: 1) “Could you make the seating chairs in the reception area more orderly?” 2) “Make the reception area more welcoming by arranging the chairs” 3) “Can you help me arrange the chairs in the reception area?” VLM checkpoints: GPT4o: gpt-4o-2024-05-13 GPT4o-mini: gpt-4o-mini-2024-07-18 Pro-1.5-Gemini: Gemini-1.5-Pro-Exp-0827 Flash-1.5-Gemini: Gemini-1.5-Flash-Exp-0827 Haiku-3-Claude: claude-3-haiku-20240307 Sonnet-3-Claude: claude-3-sonnet-20240229 Opus-3-Claude: claude-3-opus-20240229 Sonnet-3.5-Claude: claude-3-5-sonnet-20240620