Literature
Display
← All papers

BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation

Rutav Shah · Albert Yu · Yifeng Zhu · Yuke Zhu · Roberto Martín-Martín

· ICRA 2025 ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

BUMBLE combines a vision-language model, object segmentation and depth, eight parameterized robot skills, and two forms of memory. It chooses a subtask and skill, grounds the skill’s parameters in the current scene, executes, and uses feedback for the next decision.

Across seven task/building combinations with ten trials each, Table I reports 47.1% success, versus 35.7% for an adapted COME baseline. Tasks involve as many as twelve skills and about fifteen minutes of execution, excluding VLM query time. Human-provided maps, landmarks, skill implementations, and precollected failure examples are part of the system; it does not learn a building-wide controller from scratch.

I Introduction

Building-wide mobile manipulation combines semantic choices with navigation and contact: find an appropriate drink, use an elevator, open a door, move an obstruction, and select the right object among distractors. A useful system needs both high-level reasoning and precisely grounded motor parameters.

BUMBLE uses the VLM as a decision interface over an existing skill library. Short-term memory records the current rollout; long-term memory supplies earlier failed predictions and explanations. The paper tests object/layout variation across three university buildings and demonstrates recovery by changing the next skill after failures.

Refer to caption
Fig. 1: Building-wide mobile manipulation. At a building-wide scale, mobile manipulation tasks involve sequencing multiple skills like pushing obstacles or opening doors to clear robot pathways, using elevators to reach a destination floor, rearranging chairs in the workspace, and retrieving objects. We present a framework, BUMBLE, that can solve long-horizon, spatially expansive tasks across different buildings.
Refer to caption
Fig. 2: Building-wide Mobile Manipulation tasks require navigating various rooms and floors while interacting with diverse objects. To solve such long-horizon tasks, BUMBLE leverages a diverse skill library enabling navigation and interaction, with parameterized skills adaptable to different scenes. At each skill execution step (circled numbers), BUMBLE uses VLM’s reasoning capabilities to select the next skill (blue text) and skill parameters ([…]) and recover from failures (red text) to solve building-wide tasks effectively.

II Related Work

Task-and-motion planning combines symbolic reasoning with physical feasibility but depends on supplied abstractions. Language-model planners provide semantic knowledge, while visual grounding helps connect instructions to objects. BUMBLE extends VLM mobile-manipulation approaches such as COME with a broader skill set and memory.

Its interactive-navigation skills move obstacles, open doors, and operate elevators. This adds semantic choices—such as avoiding a delicate object—to geometric route and contact decisions. The VLM selects actions; conventional mapping, planning, and controllers remain responsible for executing them.

Refer to caption
Fig. 3: BUMBLE Architecture. Given a free-form text instruction, skill library (top Left), and short- and long-term memory (bottom middle and left),  BUMBLE iteratively perceives the environment through onboard RBGD sensors (top left), predicts parameterized skills (top middle and right), and executes them in the environment. The predicted skill and its parameters are executed and stored in short-term memory for iterative prediction (bottom right).

III BUMBLE: VLM-Based Building-Wide MoMa

The architecture links open-world object perception, a fixed but extensible repertoire of skills, execution history, and a VLM that makes two successive predictions at each step.

III-A Perception System, Skill Library, and Memory

Perception supplies object geometry, skills turn discrete choices into motion, and memory supplies context for recovery and avoiding repeated mistakes.

Open-world Perception System

Grounded-SAM segments interactable objects. Depth back-projection provides their point clouds and distances to the robot. This supplies reachability and manipulation information that a VLM reading only RGB would not reliably recover. Pixel-level segmentation is a representation choice, not a guarantee against depth or mask errors.

Skill Library

Eight skills span landmark navigation, approaching objects, small base adjustments, picking up, pushing floor objects, opening doors, calling an elevator, and using it. Most parameters identify scene objects or discrete directions. GoToLandmark instead selects a human-collected image associated with a known pose on a floor map.

The VLM selects the skill and its parameters; the skill implementation maps them to base or arm commands. The restored appendix gives each original prompt and the constraints of its controller, including left-arm picking, push-only doors, and force-sensed elevator-button presses.

Memory

Short-term memory records each scene, subtask, skill, parameter choice, and detected outcome within the current trial. This supports responses to failed grasps, blocked paths, or invalid depth readings.

Long-term memory consists of human-flagged earlier prediction errors, their contexts, and VLM explanations. These examples are gathered before evaluation and frozen during testing. The reported benefit is therefore use of stored lessons, not autonomous online learning across the evaluation trials.

III-B VLM-based Decision Making

The first VLM query selects a subtask and skill; the second selects grounded parameters for that skill. Execution updates the short-term history before the next decision.

Refer to caption
Fig. 4: Execution trace for a user instruction. For each decision step in the execution trace, we show the image observation with grounded skill parameters as markers, the skill name (blue text) and parameter executed, and a brief language description of the step (black) to improve readability. In trying to execute some skills, the robot fails (red text), in which case the VLM adaptively predicts the next skill.

Subtask prediction and skill selection

The query combines the instruction, skill descriptions, current RGB image, location, and both memories. Set-of-Mark labels candidate objects in the image, while text provides measured distances. The VLM uses intermediate reasoning to choose an appropriate next action.

This avoids asking the VLM to interpret raw point clouds directly. It replaces explicit task-level search with iterative decisions, but retains lower-level motion planning and offers no completeness or optimality guarantee.

Skill parameter estimation

A second prompt combines the selected subtask, previous parameter mistakes, and visual markers for candidate choices. Base moves use directional markers; object skills use skill-specific segmentation queries, such as buttons for elevator use. The chosen parameter instantiates the controller and its outcome enters memory.

TABLE I: Success rate (%) from 1010 trials in building-wide tasks

Retrieve Retrieve Rearrange
marker soda can chairs Avg.
Building B1 B2 B3 B1 B2 B3 B1 (%)
IM [72] 1010 – – 00 – – 1010 6.76.7
COME [28] 4040 3030 4040 40 3030 3030 4040 35.735.7
BUMBLE 60 40 50 40 50 40 50 47.1\mathbf{47.1}

IV Experimental Evaluation

The main evaluation covers marker retrieval and diet-soda retrieval in three buildings, plus chair rearrangement in one: seventy trials total. Each task has three natural-language specifications. Initial floors/rooms, obstacles, object brands and positions, and clutter vary; clutter ranges from five to twenty-five distractors. The reported ninety-plus hours cover the broader evaluation effort.

Inner Monologue and COME receive BUMBLE’s skill library and short-term memory so their action repertoire is comparable. Inner Monologue uses text descriptions instead of RGB; COME uses vision but lacks the stored long-term failures. GPT-4o is the main backbone. An additional manually annotated dataset of about 120 images evaluates parameter selection for three skill types, rather than full closed-loop execution.

Table I gives BUMBLE 47.1% versus COME 35.7%, an 11.4-percentage-point difference; the prose’s 12.1% margin is inconsistent with those averages. BUMBLE ties COME for soda retrieval in Building 1 rather than strictly exceeding it in every cell. Inner Monologue is evaluated only in Building 1 after poor initial performance, so its 6.7% average has a narrower scope.

Offline parameter accuracy is 80.2% for BUMBLE, 72.6% for COME, and 61.7% for Inner Monologue. Removing intermediate reasoning reduces the average to 60.7%; replacing visual markers with textual object descriptions reduces it to 49.2%. Gains are not uniform: both vision methods tie on the two Pickup rows, and the no-marker ablation reaches 81.8% on pushing versus 81.0% for BUMBLE.

The backbone study reports 44.5/47.0/39.6/64.5% across Claude Haiku 3, Sonnet 3, Opus 3, and Sonnet 3.5; Gemini Flash/Pro 1.5 give 46.3/65.3%, and GPT-4o-mini/GPT-4o give 51.8/80.2%. Stronger models can help, but the Claude sequence is not monotonic. These are offline parameter-selection results, not building-wide success rates.

The failure chart attributes many errors to VLM choices, including wrong elevator buttons, unreachable adjustments, and wrong objects in clutter. It shows 32 successes and 38 failures, while the table’s per-condition counts imply 33 successes out of 70. Both originals are retained; the discrepancy prevents treating the failure chart as a perfectly reconciled breakdown of Table I. Repositioning after invalid depth readings illustrates recovery, not guaranteed fault tolerance.

Ten participants rate ten trials per method on two unseen tasks, choosing among five outcome categories. The prose reports means of 3.7 versus 2.6, but Figure 7’s displayed category percentages imply approximately 3.42 versus 2.31. The chart shows 22 points fewer ratings in the most severe failure category and 12 points fewer recoverable-failure ratings; these should not be relabeled a general 22% satisfaction increase. Ratings assess annotated rollout images, not a broad deployment study.

Original figure
Fig. 5: Success rate (%) of VLMs in predicting skill parameters. The models are arranged with increasing capabilities, measured as per the Vision Arena [73], from left to right in each model series.
Original figure
Fig. 6: Success and Failure Categorization from 70 trials. Most failures involve VLM reasoning, such as predicting actions that lead to a collision with the environment, picking up the wrong object, especially when there are 20-25 distractors, or pressing the wrong elevator button.
Original figure
Fig. 7: Human ratings on task completion and user satisfaction (n=10n=10 participants, 1010 robot trials evaluated per participant per method) on a 55-point Likert scale. BUMBLE is rated better than COME at aligning to human preferences.

V Conclusion

BUMBLE shows how a VLM can coordinate diverse existing skills across rooms and floors, with stored failures improving some decisions. Reliability remains limited: fewer than half of the main task trials succeed, and clutter, geometry, sensors, and contact execution all matter.

Long-term context can grow expensive; retrieval or compression is future work. Human-collected landmarks remain necessary, and nonlinguistic skill interfaces need additional machinery. Greedy next-step decisions can select a merely adequate nearby object, motivating more explicit planning across multiple future steps.

VI Acknowledgments

The authors acknowledge robot infrastructure support, laboratory feedback, and funding from NSF, ONR, and DARPA TIAMAT.

VII Appendix

The appendix retains the full offline parameter table and restores method/evaluation details omitted by the source HTML conversion.

Skill BUMBLE COME Inner BUMBLE BUMBLE
Parameter Monologue w/o CoT [71] w/o SoM [70]
Pickup [Obj.] (5-10 distractors) 80.080.0 80.080.0 65.065.0 80.080.0 50.050.0
Pickup [Obj.] (20-25 distractors) 65.065.0 65.065.0 65.065.0 60.060.0 40.040.0
PushObjectOnGround [Obj., Dir.] 81.081.0 70.470.4 56.856.8 64.364.3 81.881.8
CallElevator [Button Location] 95.095.0 75.075.0 60.060.0 40.040.0 25.025.0
Average 80.2\mathbf{80.2} 72.672.6 61.761.7 60.760.7 49.249.2
TABLE II: Success Rate (%) in predicting skill parameters using OfflineSkillDataset.

VII-A Method details

Implementation details specify segmentation checkpoints, the eight skill interfaces, motion-planning software, and how failure examples are collected.

Perception System

The perception stack uses GroundingDINO checkpoint groundingdino_swinb_cogcoor for boxes and SAM-HQ checkpoint sam_hq_vit_b for masks, with code adapted from ORION.

Skill Library

Each skill exposes a semantically named interface and scene-grounded parameters. The following eight subsections retain the complete source prompts and summarize their mappings to robot commands.

goto_landmark

Select a marked landmark image on the desired floor. Its predefined 2D map pose becomes a navigation goal; global/local planners use the occupancy map and front lidar. Landmarks can include the elevator, but this skill navigates within a floor.

skill_name: goto_landmark
arguments: Selected landmark image from the
    environment from various options.
description: Navigates to the landmark in the
    environment. For instance, bedroom, kitchen,
    tool shop, etc.

navigate_to_point_on_ground

Select a segmented visible object. Its pose determines a nearby ground point, transformed to a 2D map goal for base navigation. This approaches the object rather than grasping it.

skill_name: navigate_to_point_on_ground
arguments: object
description: Moves the robot to a point near the
    selected object. This skill can be used to move
     to a point in the room to perform a task,
    example, navigating near the toaster to make a
    toast.

move_base

Choose forward, backward, left, or right for a 30 cm base adjustment. Candidate resulting positions are shown on the image and transformed into map-frame targets. The skill is intended for final positioning, not inter-room travel.

skill_name: move_base
arguments: direction
description: Moves the robot base either forward,
    backward, left, or right by 0.3 meters, w.r.t.
    the camera view. This skill should only be used
     to adjust the base of the robot in the final
    few meters of navigation and not to traverse
    between rooms.

pick_up_object

Select a segmented object within arm reach. Its geometry defines approach and grasp poses for the left gripper; inverse kinematics and the joint controller execute the motion. This skill does not move the base or lift heavy furniture.

skill_name: pick_up_object
arguments: object_of_interest
description: pick_up_object skill moves its arms to
     pick up object_of_interest. The pick_up_object
     skill can only pick up objects within arm
    reach and does not control the robot base. The
    robot cannot pick up heavy objects like chairs,
     tables, etc.

push_object_on_ground

Choose an object and forward/left/right push direction in separate VLM queries. Predicted post-push positions are marked visually. Heuristics choose a base pose, then a predefined arm trajectory executes the push.

skill_name: push_object_on_ground
arguments: object, direction
description: Pushes an object on the ground one of
    the directions. The robot can push objects that
     are 2-3m away from the robot. The skill
    decides which object and direction to push.
    Example use cases: pushing obstacle to clear
    the pathway for the robot to go forward,
    pushing objects for rearrangement, etc.

open_door

Choose the left or right arm for a push-door. The robot aligns, sets a predefined arm configuration, and drives forward using the forearm as contact. This is not a general latch-handling or pull-door skill.

skill_name: open_door
arguments: door_side
description: Opens the door by pushing on a certain
     side of the door (left or right). The robot
    moves forward with its door_side arm out to
    push open the door.

call_elevator

Select a segmented up/down button. The right wrist’s force-torque sensor detects a press threshold, after which the robot enters the elevator at a predefined map pose. The subtask supplies both current and destination floors.

skill_name: call_elevator
arguments: button position depending whether you
    want to go to a floor above or below.
description: Equips the robot with calling the
    elevator capability. The robot will push the
    button selected in the argument to call the
    elevator in the current floor. The subtask must
     indicate the current floor number and the
    destination floor number to go to. Example: ’Go
     to the second floor from first floor.’

use_elevator

Select the destination-floor button. After a force-threshold press, the robot exits and relocalizes on that floor’s 2D map. The prompt continues across PDF columns; it is restored here as a single complete block.

skill_name: use_elevator
arguments: button position of the floor to go to.
description: Equips the robot with using elevator
    capabilities. The robot will push the button
    selected in the argument. This skill is used to
     change the floor of the robot after calling
    the elevator. The subtask must indicate the
    desired floor number to go to.

ROS packages

The TIAGo implementation uses ROS gmapping for occupancy maps, AMCL for localization, and move_base with an A* global planner and a timed-elastic-band local planner. Arm motion uses the configured trajectory controller. Infrastructure is adapted from TeleMoMa.

Memory construction

Detected failures include NaN depth, infeasible inverse kinematics, and blocked collision-free paths. For long-term memory, the authors collect five annotated instances per skill, compare model predictions with ground truth, and retain only errors. This yields one to three examples for MoveBase, UseElevator, PushObjOnGround parameters and skill selection. Memory remains fixed during evaluation.

Task specifications and VLM checkpoints

The following source task wordings and dated backbone checkpoints are restored for reproducibility. These specify the tested conditions rather than current model recommendations.

Retrieve diet soda can:
1) “Could you grab me a drink that is low in calories?”
2) “Any chance you can find me a sugar-free soda?”
3) “I want something fizzy to drink, but I am on diet. Can you help me with that?”

Retrieve colored marker:
1) “I want to color the sky in my drawing. Can you get me a marker?”
2) “I want to color grass in my drawing. Can you get me a marker?”
3) “I need to color some hearts. Can you get a marker for that?”

Rearrange chairs:
1) “Could you make the seating chairs in the reception area more orderly?”
2) “Make the reception area more welcoming by arranging the chairs”
3) “Can you help me arrange the chairs in the reception area?”

VLM checkpoints:
GPT4o: gpt-4o-2024-05-13
GPT4o-mini: gpt-4o-mini-2024-07-18
Pro-1.5-Gemini: Gemini-1.5-Pro-Exp-0827
Flash-1.5-Gemini: Gemini-1.5-Flash-Exp-0827
Haiku-3-Claude: claude-3-haiku-20240307
Sonnet-3-Claude: claude-3-sonnet-20240229
Opus-3-Claude: claude-3-opus-20240229
Sonnet-3.5-Claude: claude-3-5-sonnet-20240620
Concept demonstration

Open image ↗