Literature
Display
← All papers

GeT-USE: Learning Generalized Tool Usage for Bimanual Mobile Manipulation via Simulated Embodiment Extensions

Bohan Wu · Paul de La Sayette · Li Fei-Fei · Roberto Martín-Martín

· IROS 2025 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

This edition reviews the eight-page arXiv v1 dated 29 October 2025. The four authors are affiliated with Stanford and UT Austin; Li Fei-Fei and Roberto Martín-Martín contributed equally. All six figures and the complete results table are retained.

GeT-USE learns useful tool shapes by growing simulated blocks from a robot’s wrists and testing whether they support a task. It transfers the generated shapes and success/failure experience into depth-based tool selection, grasping and manipulation modules.

On three real TIAGo tasks, it succeeds in 37/60 trials (61.7%), versus 19/60 (31.7%) for the strongest compared baseline. The thirty-point aggregate improvement concerns these tabletop tasks and candidate objects, not arbitrary tools or environments.

I Introduction

An everyday object can function as a tool when its geometry compensates for the robot’s limitations. Choosing among imperfect alternatives requires assessing both task suitability and how the object can be grasped and controlled.

Inspired by body-schema adaptation, GeT-USE separates simulated tool design from real-world execution. A task-specific builder generates useful and unhelpful geometries; downstream visual modules learn selection and control from that experience. The final pipeline uses onboard sensing, while simulation training relies on privileged state and predefined manipulation strategies.

Original Figure 1, retaining all native raster pixels.
Fig. 1: GeT-USE: Generalized Tool-Usage via Simulated Embodiment Extensions. Left: a TIAGo robot achieves a bimanual mobile manipulation task that requires using other objects as tools: “Sweep the Carrot”. In simulation, GeT-USE explores different embodiment extensions by looking at (indicated by the “camera” icon) and building on top of its two wrists (“L”, “R”) until it finds a suitable one. GeT-USE then transfers the successful strategy to the real world by learning vision-based modules to: 1) select the best available object (top right), 2) grasp it (bottom right), and 3) use it (bottom right), all based on real depth images. This methodology allows the robot to learn generalized tool usage tasks in simulation that require bimanual mobile manipulation, and zero-shot transfer to the real-world.

II Related Work

Prior work learns tool use from procedural shapes, internet models, real demonstrations, foundation models or motion planning. These approaches differ in the objects supplied, assumptions about tool suitability, visual feedback and permitted control actions.

The main comparison is with TOG-Net. Two adapted variants retain GeT-USE’s selector and six-degree-of-freedom control while replacing learned tool generation with internet or procedural tool sets. This tests the tool-generation choice more directly than the original TOG-Net comparison, which also differs in selection and action space.

The claim to novelty is a combined visual selection/grasp/manipulation pipeline for mobile bimanual tasks. Related-work limitations and novelty statements are the authors’ characterization, rather than an exhaustive proof about every prior system.

Original Figure 2, retaining all native raster pixels.
Fig. 2: The GeT-USE Framework. Below we explain training (top) and deploying in the real-world (bottom) a generalized tool usage solution with GeT-USE. Training with GeT-USE (top) is a two-step procedure: In the first step, the agent is asked to solve a simulated version of the task (top, most left) and trains a generalized tool-building policy (top, second-left), π𝑔𝑡𝑏\pi_{\mathit{gtb}}, that explores by extending its own embodiment by appending elements (blocks) until it finds an extension that can be used to perform the task. In the second step, the information of the tool-building policy is transferred to visual modules that can be used in the real world: i) a generalized tool selector (top, third-left), 𝒟𝑔𝑡𝑠\mathcal{D}_{\mathit{gts}}, trained to predict the best grasping area (first generated blocks) of the generalized tools using successful (green tick) and failed generated tools (red cross), ii) a visuomotor generalized tool grasping policy (top, second-right), π𝑔𝑡𝑔\pi_{\mathit{gtg}}, trained to grasp the tools created by π𝑔𝑡𝑏\pi_{\mathit{gtb}} using depth images as input and which success is detected automatically by a success detector, 𝒟𝑔𝑠\mathcal{D}_{\mathit{gs}}, and iii) a visuomotor generalized tool manipulation policy (top, second-right), π𝑔𝑡𝑚\pi_{\mathit{gtm}} that learns to output bimanual mobile manipulation commands to control the robot and achieve the task with tool generated by π𝑔𝑡𝑏\pi_{\mathit{gtb}}. These modules are directly transferred and applied in the real world (bottom): GeT-USE first captures a depth image of the table-top objects and generates object proposals. Each proposal is fed into GeT-USE’s tool selector, 𝒟𝑔𝑡𝑠\mathcal{D}_{\mathit{gts}} that selects the best object to use as generalized tool. GeT-USE then uses the tool-grasping and manipulation policies, π𝑔𝑡𝑔\pi_{\mathit{gtg}} and π𝑔𝑡𝑚\pi_{\mathit{gtm}}, to grasp and manipulate the selected tool to achieve the task. This process allows GeT-USE to create solutions that solve tasks requiring generalized tool usage by leveraging simulation and transferring into visual modules for real world execution.

III GeT-USE: Learning Ge neralized T ool- U sage via S imulated E mbodiment Extensions

Training has two stages. First, the builder extends the simulated wrists until a privileged scripted strategy can solve the task. Second, generated tools train a selector, a tool-grasping policy and a tool-manipulation policy for transfer to the real robot.

The paper describes the learning stages using state, observation and action spaces, dynamics, rewards, discount, initial-state distribution and finite horizon. Their original formal definitions remain below. The visual policies receive observations rather than full simulator state; each stage has its own action and reward design.

Original Figure 3, retaining all native raster pixels.
Fig. 3: Example rollouts of GeT-USE’s tool-building policy for Sweeping, Hook_and_grasp, and Decanting tasks. The bottom of each block details the tool geometry at timestep tt during the tool-building policy rollout. In Sweeping, GeT-USE builds one tool for each wrist with each timestep separated by a dashed line. The left (L) wrist is marked gray; the right (R) wrist is marked black. In Hook_and_grasp and Decanting, the robot builds only one tool on either arm. As such, GeT-USE incrementally builds complex tools using small blocks one timestep at a time.
Equation

ℳ=⟨𝒮,𝒪,𝒜,𝕋,ℛ,γ,ρ0,ℋ⟩\mathcal{M}=\langle\mathcal{S},\mathcal{O},\mathcal{A},\mathbb{T},\mathcal{R},\gamma,\rho_{0},\mathcal{H}\rangle

𝕋:𝒮×𝒜→𝒮\mathbb{T}:\mathcal{S}\times\mathcal{A}\to\mathcal{S}

γ∈[0,1)\gamma\in[0,1)

III-A Learning a tool-building policy via embodiment extensions in simulation

The builder observes two simulated depth images: the task scene and the wrist(s) being extended. Each action specifies a new block’s relative 3D position and three dimensions. Blocks are described as variable-sized cubes with a reported maximum volume of 2 cm³; successive additions can form hooks, sweepers or containers.

Terminating construction triggers a predefined task-specific manipulation strategy using simulator object locations. Its success supplies the main builder reward, with additional task-specific shaping. Geometry is therefore evaluated relative to those strategies and reward choices, not all possible ways to use an object.

The resulting tool successes and failures supervise the visual selector and provide shapes for grasp/manipulation training. The paper does not specify the complete RL optimizer, learning schedule, episode budget or total generated-tool count.

III-B Learning to select generalized tools from depth images

A failed generated tool receives an all-zero image target. For a successful tool, pixels covering the first blocks near the wrist—the intended handle—are labeled one and the rest zero. The selector regresses this binary mask with an L₂ loss from depth.

At deployment, the highest predicted pixel value scores each candidate crop. This is a learned task-affordance score, not a demonstrated calibrated success probability. Handle supervision is tied to the builder’s attachment convention, and errors in object proposals or depth can still affect selection.

Original Figure 4, retaining all native raster pixels.
Fig. 4: Example object preferences output by GeT-USE’s generalized tool selector module. GeT-USE’s generalized tool selector ranks each object based on their suitability to serve as a sweeper in Sweeping (left), a hook in Hook_and_grasp (middle), and a decanter in Decanting (right). The green “1”, “2”, and “3” in each block represent the tool selector’s preference with “1” being the highest ranked. In this way, GeT-USE “makes the best of what it has”, even when the ideal tool does not exist on the table.

III-C Learning a visuomotor generalized tool-grasping policy

A generated successful tool is detached, placed randomly in the scene and grasped by the robot’s regular parallel-jaw hands. The policy receives scene/tool depth images and joint angles and commands base, torso, head, grippers and arms in joint space.

Dense reward depends on wrist-to-tool-base relative pose; terminal success requires holding the tool in its original embodiment-extension pose. A separate depth-based grasp detector predicts this success condition and triggers the next stage. Its independent detection accuracy is not reported.

III-D Learning a visuomotor tool-manipulation policy

The manipulation policy trains with useful generated extensions, receiving a depth image each step and controlling the robot to complete the task. Binary task success and distance-based shaping provide reward.

A depth-based manipulation-success detector decides when execution is complete. Transfer depends on the grasped real object sufficiently matching the geometry, grasp pose and dynamics represented during simulated training.

IV Deploying GeT-USE Policies in Real-World

An off-the-shelf 2D object detector proposes candidate objects from the real camera view. Corresponding depth crops are scored by the selector; the highest-scoring candidate is passed, together with the scene depth, to the grasp policy. Grasp detection hands control to manipulation, which runs until its success detector fires.

The core learned modules use depth, but the complete real-world pipeline also uses RGB for object proposals and proprioception for grasping. Thus the abstract’s depth-only description does not cover every pipeline input. The paper does not report a separate robustness study for proposal or success-detector failures.

Original Figure 5, retaining all native raster pixels.
Fig. 5: Simulated and Real-World Version of the Tasks. Left to right. Three tasks in our experiments: Sweeping, Hook_and_grasp, and Decanting. Top: example tool(s) generated by GeT-USE’s trained policy next to a simulation version of the tasks used for embodiment exploration and policy training of bimanual mobile manipulation tasks. Bottom: example of the real-world version of the tasks. By learning from diverse tools and environments in simulation, GeT-USE successfully chooses the best tool in the real world, whether the ideal tool is present or not, and generalizes its policies to real-world objects, environments, and lighting conditions.

V Experimental Evaluation

Hardware is a 22-DOF TIAGo: a three-DOF mobile base, two seven-DOF arms, torso, two-DOF head and two parallel grippers. Head-camera images are 224×224 at 3 Hz. iGibson training uses sixteen 24-GB GPUs, batch 128, and six convolutional plus three fully connected layers in each visual architecture.

Each method is tested on twenty randomized trials per task, with real objects and conditions unseen during simulation training. No real-data fine-tuning is described. Evaluation uses three predefined task families and more than twenty object categories; it is not open-ended task learning.

Sweeping: six to ten candidate objects are offered without an actual sweeper. The robot selects a pusher and receiver, sweeps the target up within 600 seconds, then lifts both arms 10 cm; the target must remain contained. Builder shaping favors extension away from the wrist.

Hook and grasp: three to five candidates are offered without an actual hook. The target starts out of reach under a barrier that blocks top grasping. One tool retrieves it and the other hand lifts it 10 cm within 600 seconds. Builder shaping favors proximity of the newest block to the target.

Decanting: the robot transfers a small solid object from a source container into a selected temporary container, then into the final bowl within 600 seconds. Three to five additional candidate objects are offered. This experiment transfers a solid object; the discussion of liquids is motivation, not a liquid-pouring evaluation. Builder shaping again favors extension away from the wrist.

GeT-USE succeeds on 16/20 hooking, 11/20 sweeping and 10/20 decanting trials. TOG-Net-Internet obtains 9/6/4 successes and TOG-Net-Proc 11/2/1; original TOG-Net obtains zero. Both adapted variants include GeT-USE’s selector and six-DOF control. Training-tool counts and equal-compute comparisons are not reported.

Removing tool selection reduces the aggregate from 61.7% to 11.7%, a fifty-percentage-point drop. Restricting grasping to top-down yields 0%; restricting manipulation yields 3.3%. The corresponding drops are 61.7 and 58.3 points from the trial counts, rather than the prose’s approximate 60% and 57%. These are percentage-point differences, not relative percentage changes. Original table values remain unchanged.

The results support useful selection and flexible wrist motion in these tasks, while explanations involving internet-tool overfitting or procedural-tool limits are interpretations without separate geometry/data-budget ablations. Sixty trials per method and no repeated-training-seed uncertainty limit broader conclusions.

Failures include targets slipping under a tilted sweeper, hooks pushing targets beyond reach, and controller vibration causing missed decants. The authors attribute the first two to dynamics mismatch and the last to hardware; those explanations are not independently isolated, and all remain end-to-end system failures.

TABLE I: Successful Trials Out of 20 and Corresponding Success Rates for GeT-USE, State-of-the-Art Baselines and Ablations
GeT-USE TOG-Net-Internet TOG-Net-Proc TOG-Net GeT-USE-No-Ts GeT-USE-Topdown-Grasp GeT-USE-Topdown-Manip
Hook_and_grasp 16/20 9/20 11/20 0/20 4/20 0/20 2/20
Sweeping 11/20 6/20 2/20 0/20 2/20 0/20 0/20
Decanting 10/20 4/20 1/20 0/20 1/20 0/20 0/20
Total 37/60 (61.7%) 19/60 (31.7%) 14/60 (23.3%) 0/60 (0%) 7/60 (11.7%) 0/60 (0%) 2/60 (3.3%)
Original Figure 6, retaining all native raster pixels.
Fig. 6: All real-world objects used in our experiments for the three generalized tool usage tasks. They include useful and adversarial (useless) objects for each task. The ability to explore embodiment extensions in simulation enables GeT-USE to experiment with diverse tools and transfer vision-based modules that learned from them to the real-world to use diverse objects as tools.

VI Conclusion

Learning task-relevant extensions supplies a useful simulated curriculum for selecting and operating imperfect real objects. The demonstrated scope is rigid objects, parallel grippers and three predefined tasks.

The method assumes sufficiently accurate object dynamics. Deformable materials, liquids, richer contact behavior, dexterous hands and articulated tools such as scissors remain future work. Controller and dynamics gaps already affect the reported rigid-object tasks.

VII Acknowledgment

The authors thank Kenneth Salisbury and his laboratory’s students for supporting the research.

Concept demonstration

Open image ↗