At a glance
This edition reviews the eight-page arXiv v1 dated 29 October 2025. The four authors are affiliated with Stanford and UT Austin; Li Fei-Fei and Roberto Martín-Martín contributed equally. All six figures and the complete results table are retained.
GeT-USE learns useful tool shapes by growing simulated blocks from a robot’s wrists and testing whether they support a task. It transfers the generated shapes and success/failure experience into depth-based tool selection, grasping and manipulation modules.
On three real TIAGo tasks, it succeeds in 37/60 trials (61.7%), versus 19/60 (31.7%) for the strongest compared baseline. The thirty-point aggregate improvement concerns these tabletop tasks and candidate objects, not arbitrary tools or environments.
I Introduction
An everyday object can function as a tool when its geometry compensates for the robot’s limitations. Choosing among imperfect alternatives requires assessing both task suitability and how the object can be grasped and controlled.
Inspired by body-schema adaptation, GeT-USE separates simulated tool design from real-world execution. A task-specific builder generates useful and unhelpful geometries; downstream visual modules learn selection and control from that experience. The final pipeline uses onboard sensing, while simulation training relies on privileged state and predefined manipulation strategies.

II Related Work
Prior work learns tool use from procedural shapes, internet models, real demonstrations, foundation models or motion planning. These approaches differ in the objects supplied, assumptions about tool suitability, visual feedback and permitted control actions.
The main comparison is with TOG-Net. Two adapted variants retain GeT-USE’s selector and six-degree-of-freedom control while replacing learned tool generation with internet or procedural tool sets. This tests the tool-generation choice more directly than the original TOG-Net comparison, which also differs in selection and action space.
The claim to novelty is a combined visual selection/grasp/manipulation pipeline for mobile bimanual tasks. Related-work limitations and novelty statements are the authors’ characterization, rather than an exhaustive proof about every prior system.

III GeT-USE: Learning Ge neralized T ool- U sage via S imulated E mbodiment Extensions
Training has two stages. First, the builder extends the simulated wrists until a privileged scripted strategy can solve the task. Second, generated tools train a selector, a tool-grasping policy and a tool-manipulation policy for transfer to the real robot.
The paper describes the learning stages using state, observation and action spaces, dynamics, rewards, discount, initial-state distribution and finite horizon. Their original formal definitions remain below. The visual policies receive observations rather than full simulator state; each stage has its own action and reward design.

Equation
III-A Learning a tool-building policy via embodiment extensions in simulation
The builder observes two simulated depth images: the task scene and the wrist(s) being extended. Each action specifies a new block’s relative 3D position and three dimensions. Blocks are described as variable-sized cubes with a reported maximum volume of 2 cm³; successive additions can form hooks, sweepers or containers.
Terminating construction triggers a predefined task-specific manipulation strategy using simulator object locations. Its success supplies the main builder reward, with additional task-specific shaping. Geometry is therefore evaluated relative to those strategies and reward choices, not all possible ways to use an object.
The resulting tool successes and failures supervise the visual selector and provide shapes for grasp/manipulation training. The paper does not specify the complete RL optimizer, learning schedule, episode budget or total generated-tool count.
III-B Learning to select generalized tools from depth images
A failed generated tool receives an all-zero image target. For a successful tool, pixels covering the first blocks near the wrist—the intended handle—are labeled one and the rest zero. The selector regresses this binary mask with an L₂ loss from depth.
At deployment, the highest predicted pixel value scores each candidate crop. This is a learned task-affordance score, not a demonstrated calibrated success probability. Handle supervision is tied to the builder’s attachment convention, and errors in object proposals or depth can still affect selection.

III-C Learning a visuomotor generalized tool-grasping policy
A generated successful tool is detached, placed randomly in the scene and grasped by the robot’s regular parallel-jaw hands. The policy receives scene/tool depth images and joint angles and commands base, torso, head, grippers and arms in joint space.
Dense reward depends on wrist-to-tool-base relative pose; terminal success requires holding the tool in its original embodiment-extension pose. A separate depth-based grasp detector predicts this success condition and triggers the next stage. Its independent detection accuracy is not reported.
III-D Learning a visuomotor tool-manipulation policy
The manipulation policy trains with useful generated extensions, receiving a depth image each step and controlling the robot to complete the task. Binary task success and distance-based shaping provide reward.
A depth-based manipulation-success detector decides when execution is complete. Transfer depends on the grasped real object sufficiently matching the geometry, grasp pose and dynamics represented during simulated training.
IV Deploying GeT-USE Policies in Real-World
An off-the-shelf 2D object detector proposes candidate objects from the real camera view. Corresponding depth crops are scored by the selector; the highest-scoring candidate is passed, together with the scene depth, to the grasp policy. Grasp detection hands control to manipulation, which runs until its success detector fires.
The core learned modules use depth, but the complete real-world pipeline also uses RGB for object proposals and proprioception for grasping. Thus the abstract’s depth-only description does not cover every pipeline input. The paper does not report a separate robustness study for proposal or success-detector failures.

V Experimental Evaluation
Hardware is a 22-DOF TIAGo: a three-DOF mobile base, two seven-DOF arms, torso, two-DOF head and two parallel grippers. Head-camera images are 224×224 at 3 Hz. iGibson training uses sixteen 24-GB GPUs, batch 128, and six convolutional plus three fully connected layers in each visual architecture.
Each method is tested on twenty randomized trials per task, with real objects and conditions unseen during simulation training. No real-data fine-tuning is described. Evaluation uses three predefined task families and more than twenty object categories; it is not open-ended task learning.
Sweeping: six to ten candidate objects are offered without an actual sweeper. The robot selects a pusher and receiver, sweeps the target up within 600 seconds, then lifts both arms 10 cm; the target must remain contained. Builder shaping favors extension away from the wrist.
Hook and grasp: three to five candidates are offered without an actual hook. The target starts out of reach under a barrier that blocks top grasping. One tool retrieves it and the other hand lifts it 10 cm within 600 seconds. Builder shaping favors proximity of the newest block to the target.
Decanting: the robot transfers a small solid object from a source container into a selected temporary container, then into the final bowl within 600 seconds. Three to five additional candidate objects are offered. This experiment transfers a solid object; the discussion of liquids is motivation, not a liquid-pouring evaluation. Builder shaping again favors extension away from the wrist.
GeT-USE succeeds on 16/20 hooking, 11/20 sweeping and 10/20 decanting trials. TOG-Net-Internet obtains 9/6/4 successes and TOG-Net-Proc 11/2/1; original TOG-Net obtains zero. Both adapted variants include GeT-USE’s selector and six-DOF control. Training-tool counts and equal-compute comparisons are not reported.
Removing tool selection reduces the aggregate from 61.7% to 11.7%, a fifty-percentage-point drop. Restricting grasping to top-down yields 0%; restricting manipulation yields 3.3%. The corresponding drops are 61.7 and 58.3 points from the trial counts, rather than the prose’s approximate 60% and 57%. These are percentage-point differences, not relative percentage changes. Original table values remain unchanged.
The results support useful selection and flexible wrist motion in these tasks, while explanations involving internet-tool overfitting or procedural-tool limits are interpretations without separate geometry/data-budget ablations. Sixty trials per method and no repeated-training-seed uncertainty limit broader conclusions.
Failures include targets slipping under a tilted sweeper, hooks pushing targets beyond reach, and controller vibration causing missed decants. The authors attribute the first two to dynamics mismatch and the last to hardware; those explanations are not independently isolated, and all remain end-to-end system failures.
| GeT-USE | TOG-Net-Internet | TOG-Net-Proc | TOG-Net | GeT-USE-No-Ts | GeT-USE-Topdown-Grasp | GeT-USE-Topdown-Manip | |
|---|---|---|---|---|---|---|---|
| Hook_and_grasp | 16/20 | 9/20 | 11/20 | 0/20 | 4/20 | 0/20 | 2/20 |
| Sweeping | 11/20 | 6/20 | 2/20 | 0/20 | 2/20 | 0/20 | 0/20 |
| Decanting | 10/20 | 4/20 | 1/20 | 0/20 | 1/20 | 0/20 | 0/20 |
| Total | 37/60 (61.7%) | 19/60 (31.7%) | 14/60 (23.3%) | 0/60 (0%) | 7/60 (11.7%) | 0/60 (0%) | 2/60 (3.3%) |

VI Conclusion
Learning task-relevant extensions supplies a useful simulated curriculum for selecting and operating imperfect real objects. The demonstrated scope is rigid objects, parallel grippers and three predefined tasks.
The method assumes sufficiently accurate object dynamics. Deformable materials, liquids, richer contact behavior, dexterous hands and articulated tools such as scissors remain future work. Controller and dynamics gaps already affect the reported rigid-object tasks.
VII Acknowledgment
The authors thank Kenneth Salisbury and his laboratory’s students for supporting the research.