Overview
Guava distills high-level robot tool use into a compact vision-language model while retaining a shared perception and manipulation interface.
At a glance
Guava trains Qwen3.5-4B on 2,268 simulator-validated GPT-5.4 interaction trajectories, including 714 perturbation or intermediate-state branches. The resulting model selects semantic robot tools from images, state summaries, and execution feedback. It approaches the teacher on the tested simulation and physical tasks while reducing model-request latency. Low-level perception, geometry, grasp planning, and robot control remain delegated to the harness; this is tool-use distillation rather than an end-to-end motor policy.
1 Introduction
Repeated frontier-model calls can make interactive manipulation slow and dependent on remote inference. A smaller model is useful only if it can use a suitable action interface and recover from states outside successful demonstrations. Guava studies both: semantic tools reduce the model’s geometric burden, and branched simulated interactions broaden its experience. The contribution is the combination of interface design and validated interaction supervision, with performance bounded by the shared tools.
2 Method
The framework builds a common teacher/student harness, generates successful base and recovery interactions in simulation, and fine-tunes a compact model. A separate reinforcement-learning experiment further improves two long-horizon tasks.
2.1 Manipulation Harness
Each turn provides RGB, gripper pose and width, interaction history, and tool feedback. SAM 3 (Carion et al., 2025) and depth are used inside tools to segment queried objects and obtain geometry; depth is hidden from the VLM but required by the system. Tools expose object position/size, grasping, alignment, Cartesian motion, rotation, gripper commands, and a home pose.
A ReAct-style loop (Yao et al., 2023) alternates reasoning and tool calls. Action execution returns a new image and outcome; failures become context for the next decision. On seven tasks with fifteen trials each, the full interface reaches 88.6% for GPT-5.4 and 22.9% for base Qwen3.5-4B. Removing state/feedback or replacing semantic tools with low-level pose commands lowers performance. Per-action replanning gains 5.7 and 9.6 percentage points over the tested multi-action program interface. These results establish the interface’s value within these implementations.
Equation 1
| (1) |
| Harness configuration | GPT-5.4 | Qwen3.5-4B |
|---|---|---|
| Full | 88.6 | 22.9 |
| Image only | 56.2 | 3.8 |
| Image + gripper state | 77.1 | 10.5 |
| Low-level tools | 73.3 | 4.8 |
| Multi-action re-planning | 82.9 | 13.3 |
2.2 Data Generation Pipeline
The teacher executes randomized MuJoCo tasks while saving simulator checkpoints and interaction histories. From successful paths, compatible failures such as missed grasps, drops, displaced objects, or poor approaches generate branches that the teacher must recover from. Some branches instead start from valid intermediate states with fresh history; the 714-branch count therefore does not mean 714 injected failures.
Only successful validated continuations are retained. Simulator success predicates are supplemented by rule checks, agent-based reasoning review, and human review of ambiguous cases. Supported annotation errors are corrected and unresolved episodes removed. “Physically validated” here means executed in simulation with this quality-control process, not that every training trajectory was run on hardware.
2.3 Training a Compact Manipulation Agent
Supervised fine-tuning of Qwen3.5-4B (Qwen Team, 2026) predicts the teacher’s reasoning and next tool call from the full available history. A subsequent GRPO experiment optimizes sparse task success through the same simulator interface on selected long-horizon tasks. The main Guava-4B results refer to supervised distillation unless otherwise specified.
3 Experiments
Evaluation tests custom simulated tasks, perturbation-data benefits, physical transfer, an external benchmark, selected RL refinement, and inference overhead.
3.1 Setups
Models share the same full harness for the principal comparisons. Fine-tuning uses eight H100 80GB GPUs, frozen visual encoder/aligner, and three supervised epochs. Physical evaluation uses a Franka Research 3 and calibrated RGB-D camera. “Unseen” categories change objects, instructions, or compositions of familiar manipulation primitives; they do not imply arbitrary new control skills.
3.2 Guava-4B Acquires Generalizable Manipulation Capabilities
Simulation and recovery-data ablation. Across fifteen tasks with thirty trials each, Guava-4B achieves 87.1% success versus 22.2% for its base model and 90.4% for the teacher. On eight unseen tasks it reaches 90.8%. A size-matched base-only dataset replaces perturbation branches with ordinary trajectories, keeping both sets at 2,268 examples and the training epochs fixed. Adding branches raises unseen-task success from 77.1% to 90.8%, while seen-task success changes from 81.9% to 82.9%. This supports broader state coverage, though the proposed mechanism of weakening sequence shortcuts remains an interpretation.
Physical transfer. Nine tasks receive ten trials each without real-world model fine-tuning. Guava succeeds in 90.0% overall, versus 93.3% for GPT-5.4 and 28.9% for the base model. The teacher–student difference is three successes among ninety trials. Qualitative examples show regrasping displaced objects and responding to a controller abort; these illustrate recovery but are not a separate aggregate recovery benchmark.
External tasks and RL. On selected LIBERO-PRO perturbations (Zhou et al., 2025), average success is 0.41 for both position and task changes, versus teacher scores of 0.45 and 0.44. Scores are much lower than on the custom benchmark, despite substantial improvement over the base model. Other listed policy baselines are taken from prior reported results and use different interfaces. GRPO on two selected long-horizon tasks improves shell-game success from 63.3% to 90.0% and red-object collection from 80.0% to 86.7%, each evaluated over thirty trials.
| Task | GPT-5.4 | Qwen3.5-4B | Guava-4B | Guava-4B w/o Pert |
|---|---|---|---|---|
| Seen tasks | ||||
| Place can in box | 100.0 | 26.7 | 100.0 | 100.0 |
| Arrange by size | 100.0 | 30.0 | 90.0 | 83.3 |
| Remove cube from tray | 83.3 | 33.3 | 100.0 | 90.0 |
| Push basket | 93.3 | 0.0 | 66.7 | 86.7 |
| Close drawer | 83.3 | 63.3 | 80.0 | 83.3 |
| Shell game | 66.7 | 10.0 | 63.3 | 70.0 |
| All red objects in basket | 80.0 | 3.3 | 80.0 | 60.0 |
| Unseen objects | ||||
| Pick up carrot | 100.0 | 43.3 | 100.0 | 76.7 |
| Tomato near potato | 96.7 | 16.7 | 90.0 | 66.7 |
| Lemon in bin | 100.0 | 46.7 | 100.0 | 100.0 |
| Push pot | 83.3 | 10.0 | 86.7 | 56.7 |
| Unseen instructions | ||||
| Reverse cube stacking | 96.7 | 20.0 | 93.3 | 76.7 |
| Arrange by decreasing size | 93.3 | 23.3 | 83.3 | 63.3 |
| Unseen long-horizon compositions | ||||
| Separate food and utensils | 90.0 | 3.3 | 93.3 | 93.3 |
| Set table | 90.0 | 3.3 | 80.0 | 83.3 |
| Seen mean | 86.7 | 23.8 | 82.9 | 81.9 |
| Unseen mean | 93.8 | 20.8 | 90.8 | 77.1 |
| Overall | 90.4 | 22.2 | 87.1 | 79.3 |
| Benchmark | Pert. | Guava-4B | GPT-5.4 | Qwen3.5-4B | Cap-Agent0 | |
|---|---|---|---|---|---|---|
| libero-object | Pos | 0.60 | 0.61 | 0.13 | 0.17 | 0.22 |
| Task | 0.60 | 0.57 | 0.10 | 0.01 | 0.18 | |
| libero-goal | Pos | 0.39 | 0.44 | 0.03 | 0.38 | 0.26 |
| Task | 0.38 | 0.47 | 0.00 | 0.00 | 0.17 | |
| libero-spatial | Pos | 0.25 | 0.29 | 0.00 | 0.20 | 0.12 |
| Task | 0.26 | 0.26 | 0.00 | 0.01 | 0.14 | |
| Average | Pos | 0.41 | 0.45 | 0.06 | 0.25 | 0.20 |
| Task | 0.41 | 0.44 | 0.03 | 0.01 | 0.16 |
3.3 Inference Efficiency
Local inference on an RTX 5090 averages 0.587 seconds per model call versus 4.168 seconds for the remote GPT-5.4 comparison. Cumulative model-request time per episode is 5.64 versus 42.10 seconds, with similar call counts and 795 versus 2,389 generated tokens. These are measured inference/request costs over thirty episodes on three tasks, not complete robot execution time including perception tools and physical motions. The speedup also compares different hardware and serving conditions.
| Metric | GPT-5.4 | Guava-4B |
| Model-call latency (s) | 4.168 | 0.587 |
| Model calls/episode | 10.10 | 9.60 |
| Model-request time/episode (s) | 42.10 | 5.64 |
| Generated tokens/episode | 2389 | 795 |
3.4 Limitations
Capabilities remain limited by the tabletop toolset, single-view occlusion, and sequential observation–reasoning–action loop. The system does not address dexterous in-hand or high-frequency reactive manipulation. Even subsecond model calls can be slow for rapidly changing contact situations. Broader tools, multiple views, and lower latency are proposed extensions.
4 Related Work
The paper distinguishes high-level VLM agents using explicit tools from direct visuomotor action models. Related work supplies programmatic control, spatial objectives, learned skills, and simulated teacher data. Guava trains on interleaved reasoning, tool calls, observations, and execution feedback, preserving the same interface from teacher collection to student deployment. The resulting abstraction helps transfer while retaining the tool implementations’ constraints.
5 Conclusion
Simulation-generated interaction experience can transfer much of a frontier model’s tested tool-use capability to a smaller local model. The controlled branch-data ablation and real robot trials support this approach within familiar manipulation primitives. Broader embodiment, harder contact tasks, and more responsive interaction remain open directions.
Appendix Contents
The appendices detail tool semantics, harness ablations, perturbation eligibility, data review, and additional execution traces.
Appendix A Manipulation Harness Details
The manipulation interface delegates geometric estimation and low-level execution to reusable tools while exposing concise state and outcome feedback to the model.
A.1 Tool Implementation Details
Position tools return point-cloud centroids and optional dimensions. Grasping uses a top-down geometric or learned grasp planner; the returned “grasped” status reflects impeded gripper closure and is not an independent guarantee of secure object retention. Alignment selects top/side directions with clearances of 0.05, 0.08, or 0.15 m. Move commands specify absolute Cartesian positions while preserving orientation; rotation uses gripper-body axes. Gripper and home tools complete the interface.
A.2 Harness Study Details
The multi-action ablation can use assignments, conditionals, and bounded loops. Intermediate returns are available to the executing program, but the VLM receives no new images or automatic transcript until that program finishes or errors. It then sees the final image, gripper state, printed output, and error, with up to five rounds. The comparison therefore changes feedback timing and exposure, not only the number of actions per call.
A.3 Harness Evaluation on Additional VLMs
Additional frontier models perform strongly with the same full harness, whereas Qwen3.5-2B frequently reasons incorrectly or emits invalid tool calls. These results suggest that a useful interface helps across models but does not erase capability differences.
| Success Rate (%) | |||
| Task Name | Gemini-3.1-Pro | Claude-Sonnet-4.6 | Qwen3.5-2B Qwen Team (2026) |
| Seen tasks | |||
| place can in box | 100.0 | 100.0 | 0.0 |
| arrange by size | 100.0 | 100.0 | 0.0 |
| remove cube from tray | 100.0 | 100.0 | 0.0 |
| push basket | 66.7 | 66.7 | 0.0 |
| close drawer | 80.0 | 73.3 | 0.0 |
| shell game | 93.3 | 86.7 | 0.0 |
| place all red objects in basket | 86.7 | 100.0 | 0.0 |
| Unseen objects | |||
| pick up carrot | 100.0 | 100.0 | 26.7 |
| tomato near potato | 100.0 | 100.0 | 0.0 |
| lemon in bin | 100.0 | 100.0 | 0.0 |
| push pot | 60.0 | 60.0 | 0.0 |
| Unseen instructions | |||
| stack cube reverse order | 100.0 | 86.7 | 6.7 |
| arrange by size reverse direction | 86.7 | 100.0 | 0.0 |
| Unseen long-horizon compositions | |||
| separate food and utensils | 100.0 | 86.7 | 0.0 |
| set table | 73.3 | 73.3 | 0.0 |
| Overall | 89.8 | 88.9 | 2.2 |
Appendix B Data Pipeline Details
The data appendix specifies prompts, valid branch conditions, and checks used to retain training examples.
B.1 Prompt
The collection prompt defines observation fields, tool signatures, and the required reasoning-plus-action format. The paper refers readers to code for full prompts rather than reproducing them completely.
B.2 Perturbation Generation Details
Perturbations are stage-compatible: dropping requires a held object, and a missed-grasp intervention applies to a grasp attempt rather than a push. Fresh-history continuations from intermediate states are also counted as perturbation trajectories. The retained table lists four explicit failure types and the information required for recovery.
| Perturbation | Eligible stage | Information required for recovery |
|---|---|---|
| Missed grasp | Grasp attempt | Whether contact secured the object and how to retry |
| Transport drop | Motion while holding an object | The object’s updated location and a new retrieval action |
| Object displacement | After alignment, before the intended grasp | Whether the previous approach remains appropriate |
| Incorrect approach | Object-referenced alignment | A revised approach based on the resulting pose and feedback |
B.3 Data Cleaning
Rule checks remove invalid arguments and over-budget interactions. Agent-based review checks consistency between reasoning, instruction, actions, and feedback, including unsupported geometric claims. Text is revised only when the recorded execution supports it; humans review ambiguous outcomes or occluded final states. This introduces review effort beyond automatic simulator labeling.
Appendix C Additional Qualitative Analysis
Additional examples illustrate task-state adaptation, simulation-to-real transfer, and the model’s generated reasoning/tool-call sequences.
Execution-State Adaptation.
When initialized with a peg already grasped, the agent proceeds to insertion rather than repeating the grasp. This is qualitative evidence of using current state and context, not a quantified test over all interrupted-task conditions.
Sim2Real Transfer.
Separating high-level decisions from perception and control is a plausible contributor to transfer from simulated interactions to the real setup. The paper also notes that simulation contact errors can make individual simulated tasks harder than physical ones. Real evaluation remains necessary; neither direction of a simulation–hardware score difference proves general transfer reliability.
Representative GUAVA-4B Trajectories.
Figures 10–12 preserve full example traces for multi-object execution, failed-grasp recovery, and spatially grounded pushing. They expose the intermediate reasoning and tool calls alongside images, complementing aggregate success metrics.