Literature
Display
← All papers

Guava: An Effective and Universal Harness for Embodied Manipulation

Haowen Liu · Xirui Li · Shaoxiong Yao · Peng Shi · Tianyi Zhou · Jia-Bin Huang · Furong Huang · Jiayuan Mao

Preprint posted · arXiv (preprint) ↗

AI agents7 min text · CondensedOriginal paper ↗
Contents

Overview

Guava distills high-level robot tool use into a compact vision-language model while retaining a shared perception and manipulation interface.

At a glance

Guava trains Qwen3.5-4B on 2,268 simulator-validated GPT-5.4 interaction trajectories, including 714 perturbation or intermediate-state branches. The resulting model selects semantic robot tools from images, state summaries, and execution feedback. It approaches the teacher on the tested simulation and physical tasks while reducing model-request latency. Low-level perception, geometry, grasp planning, and robot control remain delegated to the harness; this is tool-use distillation rather than an end-to-end motor policy.

1 Introduction

Repeated frontier-model calls can make interactive manipulation slow and dependent on remote inference. A smaller model is useful only if it can use a suitable action interface and recover from states outside successful demonstrations. Guava studies both: semantic tools reduce the model’s geometric burden, and branched simulated interactions broaden its experience. The contribution is the combination of interface design and validated interaction supervision, with performance bounded by the shared tools.

Refer to caption
Figure 1: Overview. A frontier model teacher generates manipulation trajectories in simulation for training. Supervised fine-tuning distill these behaviors to Guava-4B, which operates through a manipulation harness at deployment.

2 Method

The framework builds a common teacher/student harness, generates successful base and recovery interactions in simulation, and fine-tunes a compact model. A separate reinforcement-learning experiment further improves two long-horizon tasks.

Refer to caption
Figure 2: One agentic loop turn. The agent conditions on images, end-effector state, tool feedback, and interaction history to generate reasoning and a semantic tool call. Execution returns updated feedback for the next turn.

2.1 Manipulation Harness

Each turn provides RGB, gripper pose and width, interaction history, and tool feedback. SAM 3 (Carion et al., 2025) and depth are used inside tools to segment queried objects and obtain geometry; depth is hidden from the VLM but required by the system. Tools expose object position/size, grasping, alignment, Cartesian motion, rotation, gripper commands, and a home pose.

A ReAct-style loop (Yao et al., 2023) alternates reasoning and tool calls. Action execution returns a new image and outcome; failures become context for the next decision. On seven tasks with fifteen trials each, the full interface reaches 88.6% for GPT-5.4 and 22.9% for base Qwen3.5-4B. Removing state/feedback or replacing semantic tools with low-level pose commands lowers performance. Per-action replanning gains 5.7 and 9.6 percentage points over the tested multi-action program interface. These results establish the interface’s value within these implementations.

Equation 1
ht=(ℓ,o0,u0,y0,…,ut−1,yt−1,ot),h_{t}=\bigl(\ell,o_{0},u_{0},y_{0},\ldots,u_{t-1},y_{t-1},o_{t}\bigr), (1)
Table 1: Harness study across model scales. Success rates (%) over seven tasks, 15 trials per task. Every ablated row changes one component relative to the full harness. Best and second-best results within each model column are bold and underlined, respectively.
Harness configuration GPT-5.4 Qwen3.5-4B
Full 88.6 22.9
Image only 56.2 3.8
Image + gripper state 77.1 10.5
Low-level tools 73.3 4.8
Multi-action re-planning 82.9 13.3

2.2 Data Generation Pipeline

The teacher executes randomized MuJoCo tasks while saving simulator checkpoints and interaction histories. From successful paths, compatible failures such as missed grasps, drops, displaced objects, or poor approaches generate branches that the teacher must recover from. Some branches instead start from valid intermediate states with fresh history; the 714-branch count therefore does not mean 714 injected failures.

Only successful validated continuations are retained. Simulator success predicates are supplemented by rule checks, agent-based reasoning review, and human review of ambiguous cases. Supported annotation errors are corrected and unresolved episodes removed. “Physically validated” here means executed in simulation with this quality-control process, not that every training trajectory was run on hardware.

Refer to caption
Figure 3: Reasoning and tool-use sequence example from Guava-4B. For the instruction pick up the orange, the agent aligns above the object, grasps it, lifts it off the table, and then declares completion.
Refer to caption
Figure 4: Base trajectories and recovery branches. A shared execution history branches into normal execution and an injected grasp slip followed by successful recovery. Successful base and recovery trajectories are retained for training.

2.3 Training a Compact Manipulation Agent

Supervised fine-tuning of Qwen3.5-4B (Qwen Team, 2026) predicts the teacher’s reasoning and next tool call from the full available history. A subsequent GRPO experiment optimizes sparse task success through the same simulator interface on selected long-horizon tasks. The main Guava-4B results refer to supervised distillation unless otherwise specified.

3 Experiments

Evaluation tests custom simulated tasks, perturbation-data benefits, physical transfer, an external benchmark, selected RL refinement, and inference overhead.

3.1 Setups

Models share the same full harness for the principal comparisons. Fine-tuning uses eight H100 80GB GPUs, frozen visual encoder/aligner, and three supervised epochs. Physical evaluation uses a Franka Research 3 and calibrated RGB-D camera. “Unseen” categories change objects, instructions, or compositions of familiar manipulation primitives; they do not imply arbitrary new control skills.

3.2 Guava-4B Acquires Generalizable Manipulation Capabilities

Simulation and recovery-data ablation. Across fifteen tasks with thirty trials each, Guava-4B achieves 87.1% success versus 22.2% for its base model and 90.4% for the teacher. On eight unseen tasks it reaches 90.8%. A size-matched base-only dataset replaces perturbation branches with ordinary trajectories, keeping both sets at 2,268 examples and the training epochs fixed. Adding branches raises unseen-task success from 77.1% to 90.8%, while seen-task success changes from 81.9% to 82.9%. This supports broader state coverage, though the proposed mechanism of weakening sequence shortcuts remains an interpretation.

Physical transfer. Nine tasks receive ten trials each without real-world model fine-tuning. Guava succeeds in 90.0% overall, versus 93.3% for GPT-5.4 and 28.9% for the base model. The teacher–student difference is three successes among ninety trials. Qualitative examples show regrasping displaced objects and responding to a controller abort; these illustrate recovery but are not a separate aggregate recovery benchmark.

External tasks and RL. On selected LIBERO-PRO perturbations (Zhou et al., 2025), average success is 0.41 for both position and task changes, versus teacher scores of 0.45 and 0.44. Scores are much lower than on the custom benchmark, despite substantial improvement over the base model. Other listed policy baselines are taken from prior reported results and use different interfaces. GRPO on two selected long-horizon tasks improves shell-game success from 63.3% to 90.0% and red-object collection from 80.0% to 86.7%, each evaluated over thirty trials.

Table 2: Quantitative comparison on simulation tasks. Success rates (%) over 30 trials per task. Best and second-best results in each row are bold and underlined. Pert denotes perturbation trajectories; w/o Pert is Guava-4B trained without perturbation trajectories.
Task GPT-5.4 Qwen3.5-4B Guava-4B Guava-4B w/o Pert
Seen tasks
Place can in box 100.0 26.7 100.0 100.0
Arrange by size 100.0 30.0 90.0 83.3
Remove cube from tray 83.3 33.3 100.0 90.0
Push basket 93.3 0.0 66.7 86.7
Close drawer 83.3 63.3 80.0 83.3
Shell game 66.7 10.0 63.3 70.0
All red objects in basket 80.0 3.3 80.0 60.0
Unseen objects
Pick up carrot 100.0 43.3 100.0 76.7
Tomato near potato 96.7 16.7 90.0 66.7
Lemon in bin 100.0 46.7 100.0 100.0
Push pot 83.3 10.0 86.7 56.7
Unseen instructions
Reverse cube stacking 96.7 20.0 93.3 76.7
Arrange by decreasing size 93.3 23.3 83.3 63.3
Unseen long-horizon compositions
Separate food and utensils 90.0 3.3 93.3 93.3
Set table 90.0 3.3 80.0 83.3
Seen mean 86.7 23.8 82.9 81.9
Unseen mean 93.8 20.8 90.8 77.1
Overall 90.4 22.2 87.1 79.3
Original figure
Figure 5: Real-world performance. Success rates over 10 trials per task, without real-world fine-tuning. The two panels show seen and unseen tasks, with the mean over the displayed tasks in each panel. Overall success across all nine tasks is 93.3% for GPT-5.4, 28.9% for Qwen3.5-4B, and 90.0% for Guava-4B.
Original figure
Figure 6: GRPO results. Success rates over 30 trials per task before and after RL post-training on two long-horizon tasks.
Refer to caption
Figure 7: Real-world rollouts. Selected physical executions illustrate manipulation across object configurations, instructions, and task compositions. Paired frames show stages of execution for each task.
Refer to caption
Figure 8: Qualitative recovery examples. (a) The agent retries after an object is displaced before grasp execution. (b) The agent responds to a real-world controller abort and resumes manipulation. The sequences illustrate feedback-conditioned adaptation; the controller case concerns a failure mechanism absent from simulation training.
Table 3: LIBERO-PRO success rate. We evaluate on the object, goal, and spatial suites under initial-position and task-instruction perturbations, following the LIBERO-PRO evaluation protocol adopted by CaP-X Fu et al. (2026).
Benchmark Pert. Guava-4B GPT-5.4 Qwen3.5-4B π0.5\pi_{0.5} Cap-Agent0
libero-object Pos 0.60 0.61 0.13 0.17 0.22
Task 0.60 0.57 0.10 0.01 0.18
libero-goal Pos 0.39 0.44 0.03 0.38 0.26
Task 0.38 0.47 0.00 0.00 0.17
libero-spatial Pos 0.25 0.29 0.00 0.20 0.12
Task 0.26 0.26 0.00 0.01 0.14
Average Pos 0.41 0.45 0.06 0.25 0.20
Task 0.41 0.44 0.03 0.01 0.16

3.3 Inference Efficiency

Local inference on an RTX 5090 averages 0.587 seconds per model call versus 4.168 seconds for the remote GPT-5.4 comparison. Cumulative model-request time per episode is 5.64 versus 42.10 seconds, with similar call counts and 795 versus 2,389 generated tokens. These are measured inference/request costs over thirty episodes on three tasks, not complete robot execution time including perception tools and physical motions. The speedup also compares different hardware and serving conditions.

Table 4: Inference efficiency. Means over 30 episodes on three tasks pick up carrot, reverse cube stacking, place all red objects in basket per model, except call latency, which is averaged over requests.
Metric GPT-5.4 Guava-4B
Model-call latency (s) ↓\downarrow 4.168 0.587
Model calls/episode 10.10 9.60
Model-request time/episode (s) ↓\downarrow 42.10 5.64
Generated tokens/episode ↓\downarrow 2389 795

3.4 Limitations

Capabilities remain limited by the tabletop toolset, single-view occlusion, and sequential observation–reasoning–action loop. The system does not address dexterous in-hand or high-frequency reactive manipulation. Even subsecond model calls can be slow for rapidly changing contact situations. Broader tools, multiple views, and lower latency are proposed extensions.

4 Related Work

The paper distinguishes high-level VLM agents using explicit tools from direct visuomotor action models. Related work supplies programmatic control, spatial objectives, learned skills, and simulated teacher data. Guava trains on interleaved reasoning, tool calls, observations, and execution feedback, preserving the same interface from teacher collection to student deployment. The resulting abstraction helps transfer while retaining the tool implementations’ constraints.

5 Conclusion

Simulation-generated interaction experience can transfer much of a frontier model’s tested tool-use capability to a smaller local model. The controlled branch-data ablation and real robot trials support this approach within familiar manipulation primitives. Broader embodiment, harder contact tasks, and more responsive interaction remain open directions.

Appendix Contents

The appendices detail tool semantics, harness ablations, perturbation eligibility, data review, and additional execution traces.

Appendix A Manipulation Harness Details

The manipulation interface delegates geometric estimation and low-level execution to reusable tools while exposing concise state and outcome feedback to the model.

A.1 Tool Implementation Details

Position tools return point-cloud centroids and optional dimensions. Grasping uses a top-down geometric or learned grasp planner; the returned “grasped” status reflects impeded gripper closure and is not an independent guarantee of secure object retention. Alignment selects top/side directions with clearances of 0.05, 0.08, or 0.15 m. Move commands specify absolute Cartesian positions while preserving orientation; rotation uses gripper-body axes. Gripper and home tools complete the interface.

A.2 Harness Study Details

The multi-action ablation can use assignments, conditionals, and bounded loops. Intermediate returns are available to the executing program, but the VLM receives no new images or automatic transcript until that program finishes or errors. It then sees the final image, gripper state, printed output, and error, with up to five rounds. The comparison therefore changes feedback timing and exposure, not only the number of actions per call.

A.3 Harness Evaluation on Additional VLMs

Additional frontier models perform strongly with the same full harness, whereas Qwen3.5-2B frequently reasons incorrectly or emits invalid tool calls. These results suggest that a useful interface helps across models but does not erase capability differences.

Table 5: Results of Guava harness with additional VLMs. Values are success rates (%) evaluated over 15 episodes. Best results are marked bold.
 
Success Rate (%)
Task Name Gemini-3.1-Pro Claude-Sonnet-4.6 Qwen3.5-2B Qwen Team (2026)
Seen tasks
place can in box 100.0 100.0 0.0
arrange by size 100.0 100.0 0.0
remove cube from tray 100.0 100.0 0.0
push basket 66.7 66.7 0.0
close drawer 80.0 73.3 0.0
shell game 93.3 86.7 0.0
place all red objects in basket 86.7 100.0 0.0
Unseen objects
pick up carrot 100.0 100.0 26.7
tomato near potato 100.0 100.0 0.0
lemon in bin 100.0 100.0 0.0
push pot 60.0 60.0 0.0
Unseen instructions
stack cube reverse order 100.0 86.7 6.7
arrange by size reverse direction 86.7 100.0 0.0
Unseen long-horizon compositions
separate food and utensils 100.0 86.7 0.0
set table 73.3 73.3 0.0
Overall 89.8 88.9 2.2
 

Appendix B Data Pipeline Details

The data appendix specifies prompts, valid branch conditions, and checks used to retain training examples.

B.1 Prompt

The collection prompt defines observation fields, tool signatures, and the required reasoning-plus-action format. The paper refers readers to code for full prompts rather than reproducing them completely.

System Prompt for Data Generation You are an intelligent robot arm controller. You will be shown an image of the scene and given a task to complete using the available tools. Interaction Loop In every response, think step-by-step inside <think></think> tags first, then call exactly one tool. When the task is fully complete or irrecoverable, output your thinking and end with Task complete or Task failed. Reasoning Guidelines Inside <think>, analyze current scene state - including object and gripper poses, progress of task - what has been done and what remains, result of last action - was it a success, failure, or unexpected outcome, and propose the best next action based on your reasoning... Tool Calling
<tool_call>{"name": "tool_name", "arguments": {"param": value}}</tool_call>
Gripper State The gripper’s current position [x,y,z], rotation [roll,pitch,yaw], and gripper opening % are provided at every turn. Use these values directly in your reasoning when necessary. Tool Definition ...

B.2 Perturbation Generation Details

Perturbations are stage-compatible: dropping requires a held object, and a missed-grasp intervention applies to a grasp attempt rather than a push. Fresh-history continuations from intermediate states are also counted as perturbation trajectories. The retained table lists four explicit failure types and the information required for recovery.

Table 6: Perturbations used to construct recovery supervision. Perturbations are conditioned on execution stages and will only be applied if compatible.
Perturbation Eligible stage Information required for recovery
Missed grasp Grasp attempt Whether contact secured the object and how to retry
Transport drop Motion while holding an object The object’s updated location and a new retrieval action
Object displacement After alignment, before the intended grasp Whether the previous approach remains appropriate
Incorrect approach Object-referenced alignment A revised approach based on the resulting pose and feedback

B.3 Data Cleaning

Rule checks remove invalid arguments and over-budget interactions. Agent-based review checks consistency between reasoning, instruction, actions, and feedback, including unsupported geometric claims. Text is revised only when the recorded execution supports it; humans review ambiguous outcomes or occluded final states. This introduces review effort beyond automatic simulator labeling.

Appendix C Additional Qualitative Analysis

Additional examples illustrate task-state adaptation, simulation-to-real transfer, and the model’s generated reasoning/tool-call sequences.

Execution-State Adaptation.

When initialized with a peg already grasped, the agent proceeds to insertion rather than repeating the grasp. This is qualitative evidence of using current state and context, not a quantified test over all interrupted-task conditions.

Refer to caption
Figure 9: Execution-state adaptation. When initialized with the peg already grasped, Guava-4B proceeds directly to insertion rather than repeating the completed grasp.

Sim2Real Transfer.

Separating high-level decisions from perception and control is a plausible contributor to transfer from simulated interactions to the real setup. The paper also notes that simulation contact errors can make individual simulated tasks harder than physical ones. Real evaluation remains necessary; neither direction of a simulation–hardware score difference proves general transfer reliability.

Representative GUAVA-4B Trajectories.

Figures 10–12 preserve full example traces for multi-object execution, failed-grasp recovery, and spatially grounded pushing. They expose the intermediate reasoning and tool calls alongside images, complementing aggregate success metrics.

Refer to caption
Figure 10: Long-horizon execution. Example trajectory in which Guava-4B sequentially picks and places multiple objects to complete the task.
Refer to caption
Figure 11: Recovery from a failed grasp. After an unsuccessful grasp attempt, Guava-4B observes the updated scene, retries the grasp, and continues the task.
Refer to caption
Figure 12: Spatially grounded pushing. Example trajectory requiring the agent to select an appropriate pre-contact pose and pushing direction relative to the target object.
Concept demonstration

Open image ↗