Overview
Project website. This edition follows the complete fifty-one-page v1 of 17 June 2026, retaining all twelve figures, fifteen tables, Algorithm 1, the example task specification, six skill implementations and twelve complete prompt templates. Figures 6 and 7, absent as graphics in the arXiv HTML, are restored from the PDF.
At a glance
Robotics Agent Teams (RATs) learn a persistent library of executable robot helpers through self-proposed simulated tasks before evaluation requests arrive. Task proposal, execution with verification and retries, and memory management turn successes and failures into reusable code and conditional lessons. The underlying language model is not fine-tuned.
Full-system success improves over CaP-Agent0 from 23.2% to 43.8% on LIBERO-PRO and from 21% to 38% on MolmoSpaces. Those gains combine learned skills with a stronger execution system. Plugging the library into the baseline agent also helps on average, including selected cross-environment and small hardware tests, but inappropriate reuse can reduce individual-task performance.
1 Introduction
Task-driven agents often rebuild perception and control routines for each request. RATs adds a preparatory play phase that chooses its own attainable tasks and stores useful behaviors before downstream instructions are known. The analogy to childhood play motivates exploration near a competence frontier; it is not a model or validation of children’s learning mechanisms.
The learned object is a code library and failure memory around existing control/perception APIs. Compared with Code as Policies and the task-driven CaP-Agent0, the contribution is persistent, self-directed skill acquisition and its orchestration, rather than new low-level motor learning.
2 Related Work
The paper connects intrinsic motivation, Goldilocks attention, learning from play and curriculum learning to embodied coding agents. Prior demonstration-based play learns from supplied trajectories; this system proposes and attempts its own tasks within available environments.
Persistent code resembles Voyager, while diagnosis and lessons relate to ReAct and Reflexion. RATs targets physical manipulation with grounding, verification, primitive APIs and reusable helpers. These relationships position the method without establishing superiority over every referenced system.
3 Robotics Agent Teams for Playful Agentic Robot Learning
Three cooperating teams propose tasks, execute them and maintain memory. Algorithm 1 separates preparatory play from evaluation with the frozen learned library.
Original selectable algorithm
3.1 Problem Setup and Overview
An embodied coding agent observes scene context and uses available functions to write executable policies. During play, it repeatedly selects an intrinsic task, plans and attempts it, then updates a skill library and failure memory. At evaluation, externally specified tasks replace self-proposed goals and persistent updates stop.
Skills carry executable code, interfaces, preconditions, effects, dependencies and empirical reliability. Failure memory supplies conditional corrections. The process changes accessible code/context while keeping the underlying language-model parameters fixed.
3.2 Task Proposer Team
The proposer creates a pool from the current scene, available skill metadata and recent attempts. Its main score multiplies object–skill novelty by a competence-frontier term. Novelty averages 1/√(N(o,s)+1) over object–skill pairs; the frontier is 4r̄(1−r̄), where r̄ averages Wilson-bounded skill-success estimates. The frontier peaks at 0.5 and suppresses both near-certain and apparently impossible tasks.
These are heuristic proxies: averaging skill reliabilities does not establish the probability of a composed task or its future learning benefit. Appendix A adds retry-bank bonuses and similar-failure penalties, and environment checks can veto candidates before execution.
3.3 Execution Team
The planner selects skills and orders subgoals; a visual verifier checks grounding and prerequisites. The policy writer produces bounded, instrumented Python code, and static checks catch syntax/API problems. Execution is followed by task-level and per-step verification, then failure diagnosis.
Retry routing distinguishes argument/code changes from structural plan repair. Repeated physical bottlenecks can trigger isolated practice from the reset state. Verifiers and diagnostic agents provide useful evidence but remain fallible; passing a static or visual check does not certify physical success.
3.4 Memory-Management Team
Successful end-to-end runs yield parameterized helpers, initially experimental. Failed attempts become raw episodes and compact conditional lessons. Usage outcomes determine reliability tiers and retrieval priority. Periodic curation merges or removes redundant entries; a separate Skill Proposer can draft untested helpers from recurring failures.
Thus not every stored helper has previously succeeded. Anticipatory proposals earn reliability only through later use, and growing memory counts alone do not demonstrate growing competence.
3.5 Evaluation With the Learned Skill Library
Evaluation freezes the library and disables task proposal and memory curation. Plug-in mode adds selected helper code to CaP-Agent0; full RATs execution additionally uses its planner, verification, diagnosis and bounded retries. Comparing these modes separates some library value from the execution system’s contribution.
4 Experimental Results
The experiments assess held-out simulation tasks, transfer to another environment, exploratory hardware deployment and play/execution ablations. Success rates are descriptive outcomes for the specified tasks and budgets; the paper does not supply formal significance tests or confidence intervals for the reported headline gains.
4.1 Experiment Details
The study uses LIBERO-PRO and MolmoSpaces for fifty play iterations each with gemini-3.1pro-preview. RoboSuite is held out from play. Baselines include primitive-only CaP-Agent0, OpenVLA, π₀ and π₀.₅. Library-only and full-execution results should not be conflated.
4.2 In-Domain Evaluation
LIBERO-PRO evaluates sixty tasks under ten initializations each, for six hundred trials. RATs reaches 43.8% against CaP-Agent0’s 23.2%; the largest gains are in object perturbations. MolmoSpaces evaluates forty tasks ten times each, reaching 38% versus 21%, with open/close/pick/pick-and-place rates of 20/73/37/22%.
These headline comparisons include both learned play skills and the RATs execution loop. The separate ablations provide a narrower estimate of the benefit of adding a library under a fixed test-time agent.
4.3 Cross-Environment Transfer
LIBERO-derived skills are plugged into CaP-Agent0 on seven RoboSuite tasks, fifty trials each. Success rises from 141/350 (40.3%) to 172/350 (49.1%). Two-arm lifting improves from 5 to 17 successes, although play used a single arm.
Transfer is uneven: two-arm handover decreases from 12 to 10 successes, nut assembly remains zero and wiping is already perfect. Shared or adaptable code/API abstractions support these selected transfers; the results do not establish universal cross-embodiment improvement.
Original selectable table
| Task | CaP-Agent0 | CaP-Agent0 + RATs Skills (LIBERO-PRO) | |
|---|---|---|---|
| Cube lifting | 34/50 (68.0%) | 42/50 (84.0%) | +16.0 pp |
| Cube restacking | 17/50 (34.0%) | 23/50 (46.0%) | +12.0 pp |
| Cube stacking | 23/50 (46.0%) | 30/50 (60.0%) | +14.0 pp |
| Nut assembly | 0/50 (0.0%) | 0/50 (0.0%) | 0.0 pp |
| Spill wiping | 50/50 (100.0%) | 50/50 (100.0%) | 0.0 pp |
| Two-arm handover | 12/50 (24.0%) | 10/50 (20.0%) | -4.0 pp |
| Two-arm lifting | 5/50 (10.0%) | 17/50 (34.0%) | +24.0 pp |
| Average (RoboSuite) | 141/350 (40.3%) | 172/350 (49.1%) | +8.9 pp |
| Pick up red cube | 14/40 (35.0%) | 17/40 (42.5%) | +7.5 pp |
| Place cube in bowl | 10/40 (25.0%) | 14/40 (35.0%) | +10.0 pp |
| Average (Real World) | 24/80 (30.0%) | 31/80 (38.8%) | +8.8 pp |
4.4 Ablating Play Strategy and Test-Time Execution
With fifty play iterations, CaP-Agent0 scores 23.2% without play, 24.7% with random play and 32.3% with curious play. RATs execution scores 36.3/38.3/44.3%. The stronger executor helps even without play, and curious play adds further gains under each executor.
The RATs-execution rows here use five trials per task, whereas CaP-Agent0 uses ten; the 44.3% ablation result differs from the main 43.8% sweep. This comparison supports the chosen curiosity strategy over the tested random baseline, not a general necessity theorem or equality of all inference costs.
Original selectable table
| Test-Time System | Play-Time Skills | Object | Goal | Spatial | Avg. | |||
|---|---|---|---|---|---|---|---|---|
| Pos. | Task | Pos. | Task | Pos. | Task | |||
| CaP-Agent0 | No Play | 27.0 | 31.0 | 29.0 | 16.0 | 13.0 | 23.0 | 23.2 |
| Random Play | 20.0 | 28.0 | 32.0 | 16.0 | 20.0 | 32.0 | 24.7 | |
| Curious Play | 51.0 | 47.0 | 34.0 | 20.0 | 19.0 | 23.0 | 32.3 | |
| RATs Exec. | No Play | 54.0 | 58.0 | 32.0 | 24.0 | 20.0 | 30.0 | 36.3 |
| Random Play | 54.0 | 46.0 | 34.0 | 44.0 | 24.0 | 28.0 | 38.3 | |
| Curious Play | 60.0 | 60.0 | 48.0 | 38.0 | 30.0 | 30.0 | 44.3 | |
4.5 Sim-to-Real Evaluation
For two physical tasks, forty trials each, adding LIBERO-derived skills raises red-cube pickup from 14 to 17 successes and cube-in-bowl placement from 10 to 14. The aggregate increases from 24/80 (30%) to 31/80 (38.8%): seven additional successes.
There is no additional hardware play or language-model fine-tuning. Transfer still depends on compatible low-level APIs, and two small task sets do not establish robust general sim-to-real performance. Appendix B evaluates a different, Molmo-derived library on two additional tasks.
5 Conclusion
Self-directed play can turn repeated execution experience into reusable code and lessons that improve selected downstream tasks. The evidence supports both library reuse and stronger execution, with incomplete and sometimes negative transfer. Larger physical studies, better retrieval and lower feedback costs remain necessary.
Limitations
Most evaluation is simulated, and the available scenes, objects and dynamics constrain what can be practiced. Mismatched helpers can hurt downstream performance. Frequent language-model inference and VLM verification are costly and fallible; primitive-level control APIs limit dexterity. The physical results are preliminary rather than broad deployment validation.
Acknowledgments
The authors acknowledge feedback and assistance with physical evaluation, and support for the Berkeley authors from the BAIR Commons Humanoid Intelligence Center.
Appendix
Table of Contents
Appendix A Implementation Details about RATs
The implementation appendix specifies proposal validity, retry routing, reliability management and the two distinct evaluation interfaces.
A.1 Details of Play-Time Task Proposal
LIBERO proposals use an environment catalog; Molmo proposals must refer to visible, reachable objects. Candidate ranking combines novelty and frontier scores with a retry-bank bonus and similar-failure penalty. A bridge rejects unsupported target/action families.
The Environment Creator converts selected LIBERO tasks into BDDL. Deterministic checks validate syntax, names, placements and goal predicates and try two reset seeds, with one bounded repair attempt. Invalid environments are rejected before robot execution. These restrictions and pre-existing primitive reliability constrain the nominally open-ended curriculum.
Original selectable task specification
(:language pick up the butter and put it on the plate) (:objects butter_1 - butter plate_1 - plate) (:init (On butter_1 kitchen_table_butter_init_region) (On plate_1 kitchen_table_plate_init_region)) (:goal (And (On butter_1 plate_1)))
A.1.1 Example Trace of Play-Time Task Proposal
The saved Molmo iteration-15 trace contains five visible objects, but the toilet is out of reach. The proposer sees thirteen primitives, nine learned helpers and ten previous tasks. Five candidates are scored: black-cloth picking is vetoed, two drawer objectives score 0.36, a cloth-placement fallback scores 0.19, and tissue-box lifting scores 0.6236 and is selected.
All valid candidates have novelty one here and no retry bonus or failure penalty; the difference comes from the frontier estimate. Tissue lifting uses a Wilson lower bound of about 0.1933 despite a raw 5/12 success history. Its four attempts fail, prompting an experimental centroid-adjustment helper. The trace illustrates selection mechanics, not guaranteed learnability or success.
Original selectable table
| Learned skill in prompt | Reusable capability | Raw rate | Uses | Tier |
|---|---|---|---|---|
| localize_object_with_molmo_sam3 | Localize an object from agent-view Molmo prompting and SAM3 segmentation. | 0.3125 | 32 | Exp. |
| plan_top_down_grasp_at_wrist | Move the wrist camera over an object and plan a top-down grasp from segmented point clouds. | 0.2500 | 16 | Ver. |
| get_axis_aligned_pull_direction | Compute an axis-aligned pull direction from the target toward the robot base. | 0.4286 | 7 | Ver. |
| select_grasp_for_pulling | Select a handle grasp whose approach opposes a requested pull direction. | 0.0000 | 0 | Exp. |
| execute_top_down_grasp_and_lift | Approach from above, close the gripper, and lift from a target 3D position. | 0.4167 | 12 | Ver. |
| push_surface_inward | Push a surface inward to close a drawer or door. | 1.0000 | 2 | Exp. |
| execute_grasp_from_transform | Execute a grasp pose with a collision-avoidance height offset, then lift. | 0.2000 | 5 | Exp. |
| translate_grasped_object_and_release | Translate a grasped object, release it, and retreat vertically. | 0.2000 | 5 | Exp. |
| push_object_closed | Localize an open articulation, infer its outward direction, and push inward. | 1.0000 | 1 | Exp. |
Original selectable table
| Previous task language in prompt | Success | Retries | Learned skill or failure diagnosis | |
|---|---|---|---|---|
| 1 | I wonder what happens if I lift the lever straight up into the air. | Yes | 0 | – |
| 2 | I want to see if I can push the top drawer all the way closed. | Yes | 0 | Learned push_surface_inward. |
| 3 | Let’s lift the brown sandal straight up. | Yes | 3 | – |
| 4 | I’m curious what happens when I lift the yellow button straight up. | Yes | 0 | – |
| 5 | I want to see what happens when I lift the toothpaste tube. | No | 4 | grasp_failure: argument_level. The m grasp z_offset left the fingers above the thin tube; the diagnosis suggests or m. |
| 6 | Let’s try to lift the blue clothes iron straight up and see what happens. | Yes | 0 | – |
| 7 | I wonder what happens if I lift the black videocassette straight up. | No | 4 | collision: argument_level. Subtracting m placed the grasp below the table; the diagnosis suggests subtracting or m. |
| 8 | I wonder if I can pull the walkie-talkie closer to me on the bed. | Yes | 0 | Learned two skills; execute_grasp_from_transform and translate_grasped_object_and_release. |
| 9 | I wonder if I can pull the metal ring closer to me across the table. | No | 4 | grasp_failure: rewrite_needed. The fingers missed or slipped from the knot; the diagnosis suggests targeting the center hole or subtracting m from the grasp-pose height. |
| 10 | I want to see if I can push the partially open drawer closed. | Yes | 0 | Learned push_object_closed. |
Original selectable table
| Candidate objective | Family | Surp. | Score | Result | |||||
|---|---|---|---|---|---|---|---|---|---|
| Push the top drawer of the white cabinet all the way closed. | Close | 1.0000 | 0.9000 | 0.3600 | – | 0.0000 | 0.0000 | 0.3600 | Valid |
| Pull the partially open drawer out even more. | Open | 1.0000 | 0.9000 | 0.3600 | – | 0.0000 | 0.0000 | 0.3600 | Valid |
| Pick up the black cloth from the cabinet. | Pick | – | – | – | – | – | – | 0.0000 | Vetoed |
| Lift the brown tissue box straight up into the air. | Lift | 1.0000 | 0.1933 | 0.6236 | – | 0.0000 | 0.0000 | 0.6236 | Selected |
| Put the black cloth inside the open drawer. | Place in | 1.0000 | 0.0500 | 0.1900 | – | 0.0000 | 0.0000 | 0.1900 | Valid |
A.2 Details of RATs for Execution
Execution localizes the first failed step using code slices, runtime values and before/after visual evidence. Structured simulator predicates or task-specific checks take precedence when available; a crash remains a failure. The diagnoser distinguishes wrong parameters or perception from missing prerequisites and invalid ordering.
A persistent, reset-accessible subgoal can be practiced by focused SubAgents. A visually verified local script is available to the current retry, but is not automatically permanent: end-to-end success or separately validated skill proposal governs insertion. Integration errors and subgoals requiring unavailable intermediate states should stay in the ordinary retry loop.
A.3 Details of Skill Library and Failure Memory Updates
New helpers must parse, use known APIs/dependencies and avoid duplication. Promotion to verified requires at least three uses and raw success at least 0.5; deprecation requires at least ten uses and success at most 0.2, with possible demotion after sustained poor results. Retrieval favors verified skills using Wilson bounds and hides deprecated ones.
Tiers reflect history, not a current probability guarantee: the snapshot includes verified helpers with raw rates below the original promotion threshold. Success extraction stores reusable pieces; failures store task, objects, mode, step, diagnosis and code. Every five play iterations, curation reduces redundancy, while anticipatory skill proposals remain experimental until tested.
A.4 Details of Evaluation With the Learned Skill Library
The same frozen code store supports two evaluation modes, differing in orchestration rather than whether the model’s weights are trained.
Plug-and-Play CaP-Agent0 evaluation.
CaP-Agent0 receives relevant skill signatures, descriptions and source, with executable definitions injected into its namespace. A lightweight selector can narrow large libraries. This mode omits RATs-specific planning, verification-driven retries and failure-memory retrieval, measuring a more direct plug-in benefit.
RATs evaluation.
Full RATs planning sees primitives and active learned helpers, prioritizing verified entries. The writer uses selected code while dependencies remain callable. Verification, diagnosis and bounded retry stay enabled, so this result measures the combined learned library and execution system.
Appendix B Additional Real-World Experimental Results
The additional hardware tests use MolmoSpaces-derived skills, distinct from the LIBERO-derived library tested in the main quantitative hardware table.
B.1 Additional Quantitative Results for Real-World Experiments
Thirty trials per task compare cube swapping and drawer closing. CaP-Agent0 rises from 0/30 to 7/30 on swapping and from 2/30 to 8/30 on closing. Across these tasks, 2/60 (3.3%) becomes 15/60 (25%). The larger aggregate gain is still based on two selected tasks and a small sample.
B.2 Additional Qualitative Results
Figure 4 preserves all four hardware sequences, including opening a drawer. Drawer opening is a qualitative example, not an additional quantified task in Table 8; project videos complement the paper’s snapshots.
Appendix C Details about Play Process
Saved play artifacts, memory snapshots and evaluation call logs show how skills are proposed, retained and used. These observational traces complement, but do not replace, the controlled no-play and play-strategy comparisons.
C.1 Play-Time Objectives and Knowledge Accumulation
One fifty-iteration Molmo run has forty-nine saved proposals because one artifact is missing. It covers seven interaction families and more than thirty object categories; pulling and lifting dominate. Between iterations ten and fifty, learned helpers grow from six to twenty-seven, raw failure episodes from fourteen to seventy and distilled lessons from eight to 121.
Successful shoe sliding, rail pushing and drawer pulling yield helpers later reused on other objects. Failed attempts can also motivate new experimental helpers. These counts mix reliability tiers and are not a mastery metric or a one-to-one mapping from failures to lessons.
C.2 Learned-Skill Usage in MolmoSpaces Evaluation
In four hundred evaluation trials, 391 invoke at least one learned helper; fourteen of twenty-seven stored helpers account for 5,169 calls. Object-localization wrappers contribute 2,806 calls (54.3%), while opening also uses direction and grasp planning, and closing uses push helpers.
Call frequency does not isolate causal usefulness and can grow through retries. Learned perception helpers compose existing pretrained perception APIs; the paper does not train a new vision model. Figure 9 traces failed play, helper proposals, later successful reuse and one cabinet-opening evaluation.
Original selectable table
| Task type | Trials | Success | Skill calls | Unique | Calls/trial | Most-used learned skills |
|---|---|---|---|---|---|---|
| Open | 100 | 20% | 1,900 | 11 | 19.0 | get_axis_aligned_pull_direction (614); localize_and_verify_object_point_cloud (429); localize_object_with_molmo_sam3 (254) |
| Close | 100 | 73% | 866 | 6 | 8.7 |
localize_object_with_molmo_sam3 (269);
get_axis_aligned_pull_direction (185);
push_object_closed (178); |
| Pick | 100 | 37% | 950 | 4 | 9.5 | localize_and_verify_object_point_cloud (705); execute_top_down_grasp_and_lift (139); get_highest_scoring_grasp (97) |
| Pick & Place | 100 | 22% | 1,453 | 8 | 14.5 | localize_and_verify_object_point_cloud (649); localize_object_with_molmo_sam3 (401); execute_top_down_grasp_and_lift (233) |
| Total | 400 | 38% | 5,169 | 14 | 12.9 | – |
C.3 Qualitative Comparison with Direct Code Synthesis
Matched qualitative examples compare failed direct code with successful skill-based drawer opening, shoe pickup, table closing and spanner placement. Reused localization, grasp, transport and release helpers replace repeatedly written geometry and waypoints. The figures contain condensed source-backed code excerpts, not complete standalone programs or a general causal attribution from selected successes.
C.4 Usage of LIBERO-Derived Skills in RoboSuite Evaluation
For a RoboSuite two-arm lift, selected LIBERO helpers guide policy generation toward handle localization, side-pinch geometry and coordinated lifting. In this illustrated run the final policy does not explicitly call those helpers: it reuses their structure through context. That implicit transfer differs from runtime helper-call statistics in MolmoSpaces.
Appendix D Details about the Evaluation Benchmark
Evaluation fixes the learned library and replays specified tasks. The appendix distinguishes the chosen task subsets from the number of repeated trials.
D.1 MolmoSpaces Evaluation Benchmark
The Molmo subset randomly selects ten held-out episodes for each of four task types across thirty-seven houses. Opening/closing use iTHOR; picking and pick-and-place use ProcTHOR-Objaverse. Saved episode state fixes robot initialization, object poses, referrals and cameras; simulator predicates judge articulation, target state or receptacle support. Ten repeats yield four hundred rollouts.
Original selectable table
| Task type | Scene dataset | Tasks | Main rollouts |
|---|---|---|---|
| Open | iTHOR | 10 | 100 |
| Close | iTHOR | 10 | 100 |
| Pick | ProcTHOR-Objaverse | 10 | 100 |
| pick-and-place | ProcTHOR-Objaverse | 10 | 100 |
| Total | – | 40 | 400 |
D.2 LIBERO-PRO Evaluation Benchmark
Six LIBERO-PRO suites combine object/goal/spatial categories with position swaps or task perturbations, ten tasks each. Fifty provided states would yield three thousand trials, but the main comparisons use ten states per task, six hundred rollouts. The main comparison shares task/trial budgets between baseline and learned library; the separate RATs-execution ablation uses its stated smaller repeat count.
Original selectable table
| Setting | LIBERO-PRO suite | Tasks | Reported rollouts |
|---|---|---|---|
| Object Pos | libero_object_swap | 10 | 100 |
| Object Task | libero_object_task | 10 | 100 |
| Goal Pos | libero_goal_swap | 10 | 100 |
| Goal Task | libero_goal_task | 10 | 100 |
| Spatial Pos | libero_spatial_swap | 10 | 100 |
| Spatial Task | libero_spatial_task | 10 | 100 |
| Total | – | 60 | 600 |
Appendix E Additional Studies
Additional studies separate library and executor effects, report token costs and retain representative learned code with its observed reliability metadata.
E.1 Ablation Results on MolmoSpaces
Molmo curious play improves CaP-Agent0’s mean from 21% to 25.8% and RATs execution from 32.8% to 38%. Closing improves strongly, but picking drops from 23% to 14% with CaP-Agent0 and from 45% to 37% with RATs; CaP-Agent0 pick-and-place also drops from 11% to 10%. Average gains therefore coexist with negative transfer, consistent with the paper’s retrieval limitations.
E.2 Token Cost Analysis
The token analysis explicitly uses GPT-5.5, unlike the main experiment’s stated Gemini model. Ten analyzed play iterations consume 5,121,182 tokens: diagnosis contributes 40.5%, policy writing 28.8% and failure-memory distillation 19.4%. Expensive failed retries dominate proposal cost.
The authors estimate about thirty million tokens for fifty play iterations, amortized across sixty held-out tasks, and compare a roughly compute-matched fifteen-turn baseline with a ten-turn baseline plus learned skills. Success is 26% versus 32.3%, compared with 23.2% for the ordinary ten-turn baseline. This approximate token-budget comparison favors reusable play in the tested setting; it is not an exact equality of tokens, latency or deployment cost, and amortization depends on how many later tasks reuse the library.
Original selectable table
| Agent Role | Total Tokens | Share (%) |
|---|---|---|
| Failure Diagnoser | 2,071,881 | 40.5% |
| Policy Writer | 1,472,949 | 28.8% |
| Failure Memory Distillation | 993,770 | 19.4% |
| Verifier | 194,318 | 3.8% |
| Memory Curator | 111,563 | 2.2% |
| Task Proposer | 98,505 | 1.9% |
| Planner | 78,359 | 1.5% |
| Environment Creator | 31,616 | 0.6% |
| Skill Library (Duplicate Check) | 31,377 | 0.6% |
| Skill Proposer | 30,969 | 0.6% |
| Feedback Generator | 5,875 | 0.1% |
| Total | 5,121,182 | 100.0% |
E.3 MolmoSpaces Learned Skills
All eighty-six lines preserve three of twenty-seven Molmo helpers: axis-aligned pull direction from target toward robot, inward closing based on that direction, and wrist-view top-down grasp planning. Their labels span verified, experimental and deprecated. A historical verified label coexists with 10/30 successes; the deprecated wrist helper has 6/26. These are contextual reliability records, not formal correctness guarantees.
Original downloadable source text
# MolmoSpaces representative learned-skill snippets
# Selected skills: 3 of 27
#===================================================================================================
# Skill 1/3: get_axis_aligned_pull_direction
# Interface: def get_axis_aligned_pull_direction(target_pos: np.ndarray) -> np.ndarray:
# Documentation string: Computes the axis-aligned pull direction (e.g., [1, 0, 0] or [0, -1, 0])
# pointing from the target position towards the robot base, useful for
# determining which way a drawer or cabinet door opens.
# Tier: verified; uses: 30; successes: 10
#===================================================================================================
def get_axis_aligned_pull_direction(target_pos: np.ndarray) -> np.ndarray:
import numpy as np
obs = get_observation()
robot_pos = obs["robot_base_pose"][:3]
direction = robot_pos - target_pos
direction[2] = 0
axis_idx = np.argmax(np.abs(direction))
pull_dir = np.zeros(3)
pull_dir[axis_idx] = np.sign(direction[axis_idx]) if direction[axis_idx] != 0 else 1.0
return pull_dir
#===================================================================================================
# Skill 2/3: push_object_closed
# Interface: def push_object_closed(object_name: str, push_distance: float = 0.15,
# approach_distance: float = 0.15) -> bool:
# Documentation string: Localizes an open object (like a drawer or door), determines its outward
# pull direction, and pushes it inward to close it.
# Tier: experimental; uses: 2; successes: 2
#===================================================================================================
def push_object_closed(object_name: str, push_distance: float = 0.15, approach_distance: float = 0.15) -> bool:
pos = localize_object_with_molmo_sam3(object_name)
if pos is None:
return False
pull_dir = get_axis_aligned_pull_direction(pos)
if pull_dir is None:
return False
close_gripper()
return push_surface_inward(pos, pull_dir, push_distance=push_distance, approach_distance=approach_distance)
#===================================================================================================
# Skill 3/3: plan_top_down_grasp_at_wrist
# Interface: def plan_top_down_grasp_at_wrist(object_name: str, target_world_pos: np.ndarray,
# hover_height: float = 0.15) -> np.ndarray:
# Documentation string: Moves the wrist camera to hover over a target position, uses Molmo and SAM3
# to segment the object, and plans a top-down grasp from the resulting point
# clouds.
# Tier: deprecated; uses: 26; successes: 6
#===================================================================================================
def plan_top_down_grasp_at_wrist(object_name: str, target_world_pos: np.ndarray, hover_height: float = 0.15) -> np.ndarray:
wrist_obs = inspect_at_wrist(target_world_pos, hover_height=hover_height)
if not wrist_obs["ok"]:
return None
w_rgb = wrist_obs["wrist"]["rgb"]
w_depth = wrist_obs["wrist"]["depth"]
w_K = wrist_obs["wrist"]["intrinsics"]
w_T = wrist_obs["wrist"]["pose_mat"]
pt_w = point_prompt_molmo(w_rgb, object_name)
u_w, v_w = pt_w.get(object_name, (None, None))
if u_w is None:
return None
w_masks = segment_sam3_point_prompt(w_rgb, (u_w, v_w))
if len(w_masks) == 0:
return None
w_valid_mask = w_masks[0]["mask"]
pc_segment = mask_to_world_points(w_valid_mask, w_depth, w_K, w_T)
full_mask = (w_depth > 0) & (w_depth < 2.0)
pc_full = mask_to_world_points(full_mask, w_depth, w_K, w_T)
if len(pc_segment) == 0 or len(pc_full) == 0:
return None
pc_full = subsample_point_cloud(pc_full, 10000)
pc_segment = subsample_point_cloud(pc_segment, 5000)
grasps, scores = plan_grasp_from_point_clouds(pc_full, pc_segment, object_name)
if len(grasps) == 0:
return None
best_grasp, best_score = select_top_down_grasp(grasps, scores)
return best_graspE.4 LIBERO Learned Skills
All 308 lines preserve three of forty-seven LIBERO helpers: pose-name localization with segmentation fallback; flat-object grasping with configurable offsets and a measured lift check; and carried-object placement with hover-view segmentation to correct target XY.
The placement helper’s returned success flag means its scripted release/retreat path completed; its code does not independently verify final object support. Task-level verification remains separate. Hard-coded workspace bounds and offsets also show that parameterized code reuse is still conditioned on an environment/API contract.
Original downloadable source text
# LIBERO representative learned-skill snippets
# Selected skills: 3 of 47
#===================================================================================================
# Skill 1/3: localize_object_with_pose_aliases_and_segmentation_fallback
# Interface: def localize_object_with_pose_aliases_and_segmentation_fallback(object_names,
# segmentation_query, use_multiview=True):
# Documentation string: Localizes a target object by trying one or more pose-query names, then falls
# back to language segmentation and an oriented bounding box center if pose
# queries fail.
# Tier: verified; uses: 18; successes: 16
#===================================================================================================
def localize_object_with_pose_aliases_and_segmentation_fallback(object_names, segmentation_query, use_multiview=True):
position = None
quaternion = None
method = None
point_count = None
obb = None
for object_name in object_names:
if position is None:
position, quaternion = get_object_pose(object_name, use_multiview=use_multiview)
method = "object_pose_" + object_name
if position is None:
seg = get_object_3d_points_and_masks_from_language(segmentation_query, use_multiview=use_multiview)
points = seg.get("points_3d")
if points is not None and len(points) > 0:
point_count = len(points)
obb = get_oriented_bounding_box_from_3d_points(points)
position = obb["center"]
quaternion = None
method = "segmentation_obb_" + segmentation_query
return {
"position": position,
"quaternion": quaternion,
"method": method,
"point_count": point_count,
"obb": obb
}
#===================================================================================================
# Skill 2/3: pick_flat_object_with_obb_top_down_retries
# Interface: def pick_flat_object_with_obb_top_down_retries(target_prompt, verification_names,
# approach_height=0.10, lift_height=0.05, top_clearance_attempts=(-0.02, -0.025),
# xy_offset_attempts=((-0.015, -0.015), (-0.02, 0.0)), use_multiview=True,
# min_lift_delta=0.015):
# Documentation string: Segments a flat object, computes an oriented-bounding-box top-down grasp
# pose, tries configurable top-clearance and XY-offset corrections, lifts, and
# verifies that the object rose.
# Tier: experimental; uses: 1; successes: 1
#===================================================================================================
def pick_flat_object_with_obb_top_down_retries(target_prompt, verification_names, approach_height=0.10, lift_height=0.05, top_clearance_attempts=(-0.02, -0.025), xy_offset_attempts=((-0.015, -0.015), (-0.02, 0.0)), use_multiview=True, min_lift_delta=0.015):
attempt_records = []
last_grasp_position = None
last_grasp_quaternion = None
last_lift_position = None
last_obb = None
last_segmentation = None
num_attempts = min(len(top_clearance_attempts), len(xy_offset_attempts))
for attempt_index in range(num_attempts):
open_gripper()
segmentation = get_object_3d_points_and_masks_from_language(target_prompt, use_multiview=use_multiview)
points = segmentation.get("points_3d", None)
if points is None or len(points) == 0:
attempt_records.append({
"attempt_index": attempt_index,
"top_clearance": top_clearance_attempts[attempt_index],
"xy_offset": xy_offset_attempts[attempt_index],
"grasp_position": None,
"grasp_quaternion_wxyz": None,
"lift_position": None,
"verified_position": None,
"verified_lifted": False,
})
continue
obb = get_oriented_bounding_box_from_3d_points(points)
center = obb.get("center", None)
extent = obb.get("extent", None)
if center is None:
attempt_records.append({
"attempt_index": attempt_index,
"top_clearance": top_clearance_attempts[attempt_index],
"xy_offset": xy_offset_attempts[attempt_index],
"grasp_position": None,
"grasp_quaternion_wxyz": None,
"lift_position": None,
"verified_position": None,
"verified_lifted": False,
})
continue
grasp_position = center.copy()
if extent is not None and len(extent) >= 3:
grasp_position[2] = center[2] + float(extent[2]) * 0.5 + float(top_clearance_attempts[attempt_index])
else:
grasp_position[2] = center[2] + float(top_clearance_attempts[attempt_index])
dx, dy = xy_offset_attempts[attempt_index]
grasp_position[0] = grasp_position[0] + float(dx)
grasp_position[1] = grasp_position[1] + float(dy)
grasp_quaternion = (0.0, 0.0, 1.0, 0.0)
if extent is not None and len(extent) >= 2 and float(extent[0]) > float(extent[1]):
grasp_quaternion = (0.0, -0.70710678, 0.70710678, 0.0)
goto_pose(grasp_position, grasp_quaternion, z_approach=approach_height)
close_gripper()
lift_position = grasp_position.copy()
lift_position[2] = lift_position[2] + float(lift_height)
goto_pose(lift_position, grasp_quaternion)
verified_position = None
verified_quaternion = None
for verification_name in verification_names:
verified_position, verified_quaternion = get_object_pose(verification_name, use_multiview=use_multiview)
if verified_position is not None:
break
verified_lifted = False
if verified_position is not None:
verified_lifted = float(verified_position[2]) >= float(grasp_position[2]) + float(min_lift_delta)
last_grasp_position = grasp_position
last_grasp_quaternion = grasp_quaternion
last_lift_position = lift_position
last_obb = obb
last_segmentation = segmentation
attempt_records.append({
"attempt_index": attempt_index,
"top_clearance": top_clearance_attempts[attempt_index],
"xy_offset": xy_offset_attempts[attempt_index],
"grasp_position": grasp_position.tolist() if hasattr(grasp_position, "tolist") else grasp_position,
"grasp_quaternion_wxyz": grasp_quaternion,
"lift_position": lift_position.tolist() if hasattr(lift_position, "tolist") else lift_position,
"verified_position": verified_position.tolist() if hasattr(verified_position, "tolist") else verified_position,
"verified_lifted": verified_lifted,
})
if verified_lifted:
return {
"success": True,
"grasp_position": grasp_position,
"grasp_quaternion": grasp_quaternion,
"lift_position": lift_position,
"obb": obb,
"segmentation": segmentation,
"attempt_records": attempt_records,
}
return {
"success": False,
"grasp_position": last_grasp_position,
"grasp_quaternion": last_grasp_quaternion,
"lift_position": last_lift_position,
"obb": last_obb,
"segmentation": last_segmentation,
"attempt_records": attempt_records,
}
#===================================================================================================
# Skill 3/3: place_carried_object_on_segmented_target_with_hover_correction
# Interface: def place_carried_object_on_segmented_target_with_hover_correction(target_prompt,
# initial_target_center, placement_quaternion, hover_z_offset=0.18,
# release_z_offset=0.05, retreat_z_offset=0.21, use_multiview=True):
# Documentation string: Places an already-grasped object onto a target surface/region by moving to
# an initial hover above the target, re-segmenting the target from the hover
# view, correcting XY placement, descending to release, opening the gripper,
# and retreating upward.
# Tier: verified; uses: 7; successes: 6
#===================================================================================================
def place_carried_object_on_segmented_target_with_hover_correction(target_prompt, initial_target_center, placement_quaternion, hover_z_offset=0.18, release_z_offset=0.05, retreat_z_offset=0.21, use_multiview=True):
def _coord(position, index):
if position is None:
return None
if hasattr(position, "tolist"):
position = position.tolist()
try:
return position[index]
except Exception:
return None
def _make_position_like(reference_position, x_value, y_value, z_value):
if reference_position is not None and hasattr(reference_position, "copy"):
new_position = reference_position.copy()
new_position[0] = float(x_value)
new_position[1] = float(y_value)
new_position[2] = float(z_value)
return new_position
return [float(x_value), float(y_value), float(z_value)]
def _position_with_z_offset(center, z_offset):
x = _coord(center, 0)
y = _coord(center, 1)
z = _coord(center, 2)
if x is None or y is None or z is None:
return None
return _make_position_like(center, x, y, float(z) + float(z_offset))
def _replace_xy_keep_z(base_position, xy_source_position):
bx = _coord(base_position, 0)
by = _coord(base_position, 1)
bz = _coord(base_position, 2)
sx = _coord(xy_source_position, 0)
sy = _coord(xy_source_position, 1)
if bx is None or by is None or bz is None or sx is None or sy is None:
return base_position
return _make_position_like(base_position, sx, sy, bz)
def _xy_distance(position_a, position_b):
ax = _coord(position_a, 0)
ay = _coord(position_a, 1)
bx = _coord(position_b, 0)
by = _coord(position_b, 1)
if ax is None or ay is None or bx is None or by is None:
return None
return ((float(ax) - float(bx)) ** 2 + (float(ay) - float(by)) ** 2) ** 0.5
def _position_within_table_bounds(position):
x = _coord(position, 0)
y = _coord(position, 1)
z = _coord(position, 2)
if x is None or y is None or z is None:
return False
return (-0.45 <= float(x) <= 0.45) and (-0.55 <= float(y) <= 0.55) and (0.60 <= float(z) <= 1.05)
def _segment_target_center(prompt, multiview):
segmentation = get_object_3d_points_and_masks_from_language(prompt, use_multiview=multiview)
points = None
if isinstance(segmentation, dict) and "points_3d" in segmentation:
points = segmentation["points_3d"]
if points is None:
return None, segmentation, None
try:
if len(points) == 0:
return None, segmentation, None
except Exception:
return None, segmentation, None
obb = get_oriented_bounding_box_from_3d_points(points)
center = None
if isinstance(obb, dict) and "center" in obb:
center = obb["center"]
return center, segmentation, obb
log = {
"success": False,
"target_prompt": target_prompt,
"initial_target_center": initial_target_center,
"hover_position": None,
"residual_target_center": None,
"residual_xy_distance": None,
"corrected_target_center": None,
"release_position": None,
"retreat_position": None,
"placement_quaternion": placement_quaternion,
}
if initial_target_center is None:
log["failure_reason"] = "no_initial_target_center"
return log
target_center = initial_target_center
hover_position = _position_with_z_offset(target_center, hover_z_offset)
if hover_position is None:
log["failure_reason"] = "could_not_build_hover_position"
return log
log["hover_position"] = hover_position
goto_pose(hover_position, placement_quaternion, z_approach=0.0)
residual_center, residual_segmentation, residual_obb = _segment_target_center(target_prompt, use_multiview)
log["residual_target_center"] = residual_center
log["residual_segmentation"] = residual_segmentation
log["residual_obb"] = residual_obb
log["residual_xy_distance"] = _xy_distance(target_center, residual_center)
if _position_within_table_bounds(residual_center):
target_center = _replace_xy_keep_z(target_center, residual_center)
log["corrected_target_center"] = target_center
corrected_hover_position = _position_with_z_offset(target_center, hover_z_offset)
if corrected_hover_position is not None:
log["corrected_hover_position"] = corrected_hover_position
goto_pose(corrected_hover_position, placement_quaternion, z_approach=0.0)
release_position = _position_with_z_offset(target_center, release_z_offset)
if release_position is None:
log["failure_reason"] = "could_not_build_release_position"
return log
log["release_position"] = release_position
goto_pose(release_position, placement_quaternion, z_approach=0.0)
log["open_result"] = open_gripper()
retreat_position = _position_with_z_offset(target_center, retreat_z_offset)
if retreat_position is not None:
log["retreat_position"] = retreat_position
goto_pose(retreat_position, placement_quaternion, z_approach=0.0)
log["final_target_center"] = target_center
log["success"] = True
return logAppendix F Agent I/O and Prompts
The appendix retains concrete role inputs/outputs and the full LIBERO prompt templates. Dynamic scene descriptions, images, APIs and retrieved memories remain placeholders; Molmo runs adapt environment-specific inputs. Prompt instructions express intended behavior, not proof that every generated policy follows them.
F.1 Agent I/O Summary
Table 15 assigns proposal, environment validation, planning, code generation/checking, goal and step verification, diagnosis, isolated practice and memory duties. Some checks are deterministic while others use language or vision-language models; their evidence and failure modes differ.
Original selectable table
| Agent or module | Conditioning context | Output and responsibility |
| Task Proposal | ||
| Task Proposer | Scene context, compact skill summary, recent tasks and failures | Candidate play tasks with involved objects and required skills. |
| Environment Creator | Selected task and environment catalog | Executable task instance, such as a LIBERO BDDL specification or a grounded MolmoSpaces task artifact. |
| Environment Verifier | Instantiated environment, goal predicates, and grounded scene | Pre-execution validity decision; rejects malformed or ungrounded tasks. |
| Execution | ||
| Planner | Task, initial observation, active skills, primitive APIs, and retrieved lessons | Ordered plan with step-level skill selection and predicted failure points. |
| Planner Verifier | Initial observation and proposed plan | Visual grounding verdict and structured plan-refinement feedback. |
| Policy Writer | Verified plan, selected skills, APIs, lessons, and retry feedback | Executable Python policy with step-context instrumentation. |
| Quality Checker | Generated source code and available API registry | Deterministic blocking and advisory code-review findings. |
| Goal Verifier | Terminal state, task predicates, execution status, and visual evidence when needed | Task-level success signal with state-based evidence. |
| Per-Step Verifier | Step objective, code slice, runtime values, and visual trace | Localized pass/fail verdict for each executed plan step. |
| Failure Diagnoser | Failed trajectory, goal-verifier result, and per-step verdicts | Failure category, first failed step, repair feedback, and retry-routing flags. |
| SubAgent | Isolated subgoal, relevant APIs, and environment reset function | Visually verified task-specific script for a persistent local bottleneck. |
| Memory Management | ||
| Feedback Generator | Task outcome, successful code, or failure diagnosis | Extract reusable skills after success or prepare retry feedback after failure. |
| Memory Curator | Stored failure lessons, learned skills, and their usage evidence | Merge, rewrite, deprecate, delete, or retain decisions for persistent memories. |
| Skill Proposer | Repeated failure summaries, primitives, and learned skills | Statically validated experimental helper targeting a recurring missing capability. |
F.2 Agent Prompts
The twelve exact prompts below preserve their schemas, examples and behavioral constraints, including multi-page continuations. Brief notes explain each role without replacing these source templates.
Task Proposer.
The proposer is asked for a simple, single-step exploratory manipulation grounded in skill history, a catalog, known environment limitations and a benchmark-derived reliable-pick list. It returns a task, symbolic goals, expected skills and affordance hints. This is a strongly scaffolded curriculum, even though the agent selects its own objectives.
Original downloadable source text
You are a curiosity-driven task proposer for a robot learning system. For
this run you are playing the role of a 3-4 year old child exploring the
robot's world — see one object, reach for it, do ONE simple thing.
Your job: look at the robot's skill library and task history, then propose
a SIMPLE, single-step manipulation task that would help the robot discover
or solidify a child-feasible primitive.
CURRENT SKILL LIBRARY:
{skill_context}
TASK HISTORY (previously attempted tasks and outcomes):
{task_history}
Each history entry includes: success/failure, failure_reason (e.g.,
"grasp_failure", "env_creation_failed", "code_bug"), objects_used,
fixtures_used, goal_predicates, and whether the environment was successfully
created. Use this to adapt your proposals.
AVAILABLE BUILDING BLOCKS:
{catalog}
{curriculum_hint}
KNOWN ENV LIMITATIONS (you MUST inspect your candidate task against this
list before responding — proposing a task that hits one of these wastes
a full iteration on a guaranteed crash):
{env_limitations}
PICK-PRIMITIVE RELIABILITY (the target object you pick MUST come from the
RELIABLE list below — the pick primitive is benchmark-verified to fail on
the UNRELIABLE list):
{pick_reliability}
REASONING PROCESS:
1. Review the task history — what worked, what failed, and WHY?
2. If recent tasks failed, consider whether to try a simpler variant or a
completely different approach.
3. If recent tasks succeeded, consider building on those skills with
something slightly harder — but still single-step and child-feasible.
4. Pick objects and fixtures that make the task interesting but achievable.
The target object (arg1 of an `On`/`In` goal) MUST come from the RELIABLE
list in the Pick-Primitive Reliability block.
5. Ensure the goal uses valid predicates from the catalog.
6. Do NOT use the "Stack" predicate (currently unsupported in the
simulator).
7. For kitchen scenes use LIBERO_Kitchen_Tabletop_Manipulation, for table
scenes use LIBERO_Tabletop_Manipulation.
8. **Inspect against the KNOWN ENV LIMITATIONS list above.** If your
intended (predicate, container) combo is on it, pick a different
container OR a different predicate. State in `reasoning` that you
checked the list and explain the substitution, or note "no broken
combos apply" if the list is empty for your candidate.
Respond in JSON with keys:
- "reasoning": short first-person curious sentence — explain your reasoning,
referencing task history where relevant
- "language": natural-language goal (e.g., "I want to put the mug on the
plate")
- "scene_type": problem class name from the catalog
- "objects": list of object type strings (e.g., ["porcelain_mug", "plate"])
- "fixtures": list of fixture type strings (e.g., ["wooden_cabinet",
"flat_stove"])
- "goal": list of predicate lists (e.g., [["On", "porcelain_mug_1",
"plate_1"]])
- "expected_new_skills": list of skill description strings
- "novelty_score": float 0.0 to 1.0
- "difficulty_estimate": "easy" | "medium" | "hard" (curious-child picks →
usually "easy")
- "affordance_hints": object mapping each object / fixture name you listed
above to a short string (<= 20 words) describing, from your own prior
knowledge of physical objects, which feature a gripper should target to
interact with it successfully for this task. Use neutral descriptive
language about geometry and interaction; do not prescribe specific code
primitives or phrase it as a rule. If you do not have confident prior
knowledge about an object, write an empty string for it rather than
guessing. Keys must match the object/fixture names you listed above.Environment Creator.
The creator writes a complete BDDL instance with exact catalog object types, placement regions, fixture subregions and symbolic goals. Examples explain LIBERO’s workspace-prefix conventions. The full template is retained; separate deterministic instantiation checks, not the language instructions alone, decide validity.
Original downloadable source text
You are an Environment Creator for a robot learning system. Your job is to
take a high-level task proposal and generate a valid BDDL (Behavior
Description Definition Language) specification that can be instantiated as a
MuJoCo simulation environment.
{catalog}
## BDDL Format
```
(define (problem PROBLEM_CLASS_NAME)
(:domain robosuite)
(:language NATURAL_LANGUAGE_GOAL)
(:regions
(region_name
(:target workspace_or_fixture)
(:ranges (
(x_min y_min x_max y_max)
)
)
(:yaw_rotation (
(yaw_min yaw_max)
)
)
)
...
(fixture_sub_region
(:target fixture_instance)
)
)
(:fixtures
workspace_instance - workspace_type
fixture_1 - fixture_type
...
)
(:objects
obj_1 obj_2 - object_type
obj_3 - another_type
...
)
(:obj_of_interest
relevant_obj_1
relevant_region_1
)
(:init
(On obj_1 workspace_region_1)
(On fixture_1 workspace_fixture_region)
...
)
(:goal
(And (Predicate arg1 arg2) ...)
)
)
```
## Rules
1. The problem class name MUST be one of the listed scene types (e.g.,
LIBERO_Kitchen_Tabletop_Manipulation).
2. Every object and fixture in (:init) must have a placement region defined
in (:regions).
3. Region ranges are (x_min y_min x_max y_max) relative to the workspace.
Keep ranges small (0.02 wide). Stay within [-0.45, 0.45] for x and [-0.35,
0.35] for y.
4. Fixtures that go on the workspace also need an init region and an (On
fixture workspace_region) in (:init).
5. Fixture sub-regions (like top_region for cabinet, cook_region for stove,
heating_region for microwave) use (:target fixture_instance) with NO
(:ranges) — they are predefined by the fixture.
6. Objects of interest ((:obj_of_interest)) should include the
objects/regions that appear in the goal.
7. Use And to combine multiple goal predicates.
8. Object instances are numbered: butter_1, akita_black_bowl_1,
akita_black_bowl_2, etc.
9. CRITICAL: In (:regions), region names must NOT include the workspace
prefix. LIBERO auto-prepends "{workspace}_" to region names. So define
"butter_init_region" (NOT "kitchen_table_butter_init_region"). In (:init)
and (:goal), use the FULL prefixed name: "kitchen_table_butter_init_region".
10. In (:goal), fixture sub-regions are prefixed: {fixture_instance}_{sub}
(e.g., wooden_cabinet_1_top_region).
11. CRITICAL: In (:fixtures), the workspace type is the LOWERCASE workspace
type from the catalog (e.g., "table", "kitchen_table"), NOT the problem
class name. Example: "main_table - table" or "kitchen_table -
kitchen_table".
12. For fixtures with sub-regions (cabinet, microwave, stove), you MUST also
define a "top_side" region with (:target fixture_instance) for the
microwave/cabinet — this is needed for the object state tracking.
13. CRITICAL: Use the exact object and fixture types requested in the Task
Proposal. Do NOT substitute a similar catalog item (for example, never
replace black_book with orange_juice). If the proposal names black_book,
(:objects) and (:goal) must reference black_book_1.
## Examples
### Pick and place
```
(:language pick up the butter and put it on the plate)
(:fixtures kitchen_table - kitchen_table)
(:objects butter_1 - butter plate_1 - plate)
(:goal (And (On butter_1 plate_1)))
```
### Open drawer and put object inside
```
(:language open the top drawer and put the bowl inside)
(:fixtures main_table - table wooden_cabinet_1 - wooden_cabinet)
(:objects akita_black_bowl_1 - akita_black_bowl)
(:goal (And (In akita_black_bowl_1 wooden_cabinet_1_top_region)))
```
### Turn on stove and place pot
```
(:language turn on the stove and put the moka pot on it)
(:fixtures kitchen_table - kitchen_table flat_stove_1 - flat_stove)
(:objects moka_pot_1 - moka_pot)
(:goal (And (Turnon flat_stove_1) (On moka_pot_1 flat_stove_1_cook_region)))
```
### Put object in microwave and close it (FULL EXAMPLE)
```
(define (problem LIBERO_Kitchen_Tabletop_Manipulation)
(:domain robosuite)
(:language put the butter in the microwave and close it)
(:regions
(microwave_init_region
(:target kitchen_table)
(:ranges ((-0.01 0.34 0.01 0.36)))
(:yaw_rotation ((0.0 0.0)))
)
(butter_init_region
(:target kitchen_table)
(:ranges ((0.02 -0.01 0.08 0.01)))
(:yaw_rotation ((0.0 0.0)))
)
(top_side
(:target microwave_1)
)
(heating_region
(:target microwave_1)
)
)
(:fixtures
kitchen_table - kitchen_table
microwave_1 - microwave
)
(:objects
butter_1 - butter
)
(:obj_of_interest
butter_1
microwave_1
)
(:init
(On microwave_1 kitchen_table_microwave_init_region)
(On butter_1 kitchen_table_butter_init_region)
)
(:goal
(And (In butter_1 microwave_1_heating_region) (Close microwave_1))
)
)
```
NOTE: Region names do NOT include workspace prefix — LIBERO auto-prepends
it.
NOTE: microwave uses "heating_region" (NOT "contain_region"). Always include
"top_side" for microwave/cabinet.
## Task Proposal
{task_proposal}
## Instructions
Generate a complete, valid BDDL specification for this task. Output ONLY the
BDDL text inside a code fence, nothing else. Ensure:
- All referenced regions are defined
- All objects/fixtures have init placements
- Goal predicates use correct instance names
- Region coordinates don't overlap (space objects at least 0.1 apart)Planner
Planning uses the initial scene to order prerequisites and reuses exact helper names. It also predicts a bottleneck and success probability from supplied empirical priors. Conditional failure lessons are weighted by recent success and usage, rather than treated as universally applicable rules; those predictions are not calibrated probabilities established by the paper.
Original downloadable source text
You are a robot task planner. Decompose the given task into concrete steps
using available skills.
A current agentview image of the initial scene may be attached below.
When present, INSPECT IT before planning: check the state of any
articulated elements (open vs closed), which objects are visible and
where, and whether there is clutter or occlusion. Let the scene
state shape the step ordering — if a prerequisite state is not yet
met (closed container needs opening before placement, obstacle
blocks the target, object is in a non-graspable orientation), insert
the prerequisite step BEFORE the dependent one. Stay task-agnostic:
describe what you SEE, don't assume a prototype layout.
AVAILABLE PRIMITIVES (lower-level building blocks — use only when no learned
skill fits):
{primitive_list}
Create a step-by-step plan. For each step, specify which existing
skills/primitives to use and whether a new skill needs to be written.
IMPORTANT:
- Learned skills (in the LEARNED SKILLS section below) were extracted from
past successful runs, but may have been a different task. Still, reuse them
by name whenever they match the step's intent — they encapsulate working
perception→grasp→place logic.
- When you list a skill in "relevant_skills", use its EXACT name (the `name`
field, not the Example docstring). Do not abbreviate: e.g. use
"segment_sam3_text_prompt", not "segment_sam3".
- The robot may expose two cameras: agent-view (`obs["agentview"]`) gives a
fixed wide shot, and wrist-camera (`obs["robot0_eye_in_hand"]`) is mounted
on the gripper. The default agent-view frequently mis-localizes small /
visually similar objects (cream cheese vs milk, butter, chocolate pudding).
Use the wrist view only through functions that are actually listed in
AVAILABLE PRIMITIVES.
Respond in JSON with a "steps" array AND a top-level "prediction_card"
object. Each step has: "id" (string like "step-1"), "description" (string),
"relevant_skills" (list of skill name strings), "skill_code" (string, empty
for primitives), "new_skill_needed" (boolean), "notes" (string).
The "prediction_card" must contain: "predicted_success_probability"
(0.0-1.0), "predicted_bottleneck_step", and "prediction_reasoning". Estimate
success probability using the currently available skills, scene context, and
likely execution bottlenecks.
CALIBRATION (do not anchor on 0.8): empirically, top-level success rate
across mixed LIBERO pick-and-place tasks on this stack is ~30-40%
task per iteration. A bare pick-and-place with a reliable target (mug,
milk, cookies) on an open surface lands around 0.45-0.65. A multi-step
task with an articulated fixture (open drawer → place → close) lands
around 0.10-0.25. A grasp on an unreliable target (butter,
chocolate_pudding, wine_bottle) is closer to 0.05. Use these as
priors and adjust by what's visible in the scene image. Predictions
clustered in a narrow band (e.g. always 0.75-0.85) carry no information
for downstream surprise-driven prioritization — spread them.
# === ITERATION-SPECIFIC INPUTS BELOW (vary per task; everything above is
stable for cache reuse) ===
LEARNED SKILLS FROM PRIOR SUCCESSFUL RUNS (with code). PREFER THESE OVER
BUILDING FROM PRIMITIVES:
{selected_skills}
## RECENT LESSONS FROM PAST FAILURES (advisory — gate by the metadata
footer)
{failure_lessons}
Each lesson uses a "WHEN ... | WRONG ... | DO ..." shape, followed by a
metadata footer like `(confidence=0.80, applied=5× helped=1×,
last_helped=iter4 (2 iters ago), distilled=iter2 (4 iters ago))`. Treat
lessons as priors — adopt the DO recipe when (a) the WHEN clause matches the
current task AND (b) the lesson has reliability (helped/applied >= 0.5 with
applied >= 2) or is recent (last_helped within ~2 iters). Lessons with low
reliability or `last_helped=never` / stale (>= 5 iters ago) are background
context only; don't let them shape step ordering against direct scene
evidence.
TASK: {task_description}
GOAL CONDITIONS: {goal_conditions}
OBJECT SCOPE: {object_scope}Planner Verifier.
The plan verifier checks visual misperception, missing prerequisites, impossible ordering and object/skill scope. Its JSON distinguishes pass, fail and ambiguous evidence with confidence. It inspects a plan and image without executing the policy.
Original downloadable source text
You are a strict visual + structural verifier for a robot task plan.
You are given:
- The agentview RGB image that the planner saw before writing the plan
(this is what the robot will actually start from).
- The natural-language task and goal conditions.
- The object_scope dict the planner was told to work in.
- The full plan the planner produced (ordered steps with descriptions
and relevant_skills).
- The names of skills/primitives the planner could choose from.
Your job is to decide whether the plan is consistent with what the
image shows AND structurally sound. You do NOT execute code. You
report findings only.
CHECKS (apply all four — list every violation you find, not just one):
1. Visual misperception
The plan asserts a scene state the image contradicts.
Examples (illustrative — do not transplant onto unrelated scenes):
- Plan step says "open the drawer" but the drawer is already open.
- Plan step says "pick up the milk carton" but no milk-shaped
object is visible in the image.
- Plan step describes the wrong object color/material/shape
compared to what is on the table.
2. Missing prerequisite
A step needs an earlier enabling step that is missing.
Examples:
- Plan tries to place an object inside a container the image shows
as closed, with no preceding open step.
- Plan tries to grasp an object that is obstructed by another
object the image shows on top of it, with no preceding clear/move step.
3. Wrong step ordering
Steps are present but in a physically impossible order.
Examples:
- Place-before-pick.
- Lift / transport before grasp.
- Close-container before placement.
4. Object-scope mismatch
Plan references an object that is not in `object_scope` and not
visible in the image, OR references a skill name that is not in
the available-skills list.
VERDICT RULES:
- If you find ANY high-confidence violation in checks 1-4: verdict = "fail".
- If something looks suspicious but the image is ambiguous (motion blur,
occlusion, low resolution): verdict = "ambiguous", confidence below 0.6.
- Otherwise: verdict = "pass".
CONFIDENCE: be honest. If the image is dark / blurry / a render
artifact / shows a state you cannot read with certainty, lower the
confidence. Never invent details that aren't visible.
Stay task-agnostic. Describe what you ACTUALLY SEE in the image and
what the plan ACTUALLY SAYS — do not project memorized prototype tasks.
Respond ONLY with a single JSON object matching this schema:
```json
{
"verdict": "pass" | "ambiguous" | "fail",
"confidence": 0.0,
"scene_summary": "<visible scene, fixture state, and occlusion summary>",
"issues": [
{
"kind": "visual_misperception" | "missing_prerequisite" |
"wrong_ordering" | "object_scope_mismatch",
"step_id": "<step-N from the plan, or null>",
"evidence": "<what in the image / plan supports this issue>",
"suggested_fix": "<specific step or object correction>"
}
],
"summary_for_refine_plan": "<actionable structural guidance, or empty>"
}
```
# === ITERATION-SPECIFIC INPUTS BELOW (vary per attempt) ===
TASK: {task_description}
GOAL CONDITIONS: {goal_conditions}
OBJECT SCOPE: {object_scope}
AVAILABLE SKILLS/PRIMITIVES (names only):
{available_skills}
PLAN UNDER REVIEW (ordered):
{plan_steps_block}Policy Writer.
Policy writing restricts calls to available APIs, bounds retries, records per-step outcomes and favors argument changes before structural rewrites. Detailed contracts cover ndarray truth tests, point-cloud shapes and camera-to-world transforms.
The prompt treats pre-grasp identity verification as a gate but avoids suspending a successfully grasped object solely because a later visual check is uncertain. Current retry evidence and assigned SubAgent strategy take priority over stale lessons. These are system instructions; a VLM verified flag is not independent physical ground truth despite the prompt’s wording.
Original downloadable source text
You are a deterministic policy writer for a code-as-policy robot system.
Write Python code that executes the following plan.
AVAILABLE IMPORTED FUNCTIONS (use ONLY these, not env.<method>):
{available_functions}
API DOCUMENTATION:
{api_docs}
LEARNED HELPER FUNCTION REFERENCES (already defined and callable — use them
directly):
{skill_code}
REQUIREMENTS:
- Use the primitive functions listed in AVAILABLE IMPORTED FUNCTIONS above
AND any LEARNED HELPER FUNCTION REFERENCES shown above.
- The learned helper functions are pre-defined and available in your
execution scope. Call them directly.
- Do NOT invent wrapper functions on env or low_level_env.
- Do NOT use while True loops or any unbounded retry patterns. Use bounded
for loops instead.
- Set RESULT to a structured dict describing success/failure for each step.
- Add explicit comments before each plan step.
{runtime_marker_requirement}
- Define reusable sub-functions for non-trivial skill sequences that could
be added to the skill library.
- On retries, follow the diagnoser's requested edit scale. If a top-of-retry
"## EDIT SCALE: argument-level" banner is present (or the prose says
argument-level), keep the same primitive/helper call sequence and change
only the named arguments first (target text, verify_label, pose/offset,
approach axis, release height, retry count, or gate threshold). Rewrite
helper bodies, replace primitives, or add new wrappers only when the banner
/ diagnosis says rewrite-needed because arguments cannot express the fix,
the API/function is wrong for the task, the sequence is structurally wrong,
or there is a real code/logic bug.
CRITICAL: Keep code SHORT. Stick to one or two focused function definitions
plus a brief driver block. Do NOT define throwaway helper wrappers
(`make_pose`, `offset_pose`, `find_object_name`, `verify_step`) when the
body is a single line.
IMPORTANT: Use ONLY functions from the AVAILABLE IMPORTED FUNCTIONS list and
LEARNED HELPER FUNCTION REFERENCES above.
Do NOT call any function not in those lists. Check the API DOCUMENTATION for
correct function signatures.
Extract object names from the GOAL description below.
General rules:
- Read the API DOCUMENTATION carefully for the correct function signatures
and return types.
- When retry feedback names a primitive/helper call, preserve that call and
tune its arguments before changing the overall code structure, unless the
feedback explicitly calls for a larger rewrite.
- Use bounded for loops (not while True) for retry patterns.
- For multi-step tasks (pick A then place on B), chain the patterns
sequentially.
- Never leave a successfully grasped object suspended because a pre-release
wrist check failed. Once the placement target has been localized, execute a
cautious descend, open the gripper, retreat, and then verify the final
resting state.
- NumPy array truthiness: localization primitives often return ndarrays or
lists of candidates. NEVER write `if pos:` or `if not result:` on an ndarray
— raises "truth value of an array is ambiguous". Instead use `if pos is
None` / `if len(pos) == 0` / `if result.get("verified")` on explicit
scalars. This pattern has burned many past attempts.
- **vlm_verify / verify_object_identity contract: the `verified` flag is
GROUND TRUTH for whether localization landed on the correct object. When
EVERY localization attempt for an object returns `verified=False` (or the
call errored — same effect, treat as not-verified), the policy MUST bail for
that step. Do not fabricate a position from depth at an unverified pixel —
those silently turn into wrong-object grasps that crash plan_grasp / pick
the wrong object. A failed perception for a target the proposer named is a
legitimate iter outcome — the system would rather skip than grasp the wrong
object. Conversely: this rule applies ONLY to PRE-grasp localization
verification. POST-grasp wrist / held-object checks (see the next bullet)
are diagnostics, not gates.
- **`plan_grasp_from_point_clouds(pc_full, pc_segment, label="")` SHAPE
CONTRACT**: BOTH `pc_full` AND `pc_segment` MUST be `(K, 3)` arrays of 3D
xyz POINTS (one row per point, world frame). DO NOT pass a boolean mask of
shape `(H, W)`, a 2D depth image, a flat `(N,)` array, or a `(H, W, 3)`
per-pixel xyz map directly — they all crash the server with `boolean index
did not match indexed array along dimension 1` 500-errors that the runtime
self-check has to absorb. Correct pattern: backproject depth + camera
intrinsics into a per-pixel world point cloud, reshape to `(H*W, 3)`, then
filter — `pc_full = pts_world.reshape(-1, 3)` (after dropping invalid
depths), and `pc_segment = pts_world[mask].reshape(-1, 3)` where `mask` is
the SAM3/Molmo binary mask. Verify shapes with `assert pc_full.ndim == 2 and
pc_full.shape[1] == 3` before calling.
- **`plan_grasp(depth, K, mask)` FRAME**: the returned 4×4 grasp pose is in
the **CAMERA frame of `K`/`depth`**, NOT world. If you fed agentview
depth/K, you must multiply by agentview's `pose_mat` (which is camera→world)
before passing the position to `goto_pose`/`solve_ik`: `grasp_world =
pose_mat @ grasp_cam`. Treating the raw return as world frame puts the
gripper behind the camera and silently 500s downstream IK.
- Between major steps (after a grasp, after a place, after opening a door),
write explicit per-step booleans into RESULT using only observable execution
signals from available primitives. Prefer primitive return values (for
example, `grasp_with_wrist_closeloop(... )["success"]`) or successful
completion of the intended action sequence. Do NOT invent extra verification
helpers, and do NOT hard-gate later place/release actions on a post-grasp
VLM judgment once the grasp primitive has already reported success — final
verification after release is the reliable signal. The outer retry system
and verifier will judge final task success separately.
{env_specific_patterns}
# === TASK-SPECIFIC INPUTS BELOW (vary per attempt; everything above is
stable for cache reuse) ===
SCENE: {scene_model}
GOAL: {goal_conditions}
OBJECT SCOPE: {object_scope}
PLAN:
{plan_steps}
{success_context}
{subagent_directive}
## LESSONS FROM PAST FAILURES (advisory — NOT ground truth)
{failure_context}
The block above (if non-empty) lists distilled lessons in "WHEN ... | WRONG
... | DO ..." form, each followed by a metadata footer like
`(confidence=0.80, applied=5× helped=1×, last_helped=iter4 (2 iters ago),
distilled=iter2 (4 iters ago))`. Use that footer to weigh how much to trust
each lesson.
PRIORITY ORDER when signals conflict:
0. The SUB-AGENT DIRECTIVE block above (if non-empty) — this is the
parallel sub-agent's committed approach assignment, a HARD constraint,
and dominates every other guidance source below.
1. The current attempt's RETRY CONTEXT (diagnoser feedback + visual
evidence + PREVIOUS CODE below) — this saw what actually happened on
this exact task and is ground truth for what to fix.
2. A lesson with high reliability (helped/applied >= 0.5 with applied >=
2) AND recent `last_helped` (<= 2 iters ago) — strong prior, follow its
DO recipe.
3. A lesson with low reliability (helped/applied < 0.3) OR stale
(`last_helped=never` or >= 5 iters ago) — background prior only; don't
let it override the current diagnosis.
Empty block means no relevant prior failures.
{retry_context}
Quality Checker.
After syntax/API validation, the quality-review prompt flags ungrounded trial-and-error, redundancy, missing error handling and unnecessary complexity. Its semantic findings are advisory; blocking static checks are a separate stage.
Original downloadable source text
You are a code quality reviewer for robot policy code.
Review the following code for semantic issues. The code has already passed
syntax and API validation checks.
CODE:
```python
{code}
```
AVAILABLE PRIMITIVES: {primitive_list}
TASK GOAL: {goal}
Check for these advisory issues (non-blocking, but flag them):
1. Trial-and-error patterns (random sampling loops)
2. Non-geometrically-grounded approaches
3. Redundant operations
4. Missing error handling for known failure modes
5. Overly complex solutions when simpler ones exist
Respond in JSON with keys: "issues" (list of objects with "severity",
"description", "suggestion"), "overall_assessment" (string).Goal Verifier.
The LIBERO goal verifier trusts simulator predicate states over visual guesses, identifies a blocking predicate and proposes code-specific remedies. A satisfied state short-circuits diagnosis. Suggested causal explanations and generalizable lessons remain model interpretations of the observed outcome.
Original downloadable source text
You are RATS's state-based verifier and code-level failure analyst.
Inputs you receive every attempt:
- TASK GOAL (natural language + symbolic goal predicates from BDDL).
- PER-PREDICATE STATE: which goal predicates the simulator currently judges
satisfied vs unsatisfied. This is ground truth from the LIBERO predicate
evaluator — trust it over any vision guess.
- CODE THAT JUST RAN: the policy_writer's most-recent attempt.
- ATTEMPT HISTORY: short summary of prior attempts in this iteration
(per-attempt: success bool, unsatisfied predicates, one-line failure
mode).
This shows what's already been tried so you don't repeat.
Your job:
1. From the per-predicate state, figure out WHICH unsatisfied predicate is
the root failure (abstractly: if a containment predicate is False but
its prerequisite state predicate is True, the root is the placement
step, not the state-change step).
2. From the code, identify the CODE-LEVEL antipattern that caused it
(generic patterns: "called close_gripper() but never approached the
target centroid", "used a gripper quaternion that is sideways for
this object", "called a place-primitive before any grasp completed").
3. Propose a CONCRETE fix the next attempt should apply, naming real
primitives by exact name (segment_sam3_text_prompt, plan_grasp,
inspect_at_wrist, verify_object_identity, goto_pose, close_gripper,
open_gripper, ...). Include 1-3 lines of pseudo-Python if it helps.
4. Identify what conditions of THIS failure generalize to FUTURE iterations
(e.g., "any task that requires placing a packaged item into a drawer-
like fixture" or "any cluttered scene with multiple similar containers").
This goes to failure_memory so later iterations on different tasks can
reuse the lesson.
Hard rules:
- DO NOT produce vague advice like "verify the grasp" — be CODE-specific.
- DO NOT say "vision was uncertain" without proposing an alternative call
(e.g., "fall back to point_prompt_molmo with prompt 'X package'" or
"call inspect_at_wrist(world_pos) to get a wrist close-up and
re-segment").
- If the state says the task IS satisfied, just output {"task_satisfied":
true}
and skip the rest.
- DO NOT repeat suggestions that ATTEMPT HISTORY shows have already been
tried.
Output ONLY valid JSON of this exact shape:
{
"root_cause_predicate": "<most-blocking [Predicate args], or null>",
"code_antipattern": "<one-sentence concrete code mistake>",
"fix_suggestion": "<code-level fix naming real primitives>",
"fix_pseudo_code": "<optional, 1-3 lines of pseudo-Python; '' if none>",
"generalizable_lesson": {
"condition": "<when does this lesson apply in the future>",
"antipattern": "<the antipattern, generic form>",
"remedy": "<the fix, generic form>",
"applicable_objects": [...],
"applicable_actions": [...]
},
"confidence": <float 0-1>
}Failure Diagnoser.
Diagnosis combines trajectory media, wrist images, code, numeric artifacts and previous attempts. It checks terminal stability, avoids blaming an already-working behavior without new evidence, and distinguishes physical stalls from merely slow execution.
Its output names the first failed step, failure category, repair scale and whether the plan structure is wrong. Persistent subskills can receive two or three distinct parallel practice strategies, but only if they are meaningfully executable from reset. The complete 233-line template preserves these routing rules and output schema.
Original downloadable source text
You are diagnosing a robot manipulation attempt. Use the images AND the
code/plan together — vision alone misses intent, code alone misses what
actually happened.
HOW TO ANALYZE
1. Scan the trajectory media (either an mp4 video, or a time-ordered
sequence of frames — the media manifest in the system prompt above
tells you which one this call sent). Track end-to-end: how the arm
approached each object, when close_gripper / open_gripper happened
(was the gripper near the target at that instant?), whether the arm
retreated with the object held or empty, and whether the final scene
matches the intended end state.
2. Use the wrist-camera frame (labelled "After wrist-camera frame" in
the media manifest, when present) for fine-grained grasp alignment
— it resolves "did the gripper close ON the target or next to it?",
"is the object still held?", "is the hand over the right container
opening?". Side-angle agentview cannot answer these. The wrist
frame is sent as a still image regardless of whether the trajectory
itself is video or frames; do NOT confuse it with the last image of
the input list (diagnostic artifact images may follow it).
3. If a PERCEPTION / GRASP / MOTION DIAGNOSTICS section, RAW NUMERIC
ARTIFACT FILES section, or DIAGNOSTIC ARTIFACT IMAGES are present,
use them as extra spatial evidence:
- Compare agentview vs wrist pointcloud visualizations and the listed
object centroids to decide where the robot believed the task object
was.
- Use the raw NPZ/JSON excerpts for actual numbers: point samples,
camera intrinsics/extrinsics, masks, grasp transforms/scores, selected
indices, target poses/joints, and before/after/error robot states.
The full files remain at the listed paths if another debugging pass
needs exact arrays.
- Inspect available grasp candidates, top scores, selected grasp
index/position, and approach axes. If the selected grasp was on the
wrong side, above the wrong feature, or aimed into a fixture, say so.
- Compare goto_pose / IK target positions with the segmented
pointcloud and the trajectory media. For timeouts, this is often
the best clue that the robot repeatedly pushed into a wall/door
face or took a long, unnecessary approach path.
- Pointclouds and grasp candidates are snapshots from the moment the
policy called perception. Before telling the policy writer to reuse
a prior pointcloud/centroid/grasp candidate, check whether later
arm or gripper motion could have contacted or moved the target. If
the object may have moved, was dropped, was pushed, or the
trajectory media is ambiguous, tell the writer to recompute
perception from a fresh
observation/current agentview+wrist view and reuse only the
high-level strategy or object feature, not the old numeric points.
These diagnostics are hints about the policy's internal perception;
still prefer direct visual evidence when the images contradict them.
4. Cross-check CODE vs VISION. If the code targeted a location or
object feature that the affordance hints or images suggest was the
WRONG feature (e.g. grasped the body of an object when a handle or
rim was visible; pushed a front face instead of engaging an edge
that actuates motion), call it out specifically — name the feature
the robot should have targeted instead.
Then decide the edit scale for the next policy attempt:
- Prefer an argument-level fix when the same primitive/helper call
sequence is appropriate but aimed at the wrong object, feature,
pose, offset, approach axis, release height, label, threshold, or
retry count. In `policy_feedback`, name the exact call and the
arguments to change.
- Recommend a larger rewrite/replacement only when arguments cannot
express the needed behavior, the API/function is wrong for the
task, the sequence is structurally wrong, repeated argument-level
fixes have already failed the same way, or there is a real
runtime/logic bug.
5. **Cross-reference with prior attempts**. If a sub-behavior (grasp,
approach, lift, place) was shown to work in an earlier attempt —
either because the prior diagnosis said so OR because the prior
attempt's final frame visibly shows it (e.g. the target was held
clear of its source surface) — do NOT re-attribute failure to that
sub-behavior for this attempt unless the current images show fresh
visual evidence of regression. Instead, name explicitly which
sub-behavior was already working, and focus `policy_feedback` on
the part that is still failing. Tell the policy writer to REUSE
the earlier attempt's approach for the working part (generically:
"keep the earlier grasp that lifted the target; change only the
release/placement target"). Stay abstract — do not hard-code
task-specific noun examples.
If the current or prior terminal frame shows the target held in the
gripper but not placed, treat the remaining failure as
placement/release/integration. Do NOT recommend adding a hard
"do not proceed unless wrist/held verification is true" gate; that
pattern leaves objects suspended after a successful grasp. Feedback
should instead say to complete transport, descend to the target,
open the gripper, retreat, and verify after release.
6. For EACH plan step listed in the INTENDED PLAN STEPS section below,
render a visual verdict: does the FINAL SEGMENT of the trajectory
media (the last ~1-2 seconds of video, or the last 2-3 sampled
frames when sent as a frame sequence) show that step's
natural-language intent was realized? Prefer evidence over
inference — if you cannot see whether the step succeeded, mark
`visually_satisfied: false` and note the ambiguity in the evidence
field.
**End-of-trajectory stability matters.** Mark a step
`visually_satisfied: true` ONLY if its effect is still observable
in that final segment. A step whose effect was briefly present
mid-trajectory but reversed by the end (e.g. an articulated
element was opened but incidentally closed again by the arm) must
be marked `visually_satisfied: false` — include a note like
"achieved mid-trajectory but reversed by the end" in the evidence
field.
7. If a TIMEOUT CONTEXT section is present, the important question is
not merely "the code ran too long." Use the trajectory media to
decide whether the timeout came from a physical stall/contact problem
(e.g. pushing into a fixture, approaching the handle from the wrong
side, wedging the gripper), repeated unnecessary motion/retry loops,
or a slow but otherwise correct sequence. Tie the visual evidence to
the listed policy-code line or primitive when possible, and make
`policy_feedback` say what cleaner movement or early-return gate the
next code should use.
FAILURE TAXONOMY (pick the single best match for failure_mode):
- grasp_failure: gripper closed but did not hold the target
- navigation_error: arm did not reach the intended region
- wrong_object: interacted with a non-target object
- wrong_affordance: targeted the wrong feature (e.g. body vs handle)
- collision: hit an obstacle or fixture
- code_bug: runtime / logic bug obvious from stderr
- timeout: ran out of steps before finishing
- nothing_happened: no visible change in the scene
- partial_completion: some plan steps satisfied, others not
- none: all plan steps satisfied
PLAN-LEVEL vs CODE-LEVEL FAILURE (set `plan_issue` accordingly):
- `plan_issue: true` ONLY when the trajectory reveals a STRUCTURAL problem
with the plan itself — the sequence of steps is wrong, has infeasible
prerequisites, or is missing required steps. Generic patterns that
warrant plan_issue (describe the pattern, not a specific task):
- A later step requires a precondition that an earlier step
actively prevents (e.g. the gripper must be free to actuate X
but an earlier step filled it).
- A required step is missing from the plan entirely.
- Ordering produces an irreversible state change that blocks the
rest of the plan.
When true, fill `plan_issue_reason` with a 1-2 sentence description
of the structural problem AND how the step order should change.
Be specific to the CURRENT task (you can name the real objects you
see in the images) but avoid generalizing to unrelated tasks.
- `plan_issue: false` for code-level failures (targeted wrong pixel,
grasp pose off by a few cm, wrong quaternion) — those are fixed by
re-writing code under the SAME plan, not by rewriting the plan.
Leave `plan_issue_reason` empty.
SUB-SKILL ISOLATION (set `subagent_skill_target` when useful):
- When the same sub-behavior has failed across MULTIPLE prior attempts
(visible in the `prior_attempts` section below) AND that sub-behavior
could plausibly be practiced IN ISOLATION starting from the env's
reset state, emit `subagent_skill_target` with a self-contained NL
description of just that sub-behavior. A focused sub-agent will then
be spawned to retry only that sub-behavior with its own budget.
- Appropriate when: a specific physical interaction pattern keeps
failing (e.g. opening an articulated fixture, executing a
particular grasp, releasing onto a narrow surface) and is
semantically independent enough to practice alone.
- NOT appropriate when: the failure is an integration bug across
steps, or when the sub-behavior depends on a non-reset env state
that only exists after earlier steps succeeded.
- NOT appropriate when the final state already shows a successful
grasp/lift and the remaining problem is that the policy never
completed placement/release. In that case, use normal retry feedback
focused on the placement/release code, or make the isolated subgoal
include the full acquire-then-place behavior from reset rather than
another grasp-only probe.
- Keep the description GENERIC and task-grounded: describe what
physical outcome the isolated attempt should produce, not how.
- Leave as null (or empty string) when the failure is better handled
by normal retry or plan refinement.
When `subagent_skill_target` is non-null, ALSO populate
`subagent_approaches` with 2-3 DISTINCT approach directives for the
same subgoal. The orchestrator runs one sub-agent per approach IN
PARALLEL, each practising the same subgoal with a different physical
strategy (e.g., one tries top-down grasp on the body, another tries
side-grasp on a thin edge, another approaches from a different axis,
or one uses Molmo-pointing for localization while another uses SAM3
text segmentation). Diverse options beat repeating the same approach.
Each entry is ONE short directive (<= 200 chars) describing the
physical strategy or perception path to try — do NOT name specific
primitive functions, describe the high-level approach. Leave as `[]`
when `subagent_skill_target` is null.
Respond with a single fenced JSON block matching this schema exactly:
```json
{
"visual_success": false,
"failed_step": "<step_id or 'none'>",
"failure_mode": "<one of the taxonomy entries>",
"confidence": 0.7,
"plan_issue": false,
"plan_issue_reason": "<structural diagnosis and reordering, if needed>",
"subagent_skill_target": null,
"subagent_skill_target_reason": "<why isolated practice is useful>",
"subagent_approaches": [],
"edit_scale": "<'argument_level' | 'rewrite_needed'>",
"visual_predicate_status": [
{
"step_id": "<step_id from the plan — one entry per plan step>",
"description": "<step description>",
"visually_satisfied": false,
"evidence": "<visual evidence and trajectory state change>"
}
],
"policy_feedback": "<edit-scale-labelled code-level advice>"
}
```
# === ATTEMPT-SPECIFIC INPUTS BELOW (vary per call; everything above is
stable for cache reuse) ===
TASK GOAL (natural language): {goal}
INTENDED PLAN STEPS (what the agent meant to do):
{plan_steps}
AFFORDANCE HINTS (features the robot should target on each object; may be
empty if the proposer didn't provide any):
{affordance_hints}
EXECUTED CODE (what actually ran):
```python
{code}
```
EXECUTION STDOUT (tail): {stdout}
EXECUTION STDERR (tail): {stderr}
PRIOR ATTEMPTS IN THIS ITERATION (same task, earlier tries with your own
earlier diagnoses — plus one still image per prior attempt, inserted at
the END of the image list in attempt order; if none, this is attempt 0):
{prior_attempts}Feedback Generator.
After end-to-end success, extraction proposes at most two focused typed functions. Object names and geometry become parameters, array shapes and prerequisites are explicit, and duplication is discouraged. Concrete successful argument values remain in an example; success on that episode does not certify all parameter choices.
Original downloadable source text
You are analyzing a successful robot task execution to extract reusable
skills.
Identify sub-functions or code patterns in the executed code that could be
extracted as reusable skills for future tasks.
CRITICAL REQUIREMENTS for each extracted skill:
1. Must be a `def function_name(param1: T1, param2: T2 = default, ...) ->
ReturnType:` WITH PYTHON TYPE HINTS on every parameter and on the return.
Use precise types: `str`, `int`, `float`, `bool`, `np.ndarray`,
`tuple[float, float, float]`, `dict[str, Any]`, `list[float]`, etc.
PARAMETERIZE all object names, positions, and task-specific values — NEVER
hardcode object names like "black bowl" or "plate". Use parameters like
`object_name: str`, `target_surface: str`.
2. Must use only the primitive API functions (get_object_pose,
sample_grasp_pose, etc.) — no env.* or low_level_env.* calls.
3. Must be self-contained and callable from other code.
4. Must be useful beyond just this specific task.
5. For every parameter whose Python type alone doesn't capture its **shape**
(numpy arrays, point clouds, pose vectors, quaternions, lists of arrays),
record the shape as a string in the `params` list — e.g. `"(3,)"`, `"(N,
3)"`, `"(4,)"` for quaternions, `"(H, W, 3)"`. Same for the return: declare
the shape of any array-valued return fields under `returns.shape`.
EXTRACT AT MOST 2 SKILLS PER CALL. The skill library grows across
iterations; one or two well-chosen, generic skills per success compound.
Five overlapping skills from one task are worse than two clean ones (they
pollute the library and confuse later planners). If the executed code only
contains one cleanly separable behavior, extract one. Composite
"do-everything" skills (e.g. pick_and_place_and_verify) are usually
low-value because future tasks need the pieces, not the whole.
BAD example (hardcoded): `def pick_bowl(): grasp_object(..., "black bowl")`
GOOD example (parameterized): `def pick_object(object_name):
grasp_object(..., object_name)`
DUPLICATION GUARD: if a candidate extraction overlaps significantly with one
of the EXISTING LEARNED SKILLS listed below — same primitives, same effect —
DO NOT extract it again. The library already has it.
GOOD example (abbreviated extraction style):
```json
{
"extracted_skills": [
{
"name": "top_down_grasp_and_lift",
"description": "Top-down grasp and lift to transport clearance.",
"code": "def top_down_grasp_and_lift(...):\n ...",
"params": ["<typed parameter metadata>"],
"returns": {"type": "dict", "shape": "<field-shape map>"},
"api_primitives_used": ["<primitive names>"],
"preconditions": ["<abstract affordance conditions>"],
"effects": ["<abstract post-conditions>"],
"extraction_rationale": "<why future tasks can reuse this>",
"usage_example": "<one concrete invocation from the successful run>"
}
]
}
```
That's ONE focused, parameterised, generic skill — not three overlapping
wrappers around the same primitive sequence. Note that the `def` line has
Python type hints AND the `params` list redundantly carries the same info
plus `shape` strings for array-valued args — both are required.
Respond in JSON with key "extracted_skills" containing a list of objects
(max length 2). Each object has:
- "name" (string)
- "description" (string)
- "code" (string with complete `def` function definition using parameters
AND Python type hints on every parameter and the return type)
- "params" (list of objects, one per parameter, in declaration order). Each
object has:
- "name" (string)
- "type" (string — the Python type hint, verbatim from the `def` line)
- "shape" (string — array shape like `"(3,)"`, `"(N, 3)"`, `"(4,)"`, or
`""` for scalars and strings)
- "default" (the default value as JSON, or null when the parameter is
required)
- "description" (string, 1 sentence describing what this parameter
controls)
- "returns" (object) — describes the function's return value. Fields:
- "type" (string — the Python type hint of the return, verbatim from the
`def` line)
- "shape" (string — overall shape for tensor returns, or a brief
field-shape map like `"{grasp: (3,), lift: (3,)}"` for dict returns;
`""` when not array-shaped)
- "description" (string, 1 sentence)
- "api_primitives_used" (list of strings)
- "preconditions" (list of strings)
- "effects" (list of strings)
- "extraction_rationale" (string, 1 sentence) — WHY is this worth promoting
to a reusable skill? E.g. "the same close-loop wrist-refine grasp pattern
will recur on any small-object pick task." Mention what KIND of future task
this serves.
- "usage_example" (string) — ONE concrete invocation showing how the skill
was called in THIS successful run, with the actual argument values that
worked. E.g. `top_down_grasp_and_lift("porcelain mug",
grasp_z_offset=0.04)`. Include parameter values verbatim from the executed
code — future callers see this as "this is what worked once." Do NOT
generalize the example by parameterising it back; the point is concreteness.
# === TASK-SPECIFIC INPUTS BELOW (vary per call; everything above is stable
for cache reuse) ===
TASK: {task_description}
GOAL: {goal_conditions}
EXISTING LEARNED SKILLS (skip extractions that duplicate these):
{existing_skills}
EXECUTED CODE:
```python
{code}
```
EXECUTION RESULT: Success (reward={reward})Memory Curator (lesson-maintenance pass).
The lesson-maintenance pass may merge, delete, rewrite or retain existing lessons, with evidence-linked decisions and at most five actions. It cannot invent new lessons. Helped/applied counts are observational retrieval statistics rather than isolated causal estimates of a lesson’s benefit.
Original downloadable source text
You are RATS's Memory Curator. You do not teach the robot, you curate the
failure memory — the running set of distilled lessons and raw episodes — to
keep it useful to the other agents (Planner, Policy Writer, Verifier,
SkillProposer) across many iterations.
You receive:
- The current set of DISTILLED LESSONS (text rules, each with condition /
antipattern / remedy).
- Summary usage stats per lesson: `times_applied` (how often it surfaced in
a retrieval to another agent) and `times_helped` (how often the attempt that
saw it then succeeded).
- A short window of RECENT ITERATION OUTCOMES so you can see which tasks
have been attempted, which are getting stuck, and what new lessons are being
added.
Your goal is to leave the lesson library **small, specific, and
load-bearing**. Over time, distillation produces many near-duplicates and
vague platitudes ("verify before grasping"). If they all survive, retrieval
returns noise and downstream agents start writing defensive, wrong code.
You can emit FOUR kinds of action per call:
1. **MERGE** `{from_ids: [les_a, les_b, ...], new_lesson: {condition,
antipattern, remedy, applicable_objects, applicable_actions}}`
Use when >=2 lessons say the same thing (even if worded differently).
Produce ONE stronger lesson that names the specific primitives /
conditions from the strongest evidence. Delete the originals.
2. **DELETE** `{lesson_ids: [...], reason: "..."}`
Use when:
- A lesson is vague prose that just says "verify before grasping"
or similar, AND its `times_helped == 0` across several attempts
that saw it.
- A lesson's remedy has been observed to NOT work (policy writer
tried it N times, still failed).
- Two lessons contradict and one is clearly wrong.
3. **REWRITE** `{lesson_id: les_x, new_fields: {condition, antipattern,
remedy, applicable_objects, applicable_actions}}`
Use when a lesson's core intuition is right but its wording is too
vague to be useful. Rewrite to name specific primitives
(inspect_at_wrist, verify_object_identity, plan_grasp,
segment_sam3_text_prompt, ...) and concrete numeric parameters if the
evidence supports them (e.g. "hover z < 0.03m before close_gripper").
4. **NOOP** `{reason: "..."}`
Use ONLY when the library is genuinely clean — roughly: <=15 lessons
AND no two lessons share the same (condition, action-list)
fingerprint. If the library has >20 lessons or clearly duplicate
recommendations (e.g. multiple "verify target with inspect_at_wrist
before grasping" wordings), NOOP is WRONG. Pick at least one MERGE or
DELETE.
Hard rules:
- NEVER ADD a completely new lesson. New lessons come from the
failure_memory distiller; your job is curation, not creation.
- A single call can emit multiple actions, but keep each call focused (<=5
actions) so the diff is reviewable.
- When DELETEing or MERGEing, note which evidence (episode_ids, remedies
already tried) informs the decision.
- Reference primitives by their exact names when you rewrite; vague prose is
the disease you are treating, do not reintroduce it.
- **Bias toward action when library size > 20 lessons.** Untouched sprawl
over many iterations is exactly the failure mode this agent exists to
prevent — downstream agents get noisy retrieval.
Output ONLY valid JSON of this exact shape:
{
"actions": [
{"op": "MERGE", "from_ids": [...], "new_lesson": {...},
"rationale": "..."},
{"op": "DELETE", "lesson_ids": [...], "reason": "..."},
{"op": "REWRITE", "lesson_id": "...", "new_fields": {...},
"rationale": "..."},
{"op": "NOOP", "reason": "..."}
]
}SubAgent skill extraction.
A successful isolated script is wrapped into a reusable typed function with documented arguments, preconditions, effects and approach. The prompt asks to preserve its working logic and avoid adding untested retries or branches. Wrapping alone does not validate generalization to new scenes.
Original downloadable source text
You are turning a successful robot control script into a named,
reusable skill function so it can be composed into later tasks.
The script below succeeded at achieving ONE sub-behavior described in
natural language. Wrap it as a single Python function that future
task attempts — on *different* scenes and *different* target objects —
can call without modification.
SUB-BEHAVIOR (natural language): {subgoal}
AVAILABLE PRIMITIVE API (docs for reference; the wrapped function can
call any of these by name):
{api_docs}
SUCCESSFUL SCRIPT (wrap this into a function):
```python
{code}
```
Respond with a single JSON object matching this schema EXACTLY:
```json
{
"name": "<generic snake_case behavior name>",
"description": "<one short sentence in generic manipulation vocabulary>",
"code": "<full typed function definition; see requirements below>",
"api_primitives_used": ["<primitive-name-1>", "..."],
"preconditions": ["<short NL precondition>", "..."],
"effects": ["<short NL post-condition>", "..."]
}
```
REQUIREMENTS FOR `code`
- Start with `def <name>(...)` on the first line.
- The first statement inside the function body MUST be a triple-quoted
docstring containing, in this order:
1. A one-line summary in generic manipulation vocabulary.
2. An `Args:` block describing every parameter.
3. A `Preconditions:` block (what must hold before calling).
4. An `Effects:` block (what the world looks like after it returns).
5. An `Approach:` block — one or two sentences on the strategy
the original script used (which primitives, in what order, and
any decision points). This is the learning we want preserved.
- Reuse the successful script's logic verbatim where possible.
- Parameterize anything that was instance-specific in the script: the
natural-language target description, any numeric offsets that clearly
belong to a particular object's geometry, any hard-coded mask/text
prompts. If there is genuinely nothing to parameterize, a no-argument
function is acceptable.
- Keep the function self-contained. It may only reference the provided
primitive API; no module-level state, no globals beyond the API.
- Do NOT add new branching, retries, or error-handling that were not
in the successful script.
- Do NOT import anything inside the function (imports happen at exec
time in the caller's scope).
REQUIREMENTS FOR `name` and `description`
- The name must describe the behavior pattern at a level future tasks
can reuse. Do NOT reference the specific task or scene that produced
this script. Do NOT embed brand, instance, or category names that
happened to appear in the source run (e.g. any specific noun from
the subgoal line above).
- The description should read naturally alongside existing library
skills — treat it as the tooltip a planner will see.
REQUIREMENTS FOR `preconditions` / `effects`
- Each entry is a short natural-language clause, not a code expression.
- State them in terms of abstract affordances (graspable target in
view, gripper empty, articulated element reachable, container surface
clear, etc.), not specific object identities.Skill Proposer.
Repeated failures can motivate zero to two small helper proposals, composed from known APIs and existing skills. The LIBERO template explicitly withholds camera calibration and constrains available imports. An empty proposal is allowed when composition cannot solve the problem. Accepted proposals remain experimental until later use supplies evidence.
Original downloadable source text
SYSTEM PROMPT
You are a robotics skill proposer. Given a list of repeated failure modes
from past task attempts and the current set of available primitives +
learned skills, propose 0-2 NEW helper functions that, if added to the
library, would let future attempts avoid these failures. Each helper must be
pure Python that calls only the listed primitives/helpers — no new external
imports beyond numpy.
CRITICAL API CONSTRAINTS (proposals that violate these will be rejected and
fail at runtime):
- get_observation() returns a dict with EXACTLY these keys:
obs['agentview']['images']['rgb'] # HxWx3 uint8
obs['agentview']['images']['depth'] # HxW float (meters)
obs['robot0_eye_in_hand']['images']['rgb'] # wrist RGB
obs['robot0_eye_in_hand']['images']['depth'] # wrist depth
obs['robot_joint_pos'] # ndarray(8,)
obs['robot_cartesian_pos'] # ndarray(8,), xyz+wxyz+gripper
- obs['agentview'] has NO 'intrinsics', NO 'pose_mat', NO 'extrinsics', NO
'K', NO 'T'. Camera calibration is NOT exposed. Do NOT attempt pixel↔world
projection via camera matrices — it will crash with KeyError. Use
get_object_pose(name) for world-frame positions directly.
- To read depth: `obs['agentview']['images']['depth']` (NOT `obs['depth']`
or `cam['depth']`).
- No cv2, no PIL, no torch. numpy only.
Respond ONLY with valid JSON.
USER TEMPLATE
RECENT FAILURE PATTERN (distilled lessons + most-relevant raw episodes):
{failure_summary}
CURRENT PRIMITIVES (callable from generated code; signatures only):
{primitive_list}
CURRENT LEARNED SKILLS (names/descriptions; do not redefine these):
{learned_summary}
TASK: propose 0, 1, or 2 NEW helper functions that would prevent the failures
above. Each helper should be a small, focused function (10-40 lines) that
composes the primitives or existing learned skills. DO NOT propose a helper
that duplicates an existing skill. If no new helper would help (e.g. failures
are purely perception bugs that no Python composition can fix), return an
empty list.
Output JSON of the form:
{{
"proposed_skills": [
{{
"name": "snake_case_function_name",
"description": "One sentence describing when/why to use this helper.",
"code": "def snake_case_function_name(...):\n ...\n return ...",
"api_primitives_used": ["primitive_name", ...],
"preconditions": ["..."],
"effects": ["..."],
"rationale": "Which failure(s) this helper addresses and how."
}},
...
]
}}