Literature
Display
← All papers

GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks

Kaiyuan Chen · Shuangyu Xie · Letian Fu · Justin Yu · William Pacini · Sandeep Bajamahal · Hudson Kim · Jaimyn Drake · Daehwa Kim · Haoru Xue · Jonathan Francis · Christian Juette · Peter Schaldenbrand · Muhammet Yunus Seker · Ruwan Wickramarachchi · Uksang Yoo · Guanzhi Wang · Adithyavairavan Murali · Balakumar Sundaralingam · S. Shankar Sastry · Spencer Huang · Yuke Zhu · Linxi “Jim” Fan · Ken Goldberg

Preprint posted · CoRL 2026 (accepted) ↗

Robotics15 min text · CondensedOriginal paper ↗
Contents

Overview

Agentic coding for repeated manipulation in a known workcell: typed skill graphs are generated, tested in simulation, refined, and deployed through a graph interpreter. The complete overview figure is retained with its caption.

Original overview: language and geometry inputs feed graph generation, skill-aware segments, simulation rehearsal, optimized graph and runtime executor; eight benchmark examples below.
Figure 1: GaP system architecture. Given a “Variational Automation (VA)” task specification, GaP uses a multi-agent harness for coding tools such as Claude [1] and Gemini [2] to automatically generate computation graphs that include “skill” nodes from a Modular Open Robot Skill Library (MORSL), which includes model-based procedures (e.g. ROS [3]) and model-free policies (e.g. GraspGen [4]). GaP then orchestrates self-learning using simulation (e.g. RoboSimStudio [5]) to iteratively refine the graph, which can then be interpreted without the agents on an edge device for persistent execution over time. Bottom: 8 Variational Automation Benchmark Tasks (4 Sim and 4 Real).

At a glance

GaP targets Variational Automation: repeated tasks with bounded variation in object geometry and pose, between fixed automation and unconstrained generalist robotics. A multi-agent harness composes perception, planning and control skills into a directed graph; parallel simulated rehearsals guide graph and parameter edits. Eight task benchmarks cover four simulated and four physical settings. Results favor GaP on the tested variations, but inputs, skill libraries and metrics differ across comparisons. Industrial reliability and broader dynamic manipulation remain unresolved.

1 Introduction

The central proposal is a structured coding harness, not a new end-to-end robot policy. An orchestrator decomposes the task, specialized agents build typed skill subgraphs, and separate checks evaluate the resulting program. Simulation feedback then optimizes reliability and throughput before deployment. Graph structure localizes context and permits replacing individual skills, including learned policies. Unlike a generalist household robot, the intended system repeatedly executes a specified task within a known workcell and variation envelope. The paper draws on ROS, task-and-motion planning, and CaP-X.

2 Related Work

Fixed automation repeats essentially identical instances; generalist robotics varies both tasks and environments; VA holds the task/workcell relatively fixed while objects and poses vary. Related work supplies graph execution, behavior trees, task-and-motion planning, code generation, learned vision/action models and agentic refinement. GaP combines these components through a typed, modular harness. Type checking and rehearsal improve program construction but do not establish a formal safety guarantee or field reliability over years of operation.

3 Problem Formulation

The problem specifies a bounded family of manipulation instances, an executable skill graph and a combined reliability/throughput objective. Its privileged setup information is important when interpreting baseline comparisons.

Assumptions

The robot, workcell and calibrated sensing are known. Object geometries/meshes and permitted pose distributions are provided; the simulator is instantiated from these inputs. The reported pipeline does not infer a complete unknown scene, identify physical parameters or perform domain randomization. Perception nodes still process rendered/real observations, while simulator state and contacts support evaluation.

Variational Automation Task Class

A task tuple contains language, environment, robot, object geometries, state/configuration space, a belief distribution over varying initial conditions and an objective. Sampling from the belief produces repeated instances of the same task. This formal scope is narrower than arbitrary objects, unseen workcells and unrestricted task changes.

Policy Representation via Directed Execution Graphs

A directed execution graph combines typed data dependencies with control flow. Nodes invoke skills; edges route outputs and success/failure outcomes. Independent branches can run in parallel, and subgraphs permit reusable higher-level behaviors and recovery. Learned perception, grasp models or VLA policies can be called inside this structured program.

Problem Definition

Optimize the graph for expected performance across sampled instances, balancing binary task success and throughput measured through successful cycles per time. This objective depends on the task’s success definition: the packing benchmark separately reports item yield, so its rates should not be read as probabilities of completing all six items.

4 Graph-as-Policy

The orchestrator chooses segments and their interfaces, skill agents fill the subgraphs, and a distinct checkpoint role evaluates postconditions. Static validation precedes simulated rollouts; feedback drives edits to prompts, connections, scripts and skill choices. Deployment runs the resulting graph without a coding agent in the loop, although graph nodes may still invoke learned vision/language models. The extensive original prompts show that this automation relies on detailed, hand-specified interface and skill knowledge.

4.1 Modular Open Robot Skill Library (MORSL)

MORSL wraps skills with names, descriptions, typed inputs/outputs, parameters and preconditions. Components cover detection/segmentation, grasp generation, collision-aware motion, geometry utilities, ROS and policy execution. The main text reports 51 initial skills (15 perception, 5 grasp, 8 motion, 15 geometry and 8 other). Appendix D’s finer catalog actually contains 12 composite/atomic cards and 73 primitive-method cards; these inventories should not be silently treated as the same count. All original cards are retained. GraspGen is one learned grasp source alongside geometric alternatives.

4.2 Self-Learning through Internal Simulation Rehearsal

Instantiate the supplied scene, sample parallel initial states, execute the current graph and inspect before/after states and contacts. Agents can change declared inputs, rewire connections, rewrite inline arithmetic/formatting scripts or swap skills, then rerun static and simulated checks. Proposed changes that reduce success or worsen cycle time on 100 fresh instances are rejected. Iteration ends at a plateau or a reported failure. Rehearsal supplies privileged physical feedback; it is not additional real-robot imitation data, and passing sampled checks does not guarantee all future instances.

Algorithm 1 Rehearsal-based Graph Optimization Input: 𝒯 = ⟨ ℒ , ℰ , ℛ , 𝒪 , 𝒳 , ℬ , 𝒥 ⟩ \mathcal{T}=\langle\mathcal{L},\mathcal{E},\mathcal{R},\mathcal{O},\mathcal{X},\mathcal{B},\mathcal{J}\rangle , Iterations M M , Parallel Rollouts N N Output: Optimized Task Graph 𝒢 ∗ \mathcal{G}^{*} Initialization: 𝒢 0 ← Graph ​ _ ​ Init ​ ( ℒ ) \mathcal{G}_{0}\leftarrow\mathrm{Graph\_Init}(\mathcal{L}) // Initial graph from language 𝒮 ← Build ​ _ ​ Scene ​ ( 𝒢 , { 𝒢 i } ) \mathcal{S}\leftarrow\mathrm{Build\_Scene}(\mathcal{G},\{\mathcal{G}_{i}\}) // Instantiate sim environment for j ← 1 j\leftarrow 1 to M M do // Step 1: Scene Variational Sampling { s ^ i } i = 1 N ∼ ℬ \{\hat{s}_{i}\}_{i=1}^{N}\sim\mathcal{B} 3 // Sample N N instances // Step 2: Parallel Rehearsal 4 for i ← 1 i\leftarrow 1 to N N do in parallel 7 5 τ i ← Rehearsal ⁡ ( s ^ i , 𝒢 j − 1 ) \tau_{i}\leftarrow\mathrm{Rehearsal}(\hat{s}_{i},\mathcal{G}_{j-1}) // Rollout policy F i ← Analyze _ ​ Failure ​ ( τ i , 𝒢 , { 𝒢 i } ) F_{i}\leftarrow\mathrm{Analyze_{\_}Failure}(\tau_{i},\mathcal{G},\{\mathcal{G}_{i}\}) 6 9 // Step 3: Graph Refinement 𝒢 j ← Graph ​ _ ​ Update ​ ( { F 1 , … , F N } ) \mathcal{G}_{j}\leftarrow\mathrm{Graph\_Update}(\{F_{1},\dots,F_{N}\}) 8 // Optimize via LLM 10 return 𝒢 ∗ ← 𝒢 M \mathcal{G}^{*}\leftarrow\mathcal{G}_{M}

5 Experiments

The evaluation spans grocery fulfillment, six-item packing and popcorn preparation in both simulation and reality, plus physical USB-C insertion and simulated crate washing. These are eight benchmark settings across five task families. Comparisons use different baseline families and task-specific outcome definitions.

5.1 Simulation and Real-World Variational Automation Benchmarks

Grocery variations move objects within a 20×20 cm region, swap baskets, permute target/distractor positions or combine changes. Initial layouts are generated without object overlap. Packing attempts six items and scores the fraction placed, rather than all-or-nothing six-item completion. Popcorn requires manipulating a knob, grasping the pan handle, placing it on a burner, waiting and finishing the sequence. Cable insertion varies seven ports, geometry and instruction order; crate washing uses two Franka arms with bounded crate pose changes.

5.2 Baselines

Baselines include CaP-X, π0.5, MolmoAct2 and TiPToP, plus VLA-in-GaP hybrids. VLA checkpoints are LIBERO-tuned; TiPToP uses 128 cuTAMP particles, a 60-second timeout and a warm cache. GaP and CaP-X use Gemini 3.1 Flash Lite at temperature 0.1, but CaP-X receives only language and an initial image while GaP also receives geometry and a known variation envelope. That input asymmetry limits causal conclusions about the harness alone.

5.3 Benchmarks I and II: Fulfill Grocery Orders and Pack Grocery Items

Initial graphs already achieve high grocery/packing scores, so the main reliability comparison for these tasks does not use self-learning. Separate throughput experiments subsequently refine their cycle times.

Table 1: Success Rates for 5500 trials, 100 task instances per table cell. Results on Benchmarks I-a and II-a: Fulfill Grocery Orders and Pack Grocery Items. The first two columns are the original LIBERO [36] with negligible pose variation, and LIBERO-PRO with slightly larger pose variation [71] . We introduce the VA Grocery Order Fulfillment benchmark that includes larger object positional variance (X-Y 20×\times20 c​m2cm^{2}), swapping target and basket positions (basket_swap), permuting all item locations, and everything combined (mixed_all). We also introduce the Grocery Packing benchmark that requires picking all objects and placing them in the basket. In rows 5 and 6, GaP navigates the robot wrist camera above the target object and then switches to MolmoAct2 and π0.5\pi_{0.5} VLA policies.
LIBERO LIBERO-Pro Fulfill Grocery Orders Pack Grocery Items
Method object object_swap X-Y 20×\times20 c​m2cm^{2} basket_swap permutation mixed_all fixed varied
CaP-X - 0.22 0.07 0.05 0.11 0.10 0.01 0.01
π0.5\pi_{0.5} 0.96 0.24 0.78 0.15 0.20 0.20 0.17 0.18
MolmoAct2 0.97 0.43 0.90 0.26 0.10 0.20 0.18 0.18
TipTop 0.22 0.22 0.29 0.24 0.31 0.24 0.34 0.46
π0.5\pi_{0.5} w/GaP 0.85 0.60 0.79 0.32 0.50 0.39 0.67 0.66
MolmoAct2 w/GaP 0.70 0.62 0.84 0.58 0.39 0.66 0.59 0.59
GaP 0.95 0.95 0.95 0.97 0.93 0.97 0.99 0.98

5.3.1 Simulation Results

Table 1 reports 100 instances per cell and 5,500 total. GaP scores 0.93–0.97 across grocery variants and 0.99/0.98 packing item yield. On unperturbed LIBERO-object tasks, π0.5 and MolmoAct2 score 0.96/0.97 versus GaP’s 0.95; GaP’s advantage is primarily under the tested variations. Wrapping VLAs in GaP improves many variation results, but not every condition and not uniformly by a factor of two. CaP-X’s low scores are confounded by its more limited scene inputs. Table 1 retains all individual conditions.

Table 2: Physical Benchmark on Benchmarks I-b, II-b, and III-b. We report success rate and completion time. Sub-tasks (→\rightarrow) are reported per benchmark. On average, the completion time of single pick-and-place for TipTop is 95 seconds, and GaP is 67 seconds.
Success Rate TipTop GaP
Fulfill Grocery Orders 8/25 25/25
   →\rightarrow object 3/5 5/5
   →\rightarrow 20x20 2/5 5/5
   →\rightarrow basket swap 1/5 5/5
   →\rightarrow tall basket 0/5 5/5
   →\rightarrow permutation 2/5 5/5
Pack Grocery Items 10/30 28/30
Make Popcorn 0/20 18/20

5.3.2 Real Physical Experiment Results

Across five physical grocery conditions with five trials each, GaP succeeds 25/25 versus TiPToP’s 8/25. Packing yields 28/30 versus 10/30 placed items; those denominators are item attempts, not 30 complete six-item episodes. The stated grocery breakdown is 14.1 seconds for perception and 36.4 for movement, about 50.5 seconds total; the table separately summarizes average pick/drop times of 67 versus 95 seconds. Baseline failures include perception/grasp and infeasible motion plans, so the comparison reflects full system components.

5.4 Ablation Studies

Graphless generation and a single all-in-one agent both score zero in the reported ablations, failing service/schema or static graph checks. The graphless variant must recall low-level APIs from memory, so these results also test interface support and prompt/context organization. Removing validation permits malformed initial outputs. They establish the importance of this harness’s decomposition and checking in the tested configuration, not a universal impossibility result for single-agent or free-form code policies.

5.5 Benchmark III a, b: Make Popcorn in sim and real

Popcorn rehearsal improves simulated success from 0.33 to 0.94. Early changes replace unstable learned grasp candidates with a hybrid geometric approach; later changes revise handle localization and placement offsets. The physical system succeeds 18/20, with residual IK and calibration-related grasp failures, versus 0/20 for TiPToP. The figure’s failure sequence illustrates specific edits rather than proving open-ended autonomous skill discovery.

Refer to caption
Figure 2: Self-Learning for Making Popcorn Benchmark. Pan pose variations (left) drive a 10-iteration sequence of graph updates; (blue, left axis) iteration phases shaded by class of edit.

5.6 Self-Learning to Reduce Cycle Time

Accepted edits simplify perception, tune motion settling and reuse graph outputs; the paper reports up to 57% lower cycle time without an observed success-rate penalty on fresh simulated checks. Table 3 gives learned throughput of 124.1, 16.2 and 76.6 cycles/hour for grocery, packing and popcorn, from 49.7, 7.3 and 30.5 initially. Success stays 0.97 for the first two and rises 0.33→0.94 for popcorn. Adding MORSL to TiPToP improves grocery/packing from 0.24/0.34 to 0.53/0.60 but leaves popcorn at zero, indicating both skill quality and orchestration matter. Generation uses fewer than 30 LLM calls; simulation dominates refinement at about 15 minutes/iteration and up to 150 minutes for popcorn. The brief dollar/latency claim is not a comprehensive deployment-cost analysis.

Original figure
Figure 3: Self-Learning for Cycle Time on Benchmarks I–III. Per-iteration cycle time (left axis) and success rate (right axis). Accepted graph updates are colored by edit class: perception (blue), motion parameters (green), and graph topology (purple); updates that degrade success rate or cycle time on 100 fresh task instances are rejected.
Table 3: Success rate and throughput before and after self-learning on Benchmarks I–III (simulation). TiPToP w/ MORSL replaces TiPToP’s perception and grasp planning with MORSL skills.
Success Rate ↑\uparrow Throughput (cycles/hr) ↑\uparrow
Benchmark TiPToP TiPToP w/ MORSL GaP (Initial) GaP (Learned) GaP (Initial) GaP (Learned)
I. Fulfill Grocery Orders 0.24 0.53 0.97 0.97 49.7 124.1
II. Pack Grocery Items 0.34 0.60 0.97 0.97 7.3 16.2
III. Make Popcorn 0.00 0.00 0.33 0.94 30.5 76.6

5.7 Benchmark IV: Insert USB-C Cables (Real)

A UR5 with wrist-mounted ZED Mini inserts a cable into seven USB-C ports. Five instruction families request an individual port, ascending/descending order, or odd/even ports; pose tests add ±15° and ±5 cm changes. The aggregate is 121/130 successful insertion trials, but complete ascending sequences succeed only 2/5 and descending sequences 4/5, illustrating compounding long-horizon errors. Force-guided search covers 1×1 cm in 2 mm increments; insertion requires sufficient depth under a force limit, and extraction uses lateral oscillation. Failures arise from localization outside the search grid, inadequate insertion depth and lighting/shadows. Table 4 separates per-insertion and complete-sequence rates and marks trials excluding extraction.

Refer to caption
Figure 4: Seven-ports cable insertion setup.

Variation SR (Per trial) SR (All) ET (All)
Insertion Goal Condition
Individual port∗ 5/5 - 26.0
Ascending order 32/35 2/5 247.0
Descending order 32/35 4/5 223.2
Odd index port 20/20 5/5 136.8
Even index port 14/15 4/5 104.6
Position/Orientation
-15∘15^{\circ}∗ 4/5 - 24.0
15∘15^{\circ}∗ 5/5 - 25.6
-55 cm∗ 5/5 - 29.4
55 cm∗ 4/5 - 28.0
Table 4: USB-C Cable insertion variations and results over 130 total insertion trials. Success rates (SR) and execution time in seconds (ET). ∗Excludes extraction phase.
Refer to caption
Figure 5: Left: ROS computation graph that is hand-engineered using traditional ROS nodes and topics. Middle: GaP can leverage nodes in ROS and generate a compatible graph using atomic skill sets, align to port, touch port, insert, and extract. Right: GaP can also use these nodes inside a subgraph for a long-horizon task, such as inserting the cable into even-numbered ports.

5.8 Benchmark V: Wash Crates (Simulation)

The simulated two-arm washing task varies crate yaw by ±15° and position by ±2.5 cm. GaP achieves 143/150 successes versus 148/150 for a manually engineered graph, with mean cycle times 179.13 and 176.47 seconds. Over three hours each attempts 59 cycles and completes 55 versus 58, approximately 18.33 versus 19.33 successful cycles/hour; the table rounds these values. This supports near-manual performance in the tested simulated setting, not superior reliability or a physical industrial deployment. The crate image is Figure 6 in HTML but labeled “Table 6” in the PDF; this numbering discrepancy is retained in the review record.

Table 5: Wash Crates benchmark, 150 trials. Success rate (SR), average cycle duration (s), and sustained throughput (successes/hr) over a 3-hour continuous run.
Method SR Avg(Dur) Throughput
Hand-Engineered 0.99 176 19
GaP 0.95 179 18
Refer to caption
Figure 6: Industrial crate-washing setup.

6 Conclusion

Typed graphs and simulation-guided edits make robot programs more modular and inspectable, and work well on the supplied bounded task variations. The authors explicitly state that reliability and throughput still fall short of industrial needs. Their illustrative 500-units/hour target corresponds to about 7.2 seconds per unit, not a universal specification. The tasks are mostly quasi-static; only cable insertion uses force sensing. Deformables, dynamic interactions, moving targets and stronger long-duration reliability remain future work.

Appendix

The appendices retain additional related work, ROS integration, complete generated graphs, the skill catalog, rehearsal feedback and full original agent prompts/output code.

Contents

The original appendix structure is preserved through the reader’s section navigation; each technical appendix below has edited notes and its complete original visual or code material.

Appendix A More Related Work

Additional context contrasts variable-instance automation with generalist robotics and relates structured code generation to prior graph/task construction methods such as BrickGPT. These connections motivate modularity; they do not broaden the actual experiments beyond the known workcells and bounded variations used here.

Appendix B Integrating ROS with GaP in Cable Insertion Benchmark

ROS integration exposes four insertion primitives: align, touch, insert and extract. GaP wires these into compatible computation graphs and longer instruction sequences, communicating through temporary nodes and completion/status signals. The side-by-side figure compares a hand-built ROS graph with generated single- and multi-port graphs; it is a qualitative structural comparison, not an additional controlled success-rate baseline. The appendix records all five natural-language instruction patterns and the same vision/contact limitations as the main evaluation.

Refer to caption
Figure B.1: Left: ROS computation graph that is hand-engineered using traditional ROS nodes and topics. Middle: GaP can leverage nodes in ROS and generate a compatible graph using atomic skill sets, align to port, touch port, insert, and extract. Right: GaP can also use these nodes inside a subgraph for a long-horizon task, such as inserting the cable into even-numbered ports.

Appendix C Sample Generated Graphs

Five complete generated graphs show the actual skill calls, scripts, subgraphs and success/failure routes. Their detailed interfaces reveal how task structure and reusable primitives combine.

C.1 Fulfill Grocery Orders

Grocery fulfillment perceives target and container, filters their bounding boxes, grasps through a planned approach, transports and releases. Success routes to completion; perception, grasp and motion failures route to abort. The full graph preserves the service/script labels and control/data distinctions.

Original figure

C.2 Pack Grocery Items

Packing uses the same perception–grasp–transport structure with task-specific target/container bindings and placement parameters. The shown graph is one generated workflow; benchmark yield counts the six item attempts separately.

Original figure

C.3 Fulfill Grocery Orders with VLA Policy

The VLA hybrid opens the gripper, localizes the target, moves above it and delegates closed-loop execution to a learned policy node. The surrounding graph provides structured setup and failure handling, which helps explain why hybrid results differ from raw VLA inference.

Original figure

C.4 Wash Crates

Crate washing combines crate-mask/OBB perception with two-arm approach, visual servoing, alignment and a fixed sequence of grasp, reorient, lift, flip and extension actions. The full graph shows failure routing at each stage; some motion structure and waypoints reside in supplied primitives.

Original figure

C.5 Make Popcorn

Popcorn composes repeated perception, geometric grasp, wrist/knob control, pan transport, waiting and final removal/shutdown stages. Its long graph shows why localizing edits to subgraphs matters, while preserving the task-specific mechanical sequencing supplied by the skill system.

Original figure

Appendix D MORSL Library

Every skill uses a common name, description and typed interface. Higher-level composite graphs call atomic skills, which ultimately use primitive gRPC methods. The complete appendix contains 85 skill cards: 12 composite/atomic and 73 primitive entries. Common types include poses, oriented boxes, clouds, masks, images, trajectories and joint states; listed defaults and omitted universal call fields remain in the original cards.

D.1 Composite and atomic skills

Twelve higher-level skills cover perception, tracking, alternative grasps, placement, learned-policy execution and a fixed-phase bimanual lift. Their explicit input/output contracts determine graph composability.

Perception.

The five perception/tracking cards offer single-path detection with a VLM crop tournament, multi-detector arbitration, set-of-marks selection, parent/subpart localization and streaming tracking. Depth fusion yields world-frame boxes/clouds. A source inconsistency remains: this catalog describes perception_multi as three detectors plus arbitration, whereas the appended v3 prompt describes a Molmo-only single path. The edition preserves both descriptions rather than choosing an undocumented implementation.

perception_single Description. Fast single-path detection: broad detect + a VLM crop tournament + segmentation + depth-to-3D fusion, closed by an OBB fit. Best for uncluttered scenes with visually distinct targets. Inputs. object_name (literal) Outputs. <name>_obb: OrientedBoundingBox, <name>_mask: Mask, <name>_cloud: PointCloud perception_multi Description. Three detectors run on the same observation and a VLM disambiguator picks the best mask on disagreement. Most robust, but slower; single-shot only. Inputs. object_name (literal) Outputs. <name>_obb: OrientedBoundingBox, <name>_mask: Mask perception_any Description. Lightweight one-shot set-of-marks pick. Returns “not found” cleanly when the VLM replies “none”, the right choice for clean-all-items loops. Inputs. object_name (literal) Outputs. <name>_obb: OrientedBoundingBox, <name>_mask: Mask, <name>_cloud: PointCloud perception_subpart Description. Hierarchical perception: detect the parent, crop to its box, segment the named subpart inside the crop, uncrop and fuse depth. For affordances that occupy few full-image pixels (pan handle, drawer pull, mug rim). Inputs. parent_prompt, subpart_prompt (literals) Outputs. <name>_obb: OrientedBoundingBox, <name>_mask: Mask, <name>_cloud: PointCloud track_object Description. Long-running tracker: seeds on the first frame via text prompt, then polls the observation stream and advances the tracker until the workflow signals termination, freeing tracker state on exit. Inputs. text prompt; observation stream Outputs. a stream of tracked-mask snapshots

Grasping.

Four grasp skills trade collision-aware OBB candidates, a GraspGen/geometric mixture, a geometry-locked short-axis grasp and direct IK descent. Their assumptions differ: direct IK omits collision avoidance, and short-axis grasping encodes geometric priors for elongated objects. These are selectable engineering components, not interchangeable guarantees of safe execution.

grasp_curobo_obb Description. Collision-aware grasping: plans a collision-free trajectory to one of several top-down OBB grasp candidates. The default when the OBB centroid is graspable. Inputs. target_obb: OrientedBoundingBox, target_mask: Mask Outputs. ee_pose_at_grasp: Se3Pose, grasp_pose: Se3Pose grasp_moe Description. Mixture-of-experts grasping: a diffusion sampler and a dense OBB sweep feed one discriminator; the top-K union is planned. Use when the OBB centroid is not the graspable point (bowl rim, handle, off-center). Inputs. target_obb: OrientedBoundingBox, target_mask: Mask, target_cloud: PointCloud Outputs. ee_pose_at_grasp: Se3Pose grasp_short_axis Description. Deterministic geometry-locked grasp: the finger axis snaps to the OBB’s shorter horizontal axis so the jaws close across the narrow dimension of an elongated target. An optional base OBB slides the grasp along a handle. Inputs. target_obb: OrientedBoundingBox, target_mask: Mask, base_obb: OrientedBoundingBox (optional) Outputs. ee_pose_at_grasp: Se3Pose, grasp_pose: Se3Pose grasp_direct_ik Description. Planner-free grasping: pre-rotate to the grasp orientation above the target, descend straight down, close. For uncluttered scenes or platforms without a motion planner. Inputs. target_obb: OrientedBoundingBox Outputs. (grasp completion only)

Transport, policy, and bimanual.

Transport computes a release pose, optionally grounding a named container subregion. run_policy executes learned actions in windows until a VLM termination decision or a window limit. The bimanual lift follows baked waypoints through open, approach, insert, close, lift and settle; its interface has no variable inputs. All three complete contracts are retained.

transport_to_drop Description. Moves a held object above a destination container and releases it. An optional VLM-grounded step localizes a natural-language sub-region (“the left compartment”) before the drop pose is computed. Inputs. container_obb, container_mask, target_obb, target_mask, ee_pose_at_grasp Outputs. (placement completion only) run_policy Description. Runs a learned VLA policy in closed loop, reading the observation stream each window, until a VLM termination check fires or max_windows is reached. Inputs. observation stream; max_windows Outputs. (policy rollout completion) bimanual_crate_lift Description. Two-arm crate lift: both arms side-grasp the crate’s handle bars and raise it in lock-step, in a fixed phase order (open →\to approach →\to insert →\to close →\to lift →\to settle). Inputs. (none — waypoints are baked parameters) Outputs. done: bool, final_left_world_pos, final_right_world_pos

D.2 Primitive skills (one per gRPC method)

Seventy-three primitive cards expose individual RPC methods: 11 detection/segmentation, 5 vision-language, 5 grasp generation, 12 motion/IK, 15 geometry and 25 robot/environment controls. These method-level entries explain why appendix totals differ from the main text’s initial-skill count.

Detection and segmentation.

Detection includes Grounding DINO and OWL-ViT, SAM2/3 box/point/text segmentation, stateful tracker initialization/update/cleanup and SAM3D mesh generation. Mesh reconstruction is available in the library, but the reported VA setup still assumes supplied object geometry; catalog capability is not evidence that every experiment reconstructs its environment.

grounding_dino.Detect Description. Zero-shot object detection from a text prompt; returns boxes, labels, and confidence scores. Inputs. image: Image; text_prompt (period-separated phrases); box_threshold (0.20); text_threshold (0.20) Outputs. detections: [ box: BoundingBox2D, label, score ] owlvit.Detect Description. Open-vocabulary object detection from a list of text queries; an alternative to grounding_dino. Inputs. image: Image; text_queries: string[]; score_threshold (0.05) Outputs. detections: [ box: BoundingBox2D, label, score ] sam3.SegmentText Description. Segment all instances matching a text description. Inputs. image: Image; text_prompt Outputs. masks: Mask[], scores: float[], boxes: BoundingBox2D[] sam3.SegmentPoint Description. Segment the object under a single pixel coordinate. Inputs. image: Image; pixel_x, pixel_y Outputs. masks: Mask[], scores: float[] sam3.SegmentBox Description. Segment within a bounding box, optionally refined by a foreground point. Inputs. image: Image; box: BoundingBox2D; pixel_x/pixel_y + use_point (false, optional foreground point) Outputs. masks: Mask[], scores: float[] sam2.SegmentBox Description. Box-prompted segmentation (lighter SAM2 backend). Inputs. image: Image; box: BoundingBox2D; max_masks (0 = unlimited) Outputs. masks: Mask[], scores: float[] sam2.SegmentPoint Description. Point-prompted segmentation (lighter SAM2 backend). Inputs. image: Image; pixel_x, pixel_y Outputs. masks: Mask[], scores: float[] sam3_tracker.InitTracker Description. Initialize a stateful tracker session seeded with one frame and a prompt. Inputs. image: Image (first frame); one of text / box / point (pixel_x,pixel_y,use_point); object_name Outputs. tracker_id, initial_mask: Mask, initial_box, score, object_present sam3_tracker.UpdateTracker Description. Advance an existing tracker session by one frame. Inputs. tracker_id; image: Image (new frame) Outputs. mask: Mask, box, confidence, object_present sam3_tracker.CloseTracker Description. Free a tracker session; idempotent. Inputs. tracker_id Outputs. (empty) sam3d.GenerateMesh Description. Reconstruct a triangle mesh from an RGB image + mask, scaled and posed into world frame via the OBB; writes an .obj for the rehearsal sandbox. Inputs. object_name; views: [rgb, mask, intrinsics, camera_pose]; obb: OrientedBoundingBox; view_selection; seed (42); output_dir Outputs. mesh_path, texture_path, world_pose: Se3Pose, scale: Vec3, chosen_view_index, num_vertices, num_faces

Vision–language.

Molmo supports point grounding and free-form or yes/no questions; a configurable VLM backend provides the two question interfaces. Outputs are pixels, text or booleans, so downstream code must map them into the graph’s typed geometric contracts.

molmo.PointPrompt Description. Point to a named object; returns pixel coordinates. Inputs. image: Image; object_name Outputs. pixel_x, pixel_y, found molmo.Query Description. Free-form visual question answering (Molmo backend). Inputs. prompt; image: Image (optional); use_multiview Outputs. text molmo.QueryYesNo Description. Yes/no visual question answering (Molmo backend). Inputs. prompt; image: Image (optional); use_multiview Outputs. answer: bool vlm.Query Description. Free-form visual question answering (configurable VLM backend). Inputs. prompt; image: Image (optional) Outputs. text vlm.QueryYesNo Description. Yes/no visual question answering (configurable VLM backend). Inputs. prompt; image: Image (optional) Outputs. answer: bool

Grasp generation.

GraspGen samples and scores six-degree-of-freedom candidates; Contact-GraspNet accepts depth or point clouds; M2T2 uses colored scene clouds. The cards retain candidate counts, thresholds, coordinate conventions and optional scene context. Candidate generation and feasibility filtering are distinct stages.

graspgen.PlanFromPointCloud Description. Diffusion-based 6-DoF grasp sampling from an object-segmented cloud; ranked, world-frame, fingertip convention. Inputs. target_cloud: PointCloud; scene_cloud (optional); num_grasps (200); topk_num_grasps (50); grasp_threshold (−1-1); remove_outliers (true) Outputs. grasp_poses: Se3Pose[], scores: float[], contact_points: Vec3[] graspgen.ScoreGrasps Description. Score caller-provided grasps with GraspGen’s discriminator, so OBB-sampled candidates are comparable with diffusion samples. Inputs. target_cloud: PointCloud; candidate_poses: Se3Pose[] Outputs. scores: float[] (aligned to candidates) graspnet.PlanFromDepth Description. Contact-GraspNet 6-DoF grasp detection from a depth image + instance mask. Inputs. depth: DepthImage; intrinsics: Matrix3x3; segmentation: Mask; instance_id; knobs (z_min 0.2, z_max 2.0, max_retries 7, …\dots); optional wrist camera Outputs. grasp_poses: Se3Pose[], scores: float[], contact_points: Vec3[] graspnet.PlanFromPointCloud Description. Contact-GraspNet grasp detection from precomputed clouds. Inputs. full_cloud: PointCloud; segment_cloud: PointCloud; instance_id; same knobs Outputs. grasp_poses: Se3Pose[], scores: float[], contact_points: Vec3[] m2t2.PlanFromPointCloud Description. M2T2 6-DoF grasp generation from a colored scene cloud; an alternative candidate source. Inputs. scene_cloud: PointCloud; target_cloud (optional); num_points (16384); grasp_threshold (0.035); max_grasps (200) Outputs. grasp_poses: Se3Pose[], scores: float[], contact_points: Vec3[]

Motion planning and IK.

cuRobo provides goalset, held-object, linear and directed-linear planning; phased grasp motion; IK; feasibility and trajectory collision checks. PyRoKI supplies CPU IK and Cartesian interpolation. Pure IK and unchecked linear moves do not imply collision-free motion. The full cards distinguish return success flags, trajectories, selected grasp indices and failure reasons.

curobo.PlanToGraspPoses Description. Plan a collision-free trajectory to one of several grasp poses (goalset). Inputs. world_config: WorldConfig; start_joint_position: JointState; grasp_poses: Se3Pose[]; thresholds + collision knobs Outputs. success, trajectory: Trajectory, goalset_index curobo.PlanWithGraspedObject Description. Plan a collision-free trajectory while holding a named grasped object. Inputs. world_config; start_joint_position; target_pose: Se3Pose; object_name Outputs. success, trajectory: Trajectory curobo.PlanLinear Description. GPU-accelerated straight-line Cartesian motion via per-waypoint IK, optionally constrained and collision-checked. Inputs. start_pose, end_pose: Se3Pose; start_joint_position; world_config (optional); num_waypoints (20); hold_vec_weight Outputs. success, trajectory: Trajectory, failure_reason curobo.PlanDirectedLinear Description. Constrained linear motion that holds selected Cartesian axes (project-to-target / fixed-distance / orient-in-place). Inputs. start_joint_position; start_pose, target_pose; allowed_axes; endpoint_mode; orientation_mode Outputs. success, trajectory: Trajectory, failure_reason curobo.PlanGraspMotion Description. Three-phase grasp plan (free-space approach, constrained grasp, constrained lift) so the caller can interleave gripper commands. (CuRobo v0.8.) Inputs. start_joint_position; grasp_pose: Se3Pose; approach axis+distance (0.12); lift axis+distance (0.20) Outputs. success; approach_trajectory, grasp_trajectory, lift_trajectory: Trajectory; failure_reason curobo.SolveIK Description. Pure geometric IK for a single TCP pose (no world collision). Inputs. target_pose: Se3Pose; seed_config (optional); tcp_offset; num_seeds (32) Outputs. success, joint_config: JointState curobo.PlanToPose Description. Collision-aware trajectory to a single EE target pose. Inputs. target_pose: Se3Pose; start_joint_position; world_config (optional); tcp_offset Outputs. success, trajectory: Trajectory curobo.BatchGraspFeasibility Description. Per-pose scene-collision feasibility for a batch of grasp candidates; filters blocked grasps before planning. Inputs. world_config; start_state: JointState; grasp_poses: Se3Pose[]; approach_offset_m (0.10); ignore_obstacle_names Outputs. feasible: bool[], grasp_ik_ok: bool[], approach_ik_ok: bool[], corridor_collision_fraction: float[] (all aligned to input) curobo.ValidateJointTrajectoryRobot Description. Validate joint waypoints against world + self-collision. Inputs. world_config; trajectory: Trajectory; collision knobs Outputs. success, failure_reason, first_collision_waypoint, collision_status_detail curobo.ValidateJointTrajectoryGrasped Description. As above, with a grasped object attached at the first waypoint. Inputs. world_config; trajectory; object_name; collision knobs Outputs. success, failure_reason, first_collision_waypoint, collision_status_detail pyroki.SolveIK Description. CPU differentiable IK for a target EE pose; the planner-free IK fallback. Inputs. target_pose: Se3Pose; target_link_name (panda_hand); prev_config (optional); tcp_offset (0,0,−0.10,0,-0.1) Outputs. joint_config: JointState, success pyroki.PlanLinear Description. CPU linear Cartesian trajectory between two poses. Inputs. start_pose, end_pose: Se3Pose; num_waypoints (20); dt (0.02) Outputs. trajectory: Trajectory, success

Geometry.

Geometry utilities filter clouds with DBSCAN, fit boxes, construct grasp candidates, back-project masks/pixels, transform frames, reconstruct collision meshes and calculate drop poses/distances/quaternion rotations. Defaults such as 5 mm clustering and 5 cm drop clearance are explicit parameters, not learned universal constants.

geometry_svc.FilterNoise Description. DBSCAN noise filtering of a point cloud. Inputs. point_cloud: PointCloud; eps (0.005); min_samples (10) Outputs. PointCloud geometry_svc.ComputeOBB Description. Fit an oriented bounding box to 3D points. Inputs. point_cloud: PointCloud Outputs. OrientedBoundingBox geometry_svc.FilterAndComputeOBB Description. FilterNoise + ComputeOBB in one round trip. Inputs. point_cloud: PointCloud; eps (0.005); min_samples (10) Outputs. OrientedBoundingBox geometry_svc.TopDownGraspFromOBB Description. A single top-down grasp pose from an OBB. Inputs. obb: OrientedBoundingBox; z_offset Outputs. Se3Pose geometry_svc.TopDownGraspCandidates Description. Candidate top-down grasps so a planner can pick a reachable one. Inputs. obb: OrientedBoundingBox; z_offset Outputs. poses: Se3Pose[] (primary + 90∘-rotated) geometry_svc.SelectTopDownGrasp Description. Pick the most top-down (and nearest) grasp from candidates. Inputs. grasp_poses: Se3Pose[]; gripper_position: Vec3 Outputs. Se3Pose geometry_svc.FrontGraspFromOBB Description. Front/horizontal grasp for a handle (drawers, doors), with a slide axis. Inputs. obb: OrientedBoundingBox; approach_offset (0.08); approach_hint; z_offset Outputs. grasp_pose, pre_grasp_pose: Se3Pose; approach_direction, slide_axis: Vec3 geometry_svc.DepthToPointCloud Description. Back-project a depth image to a camera-frame point cloud. Inputs. depth: DepthImage; intrinsics: Matrix3x3 Outputs. PointCloud (camera frame) geometry_svc.MaskToWorldPoints Description. Back-project a 2D mask to world-frame 3D points. Inputs. mask: Mask; depth: DepthImage; intrinsics: Matrix3x3; camera_pose: Se3Pose Outputs. PointCloud (world frame) geometry_svc.PixelToWorldPoint Description. Back-project a single pixel to a world point. Inputs. pixel_x, pixel_y; depth; intrinsics; camera_pose Outputs. Vec3 (world point) geometry_svc.TransformPoints Description. Apply a rigid transform to a set of points. Inputs. point_cloud: PointCloud; transform: Se3Pose Outputs. PointCloud geometry_svc.BuildWorldConfig Description. Reconstruct a named collision-mesh world from camera observations for any planner. Inputs. cameras: CameraObservation[]; object_masks: [name, mask, camera_index]; recon params (voxel_size 0.005, mesh_alpha 0.03); robot_joint_state (optional); target_obb (optional) Outputs. config: WorldConfig, mesh_names: string[] geometry_svc.ComputeDropPosition Description. Drop position above a container, accounting for clearance and object height. Inputs. container_obb: OrientedBoundingBox; clearance (0.05); object_z_extent Outputs. Vec3 (drop point) geometry_svc.ComputeXYDistance Description. XY-plane Euclidean distance between two points. Inputs. point_a, point_b: Vec3 Outputs. distance: double geometry_svc.RotateQuatZ90 Description. Rotate a quaternion 90∘ about Z (for grasp candidates). Inputs. quat: Quaternion (WXYZ) Outputs. Quaternion

Robot and environment control.

The 25 control/environment methods cover pose/joint motion, synchronized arms, policy actions, grippers, observations, simulator initialization/reset/stepping, ground-truth state, task completion and video capture. Ground-truth queries are available for rehearsal feedback; camera-derived policy observations and privileged evaluation state should not be conflated. Calls that directly command poses/joints have different planning assumptions from collision-checked trajectory execution.

robot_control.GetEEPose Description. Current end-effector pose in world frame. Inputs. arm_id (0) Outputs. pose: Se3Pose robot_control.GoToPose Description. Move the EE to a target pose via IK (no collision avoidance). Inputs. pose: Se3Pose; z_approach; tcp_offset (0,0,−0.10,0,-0.1); tolerance (0.01); max_steps (120); nonblocking Outputs. (empty) robot_control.GoToPoseCartesian Description. Move to a pose with smooth Cartesian interpolation. Inputs. pose: Se3Pose; lin_vel_norm (1.0); ang_vel_norm (2.0); timeout_s (20) Outputs. (empty) robot_control.ExecuteJointTrajectory Description. Execute a pre-planned joint trajectory (from CuRobo or PyRoKI). Inputs. trajectory: Trajectory; subsample (1); tolerance (0.01) Outputs. (empty) robot_control.MoveToJoints Description. Move directly to a joint configuration. Inputs. joint_config: JointState; tolerance (0.01); max_steps (120) Outputs. (empty) robot_control.GoHome Description. Move the robot to its safe home configuration. Inputs. (none) Outputs. (empty) robot_control.GoToPoseArm Description. Multi-arm: move one specified arm to a pose. Inputs. arm_id; pose: Se3Pose; z_approach; tcp_offset Outputs. (empty) robot_control.GoToPoseBoth Description. Multi-arm: move both arms to their poses simultaneously. Inputs. pose_arm0, pose_arm1: Se3Pose; z_approach; tcp_offset Outputs. (empty) robot_control.MoveToJointsBoth Description. Multi-arm: drive both arms to raw joint targets in lock-step (no IK). Inputs. joints_arm0, joints_arm1: JointState; tolerance (0.02); max_steps (300) Outputs. (empty) robot_control.ApplyPolicyAction Description. Apply one low-level policy action to the env controller (VLA rollouts). Inputs. action: double[] (e.g. 7-dim [Δ​x,Δ​y,Δ​z,Δ​rx,Δ​ry,Δ​rz,grip][\Delta x,\Delta y,\Delta z,\Delta r_{x},\Delta r_{y},\Delta r_{z},\text{grip}]); arm_id (0) Outputs. (empty) gripper.Open Description. Open the gripper and settle. Inputs. arm_id (0); settle_steps (40) Outputs. GripperState gripper.Close Description. Close the gripper and settle. Inputs. arm_id (0); settle_steps (60) Outputs. GripperState gripper.GetPosition Description. Read the current gripper opening. Inputs. arm_id (0) Outputs. GripperState (opening width) gripper.GetPose Description. Gripper pose in world frame. Inputs. arm_id (0) Outputs. pose: Se3Pose observation.GetObservation Description. Full sensor observation from all cameras and arms. Inputs. (none) Outputs. cameras: CameraObservation[] (RGB-D + intrinsics/extrinsics), arm_states: ArmState[] observation.GetCameraPose Description. A named camera’s pose in robot base frame. Inputs. camera_name (e.g. “agentview”) Outputs. pose: Se3Pose sim_bridge.Init Description. Initialize a simulation environment by name. Inputs. env_name; config_path (optional); camera_names; task_id; suite_name Outputs. success, capabilities: SimCapabilities, error_message sim_bridge.Reset Description. Reset the environment for a new episode. Inputs. seed Outputs. observation: ObservationResponse sim_bridge.StepOnce Description. Execute a single low-level MuJoCo timestep. Inputs. gripper_fraction (−1-1 = no change) Outputs. observation, reward, terminated, truncated sim_bridge.MoveToJointsBlocking Description. Move to a joint configuration, blocking until converged. Inputs. target_joints: JointState; tolerance (0.01); max_steps (120) Outputs. observation, reward, terminated, truncated sim_bridge.SetGripper Description. Set the gripper target fraction. Inputs. fraction (0 closed →\to 1 open); arm_id (0) Outputs. (empty) sim_bridge.GetState Description. Full sim state, including per-object ground-truth poses. Inputs. (none) Outputs. arm_states: ArmState[], cameras: CameraObservation[], objects: [name, pose] (ground truth) sim_bridge.CheckTaskCompletion Description. Query simulator reward and task success. Inputs. (none) Outputs. reward, task_completed, completion_rate (sub-goals satisfied) sim_bridge.EnableVideoCapture Description. Start/stop video-frame capture. Inputs. enabled; clear Outputs. (empty) sim_bridge.SaveVideo Description. Encode captured frames to an MP4 and return its path. Inputs. output_path; clear; fps (20) Outputs. success, file_path, num_frames

Appendix E Self-Learning

The self-learning appendix preserves its repeated pseudocode and the complete four-card example feedback, including the original numerical inconsistencies.

E.1 Pseudo-code

Algorithm 2 repeats the sample–rehearse–analyze–update loop with parallel rollouts. Its displayed pseudocode abstracts away the main text’s fresh-instance acceptance test; the prose supplies that additional rejection rule. Both original algorithms remain available.

Algorithm 2 Rehearsal-based Graph Optimization Input: 𝒯 = ⟨ ℒ , ℰ , ℛ , 𝒪 , 𝒳 , ℬ , 𝒥 ⟩ \mathcal{T}=\langle\mathcal{L},\mathcal{E},\mathcal{R},\mathcal{O},\mathcal{X},\mathcal{B},\mathcal{J}\rangle , Iterations M M , Parallel Rollouts N N Output: Optimized Task Graph 𝒢 ∗ \mathcal{G}^{*} Initialization: 𝒢 0 ← Graph ​ _ ​ Init ​ ( ℒ ) \mathcal{G}_{0}\leftarrow\mathrm{Graph\_Init}(\mathcal{L}) // Initial graph from language 𝒮 ← Build ​ _ ​ Scene ​ ( 𝒢 , { 𝒢 i } ) \mathcal{S}\leftarrow\mathrm{Build\_Scene}(\mathcal{G},\{\mathcal{G}_{i}\}) // Instantiate sim environment for j ← 1 j\leftarrow 1 to M M do // Step 1: Scene Variational Sampling { s ^ i } i = 1 N ∼ ℬ \{\hat{s}_{i}\}_{i=1}^{N}\sim\mathcal{B} 13 // Sample N N instances // Step 2: Parallel Rehearsal 14 for i ← 1 i\leftarrow 1 to N N do in parallel 17 15 τ i ← Rehearsal ⁡ ( s ^ i , 𝒢 j − 1 ) \tau_{i}\leftarrow\mathrm{Rehearsal}(\hat{s}_{i},\mathcal{G}_{j-1}) // Rollout policy F i ← Analyze _ ​ Failure ​ ( τ i , 𝒢 , { 𝒢 i } ) F_{i}\leftarrow\mathrm{Analyze_{\_}Failure}(\tau_{i},\mathcal{G},\{\mathcal{G}_{i}\}) 16 19 // Step 3: Graph Refinement 𝒢 j ← Graph ​ _ ​ Update ​ ( { F 1 , … , F N } ) \mathcal{G}_{j}\leftarrow\mathrm{Graph\_Update}(\{F_{1},\dots,F_{N}\}) 18 // Optimize via LLM 20 return 𝒢 ∗ ← 𝒢 M \mathcal{G}^{*}\leftarrow\mathcal{G}_{M}

E.2 Sample Feedback

Thirty parallel pan trials provide before/after object states and contacts. Perception is reported 30/30; grasp validation is 25/30; example failures distinguish an open gripper from closing without lifting. Placement feedback targets inadequate burner overlap, motivating changed release geometry. The source simultaneously lists place_pan validation as 17/30 and states that all evaluated placements fail with zero terminal coverage. Those quantities are not reconciled in the appendix, so the notes do not reinterpret 17/30 as verified terminal task success.

Summary. In the first rehearsal iteration, perception succeeds across all sampled environments, while downstream failures arise from grasp instability and incorrect pan placement. The placement node fails because the pan is released away from the burner footprint, yielding zero burner coverage.
Iteration 1 Feedback Terminal coverage: 0.00 Node Validation Feedback perceive_pan 1.00 (30/30) Perception publishes the target OBB for the pan handle. perceive_burner 1.00 (30/30) Perception publishes the container OBB for the burner. grasp_pan 0.83 (25/30) The pan is considered grasped when it remains in contact with a robot link at the exit of the grasp subgraph. place_pan 0.57 (17/30) The pan should be released by the gripper and cover at least 70% of the burner XY footprint. All evaluated placement attempts fail because the pan does not cover the burner footprint.
Representative Failure Cases • env 3, grasp_pan : grasp failed with the gripper still open. gripper_open_fraction = 1.0 , pan_z = 0.024 , contact = [ table_top ] \texttt{gripper\_open\_fraction}=1.0,\quad\texttt{pan\_z}=0.024,\quad\texttt{contact}=[\texttt{table\_top}] The subgraph returned failed ; only ee_pose_at_grasp was bound. • env 9, grasp_pan : grasp failed after gripper closure. gripper_open_fraction : 1.0 → 0.0 , pan_z = 0.024 , contact = [ table_top ] \texttt{gripper\_open\_fraction}:1.0\rightarrow 0.0,\quad\texttt{pan\_z}=0.024,\quad\texttt{contact}=[\texttt{table\_top}] The robot closed the gripper, but the pan remained on the table and the node returned failed . The elapsed time was 4.377 s . • place_pan : the evaluated placement failure trials has: pan_coverage_of_burner ≤ 0.7 \texttt{pan\_coverage\_of\_burner}\leq 0.7 Although the pan was released from the gripper, its projected bottom footprint did not sufficiently overlap with the burner footprint.
Optimization signal. The feedback suggests that the next graph update should revise the placement target and release pose for place_pan, while also improving grasp robustness for the failed grasp_pan environments.

Appendix F Sample LLM Prompts and Outputs

All four original prompt/output blocks are preserved as exact decoded plaintext with indentation. They document a particular harness snapshot, including inconsistent legacy/v3 instructions; they are not presented as verified runnable code.

F.1 Behavior Agent Prompt

The behavior agent authors only the top-level WorkflowSpec: subgraph metadata, typed bindings, edges, success exits and one error route. Discovery tools provide skill references/examples and report missing capabilities. Stage tags enable a mechanical first-iteration swap of canonical grasp/transport variants, showing an explicit refinement prior. The v3 specification restricts tool/script names, reference syntax, parallel fan-out and streaming behavior. It calls real-execution checkpoint validation a currently inactive reserved hook, so the prompt itself does not establish runtime safety monitoring.

You are a robotics task planner. Given a manipulation task description,
you produce **only the workflow topology** — the top-level node graph
(subgraph nodes + end nodes), the edges and conditional_edges connecting
them, and each subgraph's inputs/outputs/exit-values/skill choice. You do
**not** generate internal nodes/edges of any subgraph. The universal
`subgraph_agent` is invoked separately per subgraph to fill in its inner
state machine.

## Output format

A single ` ```python` fenced block (no file path) that builds the workflow
scaffold with `vos.builder.WorkflowSpec` and binds it to a module-level
variable named ``spec``:

```python
from vos.builder import WorkflowSpec, START
from vos.runtime.workflow import ServiceCall

spec = WorkflowSpec(name="<task>", description="<task prompt>")

# 1. Declare every subgraph (metadata only — nodes/edges get filled in later).
spec.declare_subgraph(
    "<sg_def_name>",                  # e.g. "target_sg"
    skill="<MORSL skill name from the Available Skills list>",
    description="<natural-language goal for this subgraph instance>",
    inputs=,
    outputs=,
    exit_success_values=["<success_exit>"],
    on_error="<failure_exit>",
    stage="<grasp|transport|place>",  # OPTIONAL; see "Stage tag" below.
)

# 2. Place the top-level subgraph nodes that reference those declarations.
spec.add_subgraph_node("<sg_node_name>", ref="<sg_def_name>")

# 3. Add end nodes — typically one success ("done") and one failure ("abort").
spec.add_end("done", status="success")
spec.add_end(
    "abort",
    status="failure",
    recovery=[
        ServiceCall("gripper.v1.Gripper", "Open", {}),
        ServiceCall("robot_control.v1.RobotControl", "GoHome", {}),
    ],
)

# 4. Wire the entry edge and conditional dispatch per subgraph node.
spec.add_edge(START, "<entry_subgraph_node>")
spec.add_conditional_edges(
    "<sg_node_name>",
    {"<exit_value>": "<next_node>", ...},
    router_field="exit",
)
```

The pipeline imports `vos.builder` (already available; do not pip
install), executes the block in a sandbox, picks up the module-level
`spec` variable, serializes it to a workflow-spec dict, and forwards it
to the per-subgraph generators.

Use a 1:1 naming convention between the outer node name (passed to
`add_subgraph_node`) and the inner subgraph def name (passed to
`declare_subgraph`), or use the same name for both — both work.

### Hard rules for the Python block

1. The block MUST end with a module-level variable named ``spec`` bound
   to a ``WorkflowSpec`` instance.
2. Imports: ``from vos.builder import WorkflowSpec, START`` and
   ``from vos.runtime.workflow import ServiceCall`` are pre-provided in
   the sandbox.
3. No I/O, no other imports.
4. Subgraph `inputs` / `outputs` MUST be dicts of `{name: proto_type_str}`.
   Each value is a **bare proto type string** (e.g. ``"OrientedBoundingBox"``,
   ``"Mask"``, ``"PointCloud"``) — never a ``Ref`` and never a nested dict.
   Cross-subgraph data flow is established implicitly by `add_edge` /
   `add_conditional_edges` plus matching input/output **names** between
   subgraphs — you do NOT wire it here.
5. Every `declare_subgraph(...)` call MUST pass BOTH `exit_success_values=`
   AND `on_error=` as keyword arguments — they are required and have no
   defaults. Omitting either raises
   `declare_subgraph() missing 2 required keyword-only arguments` and
   the whole spec rejects. Conventional values: `exit_success_values=["done"]`
   and `on_error="abort"`, paired with `spec.add_end("done", status="success")`
   and `spec.add_end("abort", status="failure", recovery=[...])`.

### Stage tag (optional but recommended)

Set ``stage="grasp"``, ``"transport"``, or ``"place"`` on every
``declare_subgraph`` that participates in the pick-and-place pipeline.
The iter-1 mechanical-swap engine
(``vos/refine/iter1_swap.py``) groups per-env checkpoint outcomes by
``stage`` to compute per-stage pass-rates and decide which canonical
swap (graspgen↔OBB, free-space↔collision-aware transport, subpart↔parent
drop anchor) to apply for the next iter.

- ``grasp`` — any subgraph whose exit is "object held in gripper"
  (skills ``grasp_moe``, ``grasp_multi``, ``grasp_curobo_obb``).
- ``transport`` — moves the held object from grasp pose to above the
  drop zone (``transport_to_drop`` when modeled as its own subgraph).
- ``place`` — the release leg: compute drop pose + drop-offset
  correction + descend/release (``transport_to_drop`` when modeled
  end-to-end, or a dedicated placement subgraph).

When a subgraph does not belong to the canonical taxonomy (e.g. a
perception or pre-grasp staging subgraph), omit ``stage``. The
iter-1 engine skips unstaged subgraphs during stage rollups; explicit
``None`` is fine.

## Hard rules

1. **Edges and conditional_edges.** For each subgraph node, every one of
   its `exit_success_values` PLUS its `on_error` symbol must appear as a
   key in the corresponding `add_conditional_edges` mapping, and every
   mapping target must be a node declared at the top level. Every end
   node must be reachable from `START`.
2. **The entry node must be a subgraph node** (not an end node).
3. **No internal nodes / edges / on_error** for any subgraph in your
   output. The universal subgraph_agent fills those in per subgraph.
4. **Inputs and outputs.** A subgraph's declared `inputs` must equal its
   skill's `required_inputs` after `<name>` substitution, and every
   input must have an upstream subgraph on some path to it that produces
   a matching output name with a matching proto type. Declared `outputs`
   must be a subset of the skill's `produces_outputs` (omit outputs
   nothing downstream consumes).
5. **Pick the right specialized variant.** When multiple variants of a
   role appear in Available Skills (e.g. `perception_multi` vs
   `perception_single`, `grasp_curobo_obb` vs `grasp_direct_ik`), read
   each skill's *When to use* guidance and pick the best fit — default
   to the more robust / collision-aware variant when both are listed.
   Skills whose required services are not deployed do not appear in
   Available Skills.

## Discovery via tools

You may call:

- `read_skill_reference(skill_name, doc_name)` to load the long-form
  rationale for a skill (the references listed in the skill's frontmatter).
- `read_skill_example(skill_name, example_name)` to load a sample
  subgraph the per-skill subgraph_agent will start from.
- `report_missing_capability(name, why)` if no skill in the catalog
  covers a step the task requires. The build aborts with a structured
  report.

Use these sparingly — the always-loaded catalog already shows skill
descriptions, tags, exit_conditions, and produces_outputs.

# Workflow v3 spec — shared core

This is the structural spec every codegen subagent shares. It defines the
top-level workflow shape, the SubgraphDef shape, node types, edge
semantics, `$ref` syntax, and validation rules.

## Top-level workflow

```json
{
  "version": 3,
  "meta": { "name": "...", "description": "..." },
  "nodes": {
    "<sg_node_name>": { "type": "subgraph", "ref": "<sg_def_name>" },
    "done":  { "type": "end", "status": "success" },
    "abort": { "type": "end", "status": "failure",
               "recovery": [ { "service": "...", "method": "...", "inputs": {} } ] }
  },
  "edges": [["START", "<first_subgraph_node>"]],
  "conditional_edges": {
    "<sg_node_name>": {
      "router_field": "exit",
      "mapping": { "<exit_value>": "<next_node>", ... }
    }
  },
  "subgraphs": { "<sg_def_name>": { /* SubgraphDef */ } }
}
```

`START` and `END` are virtual node names. The top level orchestrates
subgraph nodes and end nodes; cross-subgraph routing is via the
`conditional_edges` block reading each subgraph's `exit` value.

`meta` is an open string→string map; recognized keys include `name`,
`description`, `runtime` (`direct_real`), `observation_stream_hz`, and
`validate_checkpoints` (`"true"` opts the executor into enforcing each
subgraph's `validate=True` postcondition checkpoints at real-execution
time — **currently a no-op**; reserved seam).

## SubgraphDef shape

```json
{
  "skill":  "<MORSL skill name; echoed from your spec>",
  "inputs":  { "<name>": "<proto.type.string>" },
  "outputs": { "<name>": { "$ref": "<node>.<field>" } },
  "nodes": {
    "<node_name>": { /* NodeDef */ },
    "<terminal_marker>": { "type": "noop" }
  },
  "edges": [
    ["START", "<first_node>"],
    ["<first_node>", "<second_node>"],
    ["<terminal_marker>", "END"]
  ],
  "conditional_edges": { },
  "exit": { "router_field": null, "success_values": ["<success_exit>"] },
  "on_error": "<failure_exit_value>"
}
```

Scripts go in separate fenced blocks, namespaced under
`scripts/<subgraph_name>/`:

````python:scripts/<subgraph_name>/<script>.py
...
````

## Node types

The two production dispatch types are `tool` and `script`. `noop` and
`router` are control-flow markers. `subgraph` and `end` only appear at
the top level of the workflow (not inside subgraphs).

```json
{ "type": "tool", "tool": "gripper.Open",
  "inputs": { "settle_steps": 40 } }                       /* gRPC method */

{ "type": "tool", "tool": "sam3.SegmentBox",
  "inputs": { "image": { "$ref": "obs.rgb" }, "box": { "$ref": "perceive.box" } } }

{ "type": "tool", "tool": "run_policy",
  "inputs": { "observation_stream": { "$ref": "in.observation_stream" } } }  /* atomic MORSL skill */

{ "type": "tool", "tool": "libero_pi05",
  "inputs": { "prompt": "...", "max_windows": 25 } }       /* learned policy */

{ "type": "tool", "tool": "track_object", "streaming": true,
  "inputs": { "observation_stream": { "$ref": "in.observation_stream" } } }

{ "type": "script", "script": "scripts/<subgraph_name>/foo.py",
  "inputs": { ... } }                                      /* canonical bundle script
                                                              or rare inline helper */

{ "type": "noop" }                                          /* named terminal marker */
{ "type": "router", "script": "scripts/<sg>/route.py", "inputs": { ... } }
```

- **`tool`** — the canonical dispatch for **anything**: gRPC service
  methods (auto-registered as `<package>.<MethodName>`, e.g.
  `observation.GetObservation`, `gripper.Open`,
  `geometry_svc.FilterAndComputeOBB`), MORSL skills (registered by skill
  name, e.g. `run_policy`, `track_object`), and learned policies
  (registered by policy name). `tool:` is always a flat name; never write
  `<package>.v1.<Service>` or include a separate `method:` field. The
  runtime resolves the proto FQN internally.
- **`script`** — local Python file in the workflow folder. Prefer to
  point at a canonical bundle script (listed in the chosen skill's
  "Canonical scripts" table); only emit your own inline Python for
  ad-hoc helpers that no canonical and no tool covers.
- **`noop`** — empty body. Used as a named subgraph terminal so the node
  name becomes the subgraph's exit value when `exit.router_field=null`.
- **`router`** — Send dispatch. Function returns either a string target
  for static routing or a list of `{"to": "...", "inputs": {...}}` dicts
  for dynamic fan-out.

*Legacy node types (`type: service`, `type: skill`, `type: policy`)
have been retired. The validator rejects them with a precise migration
message. Always emit `type: tool` with the flat name instead.*

`streaming: true` is only valid on `tool` and `script` nodes. A
streaming node has **no outgoing edges** — it's a pure source. Consumers
read its latest published value via `{"$ref": "<node>"}`. The chosen
tool's bundle contract must declare `contract.streaming: true`.

## Edge semantics

- **Static edges**: `["src", "dst"]` pairs.
- **Multiple outgoing edges from one node = parallel super-step.**
- **`conditional_edges`** dispatches on a router field. From a
  non-router source the field is on the source's output (set
  `router_field` to its name). From a router source set
  `router_field: null`.
- **`exit.router_field: null`** means the subgraph's exit value is the
  name of the terminal node (the one whose edge points to `END`). For
  this to work, declare your terminals as `noop` nodes named after the
  exit values (e.g. `"found": {"type": "noop"}` with edge
  `["found", "END"]`).
- **`exit.router_field: "<field>"`** reads the field on the terminal
  node's output as the exit value.
- **`on_error`** is an optional subgraph-level catch: when any node
  raises, the runtime emits this string as the subgraph's exit value
  (bypassing the terminal-node read). The `on_error` symbol is the
  subgraph's single failure exit. It is **never** a declared node and
  **never** a `conditional_edges` mapping target — failures surface
  only by raising.

### `perception_multi`

Molmo point-prompt + SAM3 point/text segmentation + depth-to-3D fusion, ending in a FilterAndComputeOBB to produce a clean OBB and mask. Single path; molmo-only perception (no DINO/VLM).

### `perception_single`

Fast single-path object detection: DINO broad detect + VLM disambiguation + SAM3 segmentation + depth-to-3D fusion, ending in a FilterAndComputeOBB to produce a clean OBB and mask. Best for uncluttered scenes with visually distinct targets, or platforms where only DINO/VLM/SAM are deployed.

...

## Available Tools (flat catalog)

| Tool | Transport | Tags | Summary |
|------|-----------|------|---------|
| `curobo.PlanDirectedLinear` | grpc | curobo | CuRobo.PlanDirectedLinear :: PlanDirectedLinearRequest → PlanDirectedLinearResponse |
| `curobo.PlanGraspMotion` | grpc | curobo | CuRobo.PlanGraspMotion :: PlanGraspMotionRequest → PlanGraspMotionResponse |
| `curobo.PlanLinear` | grpc | curobo | CuRobo.PlanLinear :: CuRoboPlanLinearRequest → CuRoboPlanLinearResponse |
| `curobo.PlanToGraspPoses` | grpc | curobo | CuRobo.PlanToGraspPoses :: CuRoboPlanGraspRequest → CuRoboPlanGraspResponse |
| `curobo.PlanToPose` | grpc | curobo | CuRobo.PlanToPose :: CuRoboPlanToPoseRequest → CuRoboPlanToPoseResponse |
| `curobo.PlanWithGraspedObject` | grpc | curobo | CuRobo.PlanWithGraspedObject :: CuRoboPlanGraspedRequest → CuRoboPlanGraspedResponse |
...

## Task

Pick up the object on the table and place it at the designated target location. Perceive the scene to locate the object, grasp it, transport it above the target, and release it so it rests stably at the target.

F.2 Behavior Agent Output

The sample plan chains target perception, container perception, grasp and transport, then completes or recovers by opening the gripper and homing. It declares on_error="abort" but maps different failure symbols such as not_found, failed and blocked in its conditional routes. This conflicts with the preceding requirement to map the declared error symbol. The exact published example is retained, without silently correcting it or claiming it passes the current validator.

```python
from vos.builder import WorkflowSpec, START
from vos.runtime.workflow import ServiceCall

spec = WorkflowSpec(name="pick_and_place", description="Pick up the object on the table and place it at the designated target location.")

# 1. Declare subgraphs
spec.declare_subgraph(
    "target_sg",
    skill="perception_single",
    description="Locate the object on the table",
    inputs={},
    outputs={
        "target_obb": "OrientedBoundingBox",
        "target_mask": "Mask",
        "target_cloud": "PointCloud",
    },
    exit_success_values=["found"],
    on_error="abort",
)

spec.declare_subgraph(
    "container_sg",
    skill="perception_single",
    description="Locate the target location",
    inputs={},
    outputs={
        "container_obb": "OrientedBoundingBox",
        "container_mask": "Mask",
        "container_cloud": "PointCloud",
    },
    exit_success_values=["found"],
    on_error="abort",
)

spec.declare_subgraph(
    "grasp_sg",
    skill="grasp_multi",
    description="Grasp the object on the table",
    inputs={
        "target_obb": "OrientedBoundingBox",
        "target_mask": "Mask",
        "target_cloud": "PointCloud",
    },
    outputs={"ee_pose_at_grasp": "Se3Pose"},
    exit_success_values=["grasped"],
    on_error="abort",
    stage="grasp",
)

spec.declare_subgraph(
    "transport_sg",
    skill="transport_to_drop",
    description="Transport the object to the target location",
    inputs={
        "target_obb": "OrientedBoundingBox",
        "target_mask": "Mask",
        "container_obb": "OrientedBoundingBox",
        "container_mask": "Mask",
        "ee_pose_at_grasp": "Se3Pose",
    },
    outputs={},
    exit_success_values=["placed"],
    on_error="abort",
    stage="place",
)

# 2. Place subgraph nodes
spec.add_subgraph_node("target_sg", ref="target_sg")
spec.add_subgraph_node("container_sg", ref="container_sg")
spec.add_subgraph_node("grasp_sg", ref="grasp_sg")
spec.add_subgraph_node("transport_sg", ref="transport_sg")

# 3. Add end nodes
spec.add_end("done", status="success")
spec.add_end(
    "abort",
    status="failure",
    recovery=[
        ServiceCall("gripper.v1.Gripper", "Open", {}),
        ServiceCall("robot_control.v1.RobotControl", "GoHome", {}),
    ],
)

# 4. Wire edges
spec.add_edge(START, "target_sg")
spec.add_conditional_edges(
    "target_sg",
    {"found": "container_sg", "not_found": "abort"},
    router_field="exit",
)
spec.add_conditional_edges(
    "container_sg",
    {"found": "grasp_sg", "not_found": "abort"},
    router_field="exit",
)
spec.add_conditional_edges(
    "grasp_sg",
    {"grasped": "transport_sg", "failed": "abort"},
    router_field="exit",
)
spec.add_conditional_edges(
    "transport_sg",
    {"placed": "done", "blocked": "abort"},
    router_field="exit",
)
```

F.3 Skill Agent Prompt

The skill-agent prompt supplies filtered tools, canonical scripts, strict output types, reference conventions and validation rules for a single subgraph. It distinguishes successful noop exits from exception/error routing and requires outputs such as OBB, mask and cloud. Checkpoints use privileged simulator quantities rather than circular visual detections. The text first delegates checkpoint creation to another agent, then requires this builder to emit at least one; it also ends with legacy end-state instructions that conflict with the v3 rules above. Those internal version inconsistencies limit reproducibility from the prompt alone. Exact pose/box field names, WXYZ quaternions, canonical-script preference and all source code remain intact.

# Subgraph agent

You generate **one subgraph** for a v3 robotics workflow. The
coordinator chose your input spec (subgraph name, declared
inputs/outputs/exit-values, and the MORSL skill this subgraph should
use). Your output is a Python script that builds the subgraph with
the `vos.builder` library, plus any inline Python scripts referenced
by `type="script"` nodes.

## Your context window contains

1. The shared workflow spec (top-level shape, node types, edge
   semantics, `$ref` syntax, validation rules) — see
   `_workflow_spec.md`.
2. The chosen skill's SKILL.md body (this is the per-skill guidance —
   recommended node sequence, hard rules, exit-value semantics, and
   contract — including whether the skill is `streaming: true`).
3. The filtered tool catalog: only the tools listed in the skill's
   `allowed-tools` frontmatter, intersected with what is deployed.
4. The chosen skill's `canonical_scripts` list (for composite skills) —
   file references the subgraph may use as `type="script"` nodes.
5. The coordinator-supplied subgraph spec (name, inputs, outputs,
   exit values, context).

## Your job

Compose the minimal sequence of nodes and edges that:

1. Consumes declared inputs (referenced as `Ref(f"in.{name}")`).
2. Produces values bound to the declared outputs via `sg.set_outputs(...)`.
3. Names every success-path exit via `sg.add_exit(name)` (creates the
   terminal `noop` marker), and names the single failure-path exit via
   `sg.set_on_error(value)`.
4. Reaches each success-exit marker on its own path with an explicit
   edge to `END`.
5. Calls only tools in the filtered catalog and scripts in the skill's
   `canonical_scripts` list. If you need a primitive that's missing,
   call `report_missing_capability(name, why)`.

Postcondition checkpoints (`sg.add_checkpoint(...)`) are authored by a
separate `checkpoint_agent` in a follow-on pass. **Do not** declare
any checkpoints yourself — emit structure (nodes, edges,
`set_outputs(...)`, `set_on_error(...)`) only.

### Exit-value rule (HARD, single rule)

There is **one and only one** namespace collision question, and it's
answered by which field the exit value lives in:

| Form | Is it a node? | Where in code |
|---|---|---|
| Success exit (default `set_exit_router` is unset) | **YES** — call `sg.add_exit(name)` (creates a `noop` and edges to `END` are your responsibility) | `sg.add_exit("found")` + `sg.add_edge("found", END)` |
| Success exit when `set_exit_router(router_field=..., success_values=[...])` is used | **NO** — string field-values returned by the terminal node, never node names | `sg.set_exit_router(router_field="exit", success_values=["ok"])` |
| `on_error` | **NO** — single failure symbol; never declare a node with that name | `sg.set_on_error("not_found")` |

Wrong patterns the validator rejects (with rule IDs):

- **S9**: `sg.set_on_error("failed")` plus `sg.add_node("failed", type="noop")` — the `failed` node is forbidden.
- **S10**: routing `"false" -> "failed"` in a `sg.add_conditional_edges(...)` mapping — `failed` cannot be a conditional-edge target. Failure is surfaced by raising, not routing.
- **S11**: `sg.set_exit_router(..., success_values=["grasped"])` then no terminal node returns `grasped` in its `exit` field — every success value must be producible.

### Conditional-edge rules (HARD)

**Source rule:** the first argument to `sg.add_conditional_edges(src, ...)` is the source node — that node must be a real producer of the field named in `router_field`. Terminal `noop` markers (created via `add_exit`) are NOT routers; they have a single outgoing edge to `END` and must NOT be a source.

**Target rule:** every value in a `mapping` must be a **declared node name** (or `START` / `END`) AND must not equal `on_error`. The single failure exit lives only via `set_on_error` and is reached by raising, never by routing.

**How to express "this subgraph's postcondition must hold"** (e.g. "the gripper is actually holding the target after `close`"): do NOT add a node that re-checks the end state and raises, and do NOT route to `on_error`. The follow-on `checkpoint_agent` will declare the postcondition via `sg.add_checkpoint(...)` against privileged state.

**Mid-subgraph preconditions** (a check whose failure means *subsequent nodes in this subgraph cannot run*) are different — for those, insert a `type="script"` guard that raises; the raise propagates to `on_error`:

```python
# scripts/<sg>/require_cloud.py
def run(ctx, found: bool) -> None:
    if not found:
        raise RuntimeError("target not detected; cannot plan a grasp")
    return None
```
```python
sg.add_node("require_cloud", type="script", script="scripts/<sg>/require_cloud.py",
            inputs={"found": Ref("perceive.found")})
sg.add_edge("perceive", "require_cloud")
sg.add_edge("require_cloud", "compute_grasp")
sg.set_on_error("not_found")
```

Use a raising guard only for genuine mid-flow preconditions. For *postconditions* — the end-state promise of the subgraph — leave them to the `checkpoint_agent` follow-on pass; do not declare them here.

If the chosen skill's contract has `streaming: true`, the node invoking it must declare `streaming=True` and have **no outgoing edges** — it's a pure source. Other nodes consume its latest snapshot via `Ref("<this_node_name>")`. A streaming node must still be declared as an edge target from `START` (or another super-step source) so the runtime spawns it.

If an ad-hoc Python step is needed that no canonical script covers, call `request_inline_script(name, signature, purpose, body_hint)` — the coder subagent will emit the script and return its path. Reference the returned path in a `type="script"` node.

## Output format

A single ` ```python` fenced block (no file path) that builds the
subgraph by assigning to a module-level variable named ``sg``:

```python
from vos.builder import Subgraph, Ref, START, END

sg = Subgraph(name="<the subgraph name from your spec>", skill="<the chosen skill name>")

# Declare cross-subgraph inputs (from your spec):
sg.add_input("target_obb", proto_type="OrientedBoundingBox")

# Add the nodes that make up the state machine:
sg.add_node("observe", type="tool", tool="observation.GetObservation")
sg.add_node("perceive", type="script", script="scripts/<sg>/perceive.py",
            inputs={"cameras": Ref("observe.cameras"), "object": "{{target_full}}"})

# Declare success markers — these create `noop` nodes whose names equal
# the exit value. They MUST appear in the edge list with an edge to END.
sg.add_exit("found")

# Wire the edges.
sg.add_edge(START, "observe")
sg.add_edge("observe", "perceive")
sg.add_edge("perceive", "found")
sg.add_edge("found", END)

# Bind subgraph-level outputs declared by your spec, if any.
sg.set_outputs(target_obb=Ref("perceive.obb"), target_mask=Ref("perceive.mask"))

# Declare the failure-path exit symbol.
sg.set_on_error("not_found")

# Do NOT call sg.add_checkpoint(...) — the checkpoint_agent runs after
# you and authors all postconditions for the whole workflow at once.
```

The pipeline imports `vos.builder` (already available; do not pip
install), executes the block in a sandbox, picks up the module-level
`sg` variable, runs the v3 structural validator (S1–S11) against it,
and serializes to JSON.

### Hard rules for the Python block

1. The block MUST end with a module-level variable named ``sg`` bound to
   a ``Subgraph`` instance. Anything else (including stray top-level
   ``print`` calls or ``Workflow`` instances) is rejected.
2. Imports: ``from vos.builder import Subgraph, Ref, START, END`` is
   provided in the sandbox — you may re-import it (idempotent) but no
   other imports are needed.
3. No I/O: do not open files, call ``requests``, spawn threads, or
   import packages beyond ``vos.builder``. Imports outside the allow
   list are rejected.
4. No mutation of nodes/edges after they're added (the builder has no
   `remove`/`rename`/`replace` — re-author from scratch instead).

### Inline-script blocks (unchanged)

Optional ` ```python:scripts/<sg>/<file>.py` blocks for inline scripts —
**only for paths whose stem is NOT in the chosen skill's "Canonical
scripts" table above**. If a `type="script"` node points to
`scripts/<sg>/foo.py` and `foo` is a canonical-script stem, the bundle's
canonical implementation is materialized into the workflow directory
automatically; emitting your own ``` ```python:scripts/<sg>/foo.py``` ```
block **overrides** the canonical with whatever Python you write, which
is the leading cause of correctness regressions in this pipeline
(wrong proto imports, simplified math, dropped parameters).
Re-emit a canonical-stem script only after calling
`request_inline_script` to get explicit approval — and prefer to leave
the canonical alone.

Inline-script blocks are distinguished from the subgraph-builder block
by their fence info: ``` ```python:scripts/... ``` (with a path) is an
inline script, ``` ```python ``` (no path) is the subgraph builder.

## Patterns

**Linear pipeline** (most common): only the success exit is a node;
the failure exit lives in `on_error` and is NOT a node.

```python
sg.add_node("a", type="tool", tool="...")
sg.add_node("b", type="script", script="scripts/<sg>/b.py", inputs={...})
sg.add_node("c", type="tool", tool="...")
sg.add_exit("found")
for u, v in [(START, "a"), ("a", "b"), ("b", "c"), ("c", "found"), ("found", END)]:
    sg.add_edge(u, v)
sg.set_on_error("not_found")
```

**Streaming side-car** (when one of your nodes is a streaming-skill
producer that other nodes need to read continuously):

```python
sg.add_node("tracker", type="tool", tool="track_object", streaming=True,
            inputs={...})
sg.add_node("consumer", type="tool", tool="...",
            inputs={"pose": Ref("tracker")})
sg.add_exit("done")
sg.add_edge(START, "tracker")    # spawned; never blocks downstream
sg.add_edge(START, "consumer")
sg.add_edge("consumer", "done")
sg.add_edge("done", END)
sg.set_on_error("failed")
```

**Conditional branch** (router_field on a non-router source). Both
mapping targets must be **declared nodes** — never `on_error`:

```python
sg.add_node("branch", type="tool", tool="...", inputs={...})   # emits a routing field
sg.add_node("retry_step", type="tool", tool="...", inputs={...})
sg.add_exit("alt_done")
sg.add_exit("done")
sg.add_conditional_edges("branch",
    {"true": "done", "false": "retry_step"}, router_field="ok")
sg.add_edge("retry_step", "alt_done")
sg.add_edge("done", END)
sg.add_edge("alt_done", END)
sg.set_on_error("failed")
```

If the false branch should bail to the failure exit, raise instead
(see the raising-guard pattern above) — do NOT make `on_error` a mapping
target. (Postconditions go in `add_checkpoint`, not in a routing branch
or a node that re-checks-and-raises.)

**Send (dynamic fan-out)**: declare a `type="router"` node whose script
returns `[{"to": "process", "inputs": {"item": x}}, ...]`. Each Send
spawns one copy; outputs collect as a list under the router's name.

## What changes vs. legacy

In the legacy system there was a separate Python class per subagent
(perception_multi, grasp_curobo, motion, …). In MORSL, **you are
universal**: the per-skill node-flow guidance comes from the chosen
skill's SKILL.md, not from a domain-specific subagent class. Read the
SKILL.md body (it's in your context) — that's where the recommended
node flow, hard rules, and exit-value mappings live.

## Discovery tools

- `read_skill_reference(skill_name, doc_name)` — load a long-form
  reference doc bundled with the skill. Use this when the SKILL.md
  body's "See also" links the doc and you want the deeper rationale.
- `read_skill_example(skill_name, example_name)` — load a sample
  subgraph the bundle ships, useful as a starting point. (Examples
  may still be in JSON form; convert them to `vos.builder` calls when
  you reuse them.)
- `report_missing_capability(name, why)` — flag a gap; the build
  surfaces it instead of producing broken output.
- `request_inline_script(name, signature, purpose, body_hint)` —
  delegate ad-hoc Python to the coder subagent.

## Postconditions

Every subgraph MUST declare **≥ 1 `validate=True` checkpoint** (with a
non-empty `rationale`) via
`sg.add_checkpoint(name, predicate, *, diagnostics=None, rationale="",
validate=True, weight=1.0)`. **The parser rejects a subgraph with zero
`validate=True` checkpoints and re-prompts you.** Keep the total ≤ 6.
The rehearsal harness evaluates each predicate against a privileged-state
`World` snapshot at subgraph exit and feeds the results back into the
next iter's prompt — without checkpoints the refine loop has no
localized signal and cannot improve anything beyond the binary terminal
verdict.

Granularity is **subgraph-or-coarser**. A typical subgraph declares
ONE `validate=True` checkpoint (the postcondition the subgraph promised
to satisfy — e.g. `target_held`, `ee_above_target`, `target_in_container`)
plus 0-3 `validate=False` probes (`object_lifted`, `gripper_open_fraction`,
etc.) that add diagnostic richness without blocking downstream evaluation.

A `validate=True` checkpoint is also **where runtime postcondition
verification lives**: never add a node that re-checks the subgraph's end
state and raises — declare the postcondition as a `validate=True`
checkpoint on the subgraph that produced the state.

Anchor predicates to privileged quantities only — positions, contacts,
joint state, AABBs, cavity bounds. NEVER reference camera-derived
features (mask IoU, detection confidence); those are what we're trying
to validate, so using them in the predicate makes the checkpoint
circular. Use **scene-spec object ids** as literal name strings — the
`id` from `scene_spec.json` (e.g. `w.body("alphabet soup")`) is the
canonical name that flows through perception inputs, the rehearsal
`World`, and these checkpoint predicates. Never use the original noun
phrase, a free abbreviation, `{{...}}`, or `Ref(...)`.

**Important**: the `description` field of your subgraph spec may use a
paraphrased name (the coordinator should match scene-spec ids verbatim,
but treat that as best-effort context). The scene-spec ids list in your
context is the ground truth for body names. When you write a
`w.body("...")` literal in a checkpoint predicate, **pick the exact spec
id from the scene-spec context, not the word from the description**.
For example, if the description says *"Place the soup can in the basket"*
but the spec ids are `{"alphabet soup", "basket"}`, the predicate MUST
be `w.body("alphabet soup").is_in(w.body("basket"))` — using `"soup can"`
would raise `BodyNotFoundError` at runtime and the checkpoint would
silently fail.

If the chosen skill's SKILL.md has a `## Checkpoints` section, start
from the canonical checkpoint(s) it lists for that skill kind. For the
full reference — `Checkpoint` semantics, the `World` API surface
(`world.body(...)`, `body.is_grasped()`, `body.is_in(...)`, temporal
helpers, history), `validate`-vs-probe rules, and worked predicate
examples for each subgraph kind — see the loaded `_checkpoints_api.md`
include.

The checkpoints are emitted as part of the same Python builder block
that builds the subgraph; they are NOT serialized into `workflow.json`
and they do NOT change the state machine. The pipeline writes them to a
sidecar `<workflow_dir>/checkpoints/<sg>.py` module that the rehearsal
harness loads at refine time.

# Workflow v3 spec — shared core

This is the structural spec every codegen subagent shares. It defines the
top-level workflow shape, the SubgraphDef shape, node types, edge
semantics, `$ref` syntax, and validation rules.

## Top-level workflow

```json
{
  "version": 3,
  "meta": { "name": "...", "description": "..." },
  "nodes": {
    "<sg_node_name>": { "type": "subgraph", "ref": "<sg_def_name>" },
    "done":  { "type": "end", "status": "success" },
    "abort": { "type": "end", "status": "failure",
               "recovery": [ { "service": "...", "method": "...", "inputs": {} } ] }
  },
  "edges": [["START", "<first_subgraph_node>"]],
  "conditional_edges": {
    "<sg_node_name>": {
      "router_field": "exit",
      "mapping": { "<exit_value>": "<next_node>", ... }
    }
  },
  "subgraphs": { "<sg_def_name>": { /* SubgraphDef */ } }
}
```

`START` and `END` are virtual node names. The top level orchestrates
subgraph nodes and end nodes; cross-subgraph routing is via the
`conditional_edges` block reading each subgraph's `exit` value.

`meta` is an open string→string map; recognized keys include `name`,
`description`, `runtime` (`direct_real`), `observation_stream_hz`, and
`validate_checkpoints` (`"true"` opts the executor into enforcing each
subgraph's `validate=True` postcondition checkpoints at real-execution
time — **currently a no-op**; reserved seam).

## SubgraphDef shape

```json
{
  "skill":  "<MORSL skill name; echoed from your spec>",
  "inputs":  { "<name>": "<proto.type.string>" },
  "outputs": { "<name>": { "$ref": "<node>.<field>" } },
  "nodes": {
    "<node_name>": { /* NodeDef */ },
    "<terminal_marker>": { "type": "noop" }
  },
  "edges": [
    ["START", "<first_node>"],
    ["<first_node>", "<second_node>"],
    ["<terminal_marker>", "END"]
  ],
  "conditional_edges": { },
  "exit": { "router_field": null, "success_values": ["<success_exit>"] },
  "on_error": "<failure_exit_value>"
}
```

Scripts go in separate fenced blocks, namespaced under
`scripts/<subgraph_name>/`:

````python:scripts/<subgraph_name>/<script>.py
...
````

## Node types

The two production dispatch types are `tool` and `script`. `noop` and
`router` are control-flow markers. `subgraph` and `end` only appear at
the top level of the workflow (not inside subgraphs).

```json
{ "type": "tool", "tool": "gripper.Open",
  "inputs": { "settle_steps": 40 } }                       /* gRPC method */

{ "type": "tool", "tool": "sam3.SegmentBox",
  "inputs": { "image": { "$ref": "obs.rgb" }, "box": { "$ref": "perceive.box" } } }

{ "type": "tool", "tool": "run_policy",
  "inputs": { "observation_stream": { "$ref": "in.observation_stream" } } }  /* atomic MORSL skill */

{ "type": "tool", "tool": "libero_pi05",
  "inputs": { "prompt": "...", "max_windows": 25 } }       /* learned policy */

{ "type": "tool", "tool": "track_object", "streaming": true,
  "inputs": { "observation_stream": { "$ref": "in.observation_stream" } } }

{ "type": "script", "script": "scripts/<subgraph_name>/foo.py",
  "inputs": { ... } }                                      /* canonical bundle script
                                                              or rare inline helper */

{ "type": "noop" }                                          /* named terminal marker */
{ "type": "router", "script": "scripts/<sg>/route.py", "inputs": { ... } }
```

- **`tool`** — the canonical dispatch for **anything**: gRPC service
  methods (auto-registered as `<package>.<MethodName>`, e.g.
  `observation.GetObservation`, `gripper.Open`,
  `geometry_svc.FilterAndComputeOBB`), MORSL skills (registered by skill
  name, e.g. `run_policy`, `track_object`), and learned policies
  (registered by policy name). `tool:` is always a flat name; never write
  `<package>.v1.<Service>` or include a separate `method:` field. The
  runtime resolves the proto FQN internally.
- **`script`** — local Python file in the workflow folder. Prefer to
  point at a canonical bundle script (listed in the chosen skill's
  "Canonical scripts" table); only emit your own inline Python for
  ad-hoc helpers that no canonical and no tool covers.
- **`noop`** — empty body. Used as a named subgraph terminal so the node
  name becomes the subgraph's exit value when `exit.router_field=null`.
- **`router`** — Send dispatch. Function returns either a string target
  for static routing or a list of `{"to": "...", "inputs": {...}}` dicts
  for dynamic fan-out.

*Legacy node types (`type: service`, `type: skill`, `type: policy`)
have been retired. The validator rejects them with a precise migration
message. Always emit `type: tool` with the flat name instead.*

`streaming: true` is only valid on `tool` and `script` nodes. A
streaming node has **no outgoing edges** — it's a pure source. Consumers
read its latest published value via `{"$ref": "<node>"}`. The chosen
tool's bundle contract must declare `contract.streaming: true`.

## Edge semantics

- **Static edges**: `["src", "dst"]` pairs.
- **Multiple outgoing edges from one node = parallel super-step.**
- **`conditional_edges`** dispatches on a router field. From a
  non-router source the field is on the source's output (set
  `router_field` to its name). From a router source set
  `router_field: null`.
- **`exit.router_field: null`** means the subgraph's exit value is the
  name of the terminal node (the one whose edge points to `END`). For
  this to work, declare your terminals as `noop` nodes named after the
  exit values (e.g. `"found": {"type": "noop"}` with edge
  `["found", "END"]`).
- **`exit.router_field: "<field>"`** reads the field on the terminal
  node's output as the exit value.
- **`on_error`** is an optional subgraph-level catch: when any node
  raises, the runtime emits this string as the subgraph's exit value
  (bypassing the terminal-node read). The `on_error` symbol is the
  subgraph's single failure exit. It is **never** a declared node and
  **never** a `conditional_edges` mapping target — failures surface
  only by raising.

## Reference syntax

| Form | Meaning |
|---|---|
| `{"$ref": "observe"}` | Full output of the `observe` node. |
| `{"$ref": "observe.cameras"}` | Nested field walk on the output. |
| `{"$ref": "tracker"}` | Snapshot of the latest published value of the streaming `tracker` node. |
| `{"$ref": "in.<name>"}` | Cross-subgraph input from your declared `inputs` schema. |

## Validation rules

Your subgraph fails validation if:

1. Any node referenced by an edge or conditional-edge mapping is not
   declared in `nodes` (and is not `START`/`END`).
2. Any non-`END` node is unreachable from `START`.
3. Any non-streaming, non-end node has no outgoing edge or
   conditional-edges entry.
4. A streaming node has any outgoing edge or conditional-edges entry.
5. A node with `streaming: true` invokes a skill whose contract has
   `streaming: false` (or vice versa) — when the skill registry is
   available.
6. `conditional_edges` from a non-router source omits `router_field`.
7. `conditional_edges` from a router source sets `router_field` to
   non-null (router scripts return the target directly).
8. `outputs` binding references an unknown node or an end node.
9. `exit.success_values` is empty (S7).
10. `on_error` collides with a declared node (S9) or appears as a
    `conditional_edges` mapping target (S10).
11. With `router_field: null`, any name in `exit.success_values` is
    not declared as a `noop` node; OR with `router_field` set, any
    name in `exit.success_values` collides with a node name (S11).
12. Any `{"$ref": "in.<name>"}` references an input name not declared
    in your `inputs` schema (the executor-injected
    `observation_stream` is exempt).
13. `inputs.<name>` declared on a reachable subgraph has no upstream
    producer subgraph that declares an output of the same name.

Errors are fed back; fix every error and re-emit the full subgraph JSON.

## v2 → v3 mapping (for migration)

| v2 concept | v3 replacement |
|---|---|
| `begin: <state_name>` | `["START", node]` edge |
| EndState marker (`{"type": "end"}` inside states) | `noop` node + edge to `END` |
| `on_success` | static edge to next node |
| `on_failure` | subgraph-level `on_error: "<exit>"` catch (or per-node `conditional_edges`) |
| `parallel` state with `branches` | multiple outgoing edges from one node |
| `join_policy: first_success` (race) | NOT in graph — long-running concurrent participants become `streaming: true` skill nodes (consumed via `$ref` snapshots) |
| `transitions: { exit_cond: target_sg }` | top-level `conditional_edges` on the subgraph node, `router_field: "exit"` |
| EndSubgraph | top-level `end`-type node with `status` and `recovery` |

## Type mapping

| Python | Proto |
|---|---|
| `Vec3`, `Se3Pose`, `Quaternion` | `common.Vec3`, etc. |
| `list[Se3Pose]` | `repeated common.Se3Pose` |
| `PointCloud \| None` | optional `common.PointCloud` |
| `float` | `double` |
| `int` | `int64` |

Proto imports — the two `common` modules split as follows.
**`OrientedBoundingBox` lives in `geometry_pb2`, not `sensor_pb2`** —
mis-importing it from `sensor_pb2` is the leading cause of script-load
`ImportError`s in this codebase:

```python
# vos.proto.common.geometry_pb2 — pure geometry types
from vos.proto.common.geometry_pb2 import (
    Vec3, Quaternion, Se3Pose, OrientedBoundingBox,
)

# vos.proto.common.sensor_pb2 — sensor / perception payload types
from vos.proto.common.sensor_pb2 import CameraObservation, Mask, PointCloud

# Per-service request/response messages
from vos.proto.observation.v1 import observation_pb2
from vos.proto.geometry_svc.v1 import geometry_svc_pb2
from vos.proto.curobo.v1 import curobo_pb2
from vos.proto.robot_control.v1 import robot_control_pb2
from vos.proto.gripper.v1 import gripper_pb2
```

## Proto field reference

Exact field names — these trip up LLMs:

| Message | Fields |
|---|---|
| `Vec3` | `x`, `y`, `z` (all `double`) |
| `Quaternion` | `w`, `x`, `y`, `z` (WXYZ convention). Top-down gripper is `Quaternion(w=0, x=1, y=0, z=0)`. |
| `Se3Pose` | `position: Vec3`, **`rotation: Quaternion`** (NOT `orientation`) |
| `OrientedBoundingBox` | `center: Vec3`, `extent: Vec3` (half-extents), **`orientation: Quaternion`** (NOT `rotation`) |

Note the asymmetry: `Se3Pose.rotation` vs `OrientedBoundingBox.orientation`.


## Skill in scope: `perception_single`

# perception_single

Single-path perception: detect → disambiguate → segment → fuse to 3D →
extract OBB. The full pipeline runs once per camera; results across cameras
are merged via KD-tree intersection inside `perceive_dino_vlm.py`.

## When to use

- Uncluttered scenes with visually distinct targets.
- Platforms where only DINO + VLM + SAM3 + Geometry are deployed.
- When `perception_multi` is not in the available skill catalog.

## When NOT to use

- Cluttered scenes with similar nearby distractors. Prefer `perception_multi`.

## Recommended subgraph state flow

3 states:

```text
observe → perceive → filter_obb
```

State details:

> **About `object_name` below:** it is a literal Python string — the natural
> noun phrase for the object you are perceiving, drawn from this subgraph's
> description (e.g. `"alphabet soup can"`, `"basket"`, `"red bowl"`). It is
> a constant per subgraph instance, NOT a binding. **DO NOT** write
> `Ref("in.object_name")` or any other `Ref(...)`; the coordinator does
> not declare `object_name` as a subgraph input. Write the string directly,
> e.g. `"object_name": "basket"`.

1. **`observe`** — `type: tool`, `tool: "observation.GetObservation"`,
   `inputs: {}`. Auto-registered gRPC method; flat name only.
2. **`perceive`** — `type: script`, file `scripts/<sg>/perceive_dino_vlm.py`
   from this bundle. Inputs:
   `cameras=Ref("observe.cameras")`,
   `object_name="basket"` (replace with the actual target noun phrase
   from this subgraph's description),
   plus any optional fields (`object_description`, `dino_prompt`, etc.).
   Returns `{found, cloud, mask, score}`.
3. **`filter_obb`** — `type: tool`,
   `tool: "geometry_svc.FilterAndComputeOBB"`,
   `inputs={"point_cloud": Ref("perceive.cloud")}`. Returns a bare
   `OrientedBoundingBox`.

### Wiring the exit (HARD)

Use the linear edge `filter_obb → found → END`. The `perceive` script
already raises if the target isn't found, so the subgraph's
`on_error: "not_found"` catches that path automatically. Do **NOT**
add any conditional edges on `perceive` — the linear path plus
`set_on_error` is sufficient.

✅ Correct (the literal `vos.builder` calls you should emit):

```python
sg.add_node("filter_obb", type="tool",
            tool="geometry_svc.FilterAndComputeOBB",
            inputs={"point_cloud": Ref("perceive.cloud")})

# add_exit() creates the success-marker noop node AND registers the
# exit value. Do NOT also call sg.add_node("found", type="noop") — that
# would conflict with the node add_exit created.
sg.add_exit("found")

sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)

sg.set_on_error("not_found")
```

Bind the subgraph outputs (ALL THREE — required, no exceptions):

```python
sg.set_outputs(
    target_obb=Ref("filter_obb"),
    target_mask=Ref("perceive.mask"),
    target_cloud=Ref("perceive.cloud"),
)
```

(Replace `target_*` with this subgraph's actual name prefix — e.g.
`container_obb`, `container_mask`, `container_cloud` when authoring
the container subgraph.)

Note the asymmetry: `<name>_obb` references `filter_obb` with no
trailing field (FilterAndComputeOBB returns a bare OBB), while
`<name>_mask` and `<name>_cloud` walk into fields of `perceive`'s
output dict. `perceive_dino_vlm.py` already produces all three;
emitting them unconditionally lets downstream subgraphs that need any
of them (e.g. `grasp_moe` requires `<name>_cloud`) wire up without you
having to anticipate which skill they'll use.

## Hard rules

1. Subgraph-level outputs MUST emit ALL THREE: `<name>_obb`,
   `<name>_mask`, AND `<name>_cloud`. The cloud is the fused world-frame
   point cloud needed by learned-grasp skills (e.g. `grasp_moe`); emit
   it unconditionally so the downstream agent can wire it without
   round-tripping. See `references/perception_pipeline_invariants.md`.
2. `geometry_svc.FilterAndComputeOBB` returns a bare `OrientedBoundingBox`;
   bind via `Ref("filter_obb")` (no trailing field).
   See `references/geometry_calling_conventions.md`.

## Required end states

| End state | Meaning |
|---|---|
| `found` | OBB + mask bound; route to next subgraph (typically `grasp_*`). |
| `not_found` | Route to `abort` (or to `done` in clean-all-items loops). |


## See also

- `references/single_vs_multi.md` — the choice between single- and multi-
  method perception.
- `prompts/vlm_select_box.md` — the inline VLM prompt template.
- `scripts/perceive_dino_vlm.py` — the canonical perception script.

### Canonical scripts (you may emit as `type: script` states)

| Logical name | Path | Inputs | Outputs |
|--------------|------|--------|---------|
| `perceive_dino_vlm` | `scripts/perceive_dino_vlm.py` | cameras: list[CameraObservation], object_name: str, text_prompts: list[str] | None, min_points: int, min_score: float, use_multiview: bool, box_threshold: float, text_threshold: float, dino_prompt: str, object_description: str | found: bool, cloud: PointCloud, mask: Mask, score: float |

## Tools available in this subgraph

| Tool | Transport | Summary |
|------|-----------|---------|
| `geometry_svc.FilterAndComputeOBB` | grpc | Geometry.FilterAndComputeOBB :: FilterAndComputeOBBRequest → OrientedBoundingBox |
| `geometry_svc.MaskToWorldPoints` | grpc | Geometry.MaskToWorldPoints :: MaskToWorldRequest → PointCloud |
| `grounding_dino.Detect` | grpc | GroundingDino.Detect :: GroundingDinoDetectRequest → GroundingDinoDetectResponse |
| `observation.GetObservation` | grpc | Observation.GetObservation :: Empty → ObservationResponse |
| `sam3.SegmentBox` | grpc | Sam3.SegmentBox :: Sam3SegmentBoxRequest → SegmentResponse |
| `sam3.SegmentText` | grpc | Sam3.SegmentText :: Sam3SegmentTextRequest → SegmentResponse |
| `vlm.Query` | grpc | Vlm.Query :: VlmQueryRequest → VlmQueryResponse |

## Your subgraph: `container_sg`

Locate the target location

### Required outputs
Bind these from internal state fields via `{"$ref": "<state>.<field>"}` (or `{"$ref": "<state>"}` for bare-message returns):

| Name | Type |
|------|------|
| container_obb | `OrientedBoundingBox` |
| container_mask | `Mask` |
| container_cloud | `PointCloud` |

### Required end states
Your subgraph must contain exactly these end states. Each is a `{"type": "end"}` marker reached via some `on_success`/`on_failure` chain.

| End state | Meaning |
|-----------|---------|
| `found` | Target detected; OBB and mask bound in subgraph outputs. |
| `not_found` | Target not visible in any view. In clean-all-items loops route to done; in normal pick-and-place, route to abort. |

### Upstream outputs (already produced by earlier subgraphs)
| Subgraph | Output | Type |
|----------|--------|------|
| target_sg | target_obb | `OrientedBoundingBox` |
| target_sg | target_mask | `Mask` |
| target_sg | target_cloud | `PointCloud` |

Appendix G Sample Generation Outputs

The final example implements container perception as observe → canonical perception script → cloud filtering/OBB → found, with a not_found error exit and bindings for box, mask and cloud. It uses the literal object name "container" and emits no checkpoint in the shown block. This illustrates the intended typed dataflow but does not resolve the preceding prompt’s contradictory checkpoint requirements.

    ```python
from vos.builder import Subgraph, Ref, START, END

sg = Subgraph(name="container_sg", skill="perception_single")

# Add nodes for the perception pipeline
sg.add_node("observe", type="tool", tool="observation.GetObservation")

sg.add_node("perceive", type="script",
            script="scripts/perceive_dino_vlm.py",
            inputs={
                "cameras": Ref("observe.cameras"),
                "object_name": "container"
            })

sg.add_node("filter_obb", type="tool",
            tool="geometry_svc.FilterAndComputeOBB",
            inputs={"point_cloud": Ref("perceive.cloud")})

# Success marker
sg.add_exit("found")

# Wiring
sg.add_edge(START, "observe")
sg.add_edge("observe", "perceive")
sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)

# Bind outputs
sg.set_outputs(
    container_obb=Ref("filter_obb"),
    container_mask=Ref("perceive.mask"),
    container_cloud=Ref("perceive.cloud")
)

# Failure path
sg.set_on_error("not_found")
```
Concept demonstration

Open image ↗