Overview
Contents of Appendix
At a glance
AGP makes a general-purpose agent the robot's policy: it observes the scene, writes and runs programs, requests motions, and revises its plan using physical feedback. Model weights remain fixed; a documented robot interface supplies cameras, state, and motion control.
Across eight task configurations, success reaches at least 80% in seven. Reusing saved scripts and experience improves efficiency in small repeated-task studies. Execution remains slow and costly, failures persist, and the results concern specific tabletop tasks rather than unrestricted autonomous manipulation. This edition follows revision v3, posted September 28, 2026.
1 Introduction
Existing agents often generate a program before execution or select among pretrained robot skills [17] [6] [14]. AGP instead keeps the agent making decisions throughout execution, including when to observe, how to interpret evidence, which program to write, and how to recover.
The contribution is a runtime control arrangement with frozen model weights. A conventional robot controller still converts requested targets into physical motion; the agent supplies task-level reasoning, geometric computation, and adaptive action selection.
2 Related Work
2.1 Agentic Robot Systems
Code-generating systems construct and refine robot programs, while policy-orchestration systems select learned skills [22] [19] [1]. AGP gives the runtime agent responsibility for perception, motion requests, and recovery without a separately trained action policy. ASPIRE provides a useful comparison for reusable program repair [18].
2.2 Robot Manipulation
Classical task-and-motion planning uses explicit geometry [9]; diffusion policies and VLA models learn action generation from robot trajectories [7] [13]. AGP uses a pretrained MLLM agent to make control decisions through tools, complementing those approaches without updating its weights for the evaluated tasks.
2.3 Agentic Workflows Beyond Robotics
AGP brings reasoning, tool use, reflection, and local workspaces from language and coding agents into physical manipulation [28] [21] [27]. Its distinction is where these capabilities operate: they directly govern the robot's observation, geometry, planning, and control loop.
3 A General Purpose Agent as the Robot Policy
3.1 Problem Formulation
A preparation agent packages instructions, demonstration videos or goal images, interface documentation, budgets, and success criteria into a task definition. An execution agent then chooses tools using this definition, interaction history, and a workspace of scripts and observations. Equation 1 describes that decision rule with fixed model parameters.
Standard evaluation starts without task experience. Separate reuse experiments provide saved procedures; those conditions should be distinguished from the initial zero-shot trials.
Equation 1
| (1) |
3.2 Agent-Robot Bridge
A Python client exchanges requests and results with a server through workspace JSON files. The server provides camera images, calibrated depth, robot state, and motion feedback. The agent requests joint targets, gripper poses, or opening commands; Mink solves inverse kinematics for Cartesian requests [30].
A separate controller enforces motion limits and follows trajectories independently of model inference. Each arm queues its motions, while two arms can operate concurrently. The robot holds position between requests, so camera frame rate and low-level control rate do not imply a fast agent decision loop.
3.3 Runtime Programming and Execution
The agent can generate Python programs to process images, estimate geometry, compose robot calls, or inspect demonstrations. New observations and execution errors drive revisions to grasps, alignment, and strategy. Instructions encourage grouping operations and waiting for completion to reduce redundant tool calls.
Task completion is judged from physical outcomes and recorded evidence, not solely from the agent's report. Appendix E preserves example programs; Appendix F documents unsuccessful recoveries.
4 Experiments
The evaluation spans precision assembly, block structures, dice reorientation, dynamic throwing, and deformable-object manipulation on physical robots.
4.1 Experimental Setup
4.1.1 Robot and Agent
The main platform is a six-joint I2RT YAM arm with a parallel gripper; towel folding uses two arms. Wrist RGB-D and overhead RGB cameras provide observations on demand. Experiments compare Codex and Claude Code configurations with frozen weights; detailed model and effort settings remain in Table 2.
4.1.2 Tasks and Evaluation
Five task groups yield eight configurations: four-pair assembly; pyramid, two-tower, and six-block construction; dice flipping; targeted throwing; and sequential or simultaneous towel folding. Goals come from videos, images, or language.
Trials have fixed time and request budgets, with autonomous recovery allowed. Success requires predefined physical outcomes. Most time, token, and cost means include successful trials only. Throwing instead includes all ten sessions and final reporting; Appendix B specifies the accounting.
| Task | Capability | Instruction | Success | Time | Tokens | Cost | ||||||||
| (min) | (million) | (USD) | ||||||||||||
| Four pair assembly | 8/10 |
|
|
| ||||||||||
|
10/10 |
|
|
| ||||||||||
| Two towers | 10/10 |
|
|
| ||||||||||
| Six block tower | 9/10 |
|
|
| ||||||||||
| Dice flipping | 10/10 |
|
|
| ||||||||||
|
9/10 |
|
|
| ||||||||||
|
5/5 |
|
|
| ||||||||||
| Simultaneous | 3/5 |
|
|
| ||||||||||
4.2 Experimental Results
4.2.1 Main Results
AGP succeeds in 8/10 four-pair assemblies, 10/10 pyramids, 10/10 two-tower builds, 9/10 six-block towers, 10/10 dice tasks, 9/10 throws, 5/5 sequential folds, and 3/5 simultaneous folds. These are small task-specific samples.
Successful sequential folding averages 50.8 minutes and USD 24.14; four-pair assembly averages 37.2 minutes. Strong completion rates therefore coexist with substantial inference and execution overhead. Simultaneous folding remains the weakest configuration at 60%.
4.2.2 Agent and Thinking Effort Comparison
On two-pair assembly, Astra succeeds in 5/5 trials at each tested effort level. Sol and Opus also reach 5/5, while Terra reaches 1/5, Luna 0/5, and Fable 3/5. Sol has lower estimated inference cost than Astra but takes longer.
These comparisons concern complete agent configurations under shared task conditions. Successful-trial resource means condition on different numbers of successes and do not establish a general ranking across tasks.
4.2.3 Baseline Comparison
Using Astra at high effort, AGP succeeds in 23/25 trials across two-pair assembly, four-pair assembly, and pyramid construction, versus 0/25 for GaP and 3/25 for ASPIRE [6] [18]. The baselines receive matched simulation environments for program construction and refinement; AGP adapts online on the physical robot.
ASPIRE's successful frozen-program execution can be faster. Its reported total cost and time charge the entire shared preparation overhead to each successful run, without amortization. These figures answer a different question from repeated replay efficiency. Simulation-to-real mismatch is a proposed explanation for failures, not an isolated causal result.
Model Success Time Tokens Cost (min) (million) (USD) Codex GPT-6 Astra Low effort 5/5 9.9 [6.0, 19.0] 3.69 [1.44, 7.72] 4.79 [2.15, 9.56] Medium effort 5/5 9.2 [6.3, 15.4] 3.03 [1.84, 5.80] 4.09 [2.76, 7.48] High effort 5/5 9.2 [7.9, 10.6] 3.30 [2.58, 4.02] 4.47 [3.72, 5.19] GPT-5.6 Sol 5/5 14.2 [9.1, 19.3] 7.13 [4.12, 9.72] 3.94 [2.40, 5.30] GPT-5.6 Terra 1/5 19.0 [19.0, 19.0] 6.28 [6.28, 6.28] 1.98 [1.98, 1.98] GPT-5.6 Luna 0/5 – – – . Claude Code Claude Opus 5 5/5 22.2 [14.4, 31.9] 12.40 [9.05, 16.73] 9.75 [6.94, 13.03] Claude Fable 5.1 3/5 27.7 [18.3, 39.8] 9.27 [6.87, 11.10] 11.30 [8.33, 14.36]
Task Method Success Time Cost (min) (USD) Two pair assembly GaP 0/5 – – ASPIRE 2/5 2.9 / 64.8 0.00 / 18.90 AGP 5/5 9.2 4.47 Four pair assembly GaP 0/10 – – ASPIRE 0/10 – – AGP 8/10 37.2 16.62 Pyramid construction GaP 0/10 – – ASPIRE 1/10 9.2 / 271.7 0.00 / 78.13 AGP 10/10 21.6 11.69
4.3 Improving Efficiency through Experience
Astra repeats two assembly tasks for five cycles, saving programs, measurements, and corrections after each run. Later runs refresh object positions and reuse earlier procedures. Runtime falls across the sequence, mainly through reduced reasoning and programming time.
Giving Terra the same frozen experience from Astra changes two-pair assembly success from 1/5 to 4/5. Among successful trials, mean time falls from 19.0 to 12.5 minutes and tokens from 6.28M to 5.56M. The resource comparison uses one successful baseline trial versus four with experience; the ordered reuse study also lacks an empty-experience control at each cycle.
4.4 Case Study of Failure Recovery
A recovery trace shows the agent changing its pickup tilt, checking whether a tube stands after release, inspecting pin alignment, and revising insertion motions. The complete 48.6-minute session spends 68.2% of its measured time in planning/reasoning and 23.5% in robot actions. The figure links observations, programs, and requests to these decisions.
5 Conclusion
AGP demonstrates that a general-purpose runtime agent can produce manipulation behavior by combining programs with physical feedback. Saved experience offers a route to reducing overhead without weight updates.
Limits include small trial counts, restricted task and hardware coverage, slow execution, costly inference, unsuccessful recovery, and imperfect final verification. Cross-embodiment evidence is qualitative, and baseline resource comparisons depend on preparation accounting.
AI Use Statement
The authors report using generative AI for drafting and language editing and take responsibility for the manuscript's methods, results, and interpretations.
Reproducibility Statement
The appendices document interfaces, task criteria, model settings, timing boundaries, program traces, and qualitative executions. Public project materials, code, and the source paper provide further reproducibility information.
Appendix A Implementation Details
A.1 Robot Platform and Control Interface
The platform connects calibrated observations to bounded motion requests through a separate robot runtime.
Robot platform.
The YAM arm uses a parallel gripper with 95.5 mm nominal opening and a 1 Mbit/s CAN connection. A calibrated rigid transform aligns the two arms' Cartesian frames.
Observations and geometric grounding.
Wrist depth and camera calibration map pixels into robot coordinates. Overhead pixels intersect rays with an estimated-height plane. Additional views can refine geometry; the 30-frame/s camera streams are independent of the agent's decision rate.
Action interface.
Seven commands expose state, status, help, frame capture, Cartesian motion, joint motion, and gripper control. Ordinary requests enforce joint velocity/acceleration limits of 20°/s and 40°/s², translation at 0.03 m/s, orientation at 10°/s, radial targets from 0.12 to 0.65 m, and Cartesian steps no larger than 0.25 m.
Throwing uses a separate timed-motion interface at 50 Hz, with joint 4 limits of 180°/s and 360°/s². This task-specific execution capability is part of the supplied interface.
A.2 Agent Configuration and Execution Conditions
Standard trials start with empty task experience; reuse conditions receive specified saved files. Logs record agent/model versions, hosts, budgets, settings, waiting intervals, and grouped tool operations.
A.3 Model and Reasoning Effort Configurations
The Astra effort sweep fixes the environment, inputs, and robot interface. Cross-provider comparisons use each agent's native settings and therefore compare complete systems rather than an isolated model-weight change.
Appendix B Tasks and Evaluation Protocol
B.1 Task Definitions and Success Criteria
Success criteria are fixed before evaluation and require completed relationships to remain stable after release and arm withdrawal. Controller arrival tolerances and task success tolerances are separate quantities.
| Task | Capability | Instruction type | Task requirements and success criteria |
|---|---|---|---|
| Four pair assembly | Long horizon reasoning | Video | Assemble eight parts into four pairs in the demonstrated order, inferring mating relationships and insertion motions from the video and current scene. Success requires all four pairs to remain assembled, preserving completed pairs throughout subsequent operations. |
| Block construction | Long horizon reasoning | Image | Arrange six cubes into a pyramid, two towers, or a six block tower matching the goal image. Infer object identities, orientations, and support relations, and choose a placement order that maintains intermediate stability. Success requires the complete target structure. |
| Dice flipping | Action diversity | Language | Reorient six randomly placed dice to the requested upward face through grasp selection and rotation. Success requires all six dice to show the requested number after placement. |
| Targeted throwing | Action diversity | Language | Coordinate a circular joint swing with gripper release to throw an object toward the specified target. Success requires the object to separate from the moving hand, travel through the air, and come to rest within the target region. |
| Towel folding | Object diversity | Video | Use both arms to reproduce the demonstrated folds. Fold the near long edge onto the far edge, then fold both short ends inward, either right then left (Sequential) or beginning together (Simultaneous). Success requires both inward folds with reasonably aligned edges. Small wrinkles and creases are permitted, and the final table position is unrestricted. |
Assembly from video.
All methods receive the same video, extracted frames, and timestamped guide for four AutoMate part pairs, assembled in demonstration order [23].
Assembly part dimensions.
Table 5 retains mesh dimensions and mating clearances. Some nominal radial clearances are below 1 mm; actual insertion tolerance also depends on printing accuracy and angular alignment.
| Assembly | Plug overall size | Socket overall size | Hole | Shaft | ||
|---|---|---|---|---|---|---|
|
|
37.41 | 35.86 | 1.56 | 0.78 | ||
|
|
28.18 | 25.80 | 2.38 | 1.19 | ||
|
|
42.86 | 37.20 | 5.66 | 2.83 | ||
|
|
18.01 | 16.54 | 1.47 | 0.73 |
Block construction from images.
Cube poses vary across trials while each configuration retains its goal photograph, block set, and predefined placement tolerances.
Action diversity.
Dice trials vary initial and target faces. Throwing requires a permitted swing, gripper release, airborne motion, and landing in the specified target; ending inside the bowl alone is insufficient.
Bimanual towel folding.
Towel tasks provide demonstration videos and start/end images. Staged placement is allowed for arm clearance. Final assessment checks the demonstrated folds after release, allowing minor wrinkles.
B.2 Trial Protocol
Records distinguish physical failure, exhausted budgets, human intervention, and infrastructure interruption. Local computation and geometric queries are logged separately from budgeted observation and action requests.
B.3 Metric Definitions and Resource Accounting
Success and time.
Success divides fully completed trials by evaluated trials. Resource means normally use successful trials from task delivery through physical verification. No-success configurations have undefined successful-trial means. The broader timing study also includes failures and final reporting.
Interaction accounting.
One agent tool decision can group several robot requests. Interaction budgets count each observation/action request separately, so tool-call counts and robot-request counts differ.
Token usage.
Tokens sum input and output across requests and retries, counting repeated context each time and reasoning once in output. Cost estimates apply the stated API prices and caching discounts; they are study accounting, not current price quotations.
Appendix C Supplementary Coding Agent Comparison on Pyramid Construction
On pyramid construction, Sol completes 7/10 sessions and Astra 10/10. Sol's three unsuccessful sessions include two supervisor interventions and a bridge disconnection before physical execution. The table therefore includes infrastructure and intervention outcomes as failures.
| Model | Successes | Tokens (million) | Cost (USD) | Total (USD) |
|---|---|---|---|---|
| GPT-5.6 Sol | 7/10 | 18.86 | 9.22 | 87.25 |
| GPT-6 Astra | 10/10 | 9.24 | 11.69 | 116.88 |
Usage and pricing.
Cost estimates use model API prices retrieved September 9, 2026. Table 6 separates successful-session means from total cost across all ten sessions.
Interpretation.
Sol uses more tokens per successful pyramid session but has lower estimated inference cost. This does not imply better completion rate or lower elapsed time.
Appendix D Efficiency and Experience Reuse Analysis
D.1 Task Execution Time Analysis
Figure 5 decomposes all trials, including failures and final reporting, for seven configurations. Throwing lacks the required component timings and is excluded. These boundaries differ from most main-table successful-trial means.
Time categories.
Categories cover the visual loop, planning/reasoning, local tools, action service, final checking, and reporting. Overlap counts once. The visual loop includes the model response after an image, which may itself contain planning and code generation.
Observed costs.
Observation and decision-making consume substantial time. Sequential and simultaneous folding have different strategies and outcomes, so their timing difference does not isolate the effect of parallel arm motion.
D.2 Repeated Execution with Persistent Experience
Task protocols.
Each five-cycle study starts without experience, then launches fresh contexts with saved measurements, scripts, and lessons. Current images update positions. The ring task includes disassembly and reassembly; the two-pair task follows the same assembly goal across cycles.
Time measurements.
Total runtime combines task execution and subsequent experience writing. Within task time, recorded program/robot execution is assigned to tools, including overlap with model responses; remaining time estimates reasoning and programming.
What experience changes.
Most savings come from reasoning and programming as agents reuse working procedures. Two-pair tool execution actually increases slightly. Because scene positions change through an ordered sequence, a matched empty-experience control would be needed to isolate the reuse effect.
D.3 Transferring Frozen Experience to Terra
Every Terra trial receives identical frozen Astra experience. Four assemblies pass operator review; the fifth leaves one pair incomplete after recovery near a workspace boundary. Resource means compare one baseline success with four assisted successes, limiting precision.
Appendix E Runtime Program Analysis
Runtime program analysis covers 65 task executions and distinguishes retained files, inherited copies, and inline command events.
Program inventory.
The authors identify 37 working Python files across 18 sessions: 29 for perception/geometry, seven for robot-call composition, and one for visual annotation. Another 135 inline-Python events occur across 48 sessions. The 63 inherited file copies and seven exported copies are counted separately.
| Primary purpose | Files | Observed computations and operations |
|---|---|---|
| Perception and geometry | 29 | Depth measurement, color region selection, ray intersection, triangulation, surface fitting, and grasp offset estimation. |
| Robot call composition | 7 | Pose and gripper commands followed by capture, concurrent calls to different arms, compact responses, and full response logging. |
| Visual annotation | 1 | Projection of spatial reference points into camera images for inspection. |
Surface fitting and grasp offset compensation.
A plane-fitting program estimates a held pin's orientation from wrist depth using SVD, then computes grasp offsets and corrected targets. The recorded next robot request closely matches its output, linking computation to action. An arrival error prompts further adjustment rather than proving the estimate exact.
ctr = best.mean(0) _, _, v = np.linalg.svd(best - ctr) normal = v[-1] normal *= np.sign(normal[2])
Triangulation from two camera views.
A second program triangulates agent-selected pixels by finding the closest points on calibrated camera rays. Their midpoint estimates position and their separation checks consistency; calibration and correspondence errors still limit accuracy. The original program excerpts are retained.
ab = np.linalg.lstsq(
np.stack([d1, -d2], 1), t2 - t1, rcond=None
)[0]
p1 = t1 + ab[0] * d1
p2 = t2 + ab[1] * d2
Visual processing and execution helpers.
Other programs segment colors, detect circles, create demonstration montages, and dispatch concurrent arm operations before capturing observations. These traces demonstrate use, while controlled ablations would be needed to measure their contribution to success.
Appendix F Error and Recovery Study
F.1 Six Block Tower Collapse During Withdrawal
A six-block tower stands after placement but collapses during withdrawal. One cube lands beyond the reachable envelope, ending recovery despite remaining budget. The trace identifies the failure sequence without establishing the exact contact or instability that caused it.
F.2 Grasp Loss and Unstable Placement in Four Pair Assembly
Four-pair failures include a lost pin whose final location remains unresolved and an engaged assembly that tips repeatedly after release. Gripper-supported engagement is not evidence of stable completed assembly.
F.3 Insertion Misalignment and Contact During Recovery
An insertion can move both held and receiving parts, invalidating earlier alignment. One Fable trial jams and displaces parts before completing either pair. Grasp offsets and positioning error are proposed causes, not an independently measured error decomposition.
F.4 Recovery Near the Workspace Boundary
Transferred instructions fail when a receiver lies just inside the minimum radial workspace limit. Moving the requested target into the allowed region also changes its alignment with the actual receiver; subsequent recovery leaves the part tipped.
F.5 Fabric Slippage and Incomplete Fold Verification
Fabric slips and whole-towel movement leave required corners unfolded. In another trial the agent reports completion despite an incomplete corner identified by operator review, showing that final-state verification can fail independently of motion planning.
F.6 Throwing Shortfall and Bowl Displacement During Recovery
A failed throw reaches the bowl only after bouncing from the table. Retrieval then moves the bowl and leaves the object inside the displaced target. The agent reports failure because direct flight and the stationary-target constraint are unverified; final occupancy alone does not satisfy the task.
Appendix G Broader Impact
AGP could support rapid manipulation prototypes or collection of robot trajectories, but the value of such data remains to be tested through downstream policy training. Faulty perception, programs, and motion requests remain physical risks. The paper's runtime limits are implementation constraints, not evidence of safe general deployment.
Appendix H Execution Across Arm and End Effector Embodiments
A seven-joint P7 arm with a RealHand L6 dexterous hand demonstrates bottle grasping and a 2 cm lift through staged wrist/finger adjustments. Two RealSense cameras supply observations. This is qualitative evidence of interface portability, without a repeated-trial cross-embodiment success estimate.
Appendix I Qualitative Task Examples
I.1 Four Pair Assembly
The four-pair sequence shows a failed insertion, changed grasp and wrist orientation, and completion after release while preserving earlier assemblies.
I.2 Block Construction
Construction follows support relations: tiers for the pyramid, alternating levels for the two towers, and progressive alignment for the six-block tower.
I.3 Dice Flipping
Dice reorientation combines tilted grasps, low releases, and observation-guided retries. The final image verifies all six requested faces after withdrawal.
I.4 Targeted Throwing
Two retained-object motion probes guide swing and gripper timing. Because images partially occlude the flight, the assessment also uses execution records and video.
I.5 Bimanual Towel Folding
Sequential folding.
The sequential fold recovers a slipped corner and corrects an edge before the inward folds. It meets the fold-arrangement criterion despite a residual buckle and imperfect edge alignment.
Simultaneous folding.
The simultaneous case combines inward transport with staged placement for arm clearance. Both required folds remain after release, with skew and edge offsets permitted by the stated criteria.
Appendix J Baseline Resource Accounting
ASPIRE's real-only figures measure successful replay: 2.9 minutes for two-pair assembly and 9.2 minutes for pyramid construction, with no coding-model calls during replay. Charging the full shared preparation to each success yields 64.8 minutes/USD 18.90 and 271.7 minutes/USD 78.13 respectively.
The totals include dedicated simulation and robot-adaptation workers, exclude gaps, operator conversations, hardware setup, and local perception computation, and do not amortize shared preparation. Pyramid's 1/10 includes a success during adaptation rather than ten independent frozen-program evaluations. GaP has no successes, so successful-run resource means are undefined.