Literature
Display
← All papers

Agent as Policy for Robotic Manipulation

Mengzhao Jia · Yang Lin · Xixin Zhang · Zhihan Zhang · Xiaobai Liu · Meng Jiang

Preprint posted · arXiv (preprint) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

Contents of Appendix

At a glance

AGP makes a general-purpose agent the robot's policy: it observes the scene, writes and runs programs, requests motions, and revises its plan using physical feedback. Model weights remain fixed; a documented robot interface supplies cameras, state, and motion control.

Across eight task configurations, success reaches at least 80% in seven. Reusing saved scripts and experience improves efficiency in small repeated-task studies. Execution remains slow and costly, failures persist, and the results concern specific tabletop tasks rather than unrestricted autonomous manipulation. This edition follows revision v3, posted September 28, 2026.

1 Introduction

Existing agents often generate a program before execution or select among pretrained robot skills [17] [6] [14]. AGP instead keeps the agent making decisions throughout execution, including when to observe, how to interpret evidence, which program to write, and how to recover.

The contribution is a runtime control arrangement with frozen model weights. A conventional robot controller still converts requested targets into physical motion; the agent supplies task-level reasoning, geometric computation, and adaptive action selection.

Refer to caption
Figure 1: Overview of AGP. (a) Evaluated tasks include video guided assembly, image guided construction, dice flipping, targeted throwing, and bimanual towel folding. (b) A preparation agent creates a reusable task definition. At runtime, the execution agent writes and selects programs to interpret observations, compute motion targets, and request robot actions through the robot interface. After each action, the agent uses motion feedback and new observations to refine its estimates and adjust subsequent actions.

2 Related Work

2.1 Agentic Robot Systems

Code-generating systems construct and refine robot programs, while policy-orchestration systems select learned skills [22] [19] [1]. AGP gives the runtime agent responsibility for perception, motion requests, and recovery without a separately trained action policy. ASPIRE provides a useful comparison for reusable program repair [18].

2.2 Robot Manipulation

Classical task-and-motion planning uses explicit geometry [9]; diffusion policies and VLA models learn action generation from robot trajectories [7] [13]. AGP uses a pretrained MLLM agent to make control decisions through tools, complementing those approaches without updating its weights for the evaluated tasks.

2.3 Agentic Workflows Beyond Robotics

AGP brings reasoning, tool use, reflection, and local workspaces from language and coding agents into physical manipulation [28] [21] [27]. Its distinction is where these capabilities operate: they directly govern the robot's observation, geometry, planning, and control loop.

3 A General Purpose Agent as the Robot Policy

3.1 Problem Formulation

A preparation agent packages instructions, demonstration videos or goal images, interface documentation, budgets, and success criteria into a task definition. An execution agent then chooses tools using this definition, interaction history, and a workspace of scripts and observations. Equation 1 describes that decision rule with fixed model parameters.

Standard evaluation starts without task experience. Separate reuse experiments provide saved procedures; those conditions should be distinguished from the initial zero-shot trials.

Equation 1
uk∼πθ(⋅∣g,I,Hk,Wk),\lx@nobreakspace\lx@nobreakspace\lx@nobreakspace\lx@nobreakspace u_{k}\sim\pi_{\theta}(\,\cdot\mid g,I,H_{k},W_{k}),\lx@nobreakspace\lx@nobreakspace\lx@nobreakspace\lx@nobreakspace (1)

3.2 Agent-Robot Bridge

A Python client exchanges requests and results with a server through workspace JSON files. The server provides camera images, calibrated depth, robot state, and motion feedback. The agent requests joint targets, gripper poses, or opening commands; Mink solves inverse kinematics for Cartesian requests [30].

A separate controller enforces motion limits and follows trajectories independently of model inference. Each arm queues its motions, while two arms can operate concurrently. The robot holds position between requests, so camera frame rate and low-level control rate do not imply a fast agent decision loop.

3.3 Runtime Programming and Execution

The agent can generate Python programs to process images, estimate geometry, compose robot calls, or inspect demonstrations. New observations and execution errors drive revisions to grasps, alignment, and strategy. Instructions encourage grouping operations and waiting for completion to reduce redundant tool calls.

Task completion is judged from physical outcomes and recorded evidence, not solely from the agent's report. Appendix E preserves example programs; Appendix F documents unsuccessful recoveries.

4 Experiments

The evaluation spans precision assembly, block structures, dice reorientation, dynamic throwing, and deformable-object manipulation on physical robots.

4.1 Experimental Setup

4.1.1 Robot and Agent

The main platform is a six-joint I2RT YAM arm with a parallel gripper; towel folding uses two arms. Wrist RGB-D and overhead RGB cameras provide observations on demand. Experiments compare Codex and Claude Code configurations with frozen weights; detailed model and effort settings remain in Table 2.

4.1.2 Tasks and Evaluation

Five task groups yield eight configurations: four-pair assembly; pyramid, two-tower, and six-block construction; dice flipping; targeted throwing; and sequential or simultaneous towel folding. Goals come from videos, images, or language.

Trials have fixed time and request budgets, with autonomous recovery allowed. Success requires predefined physical outcomes. Most time, token, and cost means include successful trials only. Throwing instead includes all ten sessions and final reporting; Appendix B specifies the accounting.

Table 1: Main results using GPT-6 Astra as policy. Success is shown as successful trials over total trials (n/Nn/N). Time, tokens, and cost show means with [min, max] below, calculated only on successful trials (except for throwing). †\dagger Throwing instead uses all ten trials and measures total session time, including final reporting. Tokens count input and output usage, including reasoning tokens and retries. Cost uses Standard API rates with cached input pricing. Tags denote long horizon reasoning (LHR), action diversity (AD), object diversity (OD), video instruction (Vid), image instruction (Img), and language instruction (Lan).
Task Capability Instruction Success Time ↓\downarrow Tokens ↓\downarrow Cost ↓\downarrow
(n/N)(n/N) (min) (million) (USD)
Four pair assembly LHR Vid 8/10
37.2
[16.0, 59.1]
13.02
[3.96, 27.12]
16.62
[5.55, 31.50]
Block construction
Pyramid
LHR Img 10/10
21.6
[14.9, 32.2]
9.24
[6.27, 12.94]
11.69
[8.01, 15.28]
Two towers 10/10
20.6
[14.8, 29.2]
7.94
[4.94, 14.39]
9.92
[6.27, 16.85]
Six block tower 9/10
28.2
[11.6, 55.0]
11.57
[4.50, 19.85]
14.93
[6.50, 26.78]
Dice flipping AD Lan 10/10
37.9
[22.6, 56.0]
17.08
[10.16, 25.00]
21.07
[12.57, 31.20]
Targeted throwing
Potato
AD Lan 9/10
17.71†
[10.8, 28.6]
5.71
[2.58, 14.15]
8.16
[4.10, 18.97]
Towel folding
Sequential
OD Vid 5/5
50.8
[33.8, 63.3]
18.12
[10.13, 25.26]
24.14
[14.16, 30.90]
Simultaneous 3/5
21.4
[15.7, 31.2]
5.06
[4.27, 5.68]
7.46
[6.16, 8.67]

4.2 Experimental Results

4.2.1 Main Results

AGP succeeds in 8/10 four-pair assemblies, 10/10 pyramids, 10/10 two-tower builds, 9/10 six-block towers, 10/10 dice tasks, 9/10 throws, 5/5 sequential folds, and 3/5 simultaneous folds. These are small task-specific samples.

Successful sequential folding averages 50.8 minutes and USD 24.14; four-pair assembly averages 37.2 minutes. Strong completion rates therefore coexist with substantial inference and execution overhead. Simultaneous folding remains the weakest configuration at 60%.

4.2.2 Agent and Thinking Effort Comparison

On two-pair assembly, Astra succeeds in 5/5 trials at each tested effort level. Sol and Opus also reach 5/5, while Terra reaches 1/5, Luna 0/5, and Fable 3/5. Sol has lower estimated inference cost than Astra but takes longer.

These comparisons concern complete agent configurations under shared task conditions. Successful-trial resource means condition on different numbers of successes and do not establish a general ranking across tasks.

4.2.3 Baseline Comparison

Using Astra at high effort, AGP succeeds in 23/25 trials across two-pair assembly, four-pair assembly, and pyramid construction, versus 0/25 for GaP and 3/25 for ASPIRE [6] [18]. The baselines receive matched simulation environments for program construction and refinement; AGP adapts online on the physical robot.

ASPIRE's successful frozen-program execution can be faster. Its reported total cost and time charge the entire shared preparation overhead to each successful run, without amortization. These figures answer a different question from repeated replay efficiency. Simulation-to-real mismatch is a proposed explanation for failures, not an isolated causal result.

Table 2: Agent and thinking effort comparison on a two-pair assembly task. Time, tokens, and cost show means with [min, max] below, calculated only on successful trials. Models other than GPT-6 Astra uses high thinking effort.

Model Success Time Tokens Cost (n/N)(n/N) (min) (million) (USD) Codex GPT-6 Astra Low effort 5/5 9.9 [6.0, 19.0] 3.69 [1.44, 7.72] 4.79 [2.15, 9.56] Medium effort 5/5 9.2 [6.3, 15.4] 3.03 [1.84, 5.80] 4.09 [2.76, 7.48] High effort 5/5 9.2 [7.9, 10.6] 3.30 [2.58, 4.02] 4.47 [3.72, 5.19] GPT-5.6 Sol 5/5 14.2 [9.1, 19.3] 7.13 [4.12, 9.72] 3.94 [2.40, 5.30] GPT-5.6 Terra 1/5 19.0 [19.0, 19.0] 6.28 [6.28, 6.28] 1.98 [1.98, 1.98] GPT-5.6 Luna 0/5 – – – . Claude Code Claude Opus 5 5/5 22.2 [14.4, 31.9] 12.40 [9.05, 16.73] 9.75 [6.94, 13.03] Claude Fable 5.1 3/5 27.7 [18.3, 39.8] 9.27 [6.87, 11.10] 11.30 [8.33, 14.36]

Table 3: Baseline comparison. Time and cost average successful runs. ASPIRE reports real only / sim + real, with full construction and refinement overhead in the latter. Execution makes no coding model calls, so its token cost is zero. Green/red marks success above/below 50%.

Task Method Success ↑\uparrow Time ↓\downarrow Cost ↓\downarrow (n/N)(n/N) (min) (USD) Two pair assembly GaP 0/5 – – ASPIRE 2/5 2.9 / 64.8 0.00 / 18.90 AGP 5/5 9.2 4.47 Four pair assembly GaP 0/10 – – ASPIRE 0/10 – – AGP 8/10 37.2 16.62 Pyramid construction GaP 0/10 – – ASPIRE 1/10 9.2 / 271.7 0.00 / 78.13 AGP 10/10 21.6 11.69

4.3 Improving Efficiency through Experience

Astra repeats two assembly tasks for five cycles, saving programs, measurements, and corrections after each run. Later runs refresh object positions and reuse earlier procedures. Runtime falls across the sequence, mainly through reduced reasoning and programming time.

Giving Terra the same frozen experience from Astra changes two-pair assembly success from 1/5 to 4/5. Among successful trials, mean time falls from 19.0 to 12.5 minutes and tokens from 6.28M to 5.56M. The resource comparison uses one successful baseline trial versus four with experience; the ordered reuse study also lacks an empty-experience control at each cycle.

Refer to caption
Figure 2: Experience refinement and transfer. (a) Experience updates across repeated runs. (b) Transfer of frozen experience from Astra to Terra. (c, d) Experimental results for refinement and transfer, respectively.

4.4 Case Study of Failure Recovery

A recovery trace shows the agent changing its pickup tilt, checking whether a tube stands after release, inspecting pin alignment, and revising insertion motions. The complete 48.6-minute session spends 68.2% of its measured time in planning/reasoning and 23.5% in robot actions. The figure links observations, programs, and requests to these decisions.

Refer to caption
Figure 3: Failure recovery during four pair assembly. (a) After the tube falls over following an attempt to stand it upright, the agent reasons about a different grasp and introduces a tilted pickup that allows the robot to stand the tube upright successfully. Timestamps indicate elapsed time within the run. (b) Time allocation among planning and reasoning, computer program execution, robot action execution, and camera and state calls across the full run.

5 Conclusion

AGP demonstrates that a general-purpose runtime agent can produce manipulation behavior by combining programs with physical feedback. Saved experience offers a route to reducing overhead without weight updates.

Limits include small trial counts, restricted task and hardware coverage, slow execution, costly inference, unsuccessful recovery, and imperfect final verification. Cross-embodiment evidence is qualitative, and baseline resource comparisons depend on preparation accounting.

AI Use Statement

The authors report using generative AI for drafting and language editing and take responsibility for the manuscript's methods, results, and interpretations.

Reproducibility Statement

The appendices document interfaces, task criteria, model settings, timing boundaries, program traces, and qualitative executions. Public project materials, code, and the source paper provide further reproducibility information.

Appendix A Implementation Details

A.1 Robot Platform and Control Interface

The platform connects calibrated observations to bounded motion requests through a separate robot runtime.

Robot platform.

The YAM arm uses a parallel gripper with 95.5 mm nominal opening and a 1 Mbit/s CAN connection. A calibrated rigid transform aligns the two arms' Cartesian frames.

Observations and geometric grounding.

Wrist depth and camera calibration map pixels into robot coordinates. Overhead pixels intersect rays with an estimated-height plane. Additional views can refine geometry; the 30-frame/s camera streams are independent of the agent's decision rate.

Refer to caption
Figure 4: Robot platform and camera placement. The setup contains two I2RT YAM arms mounted on a shared tabletop rail. The annotations identify an arm, a linear_4310 parallel gripper, a wrist mounted Intel RealSense D405 RGB and depth camera, and the fixed overhead Logitech BRIO RGB camera. The two arms support bimanual towel folding.

Action interface.

Seven commands expose state, status, help, frame capture, Cartesian motion, joint motion, and gripper control. Ordinary requests enforce joint velocity/acceleration limits of 20°/s and 40°/s², translation at 0.03 m/s, orientation at 10°/s, radial targets from 0.12 to 0.65 m, and Cartesian steps no larger than 0.25 m.

Throwing uses a separate timed-motion interface at 50 Hz, with joint 4 limits of 180°/s and 360°/s². This task-specific execution capability is part of the supplied interface.

A.2 Agent Configuration and Execution Conditions

Standard trials start with empty task experience; reuse conditions receive specified saved files. Logs record agent/model versions, hosts, budgets, settings, waiting intervals, and grouped tool operations.

A.3 Model and Reasoning Effort Configurations

The Astra effort sweep fixes the environment, inputs, and robot interface. Cross-provider comparisons use each agent's native settings and therefore compare complete systems rather than an isolated model-weight change.

Appendix B Tasks and Evaluation Protocol

B.1 Task Definitions and Success Criteria

Success criteria are fixed before evaluation and require completed relationships to remain stable after release and arm withdrawal. Controller arrival tolerances and task success tolerances are separate quantities.

Table 4: Task definitions and success criteria. Capability identifies the primary manipulation challenge, and Instruction type identifies how the task goal or procedure is communicated. The final column specifies the required behavior and physical outcome. Success is assessed from the final state and execution video using criteria fixed before evaluation. Required assembly, support, orientation, and fold relationships must remain stable after gripper release and arm withdrawal.
Task Capability Instruction type Task requirements and success criteria
Four pair assembly Long horizon reasoning Video Assemble eight parts into four pairs in the demonstrated order, inferring mating relationships and insertion motions from the video and current scene. Success requires all four pairs to remain assembled, preserving completed pairs throughout subsequent operations.
Block construction Long horizon reasoning Image Arrange six cubes into a pyramid, two towers, or a six block tower matching the goal image. Infer object identities, orientations, and support relations, and choose a placement order that maintains intermediate stability. Success requires the complete target structure.
Dice flipping Action diversity Language Reorient six randomly placed dice to the requested upward face through grasp selection and rotation. Success requires all six dice to show the requested number after placement.
Targeted throwing Action diversity Language Coordinate a circular joint swing with gripper release to throw an object toward the specified target. Success requires the object to separate from the moving hand, travel through the air, and come to rest within the target region.
Towel folding Object diversity Video Use both arms to reproduce the demonstrated folds. Fold the near long edge onto the far edge, then fold both short ends inward, either right then left (Sequential) or beginning together (Simultaneous). Success requires both inward folds with reasonably aligned edges. Small wrinkles and creases are permitted, and the final table position is unrestricted.

Assembly from video.

All methods receive the same video, extracted frames, and timestamped guide for four AutoMate part pairs, assembled in demonstration order [23].

Assembly part dimensions.

Table 5 retains mesh dimensions and mating clearances. Some nominal radial clearances are below 1 mm; actual insertion tolerance also depends on printing accuracy and angular alignment.

Table 5: Printing dimensions and mating clearances. Assembly identifies each AutoMate part pair in demonstration order, with plug and socket meshes shown in blue and gray, respectively. Plug overall size and Socket overall size give the respective mesh dimensions as x×y×zx\times y\times z in the original mesh frame. Hole DhD_{h} is the effective bore diameter, measured by the largest cylinder that fits through the bore, including its narrowest ridges. Shaft DsD_{s} is the effective shaft diameter, measured by the smallest cylinder enclosing the mating shaft segment. The diametral clearance is Δ=Dh−Ds\Delta=D_{h}-D_{s}, and the conservative radial clearance c=Δ/2c=\Delta/2 gives the available transverse offset for parallel mating axes. All dimensions and clearances are in millimeters. The hole is in the plug mesh for the first three pairs and in the socket mesh for 00446. Clearances are computed before rounding.
Assembly Plug overall size Socket overall size Hole DhD_{h} Shaft DsD_{s} Δ\Delta cc
Original figure [-1pt]00081 52.95×52.98×27.5352.95\times 52.98\times 27.53 71.59×71.65×58.2671.59\times 71.65\times 58.26 37.41 35.86 1.56 0.78
Original figure [-1pt]00581 55.58×48.32×19.4655.58\times 48.32\times 19.46 45.52×39.42×78.8345.52\times 39.42\times 78.83 28.18 25.80 2.38 1.19
Original figure [-1pt]00007 51.08×50.94×39.6051.08\times 50.94\times 39.60 42.42×42.30×81.4442.42\times 42.30\times 81.44 42.86 37.20 5.66 2.83
Original figure [-1pt]00446 39.25×39.16×105.0439.25\times 39.16\times 105.04 31.32×31.32×90.8131.32\times 31.32\times 90.81 18.01 16.54 1.47 0.73

Block construction from images.

Cube poses vary across trials while each configuration retains its goal photograph, block set, and predefined placement tolerances.

Action diversity.

Dice trials vary initial and target faces. Throwing requires a permitted swing, gripper release, airborne motion, and landing in the specified target; ending inside the bowl alone is insufficient.

Bimanual towel folding.

Towel tasks provide demonstration videos and start/end images. Staged placement is allowed for arm clearance. Final assessment checks the demonstrated folds after release, allowing minor wrinkles.

B.2 Trial Protocol

Records distinguish physical failure, exhausted budgets, human intervention, and infrastructure interruption. Local computation and geometric queries are logged separately from budgeted observation and action requests.

B.3 Metric Definitions and Resource Accounting

Success and time.

Success divides fully completed trials by evaluated trials. Resource means normally use successful trials from task delivery through physical verification. No-success configurations have undefined successful-trial means. The broader timing study also includes failures and final reporting.

Interaction accounting.

One agent tool decision can group several robot requests. Interaction budgets count each observation/action request separately, so tool-call counts and robot-request counts differ.

Token usage.

Tokens sum input and output across requests and retries, counting repeated context each time and reasoning once in output. Cost estimates apply the stated API prices and caching discounts; they are study accounting, not current price quotations.

Appendix C Supplementary Coding Agent Comparison on Pyramid Construction

On pyramid construction, Sol completes 7/10 sessions and Astra 10/10. Sol's three unsuccessful sessions include two supervisor interventions and a bridge disconnection before physical execution. The table therefore includes infrastructure and intervention outcomes as failures.

Table 6: Supplementary coding agent comparison on pyramid construction. Both Codex models use high thinking effort, extending the comparison in Table 2. Successes count completed pyramids recorded in the session reports. Interrupted sessions count as unsuccessful. Tokens and cost are means over successful sessions for each model. Total cost covers all ten sessions for each model. Cost is estimated using API prices for each model.
Model Successes Tokens (million) Cost (USD) Total (USD)
GPT-5.6 Sol 7/10 18.86 9.22 87.25
GPT-6 Astra 10/10 9.24 11.69 116.88

Usage and pricing.

Cost estimates use model API prices retrieved September 9, 2026. Table 6 separates successful-session means from total cost across all ten sessions.

Interpretation.

Sol uses more tokens per successful pyramid session but has lower estimated inference cost. This does not imply better completion rate or lower elapsed time.

Appendix D Efficiency and Experience Reuse Analysis

D.1 Task Execution Time Analysis

Figure 5 decomposes all trials, including failures and final reporting, for seven configurations. Throwing lacks the required component timings and is excluded. These boundaries differ from most main-table successful-trial means.

Time categories.

Categories cover the visual loop, planning/reasoning, local tools, action service, final checking, and reporting. Overlap counts once. The visual loop includes the model response after an image, which may itself contain planning and code generation.

Original figure
Figure 5: Time allocation across manipulation tasks. Seven task panels show mean time allocation over all NN evaluated trials, including failures. Every bar spans 100% of its task’s total session time, including final reporting. Each component is labeled with its mean minutes and percentage of the total. The seven tasks have complete timing decompositions. Throwing is omitted because its component timings are unavailable. Displayed values are rounded.

Observed costs.

Observation and decision-making consume substantial time. Sequential and simultaneous folding have different strategies and outcomes, so their timing difference does not isolate the effect of parallel arm motion.

D.2 Repeated Execution with Persistent Experience

Task protocols.

Each five-cycle study starts without experience, then launches fresh contexts with saved measurements, scripts, and lessons. Current images update positions. The ring task includes disassembly and reassembly; the two-pair task follows the same assembly goal across cycles.

Original figure
Figure 6: Execution time over repeated tasks. Total runtime and its three components across five cycles of ring disassembly and reassembly (a–b) and two pair assembly (c–d). Times are seconds; the components sum to total runtime up to rounding. Percentages indicate the total runtime reduction from cycle 1 to cycle 5. Vertical scales differ across panels.

Time measurements.

Total runtime combines task execution and subsequent experience writing. Within task time, recorded program/robot execution is assigned to tools, including overlap with model responses; remaining time estimates reasoning and programming.

What experience changes.

Most savings come from reasoning and programming as agents reuse working procedures. Two-pair tool execution actually increases slightly. Because scene positions change through an ordered sequence, a matched empty-experience control would be needed to isolate the reuse effect.

D.3 Transferring Frozen Experience to Terra

Every Terra trial receives identical frozen Astra experience. Four assemblies pass operator review; the fifth leaves one pair incomplete after recovery near a workspace boundary. Resource means compare one baseline success with four assisted successes, limiting precision.

Appendix E Runtime Program Analysis

Runtime program analysis covers 65 task executions and distinguishes retained files, inherited copies, and inline command events.

Program inventory.

The authors identify 37 working Python files across 18 sessions: 29 for perception/geometry, seven for robot-call composition, and one for visual annotation. Another 135 inline-Python events occur across 48 sessions. The 63 inherited file copies and seven exported copies are counted separately.

Table 7: Programs constructed during execution. Primary purposes of the 37 retained Python working files. Image crops, demonstration montages, and some object detection routines also appear as inline code and are outside this file inventory.
Primary purpose Files Observed computations and operations
Perception and geometry 29 Depth measurement, color region selection, ray intersection, triangulation, surface fitting, and grasp offset estimation.
Robot call composition 7 Pose and gripper commands followed by capture, concurrent calls to different arms, compact responses, and full response logging.
Visual annotation 1 Projection of spatial reference points into camera images for inspection.

Surface fitting and grasp offset compensation.

A plane-fitting program estimates a held pin's orientation from wrist depth using SVD, then computes grasp offsets and corrected targets. The recorded next robot request closely matches its output, linking computation to action. An arrival error prompts further adjustment rather than proving the estimate exact.

ctr = best.mean(0)
_, _, v = np.linalg.svd(best - ctr)
normal = v[-1]
normal *= np.sign(normal[2])

Triangulation from two camera views.

A second program triangulates agent-selected pixels by finding the closest points on calibrated camera rays. Their midpoint estimates position and their separation checks consistency; calibration and correspondence errors still limit accuracy. The original program excerpts are retained.

ab = np.linalg.lstsq(
    np.stack([d1, -d2], 1), t2 - t1, rcond=None
)[0]
p1 = t1 + ab[0] * d1
p2 = t2 + ab[1] * d2

Visual processing and execution helpers.

Other programs segment colors, detect circles, create demonstration montages, and dispatch concurrent arm operations before capturing observations. These traces demonstrate use, while controlled ablations would be needed to measure their contribution to success.

Appendix F Error and Recovery Study

F.1 Six Block Tower Collapse During Withdrawal

A six-block tower stands after placement but collapses during withdrawal. One cube lands beyond the reachable envelope, ending recovery despite remaining budget. The trace identifies the failure sequence without establishing the exact contact or instability that caused it.

F.2 Grasp Loss and Unstable Placement in Four Pair Assembly

Four-pair failures include a lost pin whose final location remains unresolved and an engaged assembly that tips repeatedly after release. Gripper-supported engagement is not evidence of stable completed assembly.

F.3 Insertion Misalignment and Contact During Recovery

An insertion can move both held and receiving parts, invalidating earlier alignment. One Fable trial jams and displaces parts before completing either pair. Grasp offsets and positioning error are proposed causes, not an independently measured error decomposition.

F.4 Recovery Near the Workspace Boundary

Transferred instructions fail when a receiver lies just inside the minimum radial workspace limit. Moving the requested target into the allowed region also changes its alignment with the actual receiver; subsequent recovery leaves the part tipped.

F.5 Fabric Slippage and Incomplete Fold Verification

Fabric slips and whole-towel movement leave required corners unfolded. In another trial the agent reports completion despite an incomplete corner identified by operator review, showing that final-state verification can fail independently of motion planning.

F.6 Throwing Shortfall and Bowl Displacement During Recovery

A failed throw reaches the bowl only after bouncing from the table. Retrieval then moves the bowl and leaves the object inside the displaced target. The agent reports failure because direct flight and the stationary-target constraint are unverified; final occupancy alone does not satisfy the task.

Appendix G Broader Impact

AGP could support rapid manipulation prototypes or collection of robot trajectories, but the value of such data remains to be tested through downstream policy training. Faulty perception, programs, and motion requests remain physical risks. The paper's runtime limits are implementation constraints, not evidence of safe general deployment.

Appendix H Execution Across Arm and End Effector Embodiments

A seven-joint P7 arm with a RealHand L6 dexterous hand demonstrates bottle grasping and a 2 cm lift through staged wrist/finger adjustments. Two RealSense cameras supply observations. This is qualitative evidence of interface portability, without a repeated-trial cross-embodiment success estimate.

Refer to caption
Figure 7: P7 and L6 platform for the cross embodiment experiment. The annotations identify the P7 arm, RealHand L6 dexterous hand, overhead RealSense D435i, and wrist mounted RealSense D435.
Refer to caption
Figure 8: Bottle grasp and lift with the P7 arm and L6 hand. The upper panel contains 15 overhead frames read row by row, showing wrist reorientation, approach, finger closure, and lifting. The lower panel shows five corresponding stages from the wrist camera. Timestamps measure elapsed minutes and seconds from the first grasp action after stationary preparation. Frames are sampled at unequal intervals, with the last frame at 04:36. The two views are approximately aligned using their recorded host timestamps.

Appendix I Qualitative Task Examples

I.1 Four Pair Assembly

The four-pair sequence shows a failed insertion, changed grasp and wrist orientation, and completion after release while preserving earlier assemblies.

I.2 Block Construction

Construction follows support relations: tiers for the pyramid, alternating levels for the two towers, and progressive alignment for the six-block tower.

I.3 Dice Flipping

Dice reorientation combines tilted grasps, low releases, and observation-guided retries. The final image verifies all six requested faces after withdrawal.

I.4 Targeted Throwing

Two retained-object motion probes guide swing and gripper timing. Because images partially occlude the flight, the assessment also uses execution records and video.

I.5 Bimanual Towel Folding

Sequential folding.

The sequential fold recovers a slipped corner and corrects an edge before the inward folds. It meets the fold-arrangement criterion despite a residual buckle and imperfect edge alignment.

Simultaneous folding.

The simultaneous case combines inward transport with staged placement for arm clearance. Both required folds remain after release, with skew and edge offsets permitted by the stated criteria.

Refer to caption
Figure 9: Four pair assembly from a human demonstration video. The upper panel shows four demonstrated assemblies. The lower panel shows the robot assembling eight parts into four pairs, illustrating video instruction following and long horizon reasoning. Dark red bars mark failed nut insertion and displacement. Yellow bars mark grasp retries. Read each panel row by row. Timestamps give elapsed minutes and seconds in each video at unequal intervals.
Refer to caption
Figure 10: Block construction from static goal images. From top to bottom are the pyramid, two towers, and six block tower. Each occupies two rows of five images, beginning with the goal image labeled Demo and followed by nine execution frames read row by row. Timestamps give elapsed minutes and seconds in each recording at unequal intervals.
Refer to caption
Figure 11: Dice flipping across six dice. The goal photograph in the cell labeled Demo specifies six pips upward on each of three black and three light blue dice. Nine overhead execution frames show the initial faces, successive manipulations, and final state with all six target faces upward. The task requires selecting grasps and rotations that produce the requested orientation after release. Read the execution frames row by row. Timestamps show elapsed minutes and seconds in the source video.
Refer to caption
Figure 12: Targeted throwing from a language instruction. The task requires a joint driven circular swing and timed gripper release toward a bowl, illustrating the dynamic release component of action diversity. The overhead sequence shows the initial arrangement, grasp, swing, and final object position inside the bowl. The wrist sequence shows retention and separation at the gripper. Occlusion limits the visible flight trajectory. Read each grid row by row. Timestamps show elapsed minutes, seconds, and milliseconds on a common clock, with denser sampling around release and landing.
Refer to caption
Figure 13: Sequential bimanual towel folding from a human demonstration video. The upper panel contains 15 frames showing the long edge fold followed by the right and left ends folding inward. The lower panel contains 30 frames showing coordinated grasping, recovery after a corner slips, the three folds, and subsequent adjustments. The final towel preserves the demonstrated fold arrangement with a remaining buckle and edge offsets. Read each panel row by row. Timestamps show elapsed minutes and seconds in the respective source videos, with unequal sampling intervals.
Refer to caption
Figure 14: Simultaneous bimanual towel folding from a human demonstration video. The upper panel shows the long edge fold followed by both short ends folding inward together. The lower panel shows the robot execution, including recovery of a slipped corner, the long edge fold, inward transport of both ends, staged placement of the flaps, and alignment correction. The final state contains both inward folds with residual skew and edge offsets, with both arms withdrawn. Read each panel row by row. Timestamps show elapsed minutes and seconds in the respective source videos, with unequal sampling intervals.

Appendix J Baseline Resource Accounting

ASPIRE's real-only figures measure successful replay: 2.9 minutes for two-pair assembly and 9.2 minutes for pyramid construction, with no coding-model calls during replay. Charging the full shared preparation to each success yields 64.8 minutes/USD 18.90 and 271.7 minutes/USD 78.13 respectively.

The totals include dedicated simulation and robot-adaptation workers, exclude gaps, operator conversations, hardware setup, and local perception computation, and do not amortize shared preparation. Pyramid's 1/10 includes a success during adaptation rather than ten independent frozen-program evaluations. GaP has no successes, so successful-run resource means are undefined.

Concept demonstration

Open image ↗