At a glance
RRSI improves an agent’s surrounding code, prompts, tools, memory, and control flow while keeping model weights fixed. It limits edit bundles, remembers previous experiments, screens benchmark-specific changes, and accepts candidates using performance, noise, and token-cost criteria. Across eight benchmarks, the reported harnesses improve all six held-out splits. The largest 14.1-point gain occurs on the coding evolution benchmark; unseen-benchmark gains reach 4.7 points. The final workspace harness uses fewer tokens than unregularized evolution, but more than the initial harness. Code and trajectories are available.
1 Introduction
Repeatedly proposing and scoring harness changes on the same finite task set can fit benchmark details, reward evaluation noise, and accumulate costly machinery. RRSI addresses this adaptive overfitting by regulating how candidates are proposed and retained. A separate harness is evolved for each domain and then transferred unchanged to that domain’s unseen benchmarks; the paper does not demonstrate one universal harness across all domains. [23] [47] [5]
2 Preliminaries
An agent combines a frozen policy with an editable harness. The harness controls prompts, tools, memory, context, and when the policy acts or stops. Evolution repeatedly executes the agent on an “evolve” set, proposes changes from its traces, and measures candidate scores and policy-token use. Reusing these tasks makes the search adaptively dependent on earlier measurements, so its best observed score is insufficient evidence of generalization.
Equations 1–3
| (1) |
| (2) |
| (3) |
3 RRSI
RRSI leaves every harness component editable. Proposal rules constrain the amount and direction of change; selection rules constrain what survives. The intent is to retain reusable mechanisms while resisting benchmark-specific shortcuts, noisy improvements, and unjustified computation.
3.1 A Regularization View of Harness Evolution
The paper draws functional analogies to classical regularization: an edit-count limit resembles sparsity, pruning removes unproductive components, and cost-aware acceptance restrains aggregate resource growth. These are discrete search heuristics, not optimization of literal L0, L1, or L2 norm penalties. [13] [7] [31]
3.2 Regularizing the Proposal Distribution
Proposal regularization combines a shrinking edit budget, a history of tested hypotheses, and explicit exploration when recent progress stalls.
L 0 -Style Annealed Update Sparsity.
A cosine schedule reduces the number of independent edits permitted in each candidate from an initial maximum toward one. Early rounds can explore coordinated changes; later rounds make results easier to attribute to individual mechanisms.
Equation 4
| (4) |
Evidence-Aware Credit Assignment.
Each evaluated edit records its component, hypothesis, code diff, score and cost changes, and acceptance outcome. Later proposals consult both positive and negative evidence, reducing repeated tests of already unsuccessful ideas. Bundled edits still share a candidate-level score, so this history does not provide causal attribution to every individual edit. [6]
Structured Exploration.
When improvement over a recent window stays within the estimated noise band, part of the proposal budget targets component types not yet tested. This encourages exploration beyond repeated prompt rewriting without forbidding any component.
3.3 Regularizing Candidate Selection
A candidate must pass all applicable screening, stability, cost, and domain checks before competing on measured score. A high score cannot compensate for failure of one of these checks.
Leakage Screening.
A critic rejects source edits containing benchmark-specific task names, entities, answers, or values, as well as inert additions, before full evaluation. Generic mechanisms remain admissible. This targets explicit leakage in proposed code; it is not proof against every form of benchmark adaptation or backbone contamination.
Stability-Aware Acceptance.
Repeated runs of the unchanged base harness estimate a noise tolerance. A candidate must remain within that tolerance of the best accepted score so far, preventing a chain of individually small regressions from drifting steadily downward. The tolerance is empirical, rather than a general statistical guarantee for the adaptive search.
Equation 5
| (5) |
Ridge/ L 2 -Style Complexity-Aware Acceptance.
When a candidate’s measured gain exceeds the noise tolerance, its relative token-cost increase must stay below a gain-dependent allowance. Larger gains can justify more computation. Token use is the resource proxy; it does not measure every tool, simulator, or wall-clock cost.
Equations 6–7
| (6) |
| (7) |
Lasso/ L 1 -Style Structural Pruning.
Components with no positive measured gain during a recent pruning window become proposed deletion targets. This sparsifies retained structure, whereas the annealed budget limits each update. Deletions still pass through candidate evaluation and acceptance.
4 Experiments
Evaluation spans coding, professional workspace tasks, and engineering design. It asks whether gains survive new tasks and benchmarks, which regularizers matter, and how much inference the evolved harness consumes.
4.1 Experimental Setup
The evolution sets are 89 Terminal-Bench tasks, 120 Harvey LAB tasks, and 61 license-free EngDesign tasks. Held-out evaluation uses 40 additional Harvey tasks plus SWE-bench Verified, JobBench, GDPval, APEX-Agents, and Frontier-Eng. Comparisons share initial harnesses, frozen policy, evolve tasks, and candidate budgets. Main runs use Claude Opus 4.8 for acting, proposing, analysis, and criticism; coding also tests Gemini 3.5 Flash. Workspace and engineering harnesses use ReAct, an MCP gateway, a dynamic toolbelt, and context summarization. [33] [12] [10]
4.2 Main Results
With the main backbone, evolve-set gains are 6.0 points on Terminal-Bench, 1.1 on Harvey LAB, and 4.9 on EngDesign. Held-out gains are 1.8 on SWE-bench, 2.3 on Harvey, 4.7 on JobBench, 3.5 on GDPval, 3.7 on APEX, and 4.3 Medal points on Frontier-Eng. On the three unseen workspace benchmarks, the mean rises from 39.7 to 43.6 despite a smaller evolve-set improvement than prior methods. RRSI ties Meta-Harness on held-out Harvey and exceeds the compared methods on the workspace OOD scores.
Engineering tasks use deterministic simulators rather than language-model judges, so those gains do not depend on a judge’s writing preferences. Deterministic grading nevertheless leaves policy sampling and finite benchmark coverage as sources of uncertainty.
| In-Distribution | Out-of-Distribution | ||||
|---|---|---|---|---|---|
| Method | Harvey LAB (Evolve) | Harvey LAB (ID Held-out) | JobBench | GDPval | APEX-Agents |
| (no evolution) | 89.4 | 86.9 | 36.0 | 48.8 | 34.2 |
| Meta-Harness (Lee et al., 2026b) | 93.0 | 89.2 | 37.1 | 49.1 | 35.7 |
| AHE (Lin et al., 2026a) | 90.7 | 88.7 | 37.2 | 47.2 | 33.1 |
| TTHE (Nie et al., 2026) | 91.1 | 88.5 | 35.2 | 47.0 | 31.7 |
| HarnessX (Chen et al., 2026) | 91.8 | 89.1 | 36.3 | 48.5 | 34.3 |
| RRSI (ours) | 90.5 | 89.2 | 40.7 | 52.3 | 37.9 |
4.3 Analysis
Removing proposal or acceptance regularization raises the evolve-set score while reducing workspace transfer. Removing both gives a 40.3 OOD average at 3.80 million policy tokens per trial, compared with 43.6 at 2.42 million for RRSI. The initial harness costs 1.56 million tokens: RRSI is cheaper than the evolved alternatives, while still spending more than the starting system.
Separate coding evolution with Gemini 3.5 Flash improves Terminal-Bench from 64.6 to 78.7 and unseen SWE-bench from 76.8 to 79.0. Running that evolved harness with the previously unused Gemini 3.1 Flash Lite raises Terminal-Bench from 11.2 to 14.6. This supports transfer across these backbones, rather than universal independence from model choice.
| Variant | Harvey LAB (Evolve) | Harvey LAB (ID Held-out) | OOD Avg. | Tokens/trial (m) |
|---|---|---|---|---|
| (no evolution) | 89.4 | 86.9 | 39.7 | 1.56 |
| Unregularized evolution | 92.8 | 88.9 | 40.3 | 3.80 |
| w/o proposal regularizers | 90.7 | 88.8 | 41.9 | 2.69 |
| w/o acceptance regularizers | 91.5 | 88.7 | 41.0 | 3.59 |
| RRSI | 90.5 | 89.2 | 43.6 | 2.42 |
| Policy | Benchmark | RRSI | ||
|---|---|---|---|---|
| Claude Opus 4.8 | Terminal-Bench 2.1 (Evolve) | 74.2 | 80.2 | +6.0 |
| SWE-bench Verified (OOD) | 82.0 | 83.8 | +1.8 | |
| Gemini 3.5 Flash | Terminal-Bench 2.1 (Evolve) | 64.6 | 78.7 | +14.1 |
| SWE-bench Verified (OOD) | 76.8 | 79.0 | +2.2 |
| Evaluation policy | RRSI | ||
|---|---|---|---|
| Gemini 3.5 Flash (search policy) | 64.6 | 78.7 | +14.1 |
| Gemini 3.1 Flash Lite (unseen) | 11.2 | 14.6 | +3.4 |
5 Related Work
Prior work evolves complete harnesses or individual skills, memories, and preferences, generally retaining changes by benchmark score. Other concurrent approaches explicitly target generalization or preserve diverse candidates. RRSI contributes proposal and selection constraints that can operate across different editable components. [23] [25] [34] [3] [60] [32]
6 Conclusion
Automatically editing an agent’s scaffold can improve held-out performance when the search is constrained by evidence, uncertainty, and cost. The experiments support these particular short evolution runs; they do not establish unrestricted recursive improvement or changes to the underlying model’s learned weights.
Limitations
The study keeps backbone weights frozen and depends on finite evolve sets, feedback quality, search budgets, and several hyperparameters. Broader architectures, tool ecosystems, and longer runs remain untested. Judge-based workspace scores measure rubric or preference outcomes, while token comparisons concern running the resulting harness and do not establish lower total development cost. Appendix reporting also leaves some reproducibility details unresolved, noted below.
Appendix A Evaluation
Each evolved harness and its baseline are evaluated in the same time window with matching environments, judges, and trial counts. Grading differs by benchmark, so score units should not be treated as interchangeable.
A.1 Terminal-Bench 2.1
Terminal-Bench runs 89 isolated container tasks. Success requires passing hidden task tests after execution; containers are rebuilt between comparison arms. [33]
A.2 SWE-bench Verified
SWE-bench Verified requires a repository patch that fixes designated failing tests while preserving previously passing tests. The metric is the fraction of issues satisfying both requirements. [18]
A.3 Harvey LAB
Harvey LAB tasks require named deliverable files built from source documents. Gemini 3.5 Flash grades each rubric criterion independently; the aggregate weights tasks by their number of criteria, and missing files fail their criteria. The 120/40 evolve/held-out partition stays fixed. [12]
A.4 JobBench
JobBench provides professional task folders while withholding some reference material to require retrieval. The harness uses files, code, and web search; scores use weighted rubrics averaged across Gemini 3.5 Flash and Claude Opus 4.8 judges. [24]
A.5 GDPval
GDPval compares agent deliverables with supplied human-expert outputs using three judges from different model families. Both presentation orders are judged and majority preference determines the outcome. The appendix reports 185 tasks but also 204 comparisons per judge, an unresolved accounting inconsistency. Treat its reported win rate as the authors’ benchmark result, rather than inferring a trial count from those conflicting figures. [38]
A.6 APEX-Agents
APEX-Agents supplies professional sandbox worlds with MCP tools. A single rollout must satisfy the task rubric to count as a success. All 480 tasks remain in the denominator; infrastructure failures receive zero credit. [44]
A.7 EngDesign
EngDesign uses 61 tasks runnable without proprietary simulators. Task-specific deterministic checks grade physical designs; all tasks are used for evolution, leaving no internal holdout. Transfer is instead checked on Frontier-Eng. [10]
A.8 Frontier-Eng
Frontier-Eng awards 1, 0.67, or 0.33 credit against frozen gold, silver, and bronze feasible-result thresholds. The overlapping EngDesign domain is excluded. The paper retains a 47-task denominator, with only 38 tasks able to contribute credit after exclusions and unavailable environments; both arms use the same accounting. It serves only as a transfer benchmark. [4]
Appendix B Baseline Methods
Meta-Harness searches executable harness code using accumulated traces and scores. AHE organizes component observations and edit outcomes; TTHE adapts multiple executable candidates during test time; HarnessX composes typed prompts, tools, memory, and control-flow primitives. All four are compared from the same initial scaffold and candidate budget. [23] [25] [34] [3]
Appendix C Method Details
The appendix formalizes search over editable programs. The regularization analogies describe functional roles, without assuming a shared continuous parameter vector or norm-penalized objective.
C.1 Round-Level Formulation
Each round combines current feedback, edit history, the shrinking budget, exploration directives, and pruning targets to propose candidates. After screening and evaluation, the highest-scoring admissible candidate replaces the incumbent; if none qualifies, the current harness remains. The two complete algorithms and the round-level equations are retained.
Equation 8
| (8) |
C.2 Proposal-Side Bookkeeping
Atomic edit representation.
A candidate applies a subset of the round’s proposed atomic edits. The budget caps this subset’s size; the possible edit pool changes each round and remains unrestricted across the run.
Equation 9
| (9) |
Edit history and component-level summaries.
Every edit in a measured candidate inherits that candidate’s score and token changes. Only edits belonging to the selected round winner receive an acceptance flag. Component summaries track whether a type has been exercised and its best recent measured gain. This bookkeeping becomes more interpretable as fewer edits are bundled.
Equations 10–11
| (10) |
| (11) |
Structured exploration state.
The component vocabulary covers prompts, control flow, configuration, output handling, context management, tools, skills, memory, and subagents. A three-round stall reserves a proposal slot for component types with no measured edit history.
Equations 12–13
| (12) | ||||
| (13) |
Structural pruning.
A previously exercised component becomes a pruning target when no recent edit associated with it has a strictly positive gain. The proposer is asked to remove related unproductive machinery in future candidates.
Equation 14
| (14) |
C.3 Selection-Side Bookkeeping
Noise-adjusted floor.
After leakage screening, candidates must stay within the calibrated tolerance of the running best accepted score. This stability floor is mandatory before any cost or novelty trade-off.
Equation
Novelty used by the within-band rule.
Novelty counts distinct tool, skill, memory, and subagent component types that a candidate touches before any edit of that type has previously won. Prompt, configuration, context, and control-flow changes receive no novelty bonus.
Equations 15–16
| (15) |
| (16) |
Acceptance when the gain exceeds the noise tolerance.
For improvements larger than the noise tolerance, a linear allowance links permitted relative token growth to the measured score gain. It is a cost gate, not a squared-norm penalty.
Equation
Acceptance when the gain does not exceed the noise tolerance.
For gains within the noise band, a separate weighted rule combines score change, lower token cost, and structural novelty. Coding assigns zero weight to small score changes, so they cannot qualify alone; workspace and engineering runs permit positive score weight. The text says these weights appear in Table 5, but the supplied table omits them, leaving the exact within-band settings unspecified in the paper.
Equation 17
| (17) |
Domain-specific non-compensatory guards.
Engineering candidates also fail if valid-output rate drops by more than 0.03 or no-submission rate rises by more than 0.02. Coding and workspace runs impose no additional domain guard.
Final round selection.
The selector chooses the largest measured score among candidates passing the floor, the applicable cost rule, and domain guards. If none passes, it retains the incumbent and updates no accepted edit path.
Appendix D Experiments
D.1 Hyperparameter Setting
Parameters are chosen without consulting held-out tasks. Coding and workspace evolution run for 20 rounds with two trials per task; engineering uses 40 rounds with four trials. Initial edit limits are four, three, and four respectively, decreasing toward one. The noise tolerances are 0.017, 0.004, and 0.020 in fractional-score units. The complete published table preserves exploration, pruning, and gain-dependent cost settings.
| Hyperparameter | Role | Coding | Agentic workspace | Engineering design |
| evolution rounds | 20 | 20 | 40 | |
| trials per task per evaluation | 2 | 2 | 4 | |
| empirical noise tolerance | 0.017 | 0.004 | 0.020 | |
| final-round edit budget | 1 | 1 | 1 | |
| initial edit budget | 4 | 3 | 4 | |
| stall-detection window | 3 | 3 | 3 | |
| reserved exploratory proposals | 1 | 1 | 1 | |
| pruning window | 4 | 4 | 5 | |
| base cost allowance | 0.10 | 0.10 | 0.15 | |
| gain-dependent cost allowance | 44.5 | 35.4 | 24.4 |
Appendix E Qualitative Case Study
Case studies show why superficially similar changes can have different outcomes. An early coding audit is accepted with a 3.93-point gain; another is rejected with a 1.69-point gain and 26.1% added cost. A cheaper candidate loses 2.81 points and fails the stability floor. An engineering directory-recovery fix increases passes from 122/244 to 128/244 for 1.6% more tokens. These illustrate decisions from the reported runs, rather than independently proving each mechanism’s causal contribution.
| Domain / Round | Harness change | Outcome | What it illustrates |
|---|---|---|---|
| Coding, R0-A | Adds a bounded pre-completion verification audit and guidance for non-blocking polling of long-running jobs. | Accepted: points on the evolve set. | A reusable behavioral mechanism can justify a relatively broad early-round update when the gain exceeds the noise threshold. |
| Coding, R0-B | Adds a similar verification reminder and long-running-work guidance, but with a smaller measured gain and additional inference cost. | Rejected by cost rule: points, cost. | An apparent improvement is not automatically retained when it lies within the noise band and requires substantial additional computation. |
| Coding, R8-B | Pins the original task instruction into the completion gate so that the policy re-checks the literal specification before submission. | Rejected by floor: points despite cost. | Lower cost alone cannot compensate for a candidate whose performance falls below the noise-adjusted acceptance floor. |
| Engineering, R2 | Adds a bounded recovery hint for the recurring “workdir must be an existing directory” tool-use error. | Accepted: passes, tokens. | The search can retain small, task-agnostic control-flow fixes that improve reliability with little added complexity. |
Author affiliations
The authors are affiliated with Google Cloud AI Research, the University of North Carolina at Chapel Hill, Stanford University, and Washington University in St. Louis. Peng Xia contributed while a Student Researcher at Google Cloud AI Research.