At a glance
Taste-Bench tests whether a model can choose a promising direction before later work reveals the result. It contains 502 paired choices mined from engineering and research trajectories. The best reported model scores 59.7% under a domain-balanced metric requiring correct answers in both option orders. A distilled student improves held-out engineering accuracy from 30.0% to 47.9%. Advice based on that student also improves reported task success, but it uses forks mined from earlier runs on the same evaluation tasks, rather than discovering decisions online from scratch.
Introduction
Long tasks depend on intermediate choices as well as execution. The paper calls this anticipatory judgment “taste” and measures it using later trajectory outcomes as labels for earlier decisions. Parallel attempts expose divergent strategies; detours expose approaches an agent abandons and replaces. The resulting benchmark supplements end-to-end success metrics. [9] [10] [11] [12] [13]
Measuring Taste via Long-Horizon Judgment
A question contains the task, a pre-decision trajectory prefix, and two candidate directions. The model chooses without seeing later branch evidence. Labels favor the branch with the better recorded outcome. Aligned prefixes and filtering try to isolate decision quality, but observed trajectories still include downstream execution differences; a retained label is not a randomized causal estimate of that decision’s value.
Equation 1
| (1) |
Taste-Bench
The 502 questions cross two domains with two construction methods: engineering/research and parallel/detour. Candidates are written neutrally, and later outcomes remain hidden during benchmark evaluation.
Forks from parallel trajectories
Parallel questions compare attempts at the same task that take different approaches at comparable points and finish differently. The prefix comes from the supported branch. Generator checks seek a shared subproblem and evidence linking the chosen approach to the outcome.
Forks from detour trajectories
Detour questions contrast an approach that visibly failed with the later recovery. The question stops before the original mistake. Mechanical checks verify quoted evidence and event order; the generator must reject a recovery that would only be nameable with hindsight or an implausible losing option.
Mining, filtering, and composition
The source pools contain 2,677 engineering rollouts on 517 SWE-bench Pro tasks and 1,132 research runs on 47 tasks from METR’s MALT release. Of 4,657 candidate forks, 502 survive generation and filtering: 390 engineering and 112 research questions. Four judges reject questions all can solve from candidate wording alone, then retain only labels all support after seeing full outcomes. These filters shape the released difficulty and label agreement. [11] [13] [14] [15]
A two-reviewer audit includes 70 released and 30 rejected questions. After outcomes are shown, 170 of 172 explicit A/B judgments match mined labels; 28 of 200 judgments without a clear preference are excluded. This 98.8% conditional agreement is not an accuracy estimate over all released questions.
Evaluation protocol
Each question is answered twice with opposite option orders. The primary score requires both answers to be correct, penalizing order-sensitive choices; mean per-presentation accuracy is reported separately. Independent random answers score 25% under the two-answer rule, while always choosing the same position scores zero.
Experimental Results
Experiments compare model choices, difficulty by construction and evidence horizon, reasoning effort, and correspondence with an execution benchmark.
Setup
Fourteen models receive the same prompts and maximum output budget. The headline Average gives research and engineering equal weight, despite their unequal question counts. It is neither ordinary single-answer accuracy nor a pooled percentage over all 502 items.
Accuracy of current models
GPT-5.6 Sol scores 59.7% and GPT-5.5 59.5% on the domain-balanced two-order metric. Rankings vary by domain and construction. Invalid answer formats count as failures, substantially affecting some weaker rows; the full results separate these cases and include item-bootstrap intervals.

Effect of the time horizon
A model annotates how much later evidence appears necessary: already in the prefix, inferable from it, visible next step, or requiring more work. Mean accuracy across the 14 models declines from 62.3% to 21.0% across the first and last levels. These are judgment-based ordinal labels, not measured task durations, and the longest level contains 69 questions.

Effect of the reasoning budget
Across three effort settings for each of two models, the highest-minus-lowest accuracy changes are −0.2 points for Sol and +2.2 for Luna, with intervals spanning zero. Harder forks consume more reasoning tokens without a clear overall gain. This supports a limited finding for these models and settings; Luna’s research subset separately rises from 43.8% to 54.5%, so the aggregate result is not a universal absence of benefit.

Comparison with end-to-end benchmarks
For 11 models after excluding high parsing-failure rates, correlation with public SWE-bench Verified scores is r = 0.63 for the balanced Average and 0.37 for engineering alone. Confidence intervals are wide, and reasoning settings are not matched across benchmarks. The comparison suggests different rankings, rather than proving that the benchmark isolates an independent capability. [10] [29] [30]
Generalizing Taste through Distillation
A Qwen3.6-27B student learns through LoRA adapters from a teacher of the same base architecture. The teacher receives a short description of the supported candidate, while the student sees only the question. Training transfers answer-informed reasoning into the student’s parameters.
Distillation recipe
The 390 engineering questions are divided into two task-disjoint folds. Each student trains on one and is evaluated on the other. Teacher-generated reasoning and final choices provide token-distribution targets; a later calibration pass supervises answer positions on the student’s own training-fold traces. The held-out task split limits direct memorization of task labels, while every repository appears in both folds. [31] [32]
Transfer to unseen tasks
Pooled held-out predictions improve two-order accuracy from 30.0% to 47.9%, a 17.9-point gain. Mean per-presentation accuracy rises from 42.7% to 62.4%. These held-out metrics differ from the 48.6%→92.9% training-fold single-order comparison. The student remains below the strongest evaluated model on engineering questions.
End-to-end evaluation
Before a fresh executor run, notes recommend choices at forks mined from prior runs on that same held-out task. The student was trained on other tasks, but the advice-generation pipeline has access to task-specific historical forks. With a fixed Qwen3.6-27B executor on 41 selected SWE-bench Pro tasks, reported success is 14.6% without advice, 39.0% with correct-choice advice, and 33.7% with student advice. Correct advice is a comparison condition, not a proved upper bound over all possible advice. The experiment tests use of precomputed task knowledge, rather than fully online judgment on previously unexplored tasks.

Related Work
Related work covers execution benchmarks, prediction of research outcomes, process evaluation, and distillation with privileged information.
Long-horizon agent evaluation.
AgentBench, SWE-bench, and research-agent benchmarks grade completed work. Taste-Bench instead presents a choice inside a recorded task. The two evaluation styles measure related but distinct aspects of the pipeline. [9] [10] [11] [12] [13]
Judging research directions before the outcome.
Earlier studies compare research ideas, solutions, or proposals before execution. This benchmark adds a concrete task history and outcomes from later trajectories, with the trade-off that those outcomes remain imperfect counterfactual evidence. [33] [34] [35] [36]
Process evaluation and step-level judgment.
Process-supervision and agent-judging methods score steps, alternatives, or complete trajectories. Taste-Bench asks for a choice before revealing the continuation used to label it. [37] [38] [39] [40] [41] [42] [43] [44] [45]
Distillation and internalizing context.
The recipe follows knowledge and context distillation: a teacher receives privileged information and the student learns to imitate its output with less context. Here that information identifies the supported choice, and an advisor later passes the student’s decisions to an executor. [46] [47] [48] [49] [50] [51] [52]
Conclusion
Trajectory-mined choice questions provide a practical diagnostic, and answer-informed distillation improves held-out choices in the tested setting. Interpretation depends on the filtering panel, task selection, outcome-based labels, and the distinction between precomputed advice and autonomous online decisions.
Reproducibility Statement
The authors release the benchmark data and evaluation code and recorded results. The appendices retain prompts, data counts, settings, and treatment of long inputs and invalid answers.
Ethics Statement
The study uses technical agent trajectories and two human reviewers. The authors report removing credential strings before model processing and acknowledge dual-use implications of stronger long-horizon agents.
AI Use Statement
The authors disclose generative-AI assistance with synthetic data, experimental design, implementation, interpretation, writing, figures, and literature searches, while taking responsibility for the final work.
Appendix A Benchmark Construction Details
The construction appendix records source pools, exact prompts, filtering checks, and the number of questions retained at each stage.
Source trajectories
Engineering rollouts span 11 repositories and 31 collection runs from April–July 2026 using GPT-5.4/5.5. Research transcripts come from several Claude and DeepSeek models. The two research-detour passes revisit a pool of trajectories, so their input counts should not be added as distinct source runs.
Statistics of the mined forks
Parallel mining prioritizes same-model, same-setting pairs; 13 of 124 released engineering pairs nevertheless compare different models. Detour candidates require literal evidence for commitment, failure, recovery, and success in order. The table preserves task counts and prefix sizes. Comparable prefixes and these checks reduce obvious confounds without making the comparison experimentally controlled.
| Cell | Questions | Tasks | Trajectories | Steps (median) | Steps (10th–90th) | Words (median) |
|---|---|---|---|---|---|---|
| Parallel engineering | 124 | 62 | 120 | 35 | 21–57 | 38 |
| Parallel research | 48 | 8 | 32 | 56 | 6–132 | 27 |
| Detour engineering | 266 | 133 | 266 | 47 | 24–71 | 23 |
| Detour research | 64 | 22 | 64 | 30 | 3–81 | 22 |
Prompts of the generator and the judges
GPT-5.6 Sol generates candidate forks. Kimi K2.5, GPT-4.1, Llama 4 Maverick, and Mistral Large 3 form the filtering panel. Exact source prompts below specify evidence checks, neutral wording, rejection criteria, and the two filtering views.
You build parallel judgment benchmark items from real agent trajectories. Return one JSON object only. Ground every decision point and option in the supplied trajectories. Do not invent grader evidence or use prior benchmark labels. Treat rejection as a correct and preferred result whenever a clean causal fork cannot be demonstrated from exact post-decision evidence.# Goal
Find one genuine decision fork where the branch with the better native outcome made a
better task-solving decision than the worse branch. Locate the decision in the raw
trajectories and write two neutral, parallel candidate next steps.
Reject the candidate if the outcome difference is mostly a typo, crash, luck, missing
dependency, or generic execution quality rather than the chosen approach.
# Task
{query}
# Outcome interpretation
{outcome_standard}
# Branches
{branches}
# Required JSON
Return exactly one object with:
{
"is_good_sample": true or false,
"reason": "brief outcome-grounded reason",
"query": "self-contained task statement preserving important constraints",
"good_traj": "trajectory id of the passing branch",
"bad_traj": "trajectory id of the selected failing branch",
"good_start_step": integer position immediately before the good branch commits,
"bad_start_step": integer position immediately before the bad branch commits,
"good_choice": "1-3 sentence neutral next step",
"bad_choice": "1-3 sentence neutral next step",
"evidence": {
"good_action": {"step": integer, "quote": "exact excerpt from that step"},
"bad_action": {"step": integer, "quote": "exact excerpt from that step"},
"good_post_decision": [
{"step": integer, "quote": "exact excerpt showing a consequence or result"}
],
"bad_post_decision": [
{"step": integer, "quote": "exact excerpt showing a consequence or result"}
]
}
}
Requirements:
- good_traj must be the branch with the better supplied outcome; bad_traj the worse one.
- Step positions must be within their corresponding trajectories and before commitment.
- Both choices must address the same immediate subproblem from comparable decision points.
- The evidence that distinguishes the choices must occur after the proposed decision points.
If the better option is already identifiable from pre-decision differences, reject the sample.
- The prefix from good_traj before good_start_step must contain enough task context.
- Preserve important task constraints in query even if they appear early in a transcript.
- Choices must describe the actual branch decisions, at the same specificity and tone.
- Do not mention scores, pass/fail, grader results, branch names, or hindsight in choices.
- Do not put a branch's self-reported or predicted target metric in a choice when its
direction makes that choice mechanically preferable (for example, merely claiming a
lower predicted loss). Describe the decision procedure or controllable action instead.
- Do not use praise, blame, confidence, caution, or other wording that makes one option an
obvious strawman. If neutral parallel wording is not possible, reject the sample.
- In reason, identify the post-decision trajectory evidence connecting the decision to the
native outcome. If the outcome gap is not causally attributable to the decision, reject.
- Reconstruct the timeline before accepting. A score or artifact that already existed
before the proposed cut cannot be credited to the proposed next step.
- The proposed action must be new at the cut. Reject if the same action was already
executed before the cut, or if cited post-decision evidence repeats evidence already
present before the cut. Re-running the same command is not a causal decision fork.
- An experiment contributes evidence only when its output shows a completed result.
Starting a script followed by timeout, OOM, interruption, or missing output is not a
successful experiment.
- Writing a claimed or predicted objective value into the answer is not an observed task
outcome. A format, budget, or schema validator proves only those constraints; it does
not independently validate task performance. If no post-decision evidence connects the
differing action to the measured objective, reject the sample.
- Compare actual implemented actions, not plans or filenames. If both branches implement
the same underlying approach, reject even if their wording differs.
- Reject when failure is mainly submission/finalization protocol, timeout, crash, missing
dependency, hard-coded answer, task-version mismatch, or instruction noncompliance.
- Every evidence quote must be a literal excerpt from the cited trajectory step. Evidence
is checked mechanically after generation.You build judgment benchmark items from one real agent trajectory. Return one JSON object only. Ground every claim in exact excerpts from the supplied steps. Treat rejection as a correct and preferred result whenever a clean detour cannot be demonstrated, or whenever the better approach is only nameable with hindsight.# Goal
Find one genuine DETOUR in this single trajectory: the agent committed to a poor technical
approach, pursued it, hit a wall, and recovered with a better approach that worked. Write
the fork as two neutral, parallel candidate next steps.
- bad_choice = the poor approach the agent committed to first.
- good_choice = the better approach it recovered to.
# Task
{query}
# Native outcome
{outcome_note}
# Complete trajectory
trajectory_id: {trajectory_id}
{trajectory}
# Required JSON
Return exactly one object with:
{
"is_good_sample": true or false,
"reason": "brief, step-grounded reason",
"query": "self-contained task statement preserving important constraints",
"breakpoint_step": integer index where the prefix ends; the agent is AT the fork and has
NOT yet committed to the poor approach,
"good_choice": "1-3 sentence neutral next step",
"bad_choice": "1-3 sentence neutral next step",
"evidence": {
"bad_action": {"step": integer, "quote": "exact excerpt committing to the poor approach"},
"wall": {"step": integer, "quote": "exact excerpt of the observed failure that ended it"},
"recovery_action": {"step": integer, "quote": "exact excerpt committing to the recovery"},
"success": [
{"step": integer, "quote": "exact excerpt showing the recovery worked"}
]
}
}
Requirements:
- Ordering is mandatory and is checked mechanically:
breakpoint_step <= bad_action.step <= wall.step < recovery_action.step <= success[*].step.
bad_action and wall MAY cite the same step when one message contains both the commitment
and the failure it produced (common when a command and its output are narrated together).
- Every quote must be a literal excerpt from the step it cites.
- The wall must be an OBSERVED FAILURE in the trajectory: a non-zero exit, an error,
traceback, "not found", failing assertion, or an explicitly reported failing result. Your
own remark that something could be improved, a clean diff listing, or a successful
command is NOT a wall. This is checked mechanically and a fabricated wall is discarded.
- steps[0:breakpoint_step] is the prefix shown to a test-taker. It must carry enough
context to reason, and must NOT reveal the failure, the wall, or the eventual fix.
- REJECT if the recovery is only nameable after seeing the failure. Ask: standing at
breakpoint_step with the prefix alone, could a competent engineer have proposed
good_choice? "Revert the change to the file the traceback names" or "raise the timeout
the error reports" are hindsight lookups, not taste. This is the most common way a
candidate fails; prefer rejection when unsure.
- When the deciding fact is discovered mid-rollout (a missing tool, an unavailable
dependency, a broken toolchain), do NOT reject on that ground. RELOCATE the fork to after
the discovery and mine the SECOND cycle: the agent's first reaction to the now-known
constraint is bad_action, that reaction failing is the wall, and the strategy it settled
on is recovery_action. Rejecting here when a second cycle exists is a wasted candidate.
Worked example. Step 20 prints `go: command not found`. Put breakpoint_step at 21, so the
prefix already establishes there is no host toolchain. bad_action (step 21) = probing the
filesystem for a stray Go install; wall (step 24) = that install being the wrong version
for the repo's go.work; recovery_action (step 26) = building in a pinned container.
Both options are then strategies for coping with a constraint the reader can see.
- Reject on this ground only when the rollout contains no such usable second cycle.
- ACCEPT a fork where genuinely different strategies were available, INCLUDING strategy
about how to obtain a working environment or a trustworthy verification path. Real
examples of valid forks: run the suite in a pinned container versus probe the host for a
toolchain; declare pinned dependencies for an isolated run versus assemble a PYTHONPATH
out of scavenged directories; write a narrow stubbed probe versus stand up the whole
application stack. The wall being an environment failure does not disqualify the fork —
what matters is whether a competent engineer had a real decision to make.
- BOTH OPTIONS MUST BE LIVE CANDIDATES. Someone who sees only the prefix must be able to
pick either one plausibly; telling them apart has to take real technical judgment rather
than a surface cue. Neither may be a strawman. In particular, an option that merely
repeats an action the prefix already shows failing — "keep calling the binary the prefix
just reported as not found" — is a strawman, and so is any option nobody would actually
propose at that point. If the losing side is not something a competent engineer might
genuinely have chosen there, REJECT the candidate rather than writing it up. This is the
single most common reason a candidate is thrown out downstream.
- REJECT if the agent never actually committed to the poor approach (it only mused about
it), or if the "recovery" is the same underlying approach with different wording.
- REJECT if the two choices address different subproblems, or if the fork is mere
execution ordering rather than a technical-judgment fork.
- Phrase both choices in strictly parallel form: same length range, same specificity, same
tone. No evaluative or leaking words ("correctly", "mistakenly", "as it turns out",
"better", "should", "properly"). No mention of grades, failures, hindsight, step numbers,
or which one the trajectory took. Both must read as plausible candidate next steps.
- Do not put the wall's error text, file name, or symbol into good_choice when that alone
identifies it as correct.You are a reviewer at a real decision point. Choose the better next step from the task, trajectory prefix, and neutral options. End with exactly ANSWER: X.# Candidate next steps
Option A:
{choice text}
Option B:
{choice text}
Without additional task or trajectory context, which option appears to be the better
engineering decision? End with exactly ANSWER: X.# Task
{query}
# Shared trajectory before the decision
{prefix}
# Complete branch continuations
## Option A
Proposed next step:
{choice text}
Trajectory continuation through its final step:
{steps from the decision point to the end of that branch}
## Option B
Proposed next step:
{choice text}
Trajectory continuation through its final step:
{steps from the decision point to the end of that branch}
# Feedback from native grader
{outcome standard of the dataset}
Option A: passed=...; score=...
Option B: passed=...; score=...
Which option produced the better outcome under the dataset's core metric?
End with exactly ANSWER: X.Counts at each stage
The funnel retains 1,809 candidates after the generator rubric, 729 after the wording-only filter, and 502 after outcome agreement. Table 2’s “Proposed” counts describe the 1,809 rubric-approved candidates, not all 4,657 initial proposals. The two preserved rejected examples illustrate an obvious hard-coded strawman and a label dependent on an unstated timeout constraint.
| Cell | Proposed | Released |
|---|---|---|
| Parallel engineering | 562 | 124 |
| Parallel research | 186 | 48 |
| Detour engineering | 658 | 266 |
| Detour research, pass 1 | 241 | 40 |
| Detour research, pass 2 | 162 | 24 |


Appendix B Example Questions
Four complete source examples follow, one for each construction/domain cell. They contrast field selection with copying an authentication block, embedding retraining with global rescaling, a production-path change with a helper-interface change, and nonlinear residual search with a linear approximation. Prefixes and later outcomes are retained so the benchmark labels can be inspected.




Appendix C Evaluation Protocol Details
Prompts render trajectory steps and abbreviate command output beyond 700 characters. Inputs are limited to 65,536 tokens, retaining the first 25 prefix lines plus a fitting tail if necessary; main-run truncation affects one question for two models. Unparseable answers are wrong. Confidence intervals resample questions within domains, without a reported task-cluster adjustment. Full prompts and model settings remain below.
You are evaluating an autonomous agent at a technical decision point. The task, the agent's progress, and two candidate next steps are provided. Exactly one candidate is the better technical decision.# Task given to the agent
{query}
# Agent's progress so far (transcript up to the decision point)
{prefix}
# Decision point
Option A:
{candidate text}
Option B:
{candidate text}
Which option is the better technical decision?
Return exactly one line: ANSWER: X| Model | Reasoning setting |
|---|---|
| GPT-5.6 Sol | reasoning effort xhigh |
| GPT-5.5 | reasoning effort xhigh |
| GPT-5.6 Luna, GPT-5.6 Terra | default |
| GPT-5.4 Mini, GPT-5.4 Nano | reasoning effort high |
| Claude Opus 5 | default |
| Claude Sonnet 5 | adaptive thinking |
| Grok 4.5 | reasoning effort high |
| Grok 4.20 Reasoning | default |
| DeepSeek V4 Flash | reasoning effort max |
| GLM-5.2 | reasoning effort max, temperature 1.0 |
| MiniMax M3 | default |
| Mistral Medium 3.5 | reasoning effort high |
Appendix D Full Results
The full results separate model performance, cross-benchmark correlation, horizon annotation, and reasoning-effort settings.
Results per model and per cell
The full table separates domains, constructions, mean per-presentation accuracy, order consistency, and invalid responses. Mean accuracy across models is 58.1% for parallel engineering and 35.9% for detour engineering; research is not uniformly harder. Parsing failures particularly depress some low-scoring models.
| Model | Average | Eng. | Res. | P-Eng | P-Res | D-Eng | D-Res | Mean acc. | CC | CW | WC | WW | Unparsed |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | 59.7 | 56.9 | 62.5 | 75.8 | 56.2 | 48.1 | 67.2 | 65.2 | 292 | 25 | 29 | 156 | 0 |
| GPT-5.5 | 59.5 | 55.6 | 63.4 | 73.4 | 56.2 | 47.4 | 68.8 | 64.6 | 288 | 26 | 24 | 164 | 0 |
| Claude Opus 5 | 55.5 | 46.7 | 64.3 | 71.0 | 64.6 | 35.3 | 64.1 | 60.3 | 254 | 26 | 25 | 197 | 0 |
| Grok 4.5 | 54.6 | 52.1 | 57.1 | 61.3 | 56.2 | 47.7 | 57.8 | 62.0 | 267 | 40 | 43 | 152 | 10 |
| GPT-5.6 Terra | 54.0 | 50.0 | 58.0 | 70.2 | 58.3 | 40.6 | 57.8 | 62.2 | 260 | 40 | 46 | 156 | 0 |
| GLM-5.2 | 53.9 | 47.9 | 59.8 | 64.5 | 60.4 | 40.2 | 59.4 | 64.0 | 254 | 52 | 49 | 147 | 17 |
| Claude Sonnet 5 | 51.6 | 44.4 | 58.9 | 62.1 | 56.2 | 36.1 | 60.9 | 60.7 | 239 | 37 | 52 | 174 | 0 |
| GPT-5.6 Luna | 49.0 | 43.6 | 54.5 | 67.7 | 50.0 | 32.3 | 57.8 | 58.9 | 231 | 48 | 51 | 172 | 0 |
| MiniMax M3 | 45.3 | 44.1 | 46.4 | 64.5 | 56.2 | 34.6 | 39.1 | 57.2 | 224 | 61 | 51 | 166 | 3 |
| DeepSeek V4 Flash | 43.3 | 38.5 | 48.2 | 58.1 | 50.0 | 29.3 | 46.9 | 54.9 | 204 | 59 | 55 | 184 | 2 |
| GPT-5.4 Mini | 40.1 | 25.6 | 54.5 | 26.6 | 54.2 | 25.2 | 54.7 | 49.5 | 161 | 46 | 59 | 236 | 112 |
| Mistral Medium 3.5 | 37.7 | 41.5 | 33.9 | 43.5 | 41.7 | 40.6 | 28.1 | 51.8 | 200 | 70 | 67 | 165 | 24 |
| GPT-5.4 Nano | 36.6 | 32.1 | 41.1 | 46.0 | 43.8 | 25.6 | 39.1 | 48.8 | 171 | 65 | 61 | 205 | 93 |
| Grok 4.20 Reasoning | 15.7 | 22.6 | 8.9 | 28.2 | 8.3 | 19.9 | 9.4 | 29.0 | 98 | 78 | 72 | 254 | 459 |
Comparison with SWE-bench Verified
The 11-model correlation interval is [0.04, 0.89] for the balanced Average and [−0.30, 0.79] for engineering. Public SWE-bench results share a harness, but model effort settings are not fully matched to Taste-Bench, limiting causal explanations of ranking differences.
Time horizon annotation
GPT-5.5 assigns five ordinal horizon scores; the top two are merged because only 13 questions require the longest category. The four displayed levels contain 158, 219, 56, and 69 questions. Their composition differs by construction and domain, so horizon effects may include these differences.
# Task
{query}
# Trajectory before the decision
{prefix}
# Candidate next steps
Option A:
{candidate text}
Option B:
{candidate text}
# Correct answer
{letter} (established in hindsight from the completed run)
How far into the future would an observer at this decision point need to see before the correct choice becomes clearly justified? Use this scale:
0 - Explicit evidence in the prefix: a specific decisive fact is already visible before the decision (an error message, a stated constraint, a visible fix or diff) that directly rules out one option.
1 - Inferable from the prefix: no single decisive fact is visible, but weighing the hints already present in the prefix (domain judgment, combining observations) is enough to justify the correct option; no future observation is needed.
2 - Next observation settles it: the first direct observation after the decision (the output of one command or probe, the content of one opened file) would clearly justify the correct option.
3 - Quick local check needed: a completed, self-contained local verification is needed (running one unit test, a small script, a quick smoke experiment) before the correct option is clearly justified.
4 - Substantial future work needed: more than a quick local check is required - meaningful further implementation or investigation, results appearing across later stages of downstream work, or only the terminal outcome (final test suite, external grader, complete branch comparison).
Give a one-sentence justification, then end with exactly: SCORE: N| Cell | In prefix | Inferable | Next step | More work |
|---|---|---|---|---|
| Parallel engineering | 72 | 45 | 4 | 3 |
| Parallel research | 11 | 13 | 3 | 21 |
| Detour engineering | 69 | 128 | 36 | 33 |
| Detour research | 6 | 33 | 13 | 12 |
| All | 158 | 219 | 56 | 69 |
| Model | In prefix | Inferable | Next step | More work |
|---|---|---|---|---|
| () | () | () | () | |
| GPT-5.5 | 75.9 | 57.1 | 48.2 | 23.2 |
| GPT-5.6 Sol | 79.7 | 57.5 | 44.6 | 21.7 |
| Grok 4.5 | 67.1 | 53.9 | 32.1 | 36.2 |
| Claude Opus 5 | 67.1 | 47.0 | 50.0 | 24.6 |
| GPT-5.6 Terra | 73.4 | 48.9 | 41.1 | 20.3 |
| GLM-5.2 | 69.0 | 49.8 | 32.1 | 26.1 |
| Claude Sonnet 5 | 63.9 | 45.7 | 35.7 | 26.1 |
| GPT-5.6 Luna | 67.7 | 43.8 | 32.1 | 14.5 |
| MiniMax M3 | 64.6 | 41.1 | 25.0 | 26.1 |
| DeepSeek V4 Flash | 61.4 | 35.6 | 30.4 | 17.4 |
| Mistral Medium 3.5 | 54.4 | 42.5 | 14.3 | 18.8 |
| GPT-5.4 Nano | 49.4 | 31.1 | 25.0 | 15.9 |
| GPT-5.4 Mini | 46.2 | 30.1 | 16.1 | 18.8 |
| Grok 4.20 Reasoning | 32.3 | 16.4 | 14.3 | 4.3 |
| Mean over models | 62.3 | 42.9 | 31.5 | 21.0 |
Reasoning budget
Highest-minus-lowest effort intervals are [−3.2, 2.8] percentage points for Sol and [−1.6, 6.0] for Luna. No response reaches the output cap; one of 6,024 responses is unparseable. The table uses pooled accuracy across questions, unlike the domain-balanced headline Average. The research-specific Luna improvement is distinct from the overall near-null result.
| Model | Setting | Accuracy | In prefix | Inferable | Next step | More work |
|---|---|---|---|---|---|---|
| GPT-5.6 Sol | low | 56.2 | 79.1 | 53.9 | 44.6 | 20.3 |
| GPT-5.6 Sol | xhigh | 55.6 | 78.5 | 52.1 | 46.4 | 21.7 |
| GPT-5.6 Sol | max | 56.0 | 79.7 | 52.5 | 42.9 | 23.2 |
| GPT-5.6 Luna | none | 43.4 | 60.8 | 44.3 | 28.6 | 13.0 |
| GPT-5.6 Luna | high | 45.8 | 67.7 | 42.9 | 28.6 | 18.8 |
| GPT-5.6 Luna | max | 45.6 | 65.2 | 43.4 | 35.7 | 15.9 |
Appendix E Agreement of the Judges
On released parallel-engineering questions, wording-only pairwise judge agreement is 51.7%. Full-outcome agreement is unanimous by construction, because the release filter requires it; that unanimity is not an independent validation statistic.
| Cell | Judge | Proposed | Released | Agreement |
|---|---|---|---|---|
| Parallel engineering | Kimi K2.5 | 84.7 | 55.6 | 51.7 |
| GPT-4.1 | 85.8 | 63.7 | ||
| Llama 4 Maverick | 75.6 | 34.7 | ||
| Mistral Large 3 | 82.0 | 59.7 |
Appendix F Human Review
The human review checks whether the outcome-supported choice is persuasive and whether the displayed decision explains the later difference.
Sampling and annotation
The two reviewers assess the same 100 mixed released/rejected examples before and after outcome summaries. Agreement uses only explicit post-outcome A/B preferences: 170/172 match labels, and reviewers agree on 73/74 items where both give a preference. Excluding 28 uncertain or non-causal judgments and the small reviewer panel limit broader interpretation.
| Reviewer | Retained judgments | Label agreement |
|---|---|---|
| Reviewer 1 | 87 | 86/87 (98.9%) |
| Reviewer 2 | 85 | 84/85 (98.8%) |
| Combined | 172 | 170/172 (98.8%) |
Appendix G Distillation Details
The appendix gives the exact training procedure, task-disjoint folds, and the distinction between two-order evaluation and combined advice selection.
Recipe
Teacher traces are sampled at temperature 1 with a 3,072-token limit, retaining questions with usable traces in both orders. LoRA rank 16 adapts attention and MLP projections, totaling 79.7 million trainable parameters. The reported KL-based objective uses the top 100 tokens plus residual probability mass and at most 512 reasoning positions; answer-token calibration follows on training-fold student traces. Each fold takes roughly two hours on an A100 80 GB GPU. The loss is applied at selected positions; this does not by itself ensure other output distributions remain unchanged.
You are a senior software/research engineer reviewing an autonomous agent's work at a decision point. Judge the alternatives on technical merit and likely task outcome.# Task given to the agent
{query}
# Agent progress up to the decision point
{prefix}
# Candidate next steps
Option A:
{candidate text}
Option B:
{candidate text}
Think through the engineering decision carefully. After thinking, show your
choice in the "answer" field of a JSON object using only "A" or "B", for
example {"answer":"A"}.Below is a reference distilled from past agent runs on THIS EXACT task: at the current decision fork, the choice that led to success, and the choices that led to failure. Use it when relevant.
# Reference: key decisions from past runs on this task
### Decision fork (at the current point)
The right call at this point -- do this:
{text of the supported candidate}
Avoid:
- {text of the other candidate}
---
# Your task
{student user message}| Setting | Value |
|---|---|
| Base model | Qwen3.6-27B, bfloat16 |
| Adapter | LoRA, rank 16, , no dropout, on all attention and MLP projections |
| Trainable parameters | 79.7M |
| Optimizer | AdamW, gradient clipping at norm 1.0 |
| Distillation | learning rate , 2 epochs, one view per step, 624 and 568 steps |
| Distillation loss | forward KL at temperature 1.0, top-100 tokens plus one entry for the remaining mass, at most 512 reasoning positions |
| Calibration | learning rate , 1 epoch, softmax temperature 2.0, 306 and 320 steps |
| Sequence length | at most 10,240 tokens, no truncation in the training data |
| Peak GPU memory | 71 GB |
| Wall clock per fold | 2.1 h and 1.8 h for distillation |
Folds
Each fold has 195 questions, including 62 parallel and 133 detour items. Tasks, source trajectories, and prefixes do not cross folds, but all repositories occur in both, so the experiment tests new tasks within familiar repository coverage.
| Fold 1 | Fold 2 | |
|---|---|---|
| Questions | 195 | 195 |
| Parallel / detour questions | 62 / 133 | 62 / 133 |
| Tasks | 86 | 72 |
| NodeBB | 11 | 2 |
| ansible | 29 | 43 |
| element-web | 16 | 12 |
| flipt | 15 | 13 |
| vuls | 10 | 12 |
| teleport | 11 | 11 |
| openlibrary | 38 | 35 |
| navidrome | 13 | 23 |
| webclients | 7 | 16 |
| qutebrowser | 24 | 21 |
| tutanota | 21 | 7 |
Transfer results
The student answers both orders correctly on 187/390 questions versus 117/390 for the base model. To produce one advice choice, it sums answer-token log odds across both orders, giving 287/390 correct combined decisions versus 207/390. That advice-selection rate is a different metric from strict two-order accuracy.
Appendix H End-to-End Experiment Details
The advice experiment selects 41 tasks with previously mined forks across 11 repositories, totaling 98 forks. The student picks the supported candidate at 77 forks. A fixed SWE-agent executor receives advice before the problem statement, with 75 model calls and an 80-minute total limit. Official SWE-bench Pro checks grade its patch. The source reports 33.7% student-advice success but does not give an integer success count or averaging details explaining that percentage over 41 tasks. Exact advice templates and per-task fork decisions are retained below.
# Task-specific pitfall notes (from prior runs on this exact task)
These notes are distilled from earlier agent runs that attempted THIS EXACT task and either passed or failed the hidden tests. They describe decision points where runs diverged.
READ THIS FIRST — what these notes are NOT:
- They are NOT evidence about YOUR patch. They describe other runs, not this one.
- A note saying a route "passed" means it passed FOR THAT RUN. Your implementation differs in details you cannot see; the only way to know your patch works is to run checks yourself in this session.
- Do not submit because your approach matches a passed route. Submitting without running your own verification has historically produced failing patches on this exact task even when the approach was right.
SUBMISSION CONTRACT (mandatory): before you submit, you must (1) run the project's relevant tests or, if they cannot run, a focused check you construct yourself, (2) observe the actual output, and (3) fix and re-run until it passes or your budget runs out. If you have not executed a verification command and seen its output in this session, you are not ready to submit.
Work on the task in your normal way. Do not spend actions searching the codebase for the situations below, and do not restructure your plan around them. Only if you find yourself already facing one of these decisions, use the note to avoid the known trap — then verify as usual.
## Decision trap {n} [{construction}] (step {fork step})
### Situation where prior runs diverged
{situation}
### Known trap (this route produced a FAILING patch)
{candidate to avoid}
### Route that avoided the trap in a prior run
{candidate to take}
Reminder: choosing this route is not verification. After acting, produce your own evidence that the resulting behavior is correct.## Decision trap 1 [parallel] (step 79)
### Situation where prior runs diverged
Refactor Teleport's HSM/KMS test configuration under `repo/` so `lib/auth/keystore/testhelpers.go` exposes `HSMTestConfig(t *testing.T) Config`, selecting YubiHSM, CloudHSM, AWS KMS, GCP KMS, or SoftHSM from `TELEPORT_TEST_*` environment configuration and failing the test when none is available. Add reusable per-backend configuration functions returning configuration plus availability, validate missing or incomplete environment setup consistently, and replace duplicated detection in the keystore and HSM integration tests without breaking backend-specific behavior. Focus on the required test-infrastructure source changes and verify the affected Go packages.
### Known trap (this route produced a FAILING patch)
Keep the existing PKCS#11-specific alert assertion unchanged and limit the refactor to centralized configuration and availability plumbing. Finish with the focused package checks already run.
### Route that avoided the trap in a prior run
Review integration assertions for assumptions invalidated by the expanded backend selector. Replace the alert check that specifically expects “PKCS#11 HSM keys” with a backend-neutral assertion so the same test remains valid for KMS configurations.
Reminder: choosing this route is not verification. After acting, produce your own evidence that the resulting behavior is correct.| Task | Forks | Student correct |
|---|---|---|
| NodeBB (3c85b94) | 1 | 1 |
| ansible (0fd8871) | 1 | 1 |
| ansible (395e5e2) | 4 | 4 |
| ansible (7094849) | 1 | 1 |
| ansible (7765870) | 3 | 2 |
| ansible (be2c376) | 3 | 3 |
| ansible (cd9c4eb) | 4 | 3 |
| ansible (ea04e00) | 7 | 6 |
| ansible (ed6581e) | 1 | 1 |
| element-web (494d9de) | 1 | 0 |
| element-web (b7fea97) | 1 | 1 |
| element-web (ca8b1b0) | 2 | 2 |
| flipt (3ef34d1) | 3 | 1 |
| flipt (b22f5f0) | 1 | 1 |
| vuls (4c04acb) | 2 | 2 |
| vuls (abd8041) | 1 | 1 |
| vuls (dc49646) | 2 | 2 |
| vuls (e52fa8d) | 1 | 1 |
| teleport (6eaaf3a) | 2 | 2 |
| teleport (baeb269) | 1 | 1 |
| openlibrary (08ac40d) | 3 | 3 |
| openlibrary (3c48b4b) | 3 | 1 |
| openlibrary (3f580a5) | 2 | 2 |
| openlibrary (5899980) | 3 | 3 |
| openlibrary (6a117fa) | 1 | 0 |
| openlibrary (9cd47f4) | 3 | 3 |
| openlibrary (b4f7c18) | 3 | 1 |
| openlibrary (e010b2a) | 1 | 1 |
| navidrome (8383527) | 2 | 0 |
| webclients (01b519c) | 3 | 3 |
| webclients (4feccbc) | 4 | 4 |
| qutebrowser (1943fa0) | 1 | 1 |
| qutebrowser (2dd8966) | 2 | 1 |
| qutebrowser (9902914) | 1 | 1 |
| qutebrowser (996487c) | 1 | 1 |
| qutebrowser (a84ecfb) | 5 | 4 |
| qutebrowser (de4a1c1) | 4 | 2 |
| tutanota (09c2776) | 3 | 1 |
| tutanota (1e516e9) | 4 | 4 |
| tutanota (219bc8f) | 3 | 2 |
| tutanota (da4edb7) | 4 | 3 |
| Total | 98 | 77 |
Authors and resources
The authors are affiliated with City University of Hong Kong, Microsoft, and an independent researcher. Wenbo Pan’s contribution was made during a Microsoft internship. The original resource and correspondence panel is retained below.
