Literature
Display
← All papers

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

Wenbo Pan · Zhichao Liu · Shujie Liu · Jingying Zeng · Chin-Yew Lin · Xianfeng Tang · Yan Lu · Qi He · Xiaohua Jia

Preprint posted · arXiv (preprint) ↗

AI agents11 min text · CondensedOriginal paper ↗
Contents

At a glance

Taste-Bench tests whether a model can choose a promising direction before later work reveals the result. It contains 502 paired choices mined from engineering and research trajectories. The best reported model scores 59.7% under a domain-balanced metric requiring correct answers in both option orders. A distilled student improves held-out engineering accuracy from 30.0% to 47.9%. Advice based on that student also improves reported task success, but it uses forks mined from earlier runs on the same evaluation tasks, rather than discovering decisions online from scratch.

Introduction

Long tasks depend on intermediate choices as well as execution. The paper calls this anticipatory judgment “taste” and measures it using later trajectory outcomes as labels for earlier decisions. Parallel attempts expose divergent strategies; detours expose approaches an agent abandons and replaces. The resulting benchmark supplements end-to-end success metrics. [9] [10] [11] [12] [13]

Original figure
Figure 1: A decision fork from a machine-learning trajectory. The evaluated model chooses before the later losses reveal that candidate A is the better direction.
Original figure

Measuring Taste via Long-Horizon Judgment

A question contains the task, a pre-decision trajectory prefix, and two candidate directions. The model chooses without seeing later branch evidence. Labels favor the branch with the better recorded outcome. Aligned prefixes and filtering try to isolate decision quality, but observed trajectories still include downstream execution differences; a retained label is not a randomized causal estimate of that decision’s value.

Equation 1
y=arg⁡maxi∈{1,2}⁡U⁡(Ei).y=\arg\max_{i\in\{1,2\}}U(E_{i}). (1)

Taste-Bench

The 502 questions cross two domains with two construction methods: engineering/research and parallel/detour. Candidates are written neutrally, and later outcomes remain hidden during benchmark evaluation.

Forks from parallel trajectories

Parallel questions compare attempts at the same task that take different approaches at comparable points and finish differently. The prefix comes from the supported branch. Generator checks seek a shared subproblem and evidence linking the chosen approach to the outcome.

Forks from detour trajectories

Detour questions contrast an approach that visibly failed with the later recovery. The question stops before the original mistake. Mechanical checks verify quoted evidence and event order; the generator must reject a recovery that would only be nameable with hindsight or an implausible losing option.

Refer to caption
Figure 2: Construction and filtering of Taste-Bench. The parallel construction contrasts branches of the same task, while the detour construction contrasts an abandoned direction with the later recovery. The filtering stage removes trivial questions, whose answer can be guessed from the wording of the candidates alone, and undecidable questions, whose label is not clearly consistent with the full record of the task.

Mining, filtering, and composition

The source pools contain 2,677 engineering rollouts on 517 SWE-bench Pro tasks and 1,132 research runs on 47 tasks from METR’s MALT release. Of 4,657 candidate forks, 502 survive generation and filtering: 390 engineering and 112 research questions. Four judges reject questions all can solve from candidate wording alone, then retain only labels all support after seeing full outcomes. These filters shape the released difficulty and label agreement. [11] [13] [14] [15]

A two-reviewer audit includes 70 released and 30 rejected questions. After outcomes are shown, 170 of 172 explicit A/B judgments match mined labels; 28 of 200 judgments without a clear preference are excluded. This 98.8% conditional agreement is not an accuracy estimate over all released questions.

Evaluation protocol

Each question is answered twice with opposite option orders. The primary score requires both answers to be correct, penalizing order-sensitive choices; mean per-presentation accuracy is reported separately. Independent random answers score 25% under the two-answer rule, while always choosing the same position scores zero.

Experimental Results

Experiments compare model choices, difficulty by construction and evidence horizon, reasoning effort, and correspondence with an execution benchmark.

Setup

Fourteen models receive the same prompts and maximum output budget. The headline Average gives research and engineering equal weight, despite their unequal question counts. It is neither ordinary single-answer accuracy nor a pooled percentage over all 502 items.

Accuracy of current models

GPT-5.6 Sol scores 59.7% and GPT-5.5 59.5% on the domain-balanced two-order metric. Rankings vary by domain and construction. Invalid answer formats count as failures, substantially affecting some weaker rows; the full results separate these cases and include item-bootstrap intervals.

Refer to caption
Figure 3: Model performance on Taste-Bench. The first column reports the Average defined in Section 4, followed by the research and engineering subsets. Each column shows the model names again and is sorted independently from highest to lowest within that split. Solid bars show accuracy when both candidate orders are answered correctly. The dotted extension shows the mean accuracy over the two orders. Whiskers show 95% item-bootstrap intervals for the solid-bar endpoint, and the printed values report that endpoint. Unparseable responses count as incorrect.
Finding 1: model performance

Effect of the time horizon

A model annotates how much later evidence appears necessary: already in the prefix, inferable from it, visible next step, or requiring more work. Mean accuracy across the 14 models declines from 62.3% to 21.0% across the first and last levels. These are judgment-based ordinal labels, not measured task durations, and the longest level contains 69 questions.

Original figure
Figure 4: Accuracy by time horizon and by reasoning budget. Each fork is annotated with a time horizon on four levels, from in prefix to more work. Left. The dark line is the mean over the 14 models, with 95% tt-intervals of that mean, and the faint lines show the individual models. Right. Accuracy at the same levels under three reasoning-effort settings per model, where the whisker is the 95% bootstrap interval of one setting of one model.
Finding 2: evidence horizon

Effect of the reasoning budget

Across three effort settings for each of two models, the highest-minus-lowest accuracy changes are −0.2 points for Sol and +2.2 for Luna, with intervals spanning zero. Harder forks consume more reasoning tokens without a clear overall gain. This supports a limited finding for these models and settings; Luna’s research subset separately rises from 43.8% to 54.5%, so the aggregate result is not a universal absence of benefit.

Finding 3: reasoning effort

Comparison with end-to-end benchmarks

For 11 models after excluding high parsing-failure rates, correlation with public SWE-bench Verified scores is r = 0.63 for the balanced Average and 0.37 for engineering alone. Confidence intervals are wide, and reasoning settings are not matched across benchmarks. The comparison suggests different rankings, rather than proving that the benchmark isolates an independent capability. [10] [29] [30]

Original figure
Figure 5: Taste-Bench against SWE-bench Verified. Each point is one model. The dashed line is the least-squares fit, red points deviate most from it, and the band marks the four highest SWE-bench Verified scores.

Generalizing Taste through Distillation

A Qwen3.6-27B student learns through LoRA adapters from a teacher of the same base architecture. The teacher receives a short description of the supported candidate, while the student sees only the question. Training transfers answer-informed reasoning into the student’s parameters.

Refer to caption
Figure 6: Overview of distillation and advice injection. Left. We distill the reasoning of a teacher that has seen the supported candidate into a student that sees only the question. Right. At evaluation time, the student’s judgment at each fork is written as advice into the task context, and a fixed executor completes the task independently.

Distillation recipe

The 390 engineering questions are divided into two task-disjoint folds. Each student trains on one and is evaluated on the other. Teacher-generated reasoning and final choices provide token-distribution targets; a later calibration pass supervises answer positions on the student’s own training-fold traces. The held-out task split limits direct memorization of task labels, while every repository appears in both folds. [31] [32]

Transfer to unseen tasks

Pooled held-out predictions improve two-order accuracy from 30.0% to 47.9%, a 17.9-point gain. Mean per-presentation accuracy rises from 42.7% to 62.4%. These held-out metrics differ from the 48.6%→92.9% training-fold single-order comparison. The student remains below the strongest evaluated model on engineering questions.

Refer to caption
Figure 7: Distillation results. Left. On the 390 held-out engineering questions, scored on both orders as in Figure 3, the distilled student reaches a higher accuracy than the base model. Whiskers are 95% item-bootstrap intervals, and the dashed line marks the accuracy of GPT-5.6 Sol on the same questions (56.9%). Right. On held-out SWE-bench Pro tasks, the executor’s success rate with student advice reaches most of the upper bound, the dashed bar, where the advice at every fork is correct.

End-to-end evaluation

Before a fresh executor run, notes recommend choices at forks mined from prior runs on that same held-out task. The student was trained on other tasks, but the advice-generation pipeline has access to task-specific historical forks. With a fixed Qwen3.6-27B executor on 41 selected SWE-bench Pro tasks, reported success is 14.6% without advice, 39.0% with correct-choice advice, and 33.7% with student advice. Correct advice is a comparison condition, not a proved upper bound over all possible advice. The experiment tests use of precomputed task knowledge, rather than fully online judgment on previously unexplored tasks.

Finding 4: distillation and advice

Related Work

Related work covers execution benchmarks, prediction of research outcomes, process evaluation, and distillation with privileged information.

Long-horizon agent evaluation.

AgentBench, SWE-bench, and research-agent benchmarks grade completed work. Taste-Bench instead presents a choice inside a recorded task. The two evaluation styles measure related but distinct aspects of the pipeline. [9] [10] [11] [12] [13]

Judging research directions before the outcome.

Earlier studies compare research ideas, solutions, or proposals before execution. This benchmark adds a concrete task history and outcomes from later trajectories, with the trade-off that those outcomes remain imperfect counterfactual evidence. [33] [34] [35] [36]

Process evaluation and step-level judgment.

Process-supervision and agent-judging methods score steps, alternatives, or complete trajectories. Taste-Bench asks for a choice before revealing the continuation used to label it. [37] [38] [39] [40] [41] [42] [43] [44] [45]

Distillation and internalizing context.

The recipe follows knowledge and context distillation: a teacher receives privileged information and the student learns to imitate its output with less context. Here that information identifies the supported choice, and an advisor later passes the student’s decisions to an executor. [46] [47] [48] [49] [50] [51] [52]

Conclusion

Trajectory-mined choice questions provide a practical diagnostic, and answer-informed distillation improves held-out choices in the tested setting. Interpretation depends on the filtering panel, task selection, outcome-based labels, and the distinction between precomputed advice and autonomous online decisions.

Reproducibility Statement

The authors release the benchmark data and evaluation code and recorded results. The appendices retain prompts, data counts, settings, and treatment of long inputs and invalid answers.

Ethics Statement

The study uses technical agent trajectories and two human reviewers. The authors report removing credential strings before model processing and acknowledge dual-use implications of stronger long-horizon agents.

AI Use Statement

The authors disclose generative-AI assistance with synthetic data, experimental design, implementation, interpretation, writing, figures, and literature searches, while taking responsibility for the final work.

Appendix A Benchmark Construction Details

The construction appendix records source pools, exact prompts, filtering checks, and the number of questions retained at each stage.

Source trajectories

Engineering rollouts span 11 repositories and 31 collection runs from April–July 2026 using GPT-5.4/5.5. Research transcripts come from several Claude and DeepSeek models. The two research-detour passes revisit a pool of trajectories, so their input counts should not be added as distinct source runs.

Statistics of the mined forks

Parallel mining prioritizes same-model, same-setting pairs; 13 of 124 released engineering pairs nevertheless compare different models. Detour candidates require literal evidence for commitment, failure, recovery, and success in order. The table preserves task counts and prefix sizes. Comparable prefixes and these checks reduce obvious confounds without making the comparison experimentally controlled.

Table 1: Size of the released questions per cell. Steps is the number of trajectory steps before the fork, and words is the number of words in one candidate.
Cell Questions Tasks Trajectories Steps (median) Steps (10th–90th) Words (median)
Parallel engineering 124 62 120 35 21–57 38
Parallel research 48 8 32 56 6–132 27
Detour engineering 266 133 266 47 24–71 23
Detour research 64 22 64 30 3–81 22

Prompts of the generator and the judges

GPT-5.6 Sol generates candidate forks. Kimi K2.5, GPT-4.1, Llama 4 Maverick, and Mistral Large 3 form the filtering panel. Exact source prompts below specify evidence checks, neutral wording, rejection criteria, and the two filtering views.

You build parallel judgment benchmark items from real agent trajectories. Return one JSON object only. Ground every decision point and option in the supplied trajectories. Do not invent grader evidence or use prior benchmark labels. Treat rejection as a correct and preferred result whenever a clean causal fork cannot be demonstrated from exact post-decision evidence.
# Goal

Find one genuine decision fork where the branch with the better native outcome made a
better task-solving decision than the worse branch. Locate the decision in the raw
trajectories and write two neutral, parallel candidate next steps.

Reject the candidate if the outcome difference is mostly a typo, crash, luck, missing
dependency, or generic execution quality rather than the chosen approach.

# Task
{query}

# Outcome interpretation
{outcome_standard}

# Branches

{branches}

# Required JSON

Return exactly one object with:
{
  "is_good_sample": true or false,
  "reason": "brief outcome-grounded reason",
  "query": "self-contained task statement preserving important constraints",
  "good_traj": "trajectory id of the passing branch",
  "bad_traj": "trajectory id of the selected failing branch",
  "good_start_step": integer position immediately before the good branch commits,
  "bad_start_step": integer position immediately before the bad branch commits,
  "good_choice": "1-3 sentence neutral next step",
  "bad_choice": "1-3 sentence neutral next step",
  "evidence": {
    "good_action": {"step": integer, "quote": "exact excerpt from that step"},
    "bad_action": {"step": integer, "quote": "exact excerpt from that step"},
    "good_post_decision": [
      {"step": integer, "quote": "exact excerpt showing a consequence or result"}
    ],
    "bad_post_decision": [
      {"step": integer, "quote": "exact excerpt showing a consequence or result"}
    ]
  }
}

Requirements:
- good_traj must be the branch with the better supplied outcome; bad_traj the worse one.
- Step positions must be within their corresponding trajectories and before commitment.
- Both choices must address the same immediate subproblem from comparable decision points.
- The evidence that distinguishes the choices must occur after the proposed decision points.
  If the better option is already identifiable from pre-decision differences, reject the sample.
- The prefix from good_traj before good_start_step must contain enough task context.
- Preserve important task constraints in query even if they appear early in a transcript.
- Choices must describe the actual branch decisions, at the same specificity and tone.
- Do not mention scores, pass/fail, grader results, branch names, or hindsight in choices.
- Do not put a branch's self-reported or predicted target metric in a choice when its
  direction makes that choice mechanically preferable (for example, merely claiming a
  lower predicted loss). Describe the decision procedure or controllable action instead.
- Do not use praise, blame, confidence, caution, or other wording that makes one option an
  obvious strawman. If neutral parallel wording is not possible, reject the sample.
- In reason, identify the post-decision trajectory evidence connecting the decision to the
  native outcome. If the outcome gap is not causally attributable to the decision, reject.
- Reconstruct the timeline before accepting. A score or artifact that already existed
  before the proposed cut cannot be credited to the proposed next step.
- The proposed action must be new at the cut. Reject if the same action was already
  executed before the cut, or if cited post-decision evidence repeats evidence already
  present before the cut. Re-running the same command is not a causal decision fork.
- An experiment contributes evidence only when its output shows a completed result.
  Starting a script followed by timeout, OOM, interruption, or missing output is not a
  successful experiment.
- Writing a claimed or predicted objective value into the answer is not an observed task
  outcome. A format, budget, or schema validator proves only those constraints; it does
  not independently validate task performance. If no post-decision evidence connects the
  differing action to the measured objective, reject the sample.
- Compare actual implemented actions, not plans or filenames. If both branches implement
  the same underlying approach, reject even if their wording differs.
- Reject when failure is mainly submission/finalization protocol, timeout, crash, missing
  dependency, hard-coded answer, task-version mismatch, or instruction noncompliance.
- Every evidence quote must be a literal excerpt from the cited trajectory step. Evidence
  is checked mechanically after generation.
You build judgment benchmark items from one real agent trajectory. Return one JSON object only. Ground every claim in exact excerpts from the supplied steps. Treat rejection as a correct and preferred result whenever a clean detour cannot be demonstrated, or whenever the better approach is only nameable with hindsight.
# Goal

Find one genuine DETOUR in this single trajectory: the agent committed to a poor technical
approach, pursued it, hit a wall, and recovered with a better approach that worked. Write
the fork as two neutral, parallel candidate next steps.

- bad_choice = the poor approach the agent committed to first.
- good_choice = the better approach it recovered to.

# Task
{query}

# Native outcome
{outcome_note}

# Complete trajectory
trajectory_id: {trajectory_id}

{trajectory}

# Required JSON

Return exactly one object with:
{
  "is_good_sample": true or false,
  "reason": "brief, step-grounded reason",
  "query": "self-contained task statement preserving important constraints",
  "breakpoint_step": integer index where the prefix ends; the agent is AT the fork and has
                     NOT yet committed to the poor approach,
  "good_choice": "1-3 sentence neutral next step",
  "bad_choice": "1-3 sentence neutral next step",
  "evidence": {
    "bad_action": {"step": integer, "quote": "exact excerpt committing to the poor approach"},
    "wall": {"step": integer, "quote": "exact excerpt of the observed failure that ended it"},
    "recovery_action": {"step": integer, "quote": "exact excerpt committing to the recovery"},
    "success": [
      {"step": integer, "quote": "exact excerpt showing the recovery worked"}
    ]
  }
}

Requirements:
- Ordering is mandatory and is checked mechanically:
  breakpoint_step <= bad_action.step <= wall.step < recovery_action.step <= success[*].step.
  bad_action and wall MAY cite the same step when one message contains both the commitment
  and the failure it produced (common when a command and its output are narrated together).
- Every quote must be a literal excerpt from the step it cites.
- The wall must be an OBSERVED FAILURE in the trajectory: a non-zero exit, an error,
  traceback, "not found", failing assertion, or an explicitly reported failing result. Your
  own remark that something could be improved, a clean diff listing, or a successful
  command is NOT a wall. This is checked mechanically and a fabricated wall is discarded.
- steps[0:breakpoint_step] is the prefix shown to a test-taker. It must carry enough
  context to reason, and must NOT reveal the failure, the wall, or the eventual fix.
- REJECT if the recovery is only nameable after seeing the failure. Ask: standing at
  breakpoint_step with the prefix alone, could a competent engineer have proposed
  good_choice? "Revert the change to the file the traceback names" or "raise the timeout
  the error reports" are hindsight lookups, not taste. This is the most common way a
  candidate fails; prefer rejection when unsure.
- When the deciding fact is discovered mid-rollout (a missing tool, an unavailable
  dependency, a broken toolchain), do NOT reject on that ground. RELOCATE the fork to after
  the discovery and mine the SECOND cycle: the agent's first reaction to the now-known
  constraint is bad_action, that reaction failing is the wall, and the strategy it settled
  on is recovery_action. Rejecting here when a second cycle exists is a wasted candidate.
  Worked example. Step 20 prints `go: command not found`. Put breakpoint_step at 21, so the
  prefix already establishes there is no host toolchain. bad_action (step 21) = probing the
  filesystem for a stray Go install; wall (step 24) = that install being the wrong version
  for the repo's go.work; recovery_action (step 26) = building in a pinned container.
  Both options are then strategies for coping with a constraint the reader can see.
- Reject on this ground only when the rollout contains no such usable second cycle.
- ACCEPT a fork where genuinely different strategies were available, INCLUDING strategy
  about how to obtain a working environment or a trustworthy verification path. Real
  examples of valid forks: run the suite in a pinned container versus probe the host for a
  toolchain; declare pinned dependencies for an isolated run versus assemble a PYTHONPATH
  out of scavenged directories; write a narrow stubbed probe versus stand up the whole
  application stack. The wall being an environment failure does not disqualify the fork —
  what matters is whether a competent engineer had a real decision to make.
- BOTH OPTIONS MUST BE LIVE CANDIDATES. Someone who sees only the prefix must be able to
  pick either one plausibly; telling them apart has to take real technical judgment rather
  than a surface cue. Neither may be a strawman. In particular, an option that merely
  repeats an action the prefix already shows failing — "keep calling the binary the prefix
  just reported as not found" — is a strawman, and so is any option nobody would actually
  propose at that point. If the losing side is not something a competent engineer might
  genuinely have chosen there, REJECT the candidate rather than writing it up. This is the
  single most common reason a candidate is thrown out downstream.
- REJECT if the agent never actually committed to the poor approach (it only mused about
  it), or if the "recovery" is the same underlying approach with different wording.
- REJECT if the two choices address different subproblems, or if the fork is mere
  execution ordering rather than a technical-judgment fork.
- Phrase both choices in strictly parallel form: same length range, same specificity, same
  tone. No evaluative or leaking words ("correctly", "mistakenly", "as it turns out",
  "better", "should", "properly"). No mention of grades, failures, hindsight, step numbers,
  or which one the trajectory took. Both must read as plausible candidate next steps.
- Do not put the wall's error text, file name, or symbol into good_choice when that alone
  identifies it as correct.
You are a reviewer at a real decision point. Choose the better next step from the task, trajectory prefix, and neutral options. End with exactly ANSWER: X.
# Candidate next steps
Option A:
{choice text}

Option B:
{choice text}

Without additional task or trajectory context, which option appears to be the better
engineering decision? End with exactly ANSWER: X.
# Task
{query}

# Shared trajectory before the decision
{prefix}

# Complete branch continuations
## Option A
Proposed next step:
{choice text}

Trajectory continuation through its final step:
{steps from the decision point to the end of that branch}

## Option B
Proposed next step:
{choice text}

Trajectory continuation through its final step:
{steps from the decision point to the end of that branch}

# Feedback from native grader
{outcome standard of the dataset}
Option A: passed=...; score=...
Option B: passed=...; score=...

Which option produced the better outcome under the dataset's core metric?
End with exactly ANSWER: X.

Counts at each stage

The funnel retains 1,809 candidates after the generator rubric, 729 after the wording-only filter, and 502 after outcome agreement. Table 2’s “Proposed” counts describe the 1,809 rubric-approved candidates, not all 4,657 initial proposals. The two preserved rejected examples illustrate an obvious hard-coded strawman and a label dependent on an unstated timeout constraint.

Refer to caption
Figure 8: Filtering yield per cell. Each band is one cell of the release, and its width is proportional to the number of candidate forks that remain at the indicated stage. Of the 4,657 mined candidate forks, 1,809 pass the generator rubric, 729 survive the trivial filter, and 502 questions remain in the release.
Table 2: Proposed and released questions by cell. The detour research rows report the first pass and the second pass separately.
Cell Proposed Released
Parallel engineering 562 124
Parallel research 186 48
Detour engineering 658 266
Detour research, pass 1 241 40
Detour research, pass 2 162 24
Complete example removed as trivialComplete example removed as undecidable

Appendix B Example Questions

Four complete source examples follow, one for each construction/domain cell. They contrast field selection with copying an authentication block, embedding retraining with global rescaling, a production-path change with a helper-interface change, and nonlinear residual search with a linear approximation. Prefixes and later outcomes are retained so the benchmark labels can be inspected.

Complete parallel engineering exampleComplete parallel research exampleComplete detour engineering exampleComplete detour research example

Appendix C Evaluation Protocol Details

Prompts render trajectory steps and abbreviate command output beyond 700 characters. Inputs are limited to 65,536 tokens, retaining the first 25 prefix lines plus a fitting tail if necessary; main-run truncation affects one question for two models. Unparseable answers are wrong. Confidence intervals resample questions within domains, without a reported task-cluster adjustment. Full prompts and model settings remain below.

You are evaluating an autonomous agent at a technical decision point. The task, the agent's progress, and two candidate next steps are provided. Exactly one candidate is the better technical decision.
# Task given to the agent
{query}

# Agent's progress so far (transcript up to the decision point)
{prefix}

# Decision point
Option A:
{candidate text}

Option B:
{candidate text}

Which option is the better technical decision?
Return exactly one line: ANSWER: X
Table 3: Settings of the evaluated models.
Model Reasoning setting
GPT-5.6 Sol reasoning effort xhigh
GPT-5.5 reasoning effort xhigh
GPT-5.6 Luna, GPT-5.6 Terra default
GPT-5.4 Mini, GPT-5.4 Nano reasoning effort high
Claude Opus 5 default
Claude Sonnet 5 adaptive thinking
Grok 4.5 reasoning effort high
Grok 4.20 Reasoning default
DeepSeek V4 Flash reasoning effort max
GLM-5.2 reasoning effort max, temperature 1.0
MiniMax M3 default
Mistral Medium 3.5 reasoning effort high

Appendix D Full Results

The full results separate model performance, cross-benchmark correlation, horizon annotation, and reasoning-effort settings.

Results per model and per cell

The full table separates domains, constructions, mean per-presentation accuracy, order consistency, and invalid responses. Mean accuracy across models is 58.1% for parallel engineering and 35.9% for detour engineering; research is not uniformly harder. Parsing failures particularly depress some low-scoring models.

Table 4: Full results of the 14 models on Taste-Bench. P and D denote the parallel and detour constructions, and Eng and Res denote the engineering and research domains. Accuracies are percentages. Unparsed is the number of responses without a parseable answer over the 1,004 presentations.
Model Average Eng. Res. P-Eng P-Res D-Eng D-Res Mean acc. CC CW WC WW Unparsed
GPT-5.6 Sol 59.7 56.9 62.5 75.8 56.2 48.1 67.2 65.2 292 25 29 156 0
GPT-5.5 59.5 55.6 63.4 73.4 56.2 47.4 68.8 64.6 288 26 24 164 0
Claude Opus 5 55.5 46.7 64.3 71.0 64.6 35.3 64.1 60.3 254 26 25 197 0
Grok 4.5 54.6 52.1 57.1 61.3 56.2 47.7 57.8 62.0 267 40 43 152 10
GPT-5.6 Terra 54.0 50.0 58.0 70.2 58.3 40.6 57.8 62.2 260 40 46 156 0
GLM-5.2 53.9 47.9 59.8 64.5 60.4 40.2 59.4 64.0 254 52 49 147 17
Claude Sonnet 5 51.6 44.4 58.9 62.1 56.2 36.1 60.9 60.7 239 37 52 174 0
GPT-5.6 Luna 49.0 43.6 54.5 67.7 50.0 32.3 57.8 58.9 231 48 51 172 0
MiniMax M3 45.3 44.1 46.4 64.5 56.2 34.6 39.1 57.2 224 61 51 166 3
DeepSeek V4 Flash 43.3 38.5 48.2 58.1 50.0 29.3 46.9 54.9 204 59 55 184 2
GPT-5.4 Mini 40.1 25.6 54.5 26.6 54.2 25.2 54.7 49.5 161 46 59 236 112
Mistral Medium 3.5 37.7 41.5 33.9 43.5 41.7 40.6 28.1 51.8 200 70 67 165 24
GPT-5.4 Nano 36.6 32.1 41.1 46.0 43.8 25.6 39.1 48.8 171 65 61 205 93
Grok 4.20 Reasoning 15.7 22.6 8.9 28.2 8.3 19.9 9.4 29.0 98 78 72 254 459

Comparison with SWE-bench Verified

The 11-model correlation interval is [0.04, 0.89] for the balanced Average and [−0.30, 0.79] for engineering. Public SWE-bench results share a harness, but model effort settings are not fully matched to Taste-Bench, limiting causal explanations of ranking differences.

Time horizon annotation

GPT-5.5 assigns five ordinal horizon scores; the top two are merged because only 13 questions require the longest category. The four displayed levels contain 158, 219, 56, and 69 questions. Their composition differs by construction and domain, so horizon effects may include these differences.

# Task
{query}

# Trajectory before the decision
{prefix}

# Candidate next steps
Option A:
{candidate text}

Option B:
{candidate text}

# Correct answer
{letter} (established in hindsight from the completed run)

How far into the future would an observer at this decision point need to see before the correct choice becomes clearly justified? Use this scale:

0 - Explicit evidence in the prefix: a specific decisive fact is already visible before the decision (an error message, a stated constraint, a visible fix or diff) that directly rules out one option.
1 - Inferable from the prefix: no single decisive fact is visible, but weighing the hints already present in the prefix (domain judgment, combining observations) is enough to justify the correct option; no future observation is needed.
2 - Next observation settles it: the first direct observation after the decision (the output of one command or probe, the content of one opened file) would clearly justify the correct option.
3 - Quick local check needed: a completed, self-contained local verification is needed (running one unit test, a small script, a quick smoke experiment) before the correct option is clearly justified.
4 - Substantial future work needed: more than a quick local check is required - meaningful further implementation or investigation, results appearing across later stages of downstream work, or only the terminal outcome (final test suite, external grader, complete branch comparison).

Give a one-sentence justification, then end with exactly: SCORE: N
Table 5: Number of questions per time horizon level and cell.
Cell In prefix Inferable Next step More work
Parallel engineering 72 45 4 3
Parallel research 11 13 3 21
Detour engineering 69 128 36 33
Detour research 6 33 13 12
All 158 219 56 69
Table 6: Accuracy of every model per time horizon level. Accuracy requires both presentations to be answered correctly, as in Section 3.4.
Model In prefix Inferable Next step More work
(n=158n=158) (n=219n=219) (n=56n=56) (n=69n=69)
GPT-5.5 75.9 57.1 48.2 23.2
GPT-5.6 Sol 79.7 57.5 44.6 21.7
Grok 4.5 67.1 53.9 32.1 36.2
Claude Opus 5 67.1 47.0 50.0 24.6
GPT-5.6 Terra 73.4 48.9 41.1 20.3
GLM-5.2 69.0 49.8 32.1 26.1
Claude Sonnet 5 63.9 45.7 35.7 26.1
GPT-5.6 Luna 67.7 43.8 32.1 14.5
MiniMax M3 64.6 41.1 25.0 26.1
DeepSeek V4 Flash 61.4 35.6 30.4 17.4
Mistral Medium 3.5 54.4 42.5 14.3 18.8
GPT-5.4 Nano 49.4 31.1 25.0 15.9
GPT-5.4 Mini 46.2 30.1 16.1 18.8
Grok 4.20 Reasoning 32.3 16.4 14.3 4.3
Mean over models 62.3 42.9 31.5 21.0

Reasoning budget

Highest-minus-lowest effort intervals are [−3.2, 2.8] percentage points for Sol and [−1.6, 6.0] for Luna. No response reaches the output cap; one of 6,024 responses is unparseable. The table uses pooled accuracy across questions, unlike the domain-balanced headline Average. The research-specific Luna improvement is distinct from the overall near-null result.

Table 7: Accuracy under three reasoning-effort settings per model.
Model Setting Accuracy In prefix Inferable Next step More work
GPT-5.6 Sol low 56.2 79.1 53.9 44.6 20.3
GPT-5.6 Sol xhigh 55.6 78.5 52.1 46.4 21.7
GPT-5.6 Sol max 56.0 79.7 52.5 42.9 23.2
GPT-5.6 Luna none 43.4 60.8 44.3 28.6 13.0
GPT-5.6 Luna high 45.8 67.7 42.9 28.6 18.8
GPT-5.6 Luna max 45.6 65.2 43.4 35.7 15.9

Appendix E Agreement of the Judges

On released parallel-engineering questions, wording-only pairwise judge agreement is 51.7%. Full-outcome agreement is unanimous by construction, because the release filter requires it; that unanimity is not an independent validation statistic.

Table 8: Accuracy of the judges from the two candidates alone in the parallel engineering cell. Accuracies are percentages over the proposed questions and over the released questions. Agreement is the fraction of judge pairs that select the same candidate on the released questions.
Cell Judge Proposed Released Agreement
Parallel engineering Kimi K2.5 84.7 55.6 51.7
GPT-4.1 85.8 63.7
Llama 4 Maverick 75.6 34.7
Mistral Large 3 82.0 59.7

Appendix F Human Review

The human review checks whether the outcome-supported choice is persuasive and whether the displayed decision explains the later difference.

Sampling and annotation

The two reviewers assess the same 100 mixed released/rejected examples before and after outcome summaries. Agreement uses only explicit post-outcome A/B preferences: 170/172 match labels, and reviewers agree on 73/74 items where both give a preference. Excluding 28 uncertain or non-causal judgments and the small reviewer panel limit broader interpretation.

Table 9: Human review after outcome disclosure. Each reviewer assesses the same 100 questions. We exclude responses without a clear A/B preference, and label agreement is computed over the retained judgments.
Reviewer Retained judgments Label agreement
Reviewer 1 87 86/87 (98.9%)
Reviewer 2 85 84/85 (98.8%)
Combined 172 170/172 (98.8%)

Appendix G Distillation Details

The appendix gives the exact training procedure, task-disjoint folds, and the distinction between two-order evaluation and combined advice selection.

Recipe

Teacher traces are sampled at temperature 1 with a 3,072-token limit, retaining questions with usable traces in both orders. LoRA rank 16 adapts attention and MLP projections, totaling 79.7 million trainable parameters. The reported KL-based objective uses the top 100 tokens plus residual probability mass and at most 512 reasoning positions; answer-token calibration follows on training-fold student traces. Each fold takes roughly two hours on an A100 80 GB GPU. The loss is applied at selected positions; this does not by itself ensure other output distributions remain unchanged.

You are a senior software/research engineer reviewing an autonomous agent's work at a decision point. Judge the alternatives on technical merit and likely task outcome.
# Task given to the agent
{query}

# Agent progress up to the decision point
{prefix}

# Candidate next steps

Option A:
{candidate text}

Option B:
{candidate text}

Think through the engineering decision carefully. After thinking, show your
choice in the "answer" field of a JSON object using only "A" or "B", for
example {"answer":"A"}.
Below is a reference distilled from past agent runs on THIS EXACT task: at the current decision fork, the choice that led to success, and the choices that led to failure. Use it when relevant.

# Reference: key decisions from past runs on this task

### Decision fork (at the current point)

The right call at this point -- do this:
{text of the supported candidate}

Avoid:
- {text of the other candidate}


---

# Your task

{student user message}
Table 10: Distillation hyperparameters.
Setting Value
Base model Qwen3.6-27B, bfloat16
Adapter LoRA, rank 16, α=32\alpha=32, no dropout, on all attention and MLP projections
Trainable parameters 79.7M
Optimizer AdamW, gradient clipping at norm 1.0
Distillation learning rate 10−410^{-4}, 2 epochs, one view per step, 624 and 568 steps
Distillation loss forward KL at temperature 1.0, top-100 tokens plus one entry for the remaining mass, at most 512 reasoning positions
Calibration learning rate 10−510^{-5}, 1 epoch, softmax temperature 2.0, 306 and 320 steps
Sequence length at most 10,240 tokens, no truncation in the training data
Peak GPU memory 71 GB
Wall clock per fold 2.1 h and 1.8 h for distillation

Folds

Each fold has 195 questions, including 62 parallel and 133 detour items. Tasks, source trajectories, and prefixes do not cross folds, but all repositories occur in both, so the experiment tests new tasks within familiar repository coverage.

Table 11: Composition of the two task-disjoint folds.
Fold 1 Fold 2
Questions 195 195
Parallel / detour questions 62 / 133 62 / 133
Tasks 86 72
NodeBB 11 2
ansible 29 43
element-web 16 12
flipt 15 13
vuls 10 12
teleport 11 11
openlibrary 38 35
navidrome 13 23
webclients 7 16
qutebrowser 24 21
tutanota 21 7

Transfer results

The student answers both orders correctly on 187/390 questions versus 117/390 for the base model. To produce one advice choice, it sums answer-token log odds across both orders, giving 287/390 correct combined decisions versus 207/390. That advice-selection rate is a different metric from strict two-order accuracy.

Appendix H End-to-End Experiment Details

The advice experiment selects 41 tasks with previously mined forks across 11 repositories, totaling 98 forks. The student picks the supported candidate at 77 forks. A fixed SWE-agent executor receives advice before the problem statement, with 75 model calls and an 80-minute total limit. Official SWE-bench Pro checks grade its patch. The source reports 33.7% student-advice success but does not give an integer success count or averaging details explaining that percentage over 41 tasks. Exact advice templates and per-task fork decisions are retained below.

# Task-specific pitfall notes (from prior runs on this exact task)

These notes are distilled from earlier agent runs that attempted THIS EXACT task and either passed or failed the hidden tests. They describe decision points where runs diverged.

READ THIS FIRST — what these notes are NOT:
- They are NOT evidence about YOUR patch. They describe other runs, not this one.
- A note saying a route "passed" means it passed FOR THAT RUN. Your implementation differs in details you cannot see; the only way to know your patch works is to run checks yourself in this session.
- Do not submit because your approach matches a passed route. Submitting without running your own verification has historically produced failing patches on this exact task even when the approach was right.

SUBMISSION CONTRACT (mandatory): before you submit, you must (1) run the project's relevant tests or, if they cannot run, a focused check you construct yourself, (2) observe the actual output, and (3) fix and re-run until it passes or your budget runs out. If you have not executed a verification command and seen its output in this session, you are not ready to submit.

Work on the task in your normal way. Do not spend actions searching the codebase for the situations below, and do not restructure your plan around them. Only if you find yourself already facing one of these decisions, use the note to avoid the known trap — then verify as usual.

## Decision trap {n} [{construction}] (step {fork step})
### Situation where prior runs diverged
{situation}
### Known trap (this route produced a FAILING patch)
{candidate to avoid}
### Route that avoided the trap in a prior run
{candidate to take}
Reminder: choosing this route is not verification. After acting, produce your own evidence that the resulting behavior is correct.
## Decision trap 1 [parallel] (step 79)
### Situation where prior runs diverged
Refactor Teleport's HSM/KMS test configuration under `repo/` so `lib/auth/keystore/testhelpers.go` exposes `HSMTestConfig(t *testing.T) Config`, selecting YubiHSM, CloudHSM, AWS KMS, GCP KMS, or SoftHSM from `TELEPORT_TEST_*` environment configuration and failing the test when none is available. Add reusable per-backend configuration functions returning configuration plus availability, validate missing or incomplete environment setup consistently, and replace duplicated detection in the keystore and HSM integration tests without breaking backend-specific behavior. Focus on the required test-infrastructure source changes and verify the affected Go packages.
### Known trap (this route produced a FAILING patch)
Keep the existing PKCS#11-specific alert assertion unchanged and limit the refactor to centralized configuration and availability plumbing. Finish with the focused package checks already run.
### Route that avoided the trap in a prior run
Review integration assertions for assumptions invalidated by the expanded backend selector. Replace the alert check that specifically expects “PKCS#11 HSM keys” with a backend-neutral assertion so the same test remains valid for KMS configurations.
Reminder: choosing this route is not verification. After acting, produce your own evidence that the resulting behavior is correct.
Table 12: Per-task student decisions in the end-to-end experiment.
Task Forks Student correct
NodeBB (3c85b94) 1 1
ansible (0fd8871) 1 1
ansible (395e5e2) 4 4
ansible (7094849) 1 1
ansible (7765870) 3 2
ansible (be2c376) 3 3
ansible (cd9c4eb) 4 3
ansible (ea04e00) 7 6
ansible (ed6581e) 1 1
element-web (494d9de) 1 0
element-web (b7fea97) 1 1
element-web (ca8b1b0) 2 2
flipt (3ef34d1) 3 1
flipt (b22f5f0) 1 1
vuls (4c04acb) 2 2
vuls (abd8041) 1 1
vuls (dc49646) 2 2
vuls (e52fa8d) 1 1
teleport (6eaaf3a) 2 2
teleport (baeb269) 1 1
openlibrary (08ac40d) 3 3
openlibrary (3c48b4b) 3 1
openlibrary (3f580a5) 2 2
openlibrary (5899980) 3 3
openlibrary (6a117fa) 1 0
openlibrary (9cd47f4) 3 3
openlibrary (b4f7c18) 3 1
openlibrary (e010b2a) 1 1
navidrome (8383527) 2 0
webclients (01b519c) 3 3
webclients (4feccbc) 4 4
qutebrowser (1943fa0) 1 1
qutebrowser (2dd8966) 2 1
qutebrowser (9902914) 1 1
qutebrowser (996487c) 1 1
qutebrowser (a84ecfb) 5 4
qutebrowser (de4a1c1) 4 2
tutanota (09c2776) 3 1
tutanota (1e516e9) 4 4
tutanota (219bc8f) 3 2
tutanota (da4edb7) 4 3
Total 98 77

Authors and resources

The authors are affiliated with City University of Hong Kong, Microsoft, and an independent researcher. Wenbo Pan’s contribution was made during a Microsoft internship. The original resource and correspondence panel is retained below.

Resources and correspondence
Concept demonstration

Open image ↗