Overview
Tev combines procedurally generated task environments with a progress-based training reward. The overview shows results on the paper’s selected computing-and-math benchmark subsets and the complete Terminal-Bench 2.1 suite.

At a glance
Tev-Gym contains 1,170 verified task environments across seven computing-and-math domains. Each starts from a hidden correct world, renders and corrupts its public artifacts, and derives both completion checks and intermediate progress from the same specification.
Progress-potential reward shaping (PPRS) supplies learning signal when every rollout fails. Tev-27B improves ALE-CM best-of-five full success from 0.17 to 0.28 and transfers to selected Terminal-Bench tasks; a 320B model also improves. Results concern synthetic terminal workflows and small selected long-horizon evaluation sets, with different frontier harnesses and a remaining frontier-performance gap.
1 Introduction
Long workflows are difficult to generate consistently and yield sparse completion rewards. In group-relative RL, identical failing rewards provide no gradient, precisely when a weak policy needs feedback. Tev instead builds the task, public files, verifier, and progress scorer from one known-correct world.
The dataset spans software, data engineering, AI research, mathematics, security, infrastructure, and quantum computing. Seeded generators offer adjustable scale and dependency structure. Under a fixed evaluator, tasks take a median 12.4 minutes versus 0.5–2.0 minutes for comparison corpora; this is measured agent time, not human effort.
Train 9B on 264 unit-tier tasks, 27B on all 1,170, and apply the recipe to a 320B base. Model size and training coverage therefore change together; their differences do not isolate parameter scaling.

| Dataset | Size | # Domains | Agent time (min) | Oracle | Grading | Progressive | P@1 | P@8 | |
| Median | Mean | Metric | Reward | ||||||
| SWE-Smith (Yang et al., 2025a) | 50.1k | 1 | 2.0 | 2.4 | unit tests | pass/fail | ✗ | 47% | 52% |
| Terminal-Universe (Wu et al., 2026) | 32.0k | 3 | NA | NA | pytest | pass/fail | ✗ | NA | NA |
| TMAX-15K (Ivison et al., 2026) | 14.6k | 3 | 1.3 | 2.1 | unit tests | pass/fail | ✗ | 74% | 80% |
| SETA (Shen et al., 2026) | 4.5k | 4 | 1.2 | 2.1 | unit tests | test fraction | ✗ | 80% | 88% |
| Endless Terminals (Gandhi et al., 2026) | 3.3k | 2 | 0.5 | 0.7 | unit tests | pass/fail | ✗ | 83% | 89% |
| CLI-Gym (Lin et al., 2026) | 1.7k | 4 | 1.9 | 2.3 | unit tests | pass/fail | ✗ | 69% | 74% |
| Tev-Gym (ours) | 1,170 | 7 | 12.4 | 14.8 | latent world | staged pass/fail | ✓ | 29% | 40% |
2 Long-Horizon Environment Synthesis
Construct environments backward from a correct world, then enforce coverage and qualification before admitting them to training.

2.1 Latent-World-First Task Synthesis
A seeded generator first draws a valid latent world and an explicit defect plan. Code compiles its workflow into a DAG with dependencies and work estimates. Rendering produces clean heterogeneous files; round-trip checks establish that they encode the world correctly.
Declared, recoverable corruptions create the agent’s starting workspace. The prompt specifies deliverables, schemas, precedence rules, and tools. Hidden verifiers and progress scorers compare outputs with the same untouched world. Sealed metamorphic tests and stale-output counterfactuals check invariance and reject plausible wrong states.
Three worked examples extend the construction to a visual cloud-cost task, building control, and financial-statement reconstruction. In the simulator example, a pinned tool procedure supplies the reference rather than a generated record set alone.
Equation 1
| (1) |
2.2 Coverage, Calibration, and Qualification
Coverage is checked over 19 axes, including domain, scale, workflow, artifacts, and corruption type. World scale controls operational work, while dependency depth and corruption classes control required stages.
Every environment must pass eight qualification gates. Progress must respond to correct work and regressions; 14 specified gaming probes must score zero; solvers, verifier, and scorer have disjoint code closures. These checks provide concrete evidence against tested exploits, not a proof against every possible shortcut.

3 Dense Reward for Long-Horizon RL
The same hidden specification grades completion and intermediate work, enabling progress feedback without a learned critic or language-model judge.
3.1 Terminal Reward Starves Group-Relative RL
GRPO normalizes rewards within a task’s rollout group [46]. Under binary completion reward, all-fail groups have zero advantage. Their probability is (1−p)G: with eight independent rollouts it is 66% at 5% success and 92% at 1%. Even one success leaves every failure equally negative, obscuring near-misses.
Equation 2
| (2) |
3.2 A Progress Potential from the Latent World
A progress contract assigns weighted checkable nodes to the workflow. Each node’s partial score lies in [0,1], and its credited value is capped by the weakest direct or indirect prerequisite. The potential Φ sums these gated scores, preventing a correct-looking downstream artifact from bypassing incorrect inputs.
The untouched tree and tested exploits score zero; complete accepted workspaces score one. Independent scorer code checks the latent world directly. Correct intermediate states can be generated for each seed, avoiding human step labels and learned reward models.
Equation 3
| (3) |
3.3 Progress-Potential Reward Shaping (PPRS)
Each turn earns the change in progress potential. With λ=0.25 and initial potential zero, the total shaped return telescopes to the submission-success indicator plus λ times final progress. A submitted passing rollout earns 1.25; a failing or unsubmitted rollout earns at most 0.25. A complete workspace that is never submitted therefore receives only 0.25.
PPRS deliberately retains terminal potential and changes the objective, unlike policy-invariant potential shaping [40]. Successful trajectories still outrank failures, but rankings of stochastic policies can reverse. Different failing progress levels supply reward variance; groups stuck at identical progress remain uninformative.
Equation 4
| (4) |
| Model | Harness | Math & Science | ML | Coding & System | All (Avg) | ||||||||||||
| S@1 | S@5 | P@1 | P@5 | S@1 | S@5 | P@1 | P@5 | S@1 | S@5 | P@1 | P@5 | S@1 | S@5 | P@1 | P@5 | ||
| Frontier API models | |||||||||||||||||
| GPT-6 Astra | Codex | 0.58 | 0.58 | 0.38 | 0.38 | 0.48 | 0.49 | 0.20 | 0.20 | 0.81 | 0.85 | 0.20 | 0.20 | 0.62 | 0.63 | 0.28 | 0.28 |
| Claude Fable 5 | Claude Code | 0.47 | 0.51 | 0.38 | 0.38 | 0.52 | 0.57 | 0.24 | 0.40 | 0.83 | 0.90 | 0.20 | 0.20 | 0.58 | 0.63 | 0.29 | 0.33 |
| GPT-5.6 (sol) | Codex | 0.50 | 0.51 | 0.38 | 0.38 | 0.46 | 0.49 | 0.20 | 0.20 | 0.82 | 0.85 | 0.20 | 0.20 | 0.58 | 0.60 | 0.28 | 0.28 |
| GPT-5.6 (terra) | Codex | 0.51 | 0.56 | 0.38 | 0.38 | 0.25 | 0.26 | 0.00 | 0.00 | 0.75 | 0.83 | 0.20 | 0.20 | 0.50 | 0.55 | 0.22 | 0.22 |
| Claude Opus 5 | Claude Code | 0.51 | 0.53 | 0.38 | 0.38 | 0.43 | 0.46 | 0.20 | 0.20 | 0.85 | 0.87 | 0.20 | 0.20 | 0.58 | 0.60 | 0.28 | 0.28 |
| Claude Sonnet 5 | Claude Code | 0.50 | 0.51 | 0.38 | 0.38 | 0.39 | 0.56 | 0.12 | 0.40 | 0.80 | 0.87 | 0.20 | 0.20 | 0.55 | 0.62 | 0.26 | 0.33 |
| Claude Opus 4.8 | Claude Code | 0.48 | 0.51 | 0.38 | 0.38 | 0.31 | 0.33 | 0.00 | 0.00 | 0.80 | 0.86 | 0.20 | 0.20 | 0.52 | 0.55 | 0.22 | 0.22 |
| Gemini 3.7 Flash | Gemini CLI | 0.48 | 0.50 | 0.38 | 0.38 | 0.26 | 0.29 | 0.00 | 0.00 | 0.84 | 0.86 | 0.20 | 0.20 | 0.52 | 0.54 | 0.22 | 0.22 |
| Gemini 3.6 Flash | Gemini CLI | 0.48 | 0.51 | 0.35 | 0.38 | 0.25 | 0.26 | 0.00 | 0.00 | 0.81 | 0.86 | 0.20 | 0.20 | 0.51 | 0.54 | 0.21 | 0.22 |
| Gemini 3.1 Pro | Gemini CLI | 0.49 | 0.52 | 0.35 | 0.38 | 0.27 | 0.29 | 0.00 | 0.00 | 0.82 | 0.88 | 0.20 | 0.20 | 0.52 | 0.56 | 0.21 | 0.22 |
| Open-source models | |||||||||||||||||
| Qwen3.8-2.4T-A95B | Vanillux2 | 0.20 | 0.38 | 0.20 | 0.38 | 0.10 | 0.20 | 0.04 | 0.20 | 0.74 | 0.86 | 0.16 | 0.20 | 0.32 | 0.46 | 0.14 | 0.28 |
| DeepSeek-V4-pro | Vanillux2 | 0.12 | 0.25 | 0.12 | 0.25 | 0.10 | 0.20 | 0.04 | 0.20 | 0.73 | 0.88 | 0.16 | 0.20 | 0.29 | 0.41 | 0.11 | 0.22 |
| Qwen3.5-9B (base) | Vanillux2 | 0.07 | 0.12 | 0.00 | 0.00 | 0.19 | 0.26 | 0.00 | 0.00 | 0.59 | 0.73 | 0.20 | 0.20 | 0.25 | 0.33 | 0.06 | 0.06 |
| Tev-9B (ours) | Vanillux2 | 0.14 | 0.29 | 0.04 | 0.12 | 0.23 | 0.26 | 0.00 | 0.00 | 0.56 | 0.73 | 0.20 | 0.20 | 0.29 | 0.40 | 0.07 | 0.11 |
| 0.07 | 0.17 | 0.04 | 0.12 | 0.04 | 0.03 | 0.04 | 0.07 | 0.01 | 0.05 | ||||||||
| Qwen3.8-27B (base) | Vanillux2 | 0.15 | 0.25 | 0.15 | 0.25 | 0.21 | 0.32 | 0.00 | 0.00 | 0.74 | 0.82 | 0.20 | 0.20 | 0.33 | 0.43 | 0.12 | 0.17 |
| Tev-27B (ours) | Vanillux2 | 0.23 | 0.50 | 0.23 | 0.50 | 0.26 | 0.29 | 0.00 | 0.00 | 0.82 | 0.86 | 0.20 | 0.20 | 0.40 | 0.54 | 0.16 | 0.28 |
| 0.08 | 0.25 | 0.08 | 0.25 | 0.05 | 0.03 | 0.08 | 0.04 | 0.07 | 0.11 | 0.04 | 0.11 | ||||||
4 Experiments
Experiments test in-domain improvement, transfer to longer tasks, retention on a middle-horizon benchmark, and separate data/reward ablations.
4.1 Setup
Training uses DPPO, 24 rollouts for each of eight tasks per step, 65,536 response tokens per rollout, 16,384 per turn, and at most 64 tool calls through one bash tool [43]. The 27B run takes about four days on four B200 nodes or eight A100 nodes.
ALE-CM contains 18 executable computing-and-math tasks. Terminal-Bench 3.0/4.0 subsets contain 22/18 domain-related tasks runnable in the CPU-only harness; Claude assisted domain selection. High correlations with full-benchmark rankings support relevance but do not make these full-suite results.
No training prompt shares an 8-gram or 13-gram with the measured evaluation prompts. This lexical test and fresh generated artifacts do not exclude conceptual overlap or contamination in pretrained models.
4.2 Main Results
On ALE-CM, Tev-27B raises mean graded score S@1 from 0.33 to 0.40, full-success P@1 from 0.12 to 0.16, and P@5 from 0.17 to 0.28. The last matches the reported GPT-6 Astra and Claude Opus 5 value under their native harnesses; it does not establish general parity.
On the 22/18-task Terminal-Bench subsets, 27B P@1 rises 0.09→0.14 and 0.04→0.10. Tev-320B improves 0.20→0.27 and 0.29→0.37; its P@5 rises only 0.05 on 3.0 and stays unchanged on 4.0, suggesting improved reliability more than expanded task reach.
On all 89 Terminal-Bench 2.1 tasks, Tev-27B reaches 0.794 P@1 and 0.884 P@5, versus 0.764/0.864 for its base. The Kubernetes case illustrates better evidence selection and rollback judgment, but it is a selected single-rollout comparison.
| Model | TB 3.0 (22 tasks) | TB 4.0 (18 tasks) | ||||||||||||||||||
| Software | ML | Operations | Security | All | Software | ML | Operations | Security | All | |||||||||||
| P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | P@1 | P@5 | |
| Claude Fable 5 | 0.50 | 0.80 | 0.45 | 0.50 | 0.15 | 0.25 | 0.40 | 0.75 | 0.41 | 0.64 | 0.53 | 0.62 | 0.60 | 0.75 | 0.05 | 0.25 | 0.70 | 1.00 | 0.46 | 0.61 |
| Claude Opus 5 | 0.54 | 0.60 | 0.60 | 0.75 | 0.40 | 0.50 | 0.55 | 0.75 | 0.53 | 0.64 | 0.47 | 0.62 | 0.75 | 1.00 | 0.20 | 0.50 | 0.90 | 1.00 | 0.52 | 0.72 |
| GPT-5.6 (sol) | 0.32 | 0.50 | 0.65 | 0.75 | 0.10 | 0.25 | 0.70 | 0.75 | 0.41 | 0.55 | 0.33 | 0.62 | 0.70 | 1.00 | 0.00 | 0.00 | 0.80 | 1.00 | 0.39 | 0.61 |
| GPT-5.6 (terra) | 0.22 | 0.50 | 0.55 | 0.75 | 0.05 | 0.25 | 0.50 | 0.75 | 0.30 | 0.55 | 0.10 | 0.38 | 0.50 | 0.75 | 0.00 | 0.00 | 0.70 | 1.00 | 0.23 | 0.44 |
| GPT-5.6 (luna) | 0.18 | 0.30 | 0.25 | 0.50 | 0.00 | 0.00 | 0.30 | 0.50 | 0.18 | 0.32 | 0.12 | 0.25 | 0.25 | 0.50 | 0.05 | 0.25 | 0.50 | 0.50 | 0.18 | 0.33 |
| Grok 4.6 | 0.24 | 0.50 | 0.50 | 0.75 | 0.00 | 0.00 | 0.55 | 0.75 | 0.30 | 0.50 | 0.30 | 0.50 | 0.55 | 0.75 | 0.00 | 0.00 | 0.50 | 1.00 | 0.31 | 0.50 |
| Claude Sonnet 5 | 0.18 | 0.40 | 0.25 | 0.50 | 0.00 | 0.00 | 0.20 | 0.25 | 0.16 | 0.32 | 0.28 | 0.62 | 0.20 | 0.75 | 0.00 | 0.00 | 0.20 | 0.50 | 0.19 | 0.50 |
| Claude Opus 4.8 | 0.20 | 0.70 | 0.10 | 0.25 | 0.05 | 0.25 | 0.40 | 0.50 | 0.19 | 0.50 | 0.20 | 0.50 | 0.35 | 0.50 | 0.05 | 0.25 | 0.80 | 1.00 | 0.27 | 0.50 |
| Grok 4.5 | 0.14 | 0.30 | 0.45 | 1.00 | 0.15 | 0.50 | 0.50 | 0.75 | 0.26 | 0.55 | 0.12 | 0.12 | 0.35 | 0.75 | 0.00 | 0.00 | 0.60 | 1.00 | 0.20 | 0.33 |
| GLM 5.2 | 0.04 | 0.20 | 0.05 | 0.25 | 0.00 | 0.00 | 0.05 | 0.25 | 0.04 | 0.18 | 0.06∗ | 0.24∗ | 0.02∗ | 0.10∗ | 0.00∗ | 0.00∗ | 0.00∗ | 0.00∗ | 0.03∗ | 0.13∗ |
| Gemini 3.7 Flash∗ | 0.08 | 0.20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.20 | 0.50 | 0.07 | 0.18 | 0.12 | 0.25 | 0.00 | 0.00 | 0.00 | 0.00 | 0.30 | 0.50 | 0.09 | 0.17 |
| Gemini 3.6 Flash∗ | 0.04 | 0.20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.02 | 0.09 | 0.20 | 0.25 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.09 | 0.11 |
| Qwen3.8-27B (base) | 0.17 | 0.43 | 0.00 | 0.00 | 0.00 | 0.00 | 0.05 | 0.25 | 0.09 | 0.24 | 0.07 | 0.31 | 0.05 | 0.25 | 0.00 | 0.00 | 0.00 | 0.00 | 0.04 | 0.19 |
| Tev-27B | 0.15 | 0.39 | 0.19 | 0.25 | 0.00 | 0.00 | 0.20 | 0.25 | 0.14 | 0.27 | 0.17 | 0.33 | 0.12 | 0.46 | 0.00 | 0.00 | 0.00 | 0.00 | 0.10 | 0.25 |
| 0.02 | 0.04 | 0.19 | 0.25 | 0.15 | 0.05 | 0.03 | 0.10 | 0.02 | 0.07 | 0.21 | 0.06 | 0.06 | ||||||||
| GLM-5.3-Flash (base) | 0.18 | 0.60 | 0.35 | 0.50 | 0.05 | 0.25 | 0.22 | 0.50 | 0.20 | 0.50 | 0.16 | 0.50 | 0.60 | 0.75 | 0.05 | 0.25 | 0.68 | 1.00 | 0.29 | 0.56 |
| Tev-320B | 0.21 | 0.70 | 0.40 | 0.50 | 0.10 | 0.25 | 0.45 | 0.50 | 0.27 | 0.55 | 0.25 | 0.50 | 0.65 | 0.75 | 0.15 | 0.25 | 0.70 | 1.00 | 0.37 | 0.56 |
| 0.03 | 0.10 | 0.05 | 0.05 | 0.23 | 0.07 | 0.05 | 0.09 | 0.05 | 0.10 | 0.02 | 0.08 | |||||||||
| Model | Pass@1 | Pass@5 |
| GPT-6 Astra (Codex, high) | 0.874 | 0.944 |
| Fable 5 (Claude Code, xhigh) | 0.838 | 0.933 |
| Opus 4.8 (Claude Code, high) | 0.789 | 0.944 |
| Sonnet 5 (Claude Code, high) | 0.746 | 0.921 |
| Qwen3.5-9B (base) | 0.178 | 0.292 |
| Tev-9B (ours) | 0.227 | 0.348 |
| 0.049 | 0.056 | |
| Qwen3.8-27B (base) | 0.764 | 0.864 |
| Tev-27B (ours) | 0.794 | 0.884 |
| 0.030 | 0.020 |
4.3 Ablation Study
At 9B, prompt-first generation and a public-view verifier fail to improve mean graded score over the base. The latter discards 88% of sampled groups. Both use terminal rewards and should be compared with the sparse-reward arm when isolating data construction.
Keeping Tev-Gym fixed, PPRS uses undiscounted return-to-go and compares advantages at the same turn index. It improves S@1 by 0.04 over sparse reward and reduces GPU-hours per step from 16.7 to 9.0. Pooling turns weakens late credit; additive current-potential rewards encourage lingering and context exhaustion.
At 27B, the reported terminal-only run could not complete one optimizer step because nearly all groups had zero advantage. This is a result under the stated recipe and data, not proof that every sparse-reward training method must fail.
| Method | Data | Reward | GPU-h / step | ALE-CM | |||||||
| World | DAG | Hidden truth | Signal | Credit | Adv. norm. | S@1 | S@5 | P@1 | P@5 | ||
| Qwen3.5-9B (base) | – | – | – | no RL | – | – | – | 0.25 | 0.33 | 0.06 | 0.06 |
| Tev-9B (ours) | ✓ | ✓ | ✓ | + terminal | , | per turn | 9.0 | 0.29 | 0.40 | 0.07 | 0.11 |
| Data ablation: replace one construction step with an LLM; terminal reward, compare with Sparse | |||||||||||
| Public-view verifier | ✓ | ✓ | LLM | terminal | whole rollout | per rollout | 22.3 | 0.24 | 0.35 | 0.04 | 0.06 |
| Prompt-first | LLM | LLM | LLM | terminal | whole rollout | per rollout | 11.8 | 0.23 | 0.32 | 0.06 | 0.06 |
| Reward ablation: keep Tev-Gym, change how enters training | |||||||||||
| Sparse (terminal only) | ✓ | ✓ | ✓ | terminal | whole rollout | per rollout | 16.7 | 0.25 | 0.32 | 0.06 | 0.06 |
| PPRS, pooled norm. | ✓ | ✓ | ✓ | + terminal | , | per rollout | 10.9 | 0.28 | 0.35 | 0.06 | 0.06 |
| Direct reward (additive) | ✓ | ✓ | ✓ | + terminal | , | per turn | 9.7 | 0.29 | 0.35 | 0.06 | 0.06 |

5 Related Work
Prior coding corpora mine or synthesize tasks, while environment generators scale executable tool worlds. Tev’s distinction is a common latent-world source for multi-stage artifacts, exact grading, and intermediate progress. Backward construction and reward shaping are established ideas; the contribution combines them for this setting.
6 Conclusion
Verified world construction and progress feedback improve the tested open models on demanding terminal tasks. Scope remains synthetic workflows, and the hardest benchmarks retain a substantial frontier gap. New workflows require additional generator components and validation; broader coverage and simpler authoring remain future work.
Acknowledgments
The authors acknowledge Adobe Research’s support and feedback from colleagues at Adobe and UCLA.
AI use statement
Coding agents helped author generators, reference solvers, verifiers, progress scorers, and analysis code; authors reviewed the specifications. Claude assisted benchmark subset selection. AI also supported literature review and writing. Once authored, generators and reward evaluation run without model calls; no generative model grades training tasks.
Ethics statement
Training artifacts are synthetic and contain no collected personal records or proprietary source code. Security-themed tasks analyze generated artifacts in declared sandboxes. The authors acknowledge dual-use implications of stronger coding agents and report overlap checks and native-harness frontier comparisons.
Reproducibility statement
Generators, seeds, and scale tiers define reproducible environments, with reported median generation cost 30 CPU seconds. The appendix specifies qualification gates, reward equations, training settings, task lists, evaluation budgets, and three worked examples. The example listings are excerpts of package files, preserved below in full as published.
Appendix A Data Construction Details
A.1 Task Sources and Domain Taxonomy
Seven domains follow the ALE computing-and-math taxonomy [52]. Specifications come from public tasks, competition material, and quantum-computing references; new generators borrow structure and difficulty rather than copying artifacts.
Feasibility filtering favors knowable ground truth and batch/CPU execution. Near-duplicate checks and adversarial specification review precede implementation. At least two generators populate each domain; the scope excludes many live-service and heavy-toolchain workflows.
| Subdomain | Scope |
| 9.1 Software Engineering | Feature implementation, debugging, refactoring, and test authoring across real codebases, turning issues and specs into working, tested code changes. Verified by: test-suite pass (gate then score), behavioral correctness under fixed inputs, and no regression in existing tests. |
| 9.2 Data & Analytics Engineering | ETL/ELT pipelines, recommender and ranking systems, feature engineering, and analytics workflows, turning raw data into validated datasets, features, and model outputs. Verified by: output values exact or within tolerance, schema validity (gate), metric reproduction, and feature-parity against a reference. |
| 9.3 AI Engineering, Safety & CS Research | Model training and evaluation pipelines, research-paper reproduction, checkpoint and config management, and alignment/safety experiments, turning specs and code into reproduced results. Verified by: reproduced metrics within tolerance (FID, accuracy, etc.), config and alignment exactness, artifact validity, and deterministic-under-seed behavior. |
| 9.4 Mathematical & Operations Research | Combinatorial optimization, game-theoretic equilibrium computation, applied mathematics and number theory, and probabilistic estimation and filtering, implemented from specifications. Verified by: objective or solution exact or within tolerance, an optimality/feasibility certificate (gate), and reproducibility under a fixed seed. |
| 9.5 Cybersecurity & Digital Forensics | Reverse engineering and malware-config extraction, network and packet triage, crackme and key-generation analysis, and incident forensics, turning binaries and captures into extracted artifacts and findings. Verified by: extracted-value exactness against a reference (hash/string), finding-set precision and recall, and decoded-artifact correctness (gate). |
| 9.6 Infrastructure Engineering & Cloud Operations | Kubernetes migration and configuration, root-cause analysis of production incidents, OS and security hardening, and infrastructure-as-code, turning infrastructure specs and incidents into working, secured systems. Verified by: the system’s world-state (resources/health) under fixed inputs, config correctness against a reference, incident-cause exactness, and policy-check pass (gate). |
| 9.7 Quantum Computing | Quantum-circuit construction and simulation, quantum-error-correction code analysis, and circuit-level noise simulation, turning circuit and code specs into measured outcomes and logical-error estimates. Verified by: statistics and error-rates within tolerance against a reference, reproducibility under a fixed seed, and circuit/structure validity. |
A.2 Determinism, Reproducibility, and Generation Cost
Coding-agent authoring and human review are one-time costs per generator. Instantiation then uses deterministic seeded code, with median/90th-percentile generation costs of 30/82 CPU seconds including solvers, hidden-suite creation, and progress preview. These marginal costs exclude generator authoring.
A.3 Coverage Axes and Covering Array
The corpus enforces a strength-two covering array: every valid pair of values across 19 coverage axes appears in an admitted environment. Greedy selection fills uncovered pairs, reducing fixed template combinations. Files span databases, model shards, binaries, manifests, and some images. Reseeding changes entities, values, layout, and corruption placement.
A.4 Scale Ladder
Scale grows world size and processing burden without necessarily deepening the workflow DAG. For generators at all three tiers, median inputs grow from 17 files/51 KB to 20/168 KB and 28/586 KB. Checkpoint tasks also vary shard count and tensor/pipeline/expert parallelism topology.
A.5 Measuring Horizon and Difficulty
Structural horizon is the workflow’s critical path, not an agent’s potentially wasteful rollout length. Difficulty is verified success under a specified model and harness. When success saturates near zero, final potential distinguishes degrees of partial progress.
A.6 Qualification Gates
The eight gates require reward causality; zero score on 14 exploit probes; disjoint code closures; agreement between independent reference solvers; valid increasing progress with negative regression increments; metamorphic invariance and rejection of stale outputs; distinct worlds across six seeds; and a public contract without private grading data.
Solver agreement and independence reduce shared implementation failure, but the finite checks cannot establish universal specification correctness or exploit resistance. All published gate definitions remain in Table 7.
| Gate | What it enforces | Failure class it targets |
| Reward causality | each reward component drops when its cause is mutated | reward credits the wrong outcome |
| Exploit resistance | all 14 gaming probes score | verifier loophole, shortcut acceptance |
| Role independence | disjoint reference/alt/verifier/progress closures | shared bias between solver and grader |
| Solver agreement | reference and alternative solvers yield identical trees | verifier-only acceptance, unresolved spec. |
| Progress validity | ; monotone; decreases on regression | invalid process signal |
| Metamorphic invariance | meaning-preserving rewrites keep the grade; stale outputs fail | over-narrow verifier; stale-state acceptance |
| Six-seed diversity | distinct world / public-tree / task id () | degenerate environment, template repetition |
| Public-contract hygiene | no private authority in the prompt or public tree | answer leakage |
A.7 Dataset Statistics
Normalized entropy balance is 0.89 across three capability categories and 0.91 across seven domains. Median environments have six workflow stages, eight progress nodes, six reward components, two corruptions, and 12 input files. Formats total 23.
One untrained 27B probe per environment fully solves 3.2%, with mean graded score 0.06; 52% exhaust the token budget before submission. Gemini 3.6 Flash reaches 29% pass@1 and 40% pass@8 on a fixed 92-environment sample. These model/harness/sample conditions differ and should accompany difficulty comparisons.
A.8 Comparison Protocol
One fixed evaluator measures corpus difficulty at 60 tool calls and eight rollouts per task. Prior corpora use the official Terminus-2 scaffold; Tev uses its single-bash harness. Tasks that cannot faithfully build are excluded and recorded, which can bias a corpus estimate upward.
Timing uses 100 sampled Tev environments with two rollouts each and expanded 300-call/two-hour limits; neither binds. Wall time includes model calls, retries, and commands, yielding 12.4-minute median and 14.8-minute mean. Prior-corpus timing uses its difficulty runs. The comparison shares a model but not every scaffold or timing budget, and is not an estimate of human work.
Table 9 retains the generation, qualification, coverage, and reward comparison. A progress signal must grade intermediate workspace states; an end-only test fraction does not qualify.
| Corpus | Anchor | Model in generation | Validation | Difficulty control | Coverage control | Progress signal |
| SWE-Smith (Yang et al., 2025a) | repository | bug injection, task text | tests run post hoc | – | – | none |
| Terminal-Universe (Wu et al., 2026) | agent trajectory | trajectory, task text | pytest post hoc | – | – | none |
| TMAX-15K (Ivison et al., 2026) | task text | task, container, tests | none; RL soft-filters | – | topic balancing | none |
| SETA (Shen et al., 2026) | task text | task, tests | solvability post hoc | – | – | test fraction |
| Endless Terminals (Gandhi et al., 2026) | task text | task, container, tests | solvability post hoc | – | – | none |
| CLI-Gym (Lin et al., 2026) | agent trajectory | trajectory, task text | tests post hoc | – | – | none |
| Tev-Gym (ours) | latent world | generator code (once, reviewed) | 8 gates at generation | scale DAG depth | strength-2 covering array | staged |
A.9 Decontamination
Fresh seeds and public-tree hashes establish distinct generated artifacts. Across all 1,170 prompts, no measured evaluation prompt shares an 8-gram or 13-gram. A control detects 156 shared 13-grams between the Terminal-Bench sets, demonstrating that the comparator can find overlap.
Median nearest-neighbor token Jaccard is 0.13 for Terminal-Bench 3.0/4.0, 0.10 for 2.1, and 0.22 for ALE-CM. These lexical and artifact checks support lack of verbatim duplication; they do not test all semantic similarity or pretrained-model exposure.
| Evaluation set | Tasks | 13-gram hits | 8-gram hits | Max token cov. | NN Jaccard med. / max |
| ALE-CM | 18 | 0 | 0 | 0.0% | 0.22 / 0.38 |
| TB 3.0, all | 74 | 0 | 0 | 0.0% | 0.13 / 0.18 |
| TB 3.0, CM subset | 22 | 0 | 0 | 0.0% | 0.14 / 0.18 |
| TB 4.0, all | 66 | 0 | 0 | 0.0% | 0.13 / 0.18 |
| TB 4.0, CM subset | 18 | 0 | 0 | 0.0% | 0.14 / 0.18 |
| TB 2.1 | 89 | 0 | 0 | 0.0% | 0.10 / 0.16 |
| Calibration: same code, same-benchmark tasks | |||||
| ALE-CM task vs. other ALE-CM tasks | 18 | 2 | 13 | 26.0% | 0.28 / 0.34 |
| TB 3.0 CM task vs. other TB 3.0 tasks | 22 | 22 | 22 | 42.5% | 0.24 / 0.37 |
| TB 4.0 CM task vs. other TB 4.0 tasks | 18 | 18 | 18 | 42.5% | 0.25 / 0.37 |
A.10 Three Worked Training Environments
The three complete published walkthroughs separate public prompts and inputs from sealed ground truth. Cloud cost optimization demonstrates image evidence; building control and finance illustrate domains outside the seven-domain training focus. All examples run in pinned, network-disabled containers. Record-based examples derive answers directly, while building control executes a pinned simulator procedure.
Example 1: multimodal cloud cost optimization (Cloud Operations)
The cloud-cost task combines 430 resource records, pricing, and a dashboard image. Three actionable findings occur only in the image, so tabular processing alone misses them. Four explicit rules produce 148 recommendations: 74 compute downsizes, 27 database downsizes, 44 storage-tier changes, and three dashboard findings.
Reference savings are 16,395.02 USD monthly. Costs and savings are independently rounded from exact prices, so summed savings differ by 0.08 from rounded current-minus-projected totals. The verifier applies parsing/sanity gates, then scores recommendation coverage (30%), actions (30%), savings accuracy (25%), and completeness (15%). All prompts, input excerpts, reference outputs, and the dashboard follow.
# AWS cost optimization analysis (430 inventory resources) You are an AWS cost optimization analyst working on Linux. Three inputs are staged under `input/`: - `input/aws_billing_dashboard.png` - a synthetic AWS billing dashboard screenshot. It shows a per-service cost breakdown AND three 'dashboard-only' findings (an idle NAT gateway, an unattached Elastic IP, and a stale CloudWatch log group) that are NOT present in the usage CSV. Inspect the image directly to recover them. - `input/resource_usage.csv` - the EC2/RDS/S3 inventory with columns `resource_id,resource_type,name,state,instance_type,cpu_p95,mem_p95,connections_p95,size_gb,ia_access_ratio`. - `input/aws_pricing_reference.csv` - the deterministic pricing table (`sku,dimension,unit,price_usd`). Apply these rules and price every saving from the pricing table: - Downsize a running EC2 instance one tier when `cpu_p95 < 20` and `mem_p95 < 50`. - Downsize an available RDS instance one tier when `cpu_p95 < 25` and `connections_p95 < 10`. - Move an S3 bucket to Infrequent Access when `ia_access_ratio < 0.10` and `size_gb >= 100`. - Include the three dashboard-only findings (delete the idle NAT gateway, release the unattached Elastic IP, delete the stale CloudWatch log group). Write `output/optimization_report.json` with an `executive_summary` object and a `recommendations` array, and `output/savings_summary.csv` with columns `resource_id,resource_type,name,action,current_cost,projected_cost,monthly_savings`. Scoring is weighted: 30% coverage F1 over the hidden recommendation set (including the three dashboard-only resources), 30% action correctness, 25% savings accuracy (full credit within +/-20% relative error), and 15% report completeness.
input/ aws_billing_dashboard.png 900 x 560 px dashboard render: cost by service + 3 findings resource_usage.csv 430 rows: 220 EC2 instances, 80 RDS instances, 130 S3 buckets aws_pricing_reference.csv 11 SKUs: 5 EC2 tiers, 4 RDS tiers, 2 S3 storage classes

# input/aws_pricing_reference.csv (complete) sku,dimension,unit,price_usd t3.medium,ec2_compute,hour,0.041600 m5.large,ec2_compute,hour,0.096000 m5.xlarge,ec2_compute,hour,0.192000 m5.2xlarge,ec2_compute,hour,0.384000 m5.4xlarge,ec2_compute,hour,0.768000 db.t3.medium,rds_compute,hour,0.068000 db.m5.large,rds_compute,hour,0.171000 db.m5.xlarge,rds_compute,hour,0.342000 db.m5.2xlarge,rds_compute,hour,0.684000 s3-standard,s3_storage,gb_month,0.023000 s3-standard-ia,s3_storage,gb_month,0.012500 # input/resource_usage.csv (6 of 430 rows) resource_id,resource_type,name,state,instance_type,cpu_p95,mem_p95,connections_p95,size_gb,ia_access_ratio db-000067,rds,rds-svc-067,available,db.m5.xlarge,62.12,,346,, db-000048,rds,rds-svc-048,available,db.m5.xlarge,18.48,,8,, i-000161,ec2,ec2-app-161,running,m5.2xlarge,45.55,74.81,,, i-000000,ec2,ec2-app-000,running,m5.2xlarge,11.97,41.34,,, s3-000015,s3,data-bucket-015,,,,,,5923,0.052 s3-000118,s3,data-bucket-118,,,,,,19,0.596
# reference/optimization_report.json (148 recommendations; excerpt, reformatted)
{"executive_summary": {"resource_count": 430, "recommendation_count": 148,
"dashboard_only_finding_count": 3, "total_current_cost": 32629.73,
"total_projected_cost": 16234.79, "total_monthly_savings": 16395.02},
"recommendations": [
{"resource_id": "i-000003", "resource_type": "ec2", "name": "ec2-app-003",
"action": "downsize_ec2", "current_cost": 560.64, "projected_cost": 280.32,
"monthly_savings": 280.32},
{"resource_id": "i-000006", "resource_type": "ec2", "name": "ec2-app-006",
"action": "downsize_ec2", "current_cost": 560.64, "projected_cost": 280.32,
"monthly_savings": 280.32},
...
{"resource_id": "cw-loggroup-old-processor", "resource_type": "cloudwatch_log_group",
"name": "/aws/lambda/old-processor", "action": "delete_stale_log_group",
"current_cost": 64.6, "projected_cost": 0.0, "monthly_savings": 64.6},
...
{"resource_id": "nat-0af31c9d", "resource_type": "nat_gateway",
"name": "nat-gw-prod-1c", "action": "delete_idle_nat_gateway",
"current_cost": 32.41, "projected_cost": 0.0, "monthly_savings": 32.41},
...
{"resource_id": "eip-52b7e401", "resource_type": "elastic_ip",
"name": "eip-orphaned-legacy", "action": "release_unattached_eip",
"current_cost": 3.65, "projected_cost": 0.0, "monthly_savings": 3.65},
...]}
# reference/savings_summary.csv (148 rows; first 3)
resource_id,resource_type,name,action,current_cost,projected_cost,monthly_savings
i-000003,ec2,ec2-app-003,downsize_ec2,560.64,280.32,280.32
i-000006,ec2,ec2-app-006,downsize_ec2,560.64,280.32,280.32
i-000024,ec2,ec2-app-024,downsize_ec2,560.64,280.32,280.32Example 2: closed-loop building MPC (Energy, Power & Nuclear Engineering)
The building-control environment pins EnergyPlus and the full procedure: validate input hashes, edit the thermal model, run thermostat and excitation traces, identify a second-order model with ridge regression, and execute two 24-step receding-horizon controllers in closed loop.
Nine deliverables include 672-row traces, final-day actions, identification/solver logs, metrics, and a summary. References come from the sealed simulator procedure; the grader compares CSV cells and JSON leaves at tolerance 10⁻⁴. A fixed open-loop schedule or wrong-order model cannot reproduce the required closed-loop artifacts.
# EnergyPlus closed-loop building MPC (7 days at 15 minutes) Use the exact toolchain declared in `input/instance.json`. Inspect `input/model.idf`, `input/weather.epw`, `input/experiment_plan.json`, and `input/output_contract.json`. The model is the pinned EnergyPlus 22.1 one-zone example with a declared internal-mass transformation and a schedule actuator driving the `MPC Cooling Sink`. Run the deterministic thermostat baseline and supplied excitation at 15-minute resolution. Fit the declared second-order input-output thermal model over both EnergyPlus traces and preserve its training and held-out diagnostics. This is a 3R2C-comparable dynamic-order model, not a claim that physical resistance and capacitance parameters were identified. Implement two distinct cvxpy/OSQP receding-horizon policies using the declared price, peak-load, comfort, and switching weights. Deploy each through the EnergyPlus Python API: at every zone timestep read the simulated zone temperature, solve the policy, and actuate the first control. A precomputed open-loop schedule does not satisfy this task. Write the nine artifacts named in `input/output_contract.json`. The three data traces cover the complete simulation window. The two action files and comparison metrics cover the final 24 hours. RC logs must expose model coefficients, training and held-out calibration, policy-specific closed-loop error, objective weights, and solver status counts. The grader densely compares all nine artifacts to a reference generated and rerun with EnergyPlus 22.1 and cvxpy in the pinned profile.
input/ question.md the task prompt above instance.json horizon, schedules, tariff, controller weights (excerpt below) experiment_plan.json four runs, identification recipe and split, two policies output_contract.json the nine deliverables, their columns, and their rounding (below) model.idf one-zone EnergyPlus model, materialized from the pinned example weather.epw Golden, CO (NREL) TMY3 weather file, from the same pinned install
{
"kind": "energyplus_second_order_closed_loop_mpc_cosimulation", "seed": 6610,
"energyplus_version": "22.1.0", "cvxpy_version": "1.6.5", "numpy_version": "2.1.3",
"source_focus": "energyplus_1zoneuncontrolled_22_1",
"source_idf_sha256": "53e3fb38...", "source_epw_sha256": "65041e11...",
"weather_window": "golden_tmy3_july",
"run_period_start_month": 7, "run_period_start_day": 1, "run_period_days": 7,
"timestep_minutes": 15, "horizon_steps": 672,
"prediction_horizon_steps": 24, "reporting_window_steps": 96,
"cooling_capacity_kw": 2.0406, "cooling_cop": 3.0, "internal_mass_area_m2": 48.0,
"baseline_deadband_c": 0.5, "baseline_on_fraction": 0.85, "identification_ridge": 1e-08,
"occupancy": [0, 0, ...], // 672 values; 1 between 08:00 and 20:00
"is_peak_period": [0, 0, ...], // 672 values; 1 between 17:00 and 20:00
"setpoint_c": [27.5, 27.5, ...], // 24.2 while occupied, else 27.5
"comfort_min_c": [17.0, 17.0, ...], // 20.5 while occupied, else 17.0
"comfort_max_c": [28.0, 28.0, ...], // 24.5 while occupied, else 28.0
"energy_price_per_kwh": [0.061587, ...], // 672 values in [0.058, 0.250]
"excitation_schedule": [0.9, 0.1, 0.9, ...], // 672 values in {0.1, 0.9}
"policy_objectives": {
"energy_saving": {"energy_weight": 0.35, "price_weight": 1.0, "peak_weight": 0.1,
"on_peak_weight": 0.1, "comfort_weight": 2.0, "switch_weight": 0.12},
"demand_response": {"energy_weight": 0.05, "price_weight": 0.35, "peak_weight": 5.0,
"on_peak_weight": 2.0, "comfort_weight": 4.0, "switch_weight": 0.12}
}
}{
"trace_files": {"baseline": "baseline_data.csv",
"energy_saving": "mpc_energy_saving_data.csv",
"demand_response": "mpc_demand_response_data.csv"},
"trace_columns": ["step", "hour", "cooling_w", "t_zone", "setpoint", "outdoor_temp_c",
"occupancy", "cooling_fraction", "electric_kw", "price_per_kwh",
"comfort_min_c", "comfort_max_c", "one_step_predicted_zone_temp_c"],
"action_files": ["mpc_actions_energy_saving.csv", "mpc_actions_demand_response.csv"],
"rc_log_files": ["rc_log_energy_saving.json", "rc_log_demand_response.json"],
"metrics_file": "metrics_comparison.csv",
"metrics_rows": ["baseline", "mpc_energy_saving", "mpc_demand_response"],
"metrics_columns": ["cooling_kwh", "elec_kwh", "cost_usd", "peak_load_kw",
"peak_hour_avg_kw", "discomfort_dh", "calibration_rmse_c"],
"summary_file": "results_summary.json",
"numeric_rounding": {"csv": 8, "model_coefficients": 10, "summary": 8}
}Equation
Example 3: financial-statement reconstruction (Accounting & Finance)
The finance task reconstructs two years of statements from conflicting structured facts and inline-XBRL filings under an explicit precedence policy. The latent world obeys accounting identities and cross-statement ties. Corruptions affect rejected sources; the selected structured annual cash value is 246, despite original and amended filings reporting 239 and 243.
The reference contains 16 items for each year, four declared exceptions, and 32 evidence rows. A round-trip check independently reselects values from public material before publication; grading compares final files exactly. The preserved listings include the policy, conflicting source excerpts, expected statements, exceptions, and evidence schema.
# Reconstruct and reconcile annual financial statements Reconstruct the annual statements from the multi-document filing packet under `input/filings/` and the SEC-shaped `input/companyfacts.json`. Use `input/filing_index.json`, `input/account_mapping.csv`, and the authority and fallback rules in `input/reconstruction_policy.md`. Write three deliverables: - `output/financial_statements.json`, with company metadata, USD-million unit metadata, and a `periods` array containing balance sheet, income statement, and cash-flow sections. - `output/evidence.csv`, with the exact columns `period, statement, line_item, selected_value, concept, source_file, locator, citation, authority_basis`, in period and mapping order. Every locator must resolve to the selected source value. - `output/exceptions.json`, listing every cross-source conflict, explicit amendment restatement, accepted taxonomy alias, and missing-structured-fact fallback with its policy-based resolution. Preserve integer USD millions. Verify the accounting, income, and cash roll-forward identities for every period before writing the files. Do not use quarterly, unaudited, pro forma, or wrong-unit evidence.
input/ question.md the task prompt above instance.json 2 fiscal years, 16 line items, 3 deliverables reconstruction_policy.md authority and fallback rules (below) filing_navigation.md amendments carry only restated rows; ids are locators account_mapping.csv 16 line items -> statement, label, accepted concepts filing_index.json 4 filings: form, accession, filing date, status (below) companyfacts.json SEC-shaped facts: 16 us-gaap concepts x 3, whole USD filings/2025/form10-k.htm FY2025 10-K, 16 inline-XBRL rows (fact-2025-000..015) filings/2025/form10-k-a.htm FY2025 10-K/A, restates one row filings/2025/current-report.htm FY2025 8-K with quarterly (Q3) figures filings/2024/form10-k.htm FY2024 10-K
# Annual statement reconstruction policy All output values are USD millions. Companyfacts `USD` values are whole USD and must be divided by 1,000,000. Use exact fiscal-year and period-end matches, annual `FY` facts, and Forms 10-K or 10-K/A only. Ignore quarterly, non-USD, pro forma, and unaudited supplemental facts. Within Companyfacts, choose the latest `filed` date and then accession among all accepted concepts in `account_mapping.csv`. Within filing documents, a 10-K/A overrides only the rows it explicitly restates; otherwise retain the original 10-K row. When the latest structured fact and latest filing row disagree, the structured Companyfacts value controls. If no accepted structured fact exists, use the latest applicable filing row. Record every cross-source conflict, amendment restatement, taxonomy alias, and structured-fact fallback in `exceptions.json`. Every evidence row must cite an exact JSON pointer or HTML fragment that resolves to the selected value.
# account_mapping.csv (first of 16 rows)
statement,line_item,filing_label,canonical_concept,accepted_concepts,period_type
balance_sheet,cash,Cash and cash equivalents,us-gaap:CashAndCashEquivalentsAtCarryingValue,us-gaap:CashAndCashEquivalentsAtCarryingValue|jrh:CashCashEquivalentsRestrictedCashAndRestrictedCashEquivalents,instant
# filing_index.json
{"cik": "8899426957", "company": "Juniper Ridge Holdings plc", "filings": [
{"accession": "8899426957-25-000001", "document": "filings/2025/form10-k.htm",
"filed": "2026-01-30", "fiscal_year": 2025, "form": "10-K", "status": "original"},
{"accession": "8899426957-25-000002", "document": "filings/2025/form10-k-a.htm",
"filed": "2026-02-15", "fiscal_year": 2025, "form": "10-K/A",
"status": "amendment_explicit_rows_only"},
{"accession": "8899426957-25-000004", "document": "filings/2025/current-report.htm",
"filed": "2025-10-15", "fiscal_year": 2025, "form": "8-K",
"status": "current_report_decoy"},
{"accession": "8899426957-24-000001", "document": "filings/2024/form10-k.htm",
"filed": "2025-01-30", "fiscal_year": 2024, "form": "10-K", "status": "original"}]}
# companyfacts.json, node /facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD
[{"accn": "8899426957-25-000001", "form": "10-K", "fy": 2025, "fp": "FY",
"end": "2025-12-31", "filed": "2026-01-30", "val": 239000000},
{"accn": "8899426957-25-000002", "form": "10-K/A", "fy": 2025, "fp": "FY",
"end": "2025-12-31", "filed": "2026-02-15", "val": 246000000},
{"accn": "8899426957-24-000001", "form": "10-K", "fy": 2024, "fp": "FY",
"end": "2024-12-31", "filed": "2025-01-30", "val": 256000000}]
# filings/2025/form10-k.htm, balance sheet ("USD in millions")
<tr><th scope="row">Cash and cash equivalents</th><td><ix:nonFraction id="fact-2025-000"
name="us-gaap:CashAndCashEquivalentsAtCarryingValue" contextRef="FY2025" unitRef="USD"
scale="6" format="ixt:numdotdecimal">239</ix:nonFraction></td></tr>
# filings/2025/form10-k-a.htm, the only row it restates
<tr><th scope="row">Cash and cash equivalents</th><td><ix:nonFraction
id="amended-fact-2025-000" name="us-gaap:CashAndCashEquivalentsAtCarryingValue"
contextRef="FY2025" unitRef="USD" scale="6" ...>243</ix:nonFraction></td></tr>
# filings/2025/current-report.htm, 8-K with a quarterly context
<tr><th scope="row">Cash and cash equivalents</th><td><ix:nonFraction
id="current-report-fact-2025-000" name="us-gaap:CashAndCashEquivalentsAtCarryingValue"
contextRef="Q32025" unitRef="USD" scale="6" ...>230</ix:nonFraction></td></tr># reference/financial_statements.json (USD millions; reformatted)
{"cik": "8899426957", "company": "Juniper Ridge Holdings plc", "currency": "USD",
"unit": "millions", "periods": [
{"period": "2025", "period_end": "2025-12-31",
"balance_sheet": {"cash": 246, "accounts_receivable": 147, "inventory": 62,
"other_assets": 732, "assets": 1187, "liabilities": 598,
"equity": 589},
"income_statement": {"revenue": 1852, "cost_of_revenue": 666,
"operating_expense": 927, "net_income": 259},
"cash_flow": {"beginning_cash": 256, "operating_cash_flow": 365,
"investing_cash_flow": -110, "financing_cash_flow": -265,
"ending_cash": 246}},
{"period": "2024", "period_end": "2024-12-31",
"balance_sheet": {"cash": 256, "accounts_receivable": 73, "inventory": 30,
"other_assets": 546, "assets": 905, "liabilities": 508,
"equity": 397},
"income_statement": {"revenue": 1583, "cost_of_revenue": 601,
"operating_expense": 777, "net_income": 205},
"cash_flow": {"beginning_cash": 106, "operating_cash_flow": 333,
"investing_cash_flow": -146, "financing_cash_flow": -37,
"ending_cash": 256}}]}
# reference/exceptions.json (reformatted)
[{"kind": "cross_source_conflict", "period": "2025", "line_item": "accounts_receivable",
"structured_value": 147, "filing_value": 138, "selected_value": 147,
"resolution": "structured_feed_priority"},
{"kind": "amendment_restatement", "period": "2025", "line_item": "cash",
"original_value": 239, "amended_value": 243, "selected_value": 246,
"resolution": "explicit_amendment_over_original"},
{"kind": "cross_source_conflict", "period": "2025", "line_item": "cash",
"structured_value": 246, "filing_value": 243, "selected_value": 246,
"resolution": "structured_feed_priority"},
{"kind": "cross_source_conflict", "period": "2025", "line_item": "other_assets",
"structured_value": 732, "filing_value": 726, "selected_value": 732,
"resolution": "structured_feed_priority"}]
# reference/evidence.csv (2 of 32 rows)
period,statement,line_item,selected_value,concept,source_file,locator,citation,authority_basis
2025,balance_sheet,cash,246,us-gaap:CashAndCashEquivalentsAtCarryingValue,companyfacts.json,#/facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/1,"SEC Companyfacts JSON, us-gaap:CashAndCashEquivalentsAtCarryingValue, FY 2025, companyfacts.json#/facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/1",latest_exact_period_annual_structured_fact
2024,balance_sheet,cash,256,us-gaap:CashAndCashEquivalentsAtCarryingValue,companyfacts.json,#/facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/2,"SEC Companyfacts JSON, us-gaap:CashAndCashEquivalentsAtCarryingValue, FY 2024, companyfacts.json#/facts/us-gaap/CashAndCashEquivalentsAtCarryingValue/units/USD/2",latest_exact_period_annual_structured_factAppendix B Dense Reward Details
PPRS is not policy-invariant. Retaining final potential intentionally distinguishes failed trajectories. Every submitted success outranks every failure, and deterministic policies that achieve success retain that preference, but stochastic-policy rankings can reverse.
If one policy succeeds less often, shaping can prefer it only when the success-probability gap is below λ(1−p)/(1+λ), at most 0.2 for λ=0.25, and its failures make sufficiently more progress. This bound is not a guarantee that learned success remains unchanged.
In 24 measured groups—eight rollouts on 12 warehouse seeds for both base and trained 27B—none of 192 rollouts submits a passing answer. Every group nevertheless has nonzero final-potential variance. The example progress contract gates relations by correct dimensions and facts, discouraging superficially valid downstream outputs.
Total reward depends on final progress and submission, not detours; per-turn differences still assign credit and penalize regression through undiscounted return-to-go. Among all-fail groups the shaping scale cancels from normalized comparisons, so alignment of the progress scorer is essential [57]. The authors test specific exploit probes and independent code closures; no general immunity to reward hacking follows.
Equation
Appendix C Training and Evaluation Details
C.1 Training Recipe and Hyperparameters
DPPO uses group-centered advantages, a total-variation policy-ratio bound 0.1, and no KL anchor. Active sampling refills zero-variance groups until each step contains eight informative groups. Each turn receives λ times its progress increment, with completion reward only at the last turn; its advantage is return-to-go minus the group mean at that turn index. λ=0.25 is fixed across runs.
| Setting | Value |
| Objective | group-relative policy gradient with centered advantages (Eq. (2)) |
| Trust region | total-variation divergence bound on the policy ratio (DPPO loss); no KL anchor () |
| Rollouts per task () | 24 |
| Unique tasks per step | 8 (192 rollouts per step) |
| Active sampling | zero-variance groups discarded and refilled |
| Optimizer | AdamW, learning rate , constant schedule, one epoch per batch |
| Sampling temperature | 1.0 |
| Prompt / response budget | 2,048 prompt tokens; 65,536 response tokens |
| Per-turn budget | 16,384 tokens; at most 64 tool calls per rollout |
| Tools | a single bash tool in a per-task container (python:3.13-slim base image) |
| Reward | Eq. (4) with ; per-turn credit, return-to-go with , per-turn-index group baseline |
| Terminal reward | strict verifier score at submission, for a pass |
| Precision / parallelism | bf16 with fp32 LM head; ZeRO-3, sequence parallelism 4 (9B) |
| Learner / inference split | 16 learner GPUs + 16 inference engines on 32-GPU runs; 16 + 48 on the 64-GPU 27B run |
| Seed | 42 |
C.2 Evaluation Protocol
ALE-CM evaluates 18 Linux-container-compatible tasks from 23 computing-and-math tasks: eight mathematics/science, five ML, and five coding/systems. Five excluded tasks need full virtual machines. Every model is evaluated on the same selected tasks with five rollouts; Qwen uses Vanillux2 while frontier models use their native agents.
S@1 is mean graded score, P@1 the fraction of fully successful rollouts, and S@5/P@5 their best-of-five counterparts. Terminal-Bench 3.0/4.0 uses 22/18 selected domain-related CPU-compatible tasks; the 27B Terminal-Bench 2.1 evaluation uses Terminus-2, while 9B uses Vanillux2.
The 3.0/4.0 subset P@1 values correlate with full-suite leaderboard values at 0.96/0.97 across compared models, but subset scores often run higher. ALE comparisons are less controlled: full-suite headline values use one run per task and can differ in harness configuration. Tables 12–16 preserve task selection and comparison details; none establishes whole-benchmark equivalence.
| Category | Task | Title |
| Math & Science (8) | branch_bound_atsp | Branch-and-Bound ATSP Solver |
| cfr_game_theory_equilibrium | CFR Game Theory Equilibrium | |
| particle_filter_nonlinear_tracking | Particle Filter Nonlinear Tracking | |
| k3_abelian_extensions | Finite Abelian Extension Classification | |
| synthetic_causal_structure_inference | Synthetic Causal Structure Inference | |
| clustered_cyclic_code_circuit_level_simulation | Clustered-Cyclic Code Circuit Simulation | |
| ising_post_measurement_1 | Post-Measurement States for a Quantum Ising Chain | |
| go_game_reconstruction_1 | Go Game Reconstruction 1 | |
| ML (5) | dit_pipeline_cfg_alignment_fid_256_001 | Repair Standalone DiT Pipeline CFG Alignment |
| paper_reproduction_instance_1 | LCA-on-the-Line Table 2 Reproduction | |
| ranking_node_feature_parity_recovery_instance_1 | Ranking Node Feature Parity Recovery | |
| mp_checkpoint_consolidation_v2 | Checkpoint Consolidation V2 | |
| recsys_cold_start_instance_1 | Hybrid Recommender Cold-Start Benchmark | |
| Coding & System (5) | k8s_payment_api_root_cause_analysis | Kubernetes Payment API Root Cause Analysis |
| os_log_permission_guard_v1 | OS Log Permission Guard | |
| data_pipeline_etl_instance_1 | Retail ETL Warehouse Cleanup | |
| cp_test_gen_1 | Competitive Programming Test Generator – Binary String Copying | |
| cost_optimization_1 | AWS Cost Optimization Analysis |
| TB task | TB category | In TB 4.0 |
| batched-eval-parity | ML / Evaluation | yes |
| cargo-flight-dispatch | Operations / Logistics | yes |
| cli-2ph-simplex | Software / Algorithms | – |
| ctr-optimization | Operations / Marketing | yes |
| data-anonymization | Software / Data engineering | yes |
| distributed-dedup | Software / Systems | yes |
| fix-uautomizer-soundness | Software / Languages | – |
| freight-dispatch-shift | Operations / Logistics | yes |
| ico-path-patch | Security / Reverse engineering | – |
| live-database-cutover | Software / Databases | yes |
| memcached-backdoor | Security / Reverse engineering | – |
| mp-checkpoint-consolidation | ML / Inference | yes |
| mvcc-lsm-compaction | Software / Databases | yes |
| payments-pipeline-fix | Software / Systems | yes |
| pretrain-shard-corruption | ML / Training | yes |
| production-planning | Operations / Supply chain | yes |
| risk-scorer-replay | ML / Evaluation | yes |
| session-window-debug | Software / Systems | yes |
| shadow-relay | Security / Forensics | yes |
| telecom-entity-resolution | Software / Data engineering | yes |
| uefi-bootkit | Security / Forensics | yes |
| wal-recovery-ordering | Software / Databases | yes |
| Model | Subset (22) | Full (74) | Official | ||
| P@1 | P@5 | P@1 | P@5 | acc. | |
| Claude Fable 5 | 0.41 | 0.64 | 0.34 | 0.61 | 0.34 |
| Claude Opus 5 | 0.53 | 0.64 | 0.42 | 0.58 | 0.43 |
| GPT-5.6 (sol) | 0.41 | 0.55 | 0.34 | 0.55 | 0.35 |
| GPT-5.6 (terra) | 0.30 | 0.55 | 0.20 | 0.38 | 0.21 |
| GPT-5.6 (luna) | 0.18 | 0.32 | 0.13 | 0.27 | 0.14 |
| Grok 4.6 | 0.30 | 0.50 | 0.26 | 0.45 | 0.26 |
| Claude Sonnet 5 | 0.16 | 0.32 | 0.14 | 0.34 | 0.15 |
| Claude Opus 4.8 | 0.19 | 0.50 | 0.21 | 0.42 | 0.21 |
| Grok 4.5 | 0.26 | 0.55 | 0.15 | 0.34 | 0.16 |
| GLM 5.2 | 0.04 | 0.18 | 0.04 | 0.14 | 0.05 |
| Model | Subset (18) | Full (66) | Official | ||
| P@1 | P@5 | P@1 | P@5 | acc. | |
| Claude Fable 5 | 0.46 | 0.61 | 0.45 | 0.68 | 0.45 |
| Claude Opus 5 | 0.52 | 0.72 | 0.52 | 0.71 | 0.52 |
| GPT-5.6 (sol) | 0.39 | 0.61 | 0.37 | 0.61 | 0.37 |
| GPT-5.6 (terra) | 0.23 | 0.44 | 0.22 | 0.44 | 0.22 |
| GPT-5.6 (luna) | 0.18 | 0.33 | 0.17 | 0.33 | 0.17 |
| Grok 4.6 | 0.31 | 0.50 | 0.20 | 0.39 | 0.20 |
| Claude Sonnet 5 | 0.19 | 0.50 | 0.12 | 0.35 | 0.12 |
| Claude Opus 4.8 | 0.27 | 0.50 | 0.24 | 0.47 | 0.24 |
| Grok 4.5 | 0.20 | 0.33 | 0.12 | 0.27 | 0.12 |
| Model | Harness | Our ALE-CM split (18 tasks) | Official full ALE (152 tasks) | Official | ||
| S@1 | P@1 | Score | Pass rate | effort | ||
| GPT-5.6 (sol) | Codex | 0.58 | 0.28 | 0.54 | 0.31 | xhigh |
| GPT-5.6 (terra) | Codex | 0.50 | 0.22 | 0.51 | 0.28 | max |
| Claude Opus 5 | Claude Code | 0.58 | 0.28 | 0.55 | 0.32 | high |
| Claude Opus 4.8 | Claude Code | 0.52 | 0.22 | 0.45 | 0.27 | max |
| Gemini 3.1 Pro | Gemini CLI | 0.52 | 0.21 | 0.33 | 0.16 | default |
C.3 Qualitative Examples on ALE-CM
Four selected rollout comparisons illustrate Kubernetes diagnosis, quantum-state computation, branch-and-bound optimization, and finite-group enumeration. Cases were chosen where Tev beats the base rollout, and frontier agents also solve each task. Thus they explain possible behavioral changes rather than estimate average gain or unique capability. All traced decisions and scores remain in Figures 5 and 7–9 and Table 17.
| Task | Category | Best frontier rollout | Second frontier rollout | Base rollout | Ours |
| k8s_payment_api_root_cause_analysis | Coding & Sys. | 0.97 (Claude Fable 5) | 0.75 (GPT-6 Astra) | 0.70 | 0.97 |
| ising_post_measurement_1 | Math & Sci. | 1.00 (Claude Fable 5) | 1.00 (GPT-5.6 sol) | 0.00 | 1.00 |
| branch_bound_atsp | Math & Sci. | 1.00 (Claude Fable 5) | 1.00 (GPT-5.6 sol) | 0.00 | 1.00 |
| k3_abelian_extensions | Math & Sci. | 1.00 (Claude Fable 5) | 1.00 (GPT-5.6 sol) | 0.00 | 1.00 |



Appendix D Extended Related Work
Repository mining, bug injection, environment reconstruction, and executable tool-world generators provide precedents for scalable agent data. Backward construction likewise includes denoising, controlled database corruption, instruction back-translation, and procedural puzzles. Tev applies a shared latent world to mutually constraining multi-file workflows.
World-state evaluation already appears in AppWorld, τ-bench, WebArena, and OSWorld. Tev’s narrower argument is to scale environments whose latent state supplies both a completion oracle and intermediate progress, with explicit coverage and difficulty controls [74].
Alternatives to sparse reward include task curricula, expert prefixes, learned process judges, and test-pass proxies. PPRS keeps the tasks fixed and increases feedback resolution, but relies on trustworthy progress contracts and intentionally changes the optimized objective. It does not inherit classical policy-invariance guarantees.