Fixing 28 Flawed Tasks Shifted the Leaderboard by Three Ranks: A Terminal-Bench Audit Review

The Paper

E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré, "Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks", arXiv:2611.08417 [cs.AI], submitted 2026-11-05, CC BY 4.0. Affiliation: Stanford University (co-author Crystal Zhou is listed with a dual Cornell University affiliation). Corresponding author: E. Kelly Buchanan. Code and task repository: github.com/harbor-framework/terminal-bench-2-1.

This is a preprint that has not gone through peer review. All authors are at Stanford University, and the fact that this is an independent academic team with no ties to any vendor whose models are being evaluated is a point in its favor on a paper specifically about benchmark trustworthiness. That said, the acknowledgements list Anthropic, Google DeepMind, Together AI, and other companies as general funders of the Stanford DAWN and SEAMS lab projects. They did not fund this specific study, but some of the evaluated model providers (Claude, Gemini) appear in that broader funder list, which we disclose here for transparency. For this review, the session's outbound network access was blocked entirely (WebFetch failed DNS resolution for arxiv.org, example.com, and other unrelated hosts alike), so the full paper text was verified against a GitHub Actions-collected snapshot (a complete, untruncated copy of the HTML source) taken at 2026-11-06T22:09:29Z.

Study Overview

Two research questions drive this paper. First, how many task-level defects remain in an agentic benchmark that was already reviewed by humans and LM-assisted red-teaming before release, and can finding and repairing them continuously be automated? Second, once defects are fixed, how much do accuracy and rankings change, and can "harness gaps" between agents be separated into genuine capability differences versus differences in resource budget?

The authors build a three-stage audit workflow: an evidence-collector agent, an auditor agent, and a verifier agent. For each task, the auditor assigns one or more of four defect categories — no defect; instruction-verification mismatch (the instructions and the grading tests ask for different things); task drift (external state such as package versions, APIs, or web content changed after release); and environment issues (CPU, memory, or time budgets too tight even for the reference solution) — along with evidence and a proposed fix, which the verifier agent checks by re-running the reference solution. Applied to the 89 tasks of Terminal-Bench 2.0 (Merrill et al., 2026, ICLR), this workflow produced Terminal-Bench 2.1. The authors then re-evaluated 19 agent-model pairs on both TB2.0 and TB2.1, extended the time budget up to 32x to see how the efficiency frontier shifts, and tested whether the same workflow transfers to a separate benchmark, Harbor-Index v0.1.

Key Results

All figures below are read directly from the paper's tables and text (full HTML source, arxiv.org/html/2611.08417v1). The audit found defects in 28 of the 89 tasks (about 31%) — a total built from tasks already flagged by a prior audit (Terminal-Bench Verified, 11 tasks) and public issues/PRs (5 tasks), plus 14 newly identified by this audit (2 tasks overlap across sources and are counted once).

A single task can carry more than one defect label, so the shares add to over 100% (paper's Fig. 3a, Table 2). The most common category, instruction-verification mismatch, is illustrated by the query-optimize task, where the grader silently required byte-for-byte preservation of a database file — a requirement never stated in the instructions.

Re-evaluating 19 agents on the repaired Terminal-Bench 2.1 raised average accuracy by over 10 percentage points, and leaderboard positions moved by up to three ranks. The overall rank correlation stayed high (Spearman ρ=0.958, bootstrap 95% CI [0.881, 0.979]), so the broad ordering held, but individual agents deviated from the fitted trend (TB2.1 = 2.1 + 1.04 × TB2.0) by a meaningful margin — Claude Code + Opus 4.6 landed more than 7pp above the trend line and climbed three ranks, while the two weakest agents (below 40% on TB2.0) fell below the trend and actually lost accuracy on TB2.1 (consistent with 19 of the 28 repaired tasks being rated "hard," so fixes matter little to agents that rarely solved them anyway). The gains were also concentrated: five tasks with instruction-verification fixes jumped from pass rates of 15% or below to over 50%, accounting for roughly 62.7% of the total absolute pass-rate change across all modified tasks.

The second finding is that much of what looks like a "harness gap" is really a speed problem. Over half of failed trials ended in timeout, ranging from 18% under gemini-cli to 68% under claude-code. Re-testing six agent-model pairs with the time budget extended up to 32x the default changed the picture.

In the same experiment, GLM 5.1 + Claude Code solved 8 more tasks on TB2.1 and overtook GPT 5.4, whose coverage barely moved — a reminder that even efficiency rankings can shift with the benchmark version (paper's Fig. 5, 6).

Third, to check that this isn't a Terminal-Bench-specific result, the authors applied the same workflow to a separate benchmark, Harbor-Index v0.1 (a random sample of 34 tasks from an 80-task snapshot drawn from 29 source benchmarks). The workflow flagged 14 tasks, and manual review by the Harbor team confirmed 12 as genuinely defective — 86% precision. It also missed some defects: one wrong reference answer in Lab-Bench, nondeterministic grading in QCircuitBench, one HLE task admitting two plausible correct answers, and two shared grading-code bugs affecting 13 tasks in GSO and GAIA2 (some overlapping with tasks already flagged for other reasons).

Credibility Assessment

Four things support trusting these results. First, this is an independent Stanford team with no stake in any of the evaluated vendors' models. Second, inter-auditor reliability was measured directly: managed coding assistants (Claude Code + Opus 4.6, Codex + GPT-5.4) agreed on the primary defect label (including "no defect") at Cohen's kappa 0.80 across all 89 tasks. Third, the external-validity test on a separate benchmark (Harbor-Index) reached 86% precision, suggesting the workflow isn't just tuned to quirks of Terminal-Bench. Fourth, an independent prior audit (Z.ai's Terminal-Bench 2.0 Verified) reached a convergent conclusion — environment fixes alone raised Claude Opus 4.5's score from 52.4% to 57.9%, matching this paper's claim that environment defects alone can move scores substantially.

The caveats are just as clear. This is a preprint that has not been peer reviewed. The workflow's own precision (57.1%) and recall (71.4%) are measured against the final TB2.1 release, which the workflow itself helped produce — the authors themselves note this measures retrospective agreement, not independent defect discovery. Lower-cost API auditors showed moderate category-level agreement for instruction-verification mismatch (κ=0.60) and environment issues (κ=0.59), but low agreement for task drift (κ=0.21–0.41) — which is why the authors route drift detection to a deterministic re-run of the reference solution instead of trusting LLM judgment there. And while Z.ai's prior audit and this one reach the same broad conclusion, their repair philosophies are opposite: Z.ai rewrote instructions to match what the grader checks (effectively leaking part of the solution) and TB2.1 adopted that fix on only 4 of Z.ai's 11 instruction fixes, preferring in most other cases to loosen the grader instead, to preserve the task's intended difficulty. The broader industry funding of the authors' lab is disclosed above under "The Paper."

Related Research

• Zhu et al. (2025). Establishing Best Practices for Building Rigorous Agentic Benchmarks — prior work, extended here. Proposed the Agentic Benchmark Checklist (ABC), which cut performance overestimation on CVE-Bench by 33% and found estimation errors of up to 100% (relative) across ten widely used benchmarks. This paper adapts the ABC rubric for ongoing maintenance rather than one-time release review.
• Merrill et al. (2026, ICLR). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces — benchmark-origin. The original Terminal-Bench 2.0 paper that this audit targets; at release, even the best agents scored below 65%. That this benchmark had already been through human review and LM-assisted red-teaming, yet still had defects in 28 of 89 tasks, underscores that pre-release validation and continuous maintenance are different jobs.
• OpenAI (2024). Introducing SWE-bench Verified — a prior case in the same category, and a contrasting approach. 93 developers reviewed 1,699 SWE-bench tasks to produce a filtered 500-task "verified" subset. This paper classifies that as a one-shot filtering approach and positions its own method as continuous repair instead — two different philosophies (filter out defects vs. fix them) coexisting in the field.

Taken together, these three works trace a line from "how to classify benchmark defects with a checklist (ABC)" through "pre-release validation still leaves defects behind (Terminal-Bench's original paper versus this audit)" to "whether to filter defects out or repair them (SWE-bench Verified versus this paper)." This paper adds a number to that line that was previously missing: how much the leaderboard actually moves once defects are repaired — which doesn't contradict the prior work so much as fill in a gap it left open.

Reviewer's Take

First, I'd flag treating a "harness gap" as a "capability gap" as the single most dangerous habit this paper exposes. The 20pp gap for GPT 5.4 and the 30pp gap for GPT 5.4 mini mostly or substantially closed once the time budget was extended, which means a common internal practice — comparing two harnesses "under the same timeout" — may really only be measuring which one is faster under a fixed budget. Deciding to switch models or harnesses based on a single fixed-budget comparison risks the wrong conclusion.

Second, I'd weigh the external validation on Harbor-Index (86% precision) far more heavily than the workflow's own precision and recall (57.1%/71.4%). The latter is a circular check against the workflow's own output, while the former compares against human review that had no part in building the workflow. When citing a benchmark-auditing tool's performance, it matters a great deal which kind of validation produced the number.

Third, the unusually low inter-auditor agreement specifically on task drift (κ=0.21–0.41) reads as validation of a design choice rather than a weakness: the authors didn't leave this category to LLM judgment but routed it to a deterministic re-run of the reference solution instead. That's a reasonable general pattern — separate the categories that need judgment from the ones that can be checked mechanically, and design the pipeline around that split.

Practical Takeaways

• State the benchmark version — when citing a benchmark score as grounds for a model choice, note whether it's TB2.0 or TB2.1 (or equivalent). The same benchmark's version alone can move rankings by up to three spots.
• Re-test harness comparisons under a larger budget — if two harnesses show a large score gap, re-run the comparison with an extended time budget to separate a speed difference from a capability difference.
• Track timeout share separately — log what fraction of failures are timeouts (18-68% by harness in this paper) rather than lumping them in with wrong answers.
• Re-run reference solutions to catch drift automatically — for internal eval tasks that depend on external APIs or packages, periodically re-run the known-correct solution to catch false failures from state that changed outside your control.
• Don't over-trust close rankings — even with a high overall rank correlation (ρ=0.958), individual scores moved by up to 12pp, so don't settle a close model choice on benchmark score alone.

Conclusion

This paper's contribution isn't a new benchmark but a number attached to an idea: benchmarks need maintenance the way software does, and that maintenance has a measurable cost. Terminal-Bench 2.0, despite human review and red-teaming before release, still had defects in 28 of its 89 tasks (about 31%); fixing them raised accuracy by over 10 percentage points and moved rankings by up to three ranks. The finding that much of the apparent "harness gap" was really a time-budget effect means resource settings deserve to be treated as part of the measurement specification, not an afterthought.

That said, the workflow's own precision and recall are circular by construction, and its reliance on deterministic re-runs (rather than LLM judgment) for drift detection is itself a signal that full automation of benchmark auditing isn't here yet. This connects to our playbook on checkpoints and early stopping for long-horizon coding agents from a different angle — that piece is about the agent being measured; this review is about defects in the yardstick doing the measuring.

References

• Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks — arXiv abstract (source)
• Same paper, full HTML text — used for table verification
• Establishing Best Practices for Building Rigorous Agentic Benchmarks (Zhu et al., 2025) — related research source
• Terminal-Bench (Merrill et al., 2026, ICLR) — related research source
• Introducing SWE-bench Verified (OpenAI, 2024) — related research source
• 35% of Finished Products Still Fail, Yet Builds Run to Completion: Checkpoint and Early-Stop Design for Long-Horizon Coding Agents — sunny34.com blog