Passed 82.8%, Caught Bugs in Only 33.8%: What Req2Test-Bench Reveals About Agent-Generated Tests

Source

Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong, "Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements", arXiv:2610.36584 [cs.SE], submitted 2026-10-27, DOI 10.48550/arXiv.2610.36584. Affiliation: National University of Singapore (NUS), Shanghai Jiao Tong University (SJTU), Beihang University (BUAA). Benchmark code and data released (github.com/code-philia/Req2Test-Bench, huggingface.co/datasets/code-philia/Req2Test-Bench).

This is an unreviewed arXiv preprint with no stated conference or journal acceptance. All authors are university-affiliated (NUS, SJTU, BUAA) and the evaluated projects are all public open source, so this lacks the kind of conflict of interest that shows up when a company evaluates its own systems. One caveat: the LLM judge used for failure-mode labeling (Claude Opus 5.5) comes from a company unaffiliated with the authors. This session's outbound network access was blocked entirely (control checks against both example.com and arxiv.org were rejected with a 403 at the proxy's CONNECT step); the full source text was checked against a primary-source snapshot collected at 2026-10-29T22:10:13Z.

Study Overview

The paper asks three questions: ① does scaling agent effort and cost proportionally raise behavioral adequacy; ② if not, what causes the gap; and ③ what directions might narrow it. The authors define behavioral adequacy as whether a test actually exercises the behavior a requirement specifies and correctly checks the outcome, and measure it strictly via kill-completeness — whether a generated test kills every mutant (an injected code fault) that the developer's own ground-truth test kills.

Req2Test-Bench recovers 5,086 execution-validated instances from developer-authored Gherkin (Given-When-Then) requirements and their matching ground-truth test code across 8 open-source projects spanning 5 languages and 4 BDD frameworks. Given a requirement, the target interfaces, and a repository view that hides either only the target implementation (target-hidden) or all production implementations (all-hidden), an agent must generate a complete test. Generated tests are then compared against ground-truth tests on API calls, code coverage, and mutant kills. Eight agent-model-reasoning configurations were evaluated: Aider, OpenHands, Codex CLI, and Claude Code, paired with DeepSeek-V4.1-Flash and GPT-6-family models.

Key Results

Pass@1 and KillComplete@1 for all eight configurations in the target-hidden view (source Table 4):

The best-passing configuration, Codex/DS (high reasoning), passed 82.8% of instances but was kill-complete on only 33.8%. Even under a relaxed criterion that counts a test as adequate if it reproduces at least 95% of the ground-truth test's kills, Codex/DS reached only 53.9% — 28.9 points below its Pass@1. The weakest configuration, Aider (DS, reasoning off), passed only 9.7%, and 53.4% of its generations delivered no test file at all. Switching to the all-hidden view, which also hides surrounding production code, lowered Pass@1 by 2.5-9.1 points for the five high-effort configurations, but KillComplete@1 fell by only 0.1-3.7 points — surrounding code mainly helps a test pass, not reproduce real faults.

Cost per generation (source Table 5, target-hidden view) ranged roughly 30-fold, from $0.039 for Codex/GPT-6 Luna to $1.16 for Codex/GPT-6 Sol, without tracking KillComplete@1.

Only three configurations — Codex/Luna, Claude Code/DS, and Claude Code/Sonnet — sat on the cost-effectiveness Pareto frontier; the priciest configuration, Codex/Sol, had a lower KillComplete@1 than any of them. Moving from Codex/Luna to Claude Code/DS along that frontier multiplies cost by 4.0x for a 6.1-point KillComplete@1 gain, but spending beyond that buys no measurable further improvement.

Credibility Assessment

Several things support trust. The 5,086 instances were recovered and execution-validated across 8 projects, 5 languages, and 4 BDD frameworks (test levels C/I/S sum to 588+2,066+2,432=5,086, matching the total), and mutant-to-ground-truth-test mappings were confirmed through both positive and negative executions. All six authors are university-affiliated (NUS, SJTU, BUAA) and the evaluated projects are public open source, so the kind of funding or conflict-of-interest concern common to corporate self-evaluation does not apply here. Reproducibility was partly checked too — regenerating five configurations twice flipped up to 24% of individual pass verdicts and 17% of kill-complete verdicts, but average Pass@1 and KillComplete@1 moved by at most 4.5 points.

Caveats: this is an unreviewed preprint, and mutation-kill is a construct-validity limitation — it is not a perfect proxy for real fault detection. The authors acknowledge this and ran a supplementary analysis counting only check-dependent kills; the gap narrowed by only 24-37%, suggesting it isn't purely a matter of the test's setup route. The eight projects were purposively sampled, not randomly drawn. As a point of contrasting evidence, TestGenEval (below) paints a more optimistic picture using coverage alone; this paper directly rebuts that optimism, showing that 17.4-69.7% of passing tests with coverage Jaccard above 0.9 are still kill-incomplete.

Related Work

• Jimenez et al. (2024, ICLR). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — benchmark-origin. The foundational benchmark for agentic software-engineering evaluation via GitHub issue resolution. It differs by evaluating bug fixing itself rather than test generation; Req2Test-Bench specializes the same "agent works at repository scope" setting to isolate the test-generation axis.
• Jain, Synnaeve, Rozière (2025, ICLR). TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark — prior work. The closest predecessor evaluating test generation on SWE-bench repositories; it scores coverage and mutation but does not evaluate correspondence to developer-written natural-language requirements. Req2Test-Bench surfaces, at the requirement level, the gap this paper identifies: tests can have high coverage yet still fail to reproduce real faults.
• Li et al. (2024). DevBench: A Comprehensive Benchmark for Software Development — prior work (directly compared in the paper's own Table 1). Covers the full software lifecycle — design, environment setup, implementation, acceptance, and unit testing — but does not map individual requirements to individual tests as execution-validated units. Req2Test-Bench fills this gap with Gherkin-Scenario-level mapping.

All three prior works share the premise that agents operate on code and tests at repository scope, but this paper is the first to quantify the behavioral correspondence between a requirement and a test. TestGenEval's relatively optimistic, coverage-centric picture is substantially undercut once Req2Test-Bench's kill-completeness metric is applied — that tension is this paper's central claim.

Reviewer's Take

First, the abstract's headline figure — "82.8% pass" — gives the impression that coding agents are already usable for writing tests, but I think that's premature for production use. The gap between pass rate and kill-completeness (82.8% vs. 33.8%) shows that "the test runs" and "the test catches the bug" are entirely different questions; trusting an agent-written test on a green CI check alone risks missing a real defect roughly two times out of three.

Second, I'd call the finding that cost doesn't track performance (RQ2) the most practically useful result in the paper. That the most expensive configuration (Codex/Sol) had a lower KillComplete@1 than cheaper ones undercuts the intuition that "a pricier model means a safer test" for this particular task.

Third, the authors' own preliminary finding in Section 5.5 — that a single added prompt instruction to use project-specific knowledge raised KillComplete@1 by up to 15.0 points — is treated almost as an afterthought in the text, but it looks like the highest-leverage intervention in the whole paper. It suggests that tuning a harness's prompt and context design pays off more than upgrading the model.

Practical Takeaways

• Track pass rate and fault-detection rate separately — don't let a CI dashboard show only pass rate; track a generated test's fault-detection rate separately, ideally against mutation testing or a regression suite built from past bugs.
• Make setup and expected outcomes explicit in requirements — this paper's information gap (90-99% of kill-incomplete tests) traces back to requirements that left out how to set up state or what exactly to check. Add "could a third party write the same test from this requirement alone?" to your review checklist.
• Add project-knowledge instructions before upgrading models — a prompt instruction to build state through the project's own fixtures, helpers, and entry points had a larger cost-to-benefit payoff in this paper than switching models.
• Don't assume a pricier model means a safer test — check the cost-effectiveness Pareto frontier before picking a model for test generation.
• Watch for hedging patterns — loose exception handling or fallbacks can produce tests that "pass no matter what"; include detection of this pattern in code review.

Conclusion

Req2Test-Bench's contribution is separating, in concrete numbers, "the test passes" from "the test actually catches the bug a requirement specifies." Even the best configuration reproduced bugs on only about 41% of its own pass rate, and that gap barely narrowed with more spending. Authorship is entirely academic and the evaluated projects are public open source, so conflict-of-interest concerns are limited, but this remains an unreviewed preprint, and mutation-kill is not a perfect proxy for real fault detection. Organizations treating agent-written tests as a QA gate should judge them by fault-detection rate rather than pass rate, and verify against their own regression suite first. For a broader look at bringing agent-written tests into a QA gate, see the QA gate design that IDE agent testing demands.

References

• Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements — arXiv abstract (source)
• Same paper, full HTML text — used for table and figure verification
• Official implementation and data repository (GitHub)
• The QA gate design that IDE agent testing demands — sunny34.com blog