Detection 47.7%→71.9%, -25.5pp In-House: Reviewing CRBugBench on Code Review Agents

Source Document

Chenyang Yang, Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill, Carmen Badea, Christian Bird, "(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection", arXiv:2610.13668v1 [cs.SE], submitted 2026-10-09, CC BY 4.0. Affiliation: Carnegie Mellon University (first author; work done at Microsoft Research) · Microsoft · Microsoft Research. The open-source portion of the benchmark is released (github.com/malusamayo/CRBugBench); the proprietary-repo data is not.

This is an arXiv preprint without peer review (the venue field is blank). Every author is affiliated with Microsoft or Microsoft Research, and the two proprietary repositories used in the study are attributed only to "one large company," without naming it — since the authors used their own employer's internal data, we flag this as a conflict of interest. The default evaluation harness, GitHub Copilot CLI, is also a Microsoft product, which is worth noting on the same point. To their credit, the authors disclose this choice transparently through a comparison with claude-code (§4.6), and while the replication package is not public, the open-source benchmark itself is.

Study Overview

The question is whether a fully autonomous, "lights-off" software factory — where AI both writes and reviews code — is feasible today. The authors argue that existing benchmarks, which score generated reviews against human comments via text similarity, inherit the limits of human comments themselves (nitpicking, incompleteness). Instead they built a pipeline that mines bugs humans actually confirmed from repository history and traces each one back to the PR that introduced it.

CRBugBench combines bugs caught in pre-merge review comments (the "Cmt" track, 500 sampled) with bugs caught via post-merge bug-fix PRs (the "Fix" track, 500 sampled), for 1,000 tasks across 98 open-source GitHub repositories plus 800 tasks from two large proprietary company monorepos — 1,800 tasks in total. Agents explore a sandboxed repo without being told a bug exists, write a severity-ranked review to review.md, and a three-model LLM judge panel (gpt-5.5, claude-opus-4.8, gemini-3.1-flash) decides by majority vote whether a finding matches the ground truth. The metric is recall@k — whether the ground-truth bug appears among the agent's top-k findings.

Key Results

The top model, claude-opus-5 (copilot-cli harness, xhigh reasoning effort), reaches 79.6% top-5 recall on the Cmt track and 64.2% on Fix. The paper's own stated average across both tracks, 71.9%, is a sharp rise from claude-opus-4.6's 47.7% five months earlier. Average cost per task across both tracks is about $2.97, with average latency of 9.8 minutes.

Fix (missed at review, surfaced later) is generally harder than Cmt (caught before merge). Cost-performance is non-linear too: gpt-5.6-luna matched 78% of opus-5's average performance at 21% of the cost (55.8/71.9 = 77.6%, 0.62/2.97 = 20.9%).

Running the same models on large proprietary monorepos (100k+ source files on average, versus ~3k for open-source) drops performance by up to 25.5 points. Only 22.8% of proprietary-repo bugs are directly visible in the diff versus 46.1% for GitHub tasks, and this visibility gap alone explains 20% of the performance gap. Humans alone catch 50%, the best agent alone 71.9%, but combined they reach 82.1% — evidence of complementary strengths. A multivariable logistic regression (task-clustered standard errors) reports these adjusted odds ratios (OR):

Trajectory analysis: 75% of turns go to reading and searching (39% Read, 36% Probe), execution only 8%. claude-opus-5 executes code in 9.7% of turns versus 6.9% for gpt-5.6-sol, and persists longer (half-life turn 35 vs. 17). Breaking down failures: 95.6% opened the defect file, but only 80.2% produced a finding for it, and only 56.2% matched the ground truth — agents fail more at correctly diagnosing what they've read than at finding the right file (even claude-opus-5 shows the same gap: 91.9% produce a finding vs. 75% correct diagnosis).

Credibility Assessment

In favor: scale (1,800 tasks, 98 open-source repos, 5 languages), careful bug-quality control via SZZ tracing plus multi-stage filtering (a random sample of 40 was 92.5% valid, Wilson 95% CI 80.1–97.4%), a three-model LLM judge panel (human-LLM agreement κ=0.89, cross-LLM Fleiss κ=0.95), and cluster-robust standard errors with multiple-testing correction (Benjamini-Hochberg).

Caveats: every author is at Microsoft or Microsoft Research, and both the proprietary repos and the default harness (Copilot CLI) are the same company's assets — a conflict of interest. The replication package is not public. External validity is limited to "98 popular open-source repos plus two monorepos at one large company." There is counter-evidence too: Tufano et al. (2025) below, in a controlled human-AI review study, found LLM review increased low-severity comments but not high-severity detection, review time, or confidence. A rising recall number is a different question from rising real-world review value.

Related Work

With egress blocked for re-reading full source texts, we identified and verified these independently via WebSearch (the descriptions below draw on CRBugBench's own related-work section plus bibliographic details cross-checked via WebSearch; we do not cite numeric claims from these papers themselves).

• Pereira et al. (2026). CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents — prior work with similar methodology. Like CRBugBench, it traces real bugs back to introducing PRs for evaluation, but separately flags a signal-to-noise tradeoff — agents chasing every hidden issue can hurt usable precision — a useful complement to any strategy that only optimizes recall.
• Tufano et al. (2025, ICSE). Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword? — contradicting evidence. A controlled human-AI collaborative review study found LLM review increased low-severity comments but not high-severity bug detection, review time, or confidence — grounds for not equating CRBugBench's rising recall with rising real-world review value.
• Zhang et al. (2026). Code Review Agent Benchmark (c-CRAB) — methodological extension (alternative approach). Converts human review comments into executable tests and scores whether a patch addressing the review passes them. CRBugBench explicitly contrasts itself with this approach in its own §2.2, scoring recall against known bugs instead — the two benchmarks measure different things and their scores are not directly comparable.

Reviewer's Judgement

First, I judge the most practically important finding to be not the 56.2% final number but the funnel that narrows from 95.6% to 80.2% to 56.2%. The share of misses from simply failing to open the right file is small (4.4%); the much larger losses come from failing to turn that inspection into a written finding (15.4 points) or writing a finding that is simply wrong (24 points). That argues for investing in correct reasoning over code already read, not in retrieval.

Second, the finding that "copilot-cli outperforms claude-code" (§4.6) is a useful demonstration that harness choice drives outcomes — and simultaneously a reminder of the conflict of interest. Nothing in the paper rules out the possibility that Microsoft-affiliated authors choosing their own product as the default harness favored the comparison; the explanation that claude-opus-5 with claude-code incurs a cost spike from heavier sub-agent delegation (9.0 per run versus ~0) is plausible, but it also shows just how much harness choice can move a comparison.

Third, "82.1% for human+agent combined" is reasonable support for complementary collaboration over full automation, but it rests on the assumption that humans and agents each do their independent best. Factoring in the anchoring effect Tufano et al. (2025) report — reviewers who see an AI review first growing complacent — real sequential operations could plausibly fall short of 82.1%, and that caveat belongs alongside the headline number.

Putting It to Work

• Allocate review effort to PR complexity — larger, more heterogeneous diffs (OR 0.591, 0.795) detect worse, so pair those with stronger models plus human review, while small, homogeneous PRs can run on cheaper configurations.
• Call out non-functional concerns explicitly — default review skews toward severe functional bugs (OR 0.454 against usability-type issues), so ask explicitly for usability, portability, or maintainability when those matter to your team.
• Feed repository knowledge into reviews on proprietary code — the share of non-diff-visible bugs is more than double on proprietary repos (22.8% vs. 46.1%), so supply repo-wide context, past incidents, and design docs to the review agent.
• Re-evaluate on every model or harness change — comprehensive-prompt gains vary by model (nearly vanishing for opus-5), and harness traits like sub-agent delegation frequency drive cost, so periodically re-run a small benchmark built from your own repos.
• Tier models by PR characteristics — the way gpt-5.6-luna delivers 78% of opus-5's performance at 21% of its cost shows room to route cheaper or stronger models by cost/importance and lower average spend.

Conclusion

By scoring code review agents on whether they catch known real bugs rather than mimicking human comments, CRBugBench shows both a fast five-month rise from 47.7% to 71.9% detection and a reality check: that same capability drops up to 25.5 points on large proprietary repos. A fully autonomous, lights-off software factory is not yet feasible, and the authors themselves recommend keeping humans in the loop with agents as "scalable peers." Read the numbers alongside the authors' shared employer, the Microsoft-product default harness, and the unreleased replication package. For how to operationally handle the test-and-review bottleneck as agent-generated code grows, see QA Gate Design for IDE Agent Testing.

References

• (When) Should a Software Factory be Lights-off? — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and regression figures
• Pereira et al. (2026). CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents
• Tufano et al. (2025). Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?
• QA Gate Design for IDE Agent Testing — sunny34.com blog