Agents Find the File, Miss the Root Cause: A DistDebug-Bench Review

The paper

Yuxuan Jiang, Chenglin Liu, John Rose, Mason Miller, Ruthesh Thavamani, Juntao Wu, Yinwei Huang, Jiahao Xu, Peng Huang, "DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?", arXiv:2611.03523 [cs.SE], submitted 2026-11-02. Affiliation: University of Michigan. Benchmark and evaluator public (github.com/OrderLab/distdebug-bench); traces and leaderboard at distdebug.com.

This is a non-peer-reviewed preprint. Funding came from an NAIRR Pilot grant (which included OpenAI API credits) and NSF grants (CNS-2317698, CNS-2317751, CCF-2318937). Despite the OpenAI credit, the evaluation covers a wide model roster — Claude Opus 5.5, Gemini 3.5, DeepSeek, Kimi, MiMo, HY3 alongside GPT models — and the gap between the top model (GPT-6 Astra) and the runner-up (Claude Opus 5.5) is only 1.3 points, which weakens any case for funder bias.

Study Overview

The research question: can LLM agents pinpoint the actual pre-fix root-cause code for real distributed-system failures? Prior SWE-bench-style benchmarks only score whether a patch passes a test suite, which conflates "suppressing a symptom" with "fixing the real cause." The dataset is 120 curated real-world bugs across 11 widely used distributed systems (5 languages), with 218 gold root-cause blocks. Each instance gives the agent an incident report, evidence (logs, stack traces), and the pre-fix repository snapshot; the agent must name the source block(s) that explain the failure. Three independent annotators cross-checked the gold labels, and predictions are scored at four granularities — file, function, line, block. A leakage guard strips future commits and issue identifiers (flagged attempts for GPT-5.4 dropped from 102/360 to 4/360 after the fix).

Key Results

All figures are averages across the 120 shared bugs. File/function/line/block are coverage percentages; block coverage is the paper's primary localization metric. "Flaky" means the within-bug block-coverage spread across three repeated attempts exceeds 0.2.

Source: paper Table 1, averaged over 120 common bugs. Every configuration shows file coverage 15-25 points above block coverage — the gap between "reaching the relevant file" and "naming the exact causal block." Holding the model fixed (GPT-5.2-Codex) and swapping only the harness gives 56.4% (Mini-SWE-Agent), 50.0% (OpenCode), 46.5% (Codex CLI) block coverage, while block precision runs the other way — Codex CLI highest at 49.3%, Mini-SWE-Agent lowest at 43.1% — so harness choice trades recall for precision (paper Table 1).

Evidence quality drives a similar gap. When the report names the causal method, the best configuration reaches 94.8-95.8% block coverage; on "opaque" reports with neither a stack trace nor a named method (50 of 120 bugs, 42 of those with no stack trace at all), the best configuration drops to 65.0%, and to 40.5% on the seven cross-component opaque bugs (Section 4.2; small sample, exploratory). For the fixed Mini-SWE GPT-5.2-Codex baseline the same ladder is steeper: 82.8% to 44.7% to 14.3%.

Credibility Assessment

Three things support trust: gold labels were cross-checked by three independent annotators; a leakage guard plus closed-book probes show repository access matters far more than ID leakage (45.5% with repository access vs. 21.4%/17.3% without it, so training-data memorization is unlikely to explain the results); and the study holds 120 bugs fixed across 13 model configurations plus 2 models × 3 harnesses, isolating model effects from harness effects. Caveats: this is a pre-review draft, and every reported value carries a roughly 7-8 point 95% CI (n=120), so 1-2 point rank differences are not statistically meaningful. The block-coverage metric itself may be conservative — a blinded re-audit of 20 "zero coverage" GPT-5.4 diagnoses found 4 had actually identified a different, valid intervention point, suggesting the benchmark may understate true diagnostic competence in some cases.

Related Work

• Jimenez et al. (2024, ICLR). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Prior work and comparison point. Scores test-passing patch generation on 2,294 GitHub issues from 12 Python repositories. DistDebug-Bench scores root-cause diagnosis directly rather than patch success, and extends scope from single-process Python projects to 11 distributed systems across 5 languages.
• Chen et al. (2025). AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds — Prior, complementary work. A 48-task framework that injects faults into microservices and scores detection/root-cause analysis/mitigation from telemetry (metrics, traces). DistDebug-Bench instead targets precise source-code-level diagnosis, showing that "root cause analysis" can mean different evidence and precision depending on the benchmark.
• Deng et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? — Extension. Scales to 1,865 tasks across 41 repositories with hours-to-days-long tasks; the original release reports GPT-5 at 23.3% Pass@1. It raises task difficulty but still scores patch success, leaving the diagnosis-versus-repair gap that DistDebug-Bench addresses untouched.

Taken together, prior benchmarks score either patch success (SWE-bench, SWE-Bench Pro) or telemetry-driven operational response (AIOpsLab). DistDebug-Bench fills the space between them — scoring "why was this the cause" at the source-code level — and does not contradict either line of work.

Reviewer's Take

First, this benchmark's most practically useful contribution is separating file coverage from block coverage. Most vendor and leaderboard reporting collapses performance into a single patch-success number; the 16-point gap between 83.9% and 67.7% quantifies something teams building merge gates or review automation need to design around directly: reaching relevant code is not the same skill as pinpointing the cause.

Second, the flaky-bug rate (12-29%) is the number most likely to be underweighted. A 29% chance that a repeated run of the same bug produces a materially different diagnosis means a single evaluation run can paint an overly rosy or overly harsh picture of an agent's debugging reliability — teams should budget for repeated trials before trusting a verdict.

Third, the tradeoff where GPT-5.5 improves opaque-case coverage by 15.4 points over GPT-5.2-Codex while block precision falls from 43.1% to 34.7% deserves attention: a stronger model is not automatically a more useful diagnostician — gains in recall can come bundled with more irrelevant blocks.

Practical Takeaways

• Grade at the block/function level, not the file level — approve agent diagnoses based on whether they named the exact causal block, not whether they opened a relevant file.
• Run at least 3 trials before trusting a verdict — don't conclude an agent's debugging reliability from a single run; route high-variance cases to human review.
• Route by evidence quality — incidents lacking a stack trace or a named method ("opaque" reports) should escalate to a human first rather than rely on agent-only diagnosis.
• Pick a harness for your precision/recall tradeoff — choose a precision-leaning harness (Codex-CLI-style) where false positives are costly, and a recall-leaning harness where broader search matters more.
• Add tunnel-vision checkpoints — force a hypothesis re-check when the agent hasn't converged within the first two files it inspects.

Conclusion

DistDebug-Bench separates "did the agent reach the relevant code" from "did it name the actual cause," and shows that today's best agents clear 80%+ on the former but stay in the 60s on the latter. That gap widens on weak-evidence incidents, and repeated runs on the same bug often disagree — so a single aggregate success rate is not enough to evaluate a debugging agent for production use. Given the pre-review status and the roughly 7-8 point confidence intervals on a 120-bug sample, these numbers are best read as directional and re-verified against your own codebase before relying on them operationally. For the operational side of setting merge-gate thresholds for coding agents, see Full-pass-rate gate design for coding agents (sunny34.com blog).

References

• DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? — arXiv abstract (primary source)
• Full HTML text of the paper — used for table/number verification
• SWE-bench (Jimenez et al., 2024) — related work, primary source
• AIOpsLab (Chen et al., 2025) — related work, primary source
• SWE-Bench Pro (Deng et al., 2025) — related work, primary source
• Full-pass-rate gate design for coding agents — sunny34.com blog