Source
Zhongwen Luan (East China Normal University), Xiaoyu Zhang (Nanyang Technological University), Ming Hu (East China Normal University · Singapore Management University), Yue Yang (Xi'an University of Architecture and Technology), Jiongchi Yu (Nanyang Technological University · Singapore Management University), Xiaohong Chen (East China Normal University), "Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems", arXiv:2608.25920 [cs.AI], submitted 2026-08-26.
This is a preprint that has not been peer reviewed. The authors are affiliated with three universities in China and Singapore; no funding statement or conflict-of-interest disclosure is present in the paper. The full text was verified against a first-party source snapshot collected by GitHub Actions at 2026-08-28T23:06:50Z (UTC) — the session's egress was blocked to every destination at fetch time (including the control target example.com, all returning 403), so this snapshot path was used instead of WebFetch.
What the paper asks
The question is simple: when a re-run of a failed LLM multi-agent task succeeds, was the failure actually fixed, or did a different random rollout happen to pass? The authors built SymTrace, a replay framework, to tell the two apart. During execution it logs every model and tool call as a dependency graph; during replay, it injects the recorded results up to a chosen intervention point — holding that prefix fixed — and only lets execution run live from that point on. This isolates "what changed" from upstream resampling noise.
On top of SymTrace, the authors built SymFail: 167 WebArena-Verified Hard tasks plus 33 AssistantBench tasks (200 total), each run once on AG2, CrewAI, and Magentic-One, yielding 536 confirmed failures out of 600 runs. Four annotators with software-engineering backgrounds hand-labeled the failure node, category, and evidence for each case (three independent annotators, then a fourth makes the final call). Failure categories (C1–C4) are drawn from an existing taxonomy (Cemri et al., 2025).
Key results
First, reproduction rates (repk): comparing Unguided Full Rerun against SymTrace Replay, the single-attempt reproduction rate is 67.97% vs. 80.78%, and the three-attempt-consistent rate is 41.42% vs. 52.43% — replay wins throughout.
More importantly, repair success. The table below reports pass@k (at least one accepted repair within k attempts) across all 536 failures. Task-level methods (rerun, self-reflection, critic agent) get three attempts; node-level methods (random, last-node, symptom-guided) get one.
| Method | Attempts | AG2 | CrewAI | Magentic-One | Total |
|---|---|---|---|---|---|
| Last-Node Intervention | 1 | 0.58% | 1.63% | 1.66% | 1.31% |
| Random-Node Intervention | 1 | 2.34% | 4.89% | 3.87% | 3.73% |
| Critic-Agent | 3 | 4.09% | 2.72% | 4.42% | 3.73% |
| Self-Reflection | 3 | 4.68% | 2.17% | 6.08% | 4.29% |
| Unguided Full Rerun | 3 | 8.19% | 3.80% | 8.84% | 6.90% |
| Suspicious-Node Intervention | 1 | 16.37% | 25.00% | 18.78% | 20.15% |
Suspicious-Node Intervention repairs 20.15% of all failures with a single attempt — 2.92x the strongest task-level baseline (rerun, 6.90% with three attempts; the paper reports this as a 191.89% improvement). It also clears Random-Node (3.73%) and Last-Node (1.31%) by a wide margin under the same one-attempt budget, showing the gain isn't just from having an intervention point — where you place it, guided by symptom evidence, matters.
In a separate check, the authors re-ran 54 originally successful cases three times each under the same settings. Of 162 re-runs, 85 failed, and 39 of the 54 cases (72%) flipped to a failure at least once. Rerunning can break a working case about as easily as it can fix a broken one.
Credibility assessment
The verification design is a real strength. Prefix reconstruction is validated by content-hash matching (100% exactness), and the authors re-audited their own LLM-judge pipeline on a 72-case sample, reporting a 3.30% confirmed-disagreement rate and 3.67% uncertainty rate (6.96% combined upper bound) — an unusually transparent error accounting. Human annotation reliability is solid too: Fleiss' κ of 0.62 for primary category and 0.81 for failure-node type.
Caveats the authors state themselves: (1) SymFail is derived entirely from WebArena/AssistantBench-style tasks and may not capture the full range of production failure modes; (2) reproduction judging relies on automated LLM judges, which can misread long trajectories; (3) every experiment uses a single model (deepseek-v4-flash, temperature 0), so the numbers may not transfer to other models or versions. The paper's conclusion also sits in direct tension with prior work (Reflexion, CRITIC) reporting that self-reflection and critique improve success over repeated trials — the authors attribute the gap to those studies conflating repeated-sampling gains with feedback gains, but independently reproducing that claim under this paper's controlled framing remains open.
Reviewer's take
The number worth remembering isn't 20.15% — it's 39/54 (72%). Treating "it passed on rerun" as proof of a fix is exactly what this paper argues against; a team that gates releases on pass@3 alone, without a reproducibility check, is likely breaking working cases at roughly the same rate it's missing real fixes.
Second, reading 20.15% as pure evidence for "symptom detection works" over-claims. The paper is explicit that Suspicious-Node Intervention bundles target selection with repair-instruction generation and does not isolate either component's contribution. Bolt on symptom detection without equal care on the repair prompt, and this number may not reproduce.
Practical applications
- Freeze execution before you retry — snapshot/replay infrastructure that reconstructs the pre-failure state is a prerequisite for any credible "we fixed it" claim.
- Re-verify successes, not just failures — add a regression check that reruns passing cases; by this paper's numbers, 72% of "successes" can be unstable.
- Invest in node-level anomaly detection — localized, symptom-guided intervention delivered far higher repair yield per attempt than full task-level retries.
- Don't use pass@k alone as evidence of repair — it can't distinguish a real fix from a lucky resample; check whether the improvement holds on a reconstructed, fixed execution.
- Flag single-model results as unverified elsewhere — re-measure reproduction and repair rates in your own stack before porting numbers across model vendors or versions.
Conclusion
The paper's real contribution isn't the 20.15% headline — it's the method for separating a genuine fix from the confound that has always haunted rerun-based repair evaluation: random success from repeated sampling. If full-task rerun is still your default repair strategy, this paper puts numbers behind the possibility that what you're measuring is re-roll luck, not repair capability.
Adding a reproducibility check to an existing repair pipeline is an immediate, low-cost change; moving to node-level intervention is a medium-term investment that depends on having the observability infrastructure in place first.