A 60% Score Drop That Scale Can't Fix: Reviewing MCP-Injector's Fault-Tolerance Study

Source Document

Antonio Ken Iannillo, Joshua Segun Owotogbe, Roberto Natella, Francesco Avallone, Kristina Kudryavtseva, Indika Kumara, "Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance", arXiv:2610.32555 [cs.SE, cs.AI], submitted 2026-10-23. Affiliation: SnT, University of Luxembourg; Tilburg University; Gran Sasso Science Institute; University of Naples Federico II. The full text was checked against a snapshot GitHub Actions collected at 2026-10-26T22:07:58Z (a copy of the HTML full text) — the official fallback after this run re-confirmed, three times at 10-minute intervals, that session egress was fully blocked (including the unrelated control domain example.com, 403 on both).

This is a preprint without peer review, classified under cs.SE (Software Engineering) and cs.AI as an empirical evaluation paper. All six authors are academically affiliated (Luxembourg, Tilburg, Gran Sasso, Naples Federico II). The only funding disclosure is computational support from the Dutch SURF organization (allocation EINF-16024), and the authors state they declare no competing interests.

Study Overview

The paper asks two questions: how well do LLM agents calling tools over MCP (Model Context Protocol) hold up against the kinds of tool failures that actually occur in production, and does scaling the model up buy more of that resilience. Existing tool-use benchmarks assume tools always respond correctly, so they cannot answer either question.

The authors build MCP-Injector, a proxy-style middleware that interposes on the MCP tool-call path, and calibrate its fault profile empirically: they classify 57,240 issue and pull-request threads across 1,774 public MCP server repositories using an LLM (gpt-4o-2024-08-06) into failure modes (delayed response, silent drop, explicit error, incorrect output, invalid output, minor mistake) and persistence categories (permanent, intermittent, transient), then inject exactly that mined distribution. Classification reliability was checked by two independent raters on a 117-thread sample, yielding Cohen's κ=0.93 for failure mode and 0.80 for persistence. The resulting profile is dominated by permanent failures (91.3%) — a skew the authors attribute to GitHub issues over-representing reproducible, persistent bugs, and which they explicitly label a "stress profile" rather than a realistic incident rate.

The experiment is a full-factorial design crossing two model backbones (gpt-oss-20B, gpt-oss-120B), three agent stacks (MCPBench's own plan-and-execute loop, the OpenAI Agents SDK, and LangChain's create_agent), three conditions (proxy off, no-fault passthrough OP0, fault-injected OPP), 25 tasks, and three seeds — 1,350 cells in total. Scoring uses MCP-Bench tasks with an LLM-as-judge (gpt-4o-mini, median of five evaluations) across nine metrics.

Key Results

Injecting faults (OPP) drops the task-completion score from 8.08 (no-fault, OP0) to 3.23 — a 60% fall (paired difference −4.85 points, 95% CI [−5.22, −4.48]). All nine metrics fall by 54–68% (Benjamini-Hochberg corrected Wilcoxon p<0.001), and restricting to cells that completed under both conditions still shows a 57% drop — meaning the degradation is mostly quality loss in completed work, not outright non-completion.

† Counts task fulfillment only when tool appropriateness scores at least 2. Of the completed runs, 27 cells (6.0%) were "tool bypass" — a plausible-sounding answer produced without properly using the tools — and 17 of those never called a tool at all, meaning no fault could even reach them. Excluding those cells widens the task-completion drop from 60% to 67%: the reported 60% figure is, if anything, an underestimate.

The second key finding concerns model size. Under fault injection (OPP), completion rate is 95.6% [92.0, 97.6] for the 20B model versus 91.1% [86.7, 94.2] for the 120B — the larger model does worse (gap 4.4 points, 95% CI [−0.2, 9.3], Fisher's exact p=0.088 — not cleanly distinguishable from zero). Even fault-free completion is already lower for the 120B (96.9% vs. 99.1%), and the fault-attributable drop overlaps widely between the two (120B: −5.8 points [−10.4, −1.4]; 20B: −3.6 points [−7.2, −0.5]). The authors treat this as a descriptive observation, not an established scale effect.

The third finding concerns the agent stack. Completion rates under OPP ranked LangChain (95–97%) above MCPBench (91–95%) above OpenAI (88–95%). In a mixed-effects model, MCPBench loses an additional 0.93 points relative to LangChain (95% CI [−1.86, −0.01], p=0.048), and OpenAI an additional 0.84 points (p=0.074, borderline) — with no multiple-comparison correction applied, so the authors call this evidence "suggestive" rather than established. Among the 450 fault-exposed cells, the single most common outcome was partial degradation, not outright failure (56% overall, 44–71% depending on stack and model).

Credibility Assessment

This is an unreviewed arXiv preprint (submitted 2026-10-23). All six authors are academically affiliated, and the only funding disclosure is computational support from SURF, with the authors stating no competing interests. Neither the models under test (the gpt-oss family) nor the three agent frameworks are tied to the authors' institutions, so there is no sign of the "company evaluating its own product" conflict seen in some agent papers.

Three things support trust. First, the fault profile is not an arbitrary assumption but an empirical distribution mined from 57,240 real MCP issue and pull-request threads, with classification reliability independently checked (κ=0.93, 0.80). Second, the study includes two control conditions (no-fault OP0 and proxy-off) specifically to verify the middleware itself does not degrade performance. Third, every headline finding is reported with an effect size, a 95% confidence interval, and correction for multiple comparisons (Benjamini-Hochberg) where relevant.

The caveats are equally real. As the authors state in their own Limitations section, the fault profile inherits the reporting bias of open-source GitHub issues as of September 2025, and assumes a non-adversarial threat model (operational faults only, not intentional attacks). Both backbones are drawn from the same model family (gpt-oss) under the same serving setup (an Apple Mac Studio, 16K-token context), so generalizing the scale finding is premature, and the stack-level differences (p=0.048, 0.074) are themselves uncorrected for multiplicity and labeled suggestive. As contrasting evidence, the closest prior benchmark, ReliabilityBench (see Related Work), identifies rate-limiting as the most damaging fault type, whereas this paper's mined profile is dominated by explicit errors (65.6% alone) — the field has not yet converged on which failure type matters most.

Related Work

• Gupta (2026). ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions — the prior benchmark this paper cites directly as its closest comparison (prior). At a smaller scale (two models, two architectures, four domains, 1,280 episodes), it injects timeouts, rate limits, partial responses, and schema drift, and reports that rate-limiting faults are most damaging and that ReAct holds up better than Reflexion (independently confirmed via WebSearch — not a figure this paper reproduces). This paper differs by deriving its fault distribution empirically from 57,240 real issue/PR threads rather than by design, and by finding that agent-stack differences matter more than model scale.
• (2026). Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy — a practitioner-validated taxonomy of MCP faults that shares this paper's "observe real failures, then classify" approach (prior). It ranks tool discovery/registration failures and server platform incompatibility as the most severe categories (independently confirmed via WebSearch), while this paper's six operators (delay, silent drop, explicit error, incorrect output, invalid output, minor mistake) only cover the "post-connection" stage after a tool is already registered — a narrower scope. Combining the two taxonomies would cover the MCP failure lifecycle more completely.
• Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning — a self-correction technique this paper names directly in its own related work (closer to contradiction). Reflexion has agents verbally reflect on failure and carry that reflection into the next attempt, reportedly improving performance on HumanEval and other benchmarks (independently confirmed via WebSearch). But this paper's case studies repeatedly show agents treating every failure as transient, retrying with minor parameter tweaks rather than diagnosing persistence — if an agent can reflect but still cannot tell a permanent fault from a transient one, Reflexion-style self-correction's implicit premise (that tools are ultimately reliable) breaks down.

Every number cited for this paper's own findings (completion rates, score drops) is reproduced directly from its own tables. The specific figures for the three papers above were independently confirmed via WebSearch rather than reproduced by this paper's authors, so they are not drawn from the same benchmark suite and should not be compared head-to-head.

Reviewer's Judgement

First, in this reviewer's judgment, the most important finding is not the "60% drop" headline but the tool-bypass pattern: agents that never properly used a tool still scored well 6% of the time, and excluding those cells widens the real drop to 67%. Current LLM-as-judge scoring cannot distinguish an honest failure ("I couldn't use the tool, so I don't know") from a confident-sounding guess — and the latter is the more dangerous failure mode in production.

Second, it would be an overreach to generalize "scale doesn't buy fault tolerance" into "model scale is irrelevant to reliability." This experiment compares only two sizes within one model family, and the confidence interval straddles zero (p=0.088) — both "120B is worse" and "no real difference" remain consistent with the data. The conclusion this paper actually supports is narrower: don't assume a bigger model will automatically handle faults better, and invest in harness-level error handling instead.

Putting It to Work

• Audit the harness before upgrading the model — a bigger model does not bring fault tolerance with it. The completion-rate gap between a harness with retry/fallback logic and one without (up to roughly 9 points) exceeded the gap between model sizes.
• Prioritize permanent failures first — 91.3% of the mined MCP ecosystem incidents are permanent, and 65.6% of all incidents are explicit errors alone. Rather than raising retry limits, build logic that quickly identifies a permanent error and switches tools.
• Instrument tool bypass separately — response-quality scores alone cannot catch a confident-sounding answer produced without tool grounding. Log tool-call presence and appropriateness as metrics distinct from task completion to catch false positives.
• Verify middleware non-intrusiveness first — before adopting a fault-injection or observability middleware, measure the score gap between "off" and "passthrough" conditions to confirm the test apparatus itself isn't degrading performance.
• Choose a harness by its failure-response profile — plan-and-execute stacks (MCPBench-style) lean toward switching tools, while message-loop stacks (OpenAI SDK, LangChain) lean toward giving up and answering anyway. Pick based on which behavior you actually want when a fault hits in production.

Conclusion

This paper's contribution is twofold: a reproducible tool, MCP-Injector, that measures MCP tool-calling agents' fault tolerance against a real, empirically mined failure distribution rather than arbitrary synthetic faults, and a 1,350-cell rebuttal of the assumption that a bigger model automatically becomes safer. A 60% drop in task-completion score — a meaningful share of it traceable to agents bluffing their way through without using tools — is a failure mode worth checking before any production deployment. The limits are real too: both backbones share one model family, and the stack-level differences are suggestive rather than statistically settled. The principle of recovering from tool-call failures in small, verifiable increments connects to the operational checklist in Designing Multi-Tool-Call Partial-Failure Recovery.

References

• Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and figures
• Designing Multi-Tool-Call Partial-Failure Recovery — sunny34.com blog