Completion 75.8%→53.5%: The MCP Paradox in a Baseline Assessment of the Agentic Web Stack

Source Document

Vagner Figueredo de Santana, Diogo Cortiz, "A Baseline Assessment for the Agentic Web Technology Stack", arXiv:2610.13703v1 [cs.AI], submitted 2026-10-09, CC BY 4.0. Affiliation: NIC.br (Brazilian Network Information Center); one co-author is also at PUC-SP. Code, experiment configuration, and logs released in full (github.com/cewebbr/mastif).

This is an arXiv preprint without peer review; the venue field is blank. NIC.br is Brazil's non-profit internet-governance body, and it also funded this work — there is no outside sponsor, but every author is employed by the funding institution, which we flag as a conflict of interest. Releasing code, configuration, and logs in full is a point in favor of reproducibility.

Study Overview

The question is simple: for an LLM agent, what actually decides outcomes — the model, the protocol, or the framework? The authors built an evaluation tool, MASTIF, and ran 3 models (Mistral-7B-Instruct, Qwen2.5-7B, Llama-4-Scout-17B-16E) × 3 communication modes (standard calls, MCP, A2A) × 4 frameworks (LangChain, LangGraph, CrewAI, Semantic Kernel) against a stratified sample of 300 Mind2Web tasks (out of 2,350). That is 3×3×4×300 = 10,800 task executions, matching the paper's own equation. Judging used GPT-4o-mini, a different model family from every system under test, to limit self-preference bias.

Key Results

Task completion rate (TCR) differed significantly across all three factors (Kruskal-Wallis: model H=1203.3, protocol H=428.8, framework H=6039.3, all p<0.001). Averaging Table 2's 36 cells by model (our own calculation, not a number the paper states directly) shows the MCP condition scoring lowest for every model.

The sharpest drop is Qwen2.5-7B: standard+LangChain scores 79.5% (σ0.157, the single highest cell in Table 2), but MCP+LangChain falls to 53.7% (σ0.278). In the opposite direction, tool-use success (Table 4 — only Qwen2.5-7B externalized tool calls at all) peaks at 68.1% under MCP+Semantic Kernel, the highest of any combination. MCP shows a two-sided effect: it drives more tool use while completing fewer tasks. The paper's own "best stack per metric" table (Table 6) names Qwen2.5-7B, standard, Semantic Kernel as best for completion by mean rank — a different question from, and not the same as, the single highest raw cell (Qwen2.5-7B, standard, LangChain at 79.5%). The two should not be read as the same claim.

Credibility Assessment

In favor: 300 tasks per condition, 10,800 executions in total, with Shapiro-Wilk, Kruskal-Wallis, and Dunn's post-hoc all reported; full code and logs released; and a judge model from a separate family. Caveats: as the authors state (§7.4), the protocol comparison only simulates "how messages are exchanged" rather than running real MCP/A2A servers, so schema-injection and handshake costs of an actual deployment may not be captured. Only one model externalized tool calls at all, so the tool-use results confound model effects with protocol/framework effects. The funding conflict of interest noted above stands. We found no peer-reviewed counter-evidence, but the related literature below points in the same direction — adding a protocol layer is not free.

Related Work

• Ehtesham et al. (2025). A Survey of Agent Interoperability Protocols — prior work. Argued that no single protocol suffices across contexts; this paper supplies the numbers behind that claim.
• Kapoor et al. (2024). AI Agents That Matter — prior methodological work. Called for cost-controlled agent evaluation; this paper follows through by reporting tokens and convergence time as separate metrics.
• Guo et al. (2025). A Measurement Study of Model Context Protocol Ecosystem — extension at ecosystem scale. Found that more than half of 8,060 measured MCP servers were low-quality or abandoned, which may offer an ecosystem-level explanation for this paper's MCP completion drop (the two studies measure different objects).

Reviewer's Judgement

First, the most practically important finding is that MCP helps tool use while hurting completion. The authors note this in §6.3 but understate it — separating "picked the right tool" from "finished the task" is exactly where this paper earns its keep.

Second, Table 6's "best stack per metric" invites misreading on its own. The factor-level, mean-rank winner (Semantic Kernel) differs from the single-cell winner (LangChain); a team that picks a framework from Table 6 alone ends up with the runner-up, not the top performer.

Putting It to Work

• Treat protocols as a choice, not a default — don't add MCP or A2A because they are "the standard"; measure completion and tool-use rate on your own tasks first.
• Evaluate model × protocol together — the size of the protocol effect varied by model (Qwen -22.4 pts, Llama -8.7 pts, Mistral -3.5 pts). Re-test when you swap models.
• Check what a "best combo" table is ranking — factor-level mean rank and single-combination raw value are different claims; report both the same way in your own benchmarks.
• Re-verify protocol overhead against real servers — this experiment only simulated message formatting, so schema-injection and handshake costs of an actual deployment need separate measurement.

Conclusion

The contribution here is not a new technology but a stance: treat the agentic web stack as something to test, not something to assume. Across 10,800 executions, model, protocol, and framework all mattered statistically, and MCP in particular raised tool use while lowering completion for every model tested. Given the message-only protocol simulation and the funder-author overlap, the numbers are best read as a direction to verify on your own tasks before adopting. For the operational side of MCP reliability, see Reliability Design for Locally Deployed MCP Tool-Calling Agents.

References

• A Baseline Assessment for the Agentic Web Technology Stack — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and statistics
• Ehtesham et al. (2025). A Survey of Agent Interoperability Protocols
• Kapoor et al. (2024). AI Agents That Matter
• Reliability Design for Locally Deployed MCP Tool-Calling Agents — sunny34.com blog