Configuration Beats the Model: Reviewing an MCP Tool-Calling Reliability Benchmark

Source Document

Leonardo Liparulo, Francesco Pierri, "Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling", arXiv:2608.26199 [cs.AI], submitted 2026-08-25, DOI 10.48550/arXiv.2608.26199. Affiliation: Politecnico di Milano. The acknowledgments state the work was carried out during one author's internship at Huawei, under the guidance of a Huawei-employed supervisor.

A session-level egress failure blocked direct access to the arXiv source, so the full text was cross-checked against a primary-source snapshot GitHub Actions collected at 2026-08-31T22:04:49Z (UTC). This is a preprint without peer review. The evaluation target is not the production API of a real industrial hardware-design tool (in the vein of Kactus2) but an MCP (Model Context Protocol) server replica of its data model and dependency rules — a limitation the authors state themselves in the paper's closing section. The conflict of interest is understated but real: the work itself is a product of a Huawei internship and the supervising figure is at the same company, while on the other side all seven evaluated models are open source with no favored commercial model, and industrial experts independently validated the task design and expected call sequences.

Study Overview

The paper asks two questions: can locally run open-source LLM agents reliably perform dependency-ordered, repetitive hardware-design operations (create a component → add ports → wire a subcomponent) through MCP tool calling alone, and is reliability determined by model choice or by agent configuration (prompt, tool description, context scope, architecture)? Local deployment is the industrial reality here — component specifications and naming conventions can reveal unreleased-product details, making hosted commercial APIs impractical.

The authors built a 14-tool MCP server themselves and designed six core task suites — Easy, Medium, Hard, History, Errors, Cross (40 tasks each) — plus two multi-server suites, Easy-Noise and Hard-Noise (60 tasks each), validated by industrial experts. Evaluation uses four complementary metrics rather than a single accuracy number: Expected Call Coverage (ECC, share of the annotated work completed), Extraneous Valid Call Ratio (EVCR), Tool Failure Rate (TFR), and No-Call Accuracy (NCA). The authors' rationale for call-level scoring is that stateful tool-calling agents can hide partial failure behind a plausible final text output.

Key Results

Comparing seven models (Llama 3.1 8B, Gemma 4 E4B/26B/31B, Qwen 3.5/3.6 27B, GPT-OSS 20B — all 4-bit quantized, run locally through Ollama) across the six core suites gives each model's best-observed configuration below. Config notation is history scope (R = accumulates across the session, T = resets per task) / system prompt (MD = Markdown, none = tool schemas only, fs = few-shot) / tool description (C = comprehensive, M = minimal).

These are each model's best-configuration numbers, not out-of-the-box performance. The same Gemma 4 E4B drops from 0.811 ECC to as low as 0.168 under a poor configuration. The gap by task type is just as large: Llama 3.1 8B averages 0.753 ECC on independent tasks (Easy, Medium, Hard) but only 0.139 on tasks requiring state carried across turns (History, Cross) — an 82% drop. In the prompt and tool-description experiments, comprehensive tool descriptions won the best ECC in 35 of 42 model–suite combinations (83%), and switching to minimal descriptions roughly doubled tool failure rate for every model. Few-shot prompting, by contrast, moved most models little (within 0.04 ECC) but caused a sharp inaction collapse for Gemma 4 31B (0.956 → 0.571) and Gemma 4 E4B (0.731 → 0.179) — a failure mode where the model stops acting rather than acting incorrectly.

The architecture comparison (ReAct single-agent vs. Plan-and-Act multi-agent decomposition, with Gemma 4 26B as planner) split by worker strength.

Decomposition helped the weak worker (Llama 3.1 8B) substantially, but putting the strong worker (Gemma 4 26B) through the same six suites reduced coverage — a strong model does better retaining full session context. Gemma 4 26B did benefit from decomposition, however, on the longer Hard-Noise suite (60 tasks, sharing the tool context with other MCP servers), where coverage rose from 0.858 to 0.950. The authors read this as multi-agent decomposition paying off only under one of two conditions: a weak worker, or a long session.

Credibility Assessment

Three things support the numbers. Repeated runs across temperatures 0/0.5/1.0 and ten repeated runs of key configurations confirm ranking and value stability (Gemma 4 26B's ECC stays within 0.881–0.896 regardless of temperature), and the rare few-shot collapses were confirmed by repetition rather than being sampling noise. Industrial experts validated task design and expected call sequences. And the pattern — model sets the achievable range, configuration determines whether it's reached — recurs consistently across a reasonably broad grid of seven models and eight task suites.

The caveats are equally clear. This is a preprint without peer review, and the acknowledgments disclose that the research itself was produced during a Huawei internship under a Huawei-employed supervisor. The benchmark server and task data encode a partner's proprietary information and cannot be released, so external replication is not possible — unlike comparison work that ships its code and data. By the authors' own account, the context-management experiments cover only a binary accumulate-or-reset choice and do not test intermediate options such as summarizing memory. EVCR does not weight the severity of extraneous calls, so a trivial unnecessary parameter counts the same as a serious wiring error. Response latency was not measured, so the speed advantage of smaller models is not reflected in the results.

Reviewer's Judgement

First, the most operationally valuable result here is not the 0.990 top ECC but the observation that configuration can reverse a model's standing. Gemma 4 E4B reaching 0.811 at its best configuration and only 0.168 at a poor one is a warning against the common sequence of picking a model off a leaderboard and only loosely tuning the prompt afterward. Model rankings should not be trusted without at least a minimal search across prompt, tool description, and history scope.

Second, the asymmetric risk of few-shot prompting is an easy trap to generalize past. Applying the same few-shot template to every model on the assumption that "examples usually help" turns failure into silent inaction for the models prone to collapse — which is harder to debug than an ordinary wrong answer. Every model swap should get its own few-shot A/B check before deployment.

Third, the fact that comprehensive tool descriptions won 83% of best configurations is a useful anchor when weighing token cost against reliability. Trimming 14 tool descriptions to the minimum saves only about 2,000 tokens, and in exchange tool failure rate roughly doubles. Cutting tool descriptions first to save cost does not match the payoff this benchmark shows.

Putting It to Work

• Search configuration before ranking models — run at least a minimal sweep of prompt, tool description, and history scope before trusting a leaderboard ranking for a new model.
• Write detailed tool descriptions — descriptions that state parameter semantics, constraints, and failure conditions are the most consistent lever for lowering failure rate.
• A/B few-shot prompting per model — check each individual model for inaction collapse before shipping a few-shot template.
• Reserve multi-agent decomposition for weak workers or long sessions — applying it indiscriminately to a strong model in a short session can reduce coverage.
• Benchmark stateful tasks separately — independent-task scores do not predict a model's ability to recover state across turns.

Conclusion

The paper's conclusion is not that a newer model does better, but that the same model's performance swings sharply with configuration. Even the top model, Gemma 4 31B, shows signs it would look different under a poor configuration, and the weak model, Llama 3.1 8B, recovered much of its state-dependent-task performance through multi-agent decomposition and history-scope tuning alone. Given the unreviewed-preprint status, the internship-linked conflict of interest, and the non-public benchmark's replicability limits, the numbers are best read as a directional signal. Bringing MCP into production also means checking the protocol's own churn, not just tool-calling reliability — that continues in MCP Roadmap: A Readiness Checklist for Server-Initiated Events.

References

• Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and metric values (via snapshot)
• MCP Roadmap: A Readiness Checklist for Server-Initiated Events — sunny34.com blog