Only 5 of 1,000 Were Real JSON: A Sub-2B MCP Tool-Calling Reliability Review

Source Document

Abrar Shahriar, Qurat-Ul-Ain Mastoi, "Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment", arXiv:2609.07370 [cs.CL], submitted 2026-09-07. Affiliation: University of the West of England, Bristol. Notebook, prompts and detailed responses released (github.com/Abrar051/sml_mcp_benchmark).

This is a preprint without peer review. Both authors share the same university affiliation and the paper lists no funding source, but since it compares five open-weight checkpoints rather than promoting a single vendor's model, commercial conflict of interest appears low. Session-level egress was blocked during this run, so the full text was not fetched live; tables and figures were checked against a snapshot GitHub Actions collected from the arXiv HTML full text at 2026-09-10T23:15:12Z (9 days prior).

Study Overview

The paper asks two questions: how reliably can open-weight small language models under 2B parameters produce MCP-style JSON tool calls, and what CPU resource cost accompanies that reliability? Five mock tools (weather, web search, calculation, email, to-do) each get 20 prompts, for 100 total. Every prompt states the correct tool and argument values explicitly, so this is closer to a copying-and-serialization floor test than a test of intent recognition.

Five checkpoints — Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, Qwen2.5-1.5B — were loaded in CPU FP32 and run on all 100 prompts under both greedy decoding and temperature-0.7/top-p-0.9 sampling (1,000 generations total, seed 42). The standard scoring path strips Markdown fences, extracts the brace-delimited substring, and parses it as JSON ("recovered" validity), then checks tool name, argument presence, and value match (relative tolerance 10⁻³) in sequence. A separate audit re-parses the complete raw response with no recovery step to measure strict whole-response JSON validity.

Key Results

Recovered-call success ranged from 0% to 79% depending on model and decoding. Confidence intervals are two-sided 95% Wilson intervals.

The last column is the striking part. Even under the best condition (Qwen2.5-1.5B, sampling), only 5 of the 1,000 raw responses (0.5%) parse as JSON outright — everything else needs the fence-stripping/brace-extraction step. The by-tool breakdown (Table 4) shows aggregate rates can hide category failures: Qwen2.5-1.5B greedy scores 100% on calculation, to-do and email, 75% on search, and exactly 0% on weather (rising to 70% under sampling). We verified that the five per-tool rates average to each condition's aggregate success rate exactly.

Decoding strategy cuts in opposite directions by model: Qwen2.5-0.5B drops 72%→32% under sampling (paired McNemar p=1.62×10⁻⁷; 50 successes exist only under greedy, 10 only under sampling), while Qwen2.5-1.5B shows no significant difference (75%→79%, p=.608, discordant counts 15 vs. 19).

A separate three-run CPU resource probe (shared notebook process, one warm-up, three 50-token generations) gave:

Qwen2.5-0.5B trails Qwen2.5-1.5B by only 3 points on greedy success (72% vs. 75%) while running about 65% faster (10.627s vs. 30.782s) and using about 54% less RSS (3,637 vs. 7,960 MiB) — both figures independently recomputed and confirmed. As the authors note, RSS here reflects a shared process rather than isolated model weight size (Pythia, at 1.4B parameters, implausibly shows the lowest RSS of all five). This is a platform-agnostic CPU baseline, not a measurement on any specific board.

Credibility Assessment

Three things earn trust: Wilson intervals and McNemar tests are reported together, per-tool averages reconcile exactly with aggregate rates across the tables we checked, and the LangGraph integration failure (all 10 attempts: "Agent error: 'id'") is honestly separated from model capability as a framework-integration issue.

The caveats are real. Every prompt exposes the correct answer, so ambiguous requests, distractor tools and changing state are untested, and tools were simulated rather than actually invoked. Sampling ran only once per prompt, so run-to-run variance is unknown. Our audit also surfaced an internal inconsistency: Section 4.4 reports "658 failed notebook evaluations," but subtracting Table 2's success counts from 1,000 gives 732 — a 74-count gap with no reconciliation in the text. We did not cite that failure-mode breakdown (89.4% recovery failures, etc.) for this reason.

Related Work (Academic Cross-Check)

Compiled from bibliographic entries in this paper's own reference list. Session egress was blocked, so these three were not independently re-verified.

• Liu et al. (2024). MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — prior work. Showed architecture, not parameter count, drives on-device quality for general language ability; this paper reproduces the same pattern for tool-calling specifically (0.5B Qwen far outperforms ~1.4B Phi, Pythia, TinyLlama).
• Erdogan et al. (2024). TinyAgent: Function Calling at the Edge — extension/contrast. Task-specific fine-tuning, tool retrieval and quantization raised on-device function-calling reliability, suggesting this paper's low scores for untuned checkpoints (0–79%) reflect the absence of training rather than a hard SLM ceiling.
• Fan et al. (2025). MCPToolBench++ — benchmark-origin. Defines what a complete MCP benchmark requires (tool discovery, multi-step execution, live results); this review's subject covers only the narrow slice of that scope (single-shot JSON generation with the answer given), which bounds its external validity.

All three point the same direction as this paper's own conclusion: untuned small models are not inherently incapable so much as under-trained for the task.

Reviewer's Judgement

First, the number practitioners should watch is not 79% but "5 of 1,000." Without the recovery parser, even the best condition yields machine-consumable JSON only 0.5% of the time — the recovery layer is a core system component, not an optional convenience.

Second, sampling cut Qwen2.5-0.5B's success by 40 points but made no significant difference for the 1.5B model. The assumption that temperature 0.7 is a broadly safe default can invert entirely between checkpoints, so each model/decoding pair needs its own regression test before deployment.

Third, Qwen2.5-1.5B greedy scoring exactly 0% on weather alone shows how an aggregate rate can conceal a deterministic failure category that a single summary number would miss.

Fourth, the 74-count gap between Section 4.4's failure total and what Table 2 implies stands out against how well the paper's other tables reconcile. Until that specific breakdown is recomputed, we would not cite it.

Putting It to Work

• Treat the recovery layer as a system component — make fence-stripping, brace extraction and retry explicit in code, and track pre- and post-recovery parse rates as separate deployment metrics.
• Validate each model/decoding pair separately — don't apply temperature 0.7 as a universal default; regression-test greedy and sampling per candidate checkpoint.
• Break out success by tool — dashboard per-tool and per-argument success rates, not just the aggregate, to catch deterministic failure categories early.
• Weigh fine-tuning and constrained decoding — treat untuned-checkpoint performance as a floor, and evaluate TinyAgent-style fine-tuning or schema-constrained decoding separately.
• Never execute without a gate — since no tool was actually called in this benchmark, route small-model outputs through schema validation, permission checks and confirmation for consequential actions before execution.

Conclusion

The contribution here is not "sub-2B models are good enough" but a controlled quantification of how much reliability swings — 0% to 79% — with checkpoint, decoding and recovery layer. Even the best condition has too high a failure rate for unsupervised autonomous execution, and strict recovery-free JSON validity stays under 1%. Given the preprint status and one unreconciled table, treat these numbers as directional and re-verify on your own pipeline. For the operational side of local MCP reliability design, see When You Can't Swap the Model, Change the Settings.

References

• Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling — arXiv abstract (source)
• Full HTML text of the same paper — used via snapshot to verify tables and figures
• MobileLLM (Liu et al., 2024) — related work
• TinyAgent (Erdogan et al., 2024) — related work
• MCPToolBench++ (Fan et al., 2025) — related work
• When You Can't Swap the Model, Change the Settings — sunny34.com blog