The floor your edge harness actually has to clear
A recent arXiv preprint ran five sub-2B open-weight checkpoints (Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, Qwen2.5-1.5B) through 100 MCP-style JSON tool-call prompts each, spanning five mock tools (weather, search, calculator, email, todo). Every prompt spells out the correct tool name and arguments up front, so this is less a test of intent understanding than a floor test of "copy the instruction into valid serialized JSON." Run on CPU FP32 under greedy decoding and temperature-0.7 sampling, recovery-parsed success rates ranged from 0% to 79% depending on the model/decoding pair.
Without a recovery layer, only 0.5% is machine-readable as-is
The more striking number isn't the 79% ceiling. Under a strict-JSON standard — parsing the raw response with no markdown-fence stripping or brace extraction — only 5 of 1,000 generations across the whole experiment, 0.5%, parsed cleanly. Everything else needed a recovery pass to be salvageable. For anyone building a harness, that means the recovery parser isn't optional post-processing; it's a core pipeline component. Qwen2.5-1.5B greedy hit 100% on calculator, todo, and email yet exactly 0% on weather alone (recovering to 70% under sampling) — a deterministic failure that an aggregate success rate alone would hide.
From design to ops: a recovery-and-sizing gate checklist for edge harnesses
Declare target metrics before writing any code. Track recovery-parsed success rate as the canonical metric per (model, decoding) pair, and log the confidence interval alongside it the way the paper does for Qwen2.5-1.5B sampling (79%, 95% CI 70.0–85.8%). Any pair whose strict-JSON rate sits in the low single digits should be excluded from autonomous execution unless a recovery layer sits in front of it.
The most common failure mode is that a single decoding choice can flip your success rate. Qwen2.5-0.5B dropped from 72% greedy to 32% sampling — a 40-point swing (McNemar's test, p=1.62×10⁻⁷) — while Qwen2.5-1.5B moved from 75% to 79% with no significant difference (p=.608). Applying the "temperature 0.7 is generally safe" assumption uniformly across checkpoints can produce opposite outcomes, so regression tests need to run per model-decoding pair, not per model family. In the same experiment, all 10 LangGraph integration attempts failed with "Agent error: 'id'" — a framework-integration defect unrelated to model output quality, which is why regression suites should separate model-quality tests from framework-integration tests.
Build the operational checklist in three layers. Codify the recovery parser (fence stripping, brace extraction, retry) as an explicit pipeline stage and log pre- and post-recovery success rates as separate deployment metrics. Break out success rates by tool and argument type on a dashboard so deterministic failures like the weather-tool zero surface early. And because this benchmark never actually invoked any tool — it only simulated calls — treat every small-model-generated call as untrusted until it clears schema validation and a permission check; never execute on receipt without that gate.
Sizing decisions should weigh latency and memory against success rate together. In the same paper's CPU measurements, Qwen2.5-0.5B trades 3 points of success rate versus 1.5B (72% vs. 75%, greedy) for roughly 65% lower average latency (10.6s vs. 30.8s) and about 54% lower memory (RSS 3,637MiB vs. 7,960MiB). That trade-off is reasonable for latency-sensitive, low-stakes routing — but only holds under a fixed greedy-decoding assumption, which your regression suite needs to state explicitly.
Run the improvement loop per checkpoint-decoding pair, not per model family, since decoding sensitivity flipped in opposite directions across otherwise similar checkpoints. Keep in mind this is a pre-peer-review preprint, and the authors themselves flag an internal inconsistency: Section 4.4 reports 658 failures, but subtracting Table 2's successes from 1,000 yields 732 — a 74-case gap. Given that, cite the headline figures (0.5%, 79%, the 40-point swing) as directional signals to re-verify in your own pipeline, rather than importing the paper's detailed failure-mode breakdown wholesale.
Checklist you can use today
Before handing MCP tool calling to a small edge-deployed model, lock the recovery parser in as a mandatory pipeline stage, treat per-(model, decoding) success rate and its confidence interval as your deployment metric, and watch per-tool success separately so deterministic failures surface early. Document that the 0.5B latency/memory trade-off only holds under fixed greedy decoding — state that assumption explicitly in the regression suite.
Reference links
Tools — Model Context Protocol official specification
Ask AI about this article
The assistant has read this article. Ask anything — it answers from the text and says so when something isn't in it.
Loading the chat…