Why Configuration Beats the Model When You Can't Swap It
In domains like hardware design automation, component specs and naming conventions can leak unreleased product details, which rules out hosted commercial APIs. Teams are left running 4-bit quantized open-source models locally instead. Researchers at Politecnico di Milano benchmarked seven local models on MCP tool calling and found that the same Gemma 4 E4B model swung from an expected-call coverage (ECC) of 0.811 down to 0.168 purely based on configuration. When the model is fixed, history scope, system prompt, and tool description become the only levers left to pull.
Why Comprehensive Descriptions Won Most Matchups
Across 42 model-task-group combinations, the configuration with the highest ECC used comprehensive tool descriptions — ones that spelled out parameter meaning, constraints, and failure conditions — in 35 cases (83%). Trimming descriptions to a minimum roughly doubled the tool failure rate across every model, while the tokens saved by shortening all 14 tool descriptions came to only about 2,000. The failure-rate spike costs far more than the token budget it frees up.
From Design to Deployment: A Local MCP Agent Configuration Checklist
Before wiring in a new local model, declare the acceptance bar in code first. A reasonable floor is expected-call coverage (ECC) at or above 0.85, tool failure rate (TFR) at or below 5%, and excess-call rate (EVCR) at or below 5%. Keep history- and cross-session tasks that require restoring prior state in a separate golden set from independent tasks — Llama 3.1 8B averaged 0.753 ECC on independent tasks but dropped 82% to 0.139 on tasks requiring state recovery, so a single shared bar buries that session-dependent failure inside the average.
The assumption that few-shot examples generally help does not hold uniformly across models. Most models saw ECC shift by less than 0.04 with or without few-shot prompting, but Gemma 4 31B collapsed from 0.956 to 0.571 and Gemma 4 E4B from 0.731 to 0.179. This isn't wrong answers — it's the model going silent and stopping action altogether, which response logs alone won't explain.
Before adding a new model to the pipeline, A/B-test both a with-few-shot and without-few-shot variant and set the default from that comparison. After deployment, track no-call accuracy (NCA) as its own metric, and write the alert rule so that a rising no-response rate points the on-call engineer to suspect the few-shot prompt first.
Minimizing tool descriptions saves tokens but roughly doubles the failure rate across nearly every model. That failure accumulates quietly — the agent still responds, but drops a parameter or references the wrong component, which monitoring that only checks the final text output will miss entirely.
That's why a state-changing tool-calling agent needs call-level grading, not just final-response grading. Standard log fields — model ID, history scope (session-cumulative vs. per-task reset), system prompt type, tool description detail, and the ECC/EVCR/TFR/NCA quartet — let you compare the impact of any configuration change immediately. In a local deployment, add a pre-deploy checklist item for masking confidential fields, like component specs and naming conventions, out of those logs.
Multi-agent decomposition is not a universal fix. The weak worker, Llama 3.1 8B, improved from 0.554 with a single ReAct agent to 0.718 under Plan-and-Act decomposition, with history tasks jumping from 0.113 to 0.650. But putting the strong worker, Gemma 4 26B, through the same decomposition on the same six core task groups actually lowered coverage — it only helped again on the 60-task Hard-Noise group with a longer session, moving from 0.858 to 0.950. Decomposition earns its cost only under weak-worker or long-session conditions, so gate it on both.
Repeated runs across temperatures 0, 0.5, and 1.0, plus ten reruns, showed the strong models' rankings holding steady — Gemma 4 26B's ECC stayed within 0.881–0.896 regardless of temperature — but that stability doesn't mean one configuration sweep is enough forever. Pull the weekly failure log for combinations with high rates of no-response collapse, lost session state, or excess calls, feed them back into the golden set, and log configuration changes separately from model swaps so next week's metric shift can be traced to the right cause.
Takeaways at a Glance
If confidentiality or local deployment constraints mean you can't freely swap models, put history scope, system prompt, and tool description ahead of model choice as the first thing you explore. Don't skimp on tool descriptions, A/B-test few-shot per model, and only turn on multi-agent decomposition for weak workers or long sessions. With call-level grading and per-axis log fields in place, the same model can land anywhere between an ECC of 0.168 and 0.990 — and configuration decides which.
References
Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling — arXiv