Source Document
Bartolomeo Bogliolo, "From SQL Generation to Tool Selection: A Domain-Oriented Pattern for MCP Servers", arXiv:2608.22063 [cs.AI], submitted 2026-08-22, DOI 10.48550/arXiv.2608.22063. No institutional affiliation is listed; published solo. Harness code and benchmark artifacts released on GitHub (github.com/meob/mcp-blueprint, github.com/meob/mcp-blueprint-benchmark).
This is a preprint without peer review. The author is also the developer of MCP Blueprint, the reference implementation behind one of the three server designs compared in the benchmark — so the tool the author built scored highest on a benchmark the same author designed. No funding source is disclosed. Counting the other way: the harness code, task definitions, scoring rules, frozen packs, and per-cell raw results (one JSON per check) are all public, leaving a path to external replication. Live access to the source was blocked for this session, so we checked the tables against the arXiv HTML full-text snapshot the site's automated pipeline fetched at 2026-08-25T21:48:41Z.
Study Overview
The research question is singular: when LLM agents reach an enterprise database through MCP (Model Context Protocol), how do accuracy, cost, and the model tier required change if the model picks from a small set of predefined domain tools instead of generating SQL on every query? The author formalizes this as the "Domain-Oriented Tooling Pattern" and names the observation that replacing SQL synthesis with intent classification lets smaller models handle the same requests "Model Demotion."
Validation runs on a reproducibility benchmark built with the reference implementation, MCP Blueprint. The Sakila sample database (customers, rental history) is exposed through three MCP server designs — (A) raw SQL, where the DDL of six tables sits in the system prompt alongside a single SQL-execution tool; (B) a verticalized pack of five domain tools (customer account summary, rental history, film recommendation, stock lookup, customer search) that encapsulate joins and business rules in server-side SQL; and (C) a generic pack of four table-oriented tools carrying only minimal descriptions. Four locally hosted models spanning 3B–8B (llama3.2:3b, qwen2.5:3b, qwen2.5:7b, llama3.1:8b) ran seventeen customer-facing tasks three times each at temperature 0, seed 42 — a planned 4×3×17×3 = 612 cells, of which 609 completed (99.5%); three cells (0.5%) were lost to an MCP server startup pipe hang. The run was recorded on 2026-08-20 against commit b386d58. Scoring uses rule-based checks against gold answers computed live from the database, not an LLM judge.
Key Results
Pooled mean accuracy was 0.939 for the verticalized pack (B), 0.666 for raw SQL (A), and 0.605 for the generic pack (C). B's fully-correct rate reached 85% (174/204) versus 33% (67/201) for A and 31% (63/204) for C, while C carried the highest zero-score rate at 9% (19 cells) — exposing tools does not automatically help; a shallow design can land below raw SQL.
| Model | A (Raw SQL) | B (Verticalized) | C (Generic) |
|---|---|---|---|
| llama3.2:3b | 0.583 (6/51) | 0.929 (42/51) | 0.419 (0/51) |
| qwen2.5:3b | 0.684 (17/48) | 0.902 (42/51) | 0.602 (14/51) |
| qwen2.5:7b | 0.647 (20/51) | 0.958 (45/51) | 0.631 (19/51) |
| llama3.1:8b | 0.750 (24/51) | 0.966 (45/51) | 0.769 (30/51) |
B led every model (never dropping below 0.90), and the smallest model gained the most: llama3.2:3b went from 6/51 fully-correct (0.583) under raw SQL to 42/51 (0.929) with domain tools — the empirical evidence the author cites for Model Demotion.
Cost figures need their basis kept straight. Mean tokens per cell (total-token basis) were close across designs — B 3,056, C 2,894, A 3,953 — but the gap widens once cost is measured per correct answer.
| Design | Tokens per correct answer | Seconds per correct answer |
|---|---|---|
| B (Verticalized) | 3,582 | 5.2s |
| C (Generic) | 9,372 | 20.7s |
| A (Raw SQL) | 11,858 | 51.9s |
Per-model reductions in cost per correct answer (verticalized versus raw SQL) were roughly 11.6x for llama3.2:3b, 3.7x for qwen2.5:3b, 2.0x for qwen2.5:7b, and 2.3x for llama3.1:8b. The paper's central economic claim: the cheapest fully-correct cell in the study comes from a 3B model with domain tools, not an 8B model writing SQL.
Credibility Assessment
Three things earn trust. Scoring is rule-based against live database state rather than an LLM judge, leaving little room for evaluator bias. The direction — B beating both A and C — holds across all four models. And the harness, tasks, scoring rules, frozen packs, and per-cell raw results are all public, so external replication is possible.
The caveats are equally well documented by the paper itself. First, the verticalized pack (B) and the scoring rules were developed against the same seventeen tasks, with no held-out task set evaluated — part of the accuracy gap may reflect task-aware tool design rather than open-ended generalization. Second, Sakila is a small schema whose six-table DDL fits entirely in context, which if anything favors raw SQL (A); the author expects the gap to widen on enterprise schemas whose DDL cannot fit in context, but this is untested. Third, the study covers only local 3B–8B models at a fixed decoding setting (temperature 0), so whether the raw-SQL gap narrows on frontier cloud models is unmeasured. Fourth, there is a conflict of interest in that the author is the developer of the reference implementation behind the benchmark's top-scoring design. As a contrasting view, text-to-SQL research (Spider, BIRD, DIN-SQL, and similar work) pushes to improve SQL generation quality itself, whereas this paper removes the generation step from the serving path entirely — the paper frames the two as complementary rather than competing.
Reviewer's Judgement
First, the most operationally significant result here is not the 0.939 peak but the fact that the generic pack (C) landed below raw SQL (A) at 0.605. The assumption that "wrapping it in an MCP tool made it safer" does not hold in this benchmark, and treating tool existence and tool design quality as the same thing can make things worse, not better.
Second, reading Model Demotion purely as a cost-cutting strategy undersells the result. An up-to-11.6x reduction in cost per correct answer is not just "you can use a cheaper model" — it means that once domain rules are encapsulated in server-side SQL, the recurring cost of the model re-understanding the schema and reconstructing joins on every request disappears. For organizations where a human has to re-explain the schema every time someone looks at the data, that reuse effect may be a bigger lever than downgrading the model tier.
Third, the task-aligned scoring limitation deserves to be taken as seriously as the author states it. Because the tools and the scoring criteria were co-designed within the same seventeen tasks, 0.939 should not be misread as a ceiling that transfers unchanged to novel, unseen requests. Applying this in practice requires separately measuring how well the tool coverage matches your domain's actual recurring questions before assuming this gap will reproduce.
Putting It to Work
- Encapsulate recurring questions first — inventory what users actually ask repeatedly (customer lookup, rental history, stock checks), then build domain tools with the joins and business rules for those questions baked into server-side SQL.
- Treat tool design as an API contract — human-readable parameter names, descriptions crafted to steer routing. C's failure shows that "a tool exists" and "the tool is well designed" are not the same thing.
- Verify tool existence and tool design separately — before shipping a new MCP server, check with your own data whether a shallow generic pack actually beats raw SQL. A bad tool can be worse than no tool.
- Leave an escape hatch for uncovered requests — route exceptions to a human-in-the-loop path where a domain engineer authors and commits a new YAML/SQL definition, rather than falling back to raw SQL by default.
- Re-evaluate model tier by cost per correct answer — compare tokens and latency per correct answer, not per cell, and measure on your own traffic whether domain tools let you drop to a smaller model.
Conclusion
This paper's real contribution is not a new SQL generation technique but a quantified demonstration that "exposing a tool" and "designing a tool well" produce very different outcomes on an MCP server. A well-designed domain tool pack delivered 0.939 accuracy and up to 11.6x lower cost per correct answer, while a shallow generic pack landed below raw SQL at 0.605. The limitations remain real, though: tools and scoring were co-designed within the same seventeen tasks, only a small schema and local small models were tested, and the author is the developer of the compared reference implementation. For the schema-level detail of MCP tool design, see When Search Becomes a Function Call: Designing Vectorize as an MCP Tool.
References
- From SQL Generation to Tool Selection: A Domain-Oriented Pattern for MCP Servers — arXiv abstract (source)
- Full HTML text of the same paper — used to verify tables and figures (via snapshot)
- When Search Becomes a Function Call: Designing, Failing, and Operating Vectorize as an MCP Tool — sunny34.com blog