Formulation Works, Code Breaks: How an MCP Documentation Server Nearly Quadrupled a Scheduling Agent's Success Rate

Source Document

Ángel Sánchez-Fernández, Javier Pernas-Álvarez, Diego Crespo-Pereira, "Agentic AI-Assisted Modeling for Production Scheduling: Assessment in Constraint Programming", arXiv:2610.10184 [cs.AI, cs.SE], submitted 2026-10-07, DOI 10.48550/arXiv.2610.10184. Affiliation: CITENI, Universidade da Coruña (Ferrol Industrial Campus). Prompts, benchmark data and generated scripts released on GitHub (angelsnfn/AgenticProductionScheduling).

This is a preprint without peer review, from a single institution. The three evaluated LLMs (Gemini 2.5 Pro, GPT-5, DeepSeek V4 Pro) are all third-party commercial models, so there is no own-model bias. The authors' disclosure states no conflict of interest and no specific funding for the project. Because this session's egress was blocked for the full retry window (3 attempts at 10-minute intervals, confirmed against the example.com control, both returning 403), the source text was cross-checked against a full-text HTML snapshot GitHub Actions collected at 2026-10-11T22:12:17Z.

Study Overview

The question: can general-purpose LLMs, with no task-specific training and orchestrated only as agents, formulate and implement constraint-programming (CP) scheduling models end to end — from a natural-language description to a running script — for industrial production scheduling? Prior LLM-for-optimization work skews toward mixed-integer linear programming (MILP), and the few CP-focused efforts (e.g., ConstraintLLM) mostly rely on dedicated fine-tuning.

Three workflows are compared under controlled conditions: (a) a multi-agent pipeline that separates formulation from coding, (b) a single agent that does both, and (c) a direct LLM call with no tool access. Workflows (a) and (b) connect to an MCP server that exposes docplex.cp (the CP Optimizer Python API) documentation through symbol-level search — rather than loading the entire 77k-token corpus, a two-stage "discovery → precision" tool pair retrieves only the roughly 20 of 333 symbols a given formulation needs. The design spans three LLMs (Gemini 2.5 Pro, GPT-5, DeepSeek V4 Pro) × six industry-style scheduling problems of increasing difficulty (PC1–PC6: flow shop, job shop, flexible job shop, three resource-constrained AGV/picking warehouse variants) × three replicates = 162 runs. Each generated script is executed once, unmodified, and judged against the reference optimum at two levels: formulation+coding accuracy (Acc_F+C) and, after correcting only implementation errors, formulation accuracy (Acc_F).

Key Results

The two metrics diverge sharply. Formulation accuracy (Acc_F) barely depends on the workflow, while formulation+coding accuracy (Acc_F+C) depends heavily on the agent architecture — the bottleneck is not the model itself but turning it into working code.

The two resource-constrained warehouse problems (PC5, PC6; 2,000+ constraints) behave differently: formulation accuracy itself falls to 44.4% (PC5) and 14.8% (PC6), barely moving across workflows — a formulation-level ceiling, not an implementation problem.

Of the 162 generated scripts, 58 reach the reference objective as generated, and 81 fail with an execution error. After correcting implementation errors only, 121 reach the reference value. Of 159 distinct implementation errors, only 43 (about 27%) are hallucinations in the strict sense (calls to symbols that don't exist); the larger share (78) misuse an existing API element — most often importing a variable constructor from the wrong module.

Credibility Assessment

Grounds for trust: a controlled design holding the LLM and prompt fixed while swapping the workflow, 162 runs across six problems of varying difficulty, reference models independently checked against every constraint (not just solver output), and full release of prompts, data and generated code. Caveats: a single-institution preprint with sizeable per-cell variance in tokens and time (e.g., GPT-5 input tokens on PC1: 414.2k ± 173.7k), three LLMs accessed as commercial black boxes via OpenRouter whose results can shift with provider-side updates, and a "$0.3–1 per model" cost estimate qualified as based on "current list prices." While cross-checking, a sentence near the middle of the results section states the multi-agent workflow "reaches up to 67.7%" on PC1–PC4, but the abstract, the conclusion, and an independent recomputation from Table 8's per-cell values all agree on 80.6%. The 67.7% figure appears to be a wording error in the source text; this review cites only the doubly-confirmed 80.6%.

Related Academic Literature

• Szeider et al. (2025). CP-Agent (arXiv:2508.07468) — prior work. A ReAct-based coding agent reportedly solves all 101 problems of the clarified CP-Bench. This is the figure the reviewed paper itself cites; the gap between that clean academic benchmark's perfect score and 59.3% on six industry-style problems here is exactly the "benchmark performance is not deployment readiness" point this paper is built around.
• Shi et al. (2025). ConstraintLLM (arXiv:2510.05774, EMNLP 2025) — prior work, contrasting approach. Dedicated fine-tuning plus Tree-of-Thought and constraint-aware retrieval roughly doubles accuracy on its own IndusCP benchmark (140 tasks). The reviewed paper takes the opposite route — no fine-tuning, only agentic retrieval — making the specialization-vs-generality trade-off directly visible.
• AhmadiTeshnizi et al. (2024). OptiMUS-0.3 (arXiv:2407.19633) — prior work, different paradigm. A modular agent that retrieves similar constraint examples via RAG to build MILP models, reporting roughly 22%/24% accuracy on easy/hard NLP4LP problems. Different optimization paradigm (MILP vs. CP), but the same design principle — reducing hallucination through retrieval — underlies the reviewed paper's MCP server.

Reviewer's Assessment

First, the most useful conclusion here is not the 59.3% headline but the finding that formulation and implementation are distinct capabilities. Formulation accuracy sitting near 77–78% regardless of workflow means today's general-purpose LLMs already reason reasonably well about the model itself; the investment that pays off is not a better model but better scaffolding for the implementation step.

Second, the collapse of formulation accuracy itself on PC5/PC6 is a more important operational signal than 80.6%. Once constraints exceed roughly 2,000 and optional-interval, integer-assignment logic pile up, the formulation itself — not just the code — becomes unreliable, which argues for expert review starting at the formulation stage rather than after generation.

Third, surfacing the 67.7%/80.6% inconsistency found during cross-checking is not a knock on the authors — it is a reminder that even a careful, fully open paper can carry a wrong sentence next to a correct table, and a reader encountering only a secondhand summary has no way to catch it.

Practical Takeaways

• Two-stage retrieval MCP server — instead of loading full API docs, split "discovery" (concept → candidate symbols) from "precision" (complete signatures) to cut both token cost and hallucination.
• Separate formulation and implementation agents — splitting the two roles costs more than a single agent (66% more tokens) but buys a larger accuracy gain (26 points).
• Set an automation boundary by constraint complexity — trust automated generation under a few hundred variables/constraints; require expert review once constraints exceed ~2,000 with mixed integer-assignment logic.
• Add a minimal auto-repair loop — most failures stop loudly with an explicit error rather than silently producing a wrong schedule, so a single log-driven repair pass recovers a large share of them.
• Pick models on token efficiency, not just accuracy — token consumption can differ by up to 7x between models in the same workflow; benchmark before switching providers.

Conclusion

This paper's contribution is not a blanket claim that agents model better, but a clean split of that ability into formulation and implementation. General-purpose LLMs already formulate reasonably well; the bottleneck is calling the solver API correctly, and an on-demand documentation server plus role separation substantially narrows that bottleneck. The gain is clear only up to a few hundred constraints — it barely reaches the warehouse-scale problems where formulation itself is hard. Given the single-institution, pre-peer-review status and the inconsistency found within the text itself, the right reading is not "automate scheduling modeling now" but a data point for where to draw the line between automation and expert review. For designing MCP tool trust more broadly, see our guide to reliability design for locally deployed MCP tool-calling agents.

References

• Agentic AI-Assisted Modeling for Production Scheduling: Assessment in Constraint Programming — arXiv abstract (source)
• Full HTML text of the same paper — used for table and figure cross-checks (checked against a 2026-10-11T22:12:17Z snapshot)
• Reliability design for locally deployed MCP tool-calling agents — sunny34.com blog