Past 2,000 Constraints, Even the Formalization Breaks: Setting the Automation Boundary for Agentic Production Scheduling

Constraint Programming Scheduling: Two Skills the Agent Pulled Apart

Researchers at CITENI, University of A Coruña ran 162 executions across Gemini 2.5 Pro, GPT-5, and DeepSeek V4 Pro, and found that the ability to formalize a production-scheduling constraint-programming (CP) model (Acc_F) stayed in the 77-78% range regardless of workflow. The ability to turn that formalization into working code (Acc_F+C), though, swung nearly fourfold by workflow: 14.8% for a direct LLM, 33.3% for a single agent with MCP, and 59.3% for a role-split multi-agent setup with MCP.

That split matters operationally. Upgrading the model barely moves formalization accuracy, but how you design the coding stage decides the overall success rate. The investment that pays off is not a smarter model — it's tooling that helps the agent implement code correctly.

How an MCP Document Server Cuts Hallucination

The paper's multi-agent setup never dumps the entire CP Optimizer Python interface (docplex.cp) documentation — roughly 77,000 tokens across 333 symbols — into the prompt. Instead, a two-stage "search then detail" MCP server pulls in only the roughly 20 symbols actually needed. Of 81 runs that halted on execution errors, 58 reached the correct answer once only the implementation was patched, and among 159 individual errors, just 43 (27%) were hallucinations calling functions that don't exist. The remaining 78 (49%) were misuse of real APIs, most often importing a constructor from the wrong module.

Misuse outnumbering hallucination changes where design effort should go. A search tool that points to the right symbol location strips out more errors than a filter built to catch hallucinations.

The Ontology Angle: What Do Constraints and Resources Connect To?

A scheduling model's variables and constraints are, in the end, a code translation of real plant entities — equipment, jobs, time windows, resources. Without that mapping structured in advance — through an asset-and-process semantic layer such as ISO 15926 or the Asset Administration Shell (IEC 63278) — the agent has to re-derive the relationships for every constraint, and that reasoning load grows with the constraint count. The resource-constrained warehouse problems (PC5, PC6) show the cost directly: once constraints passed 2,000, formalization accuracy itself fell from 44.4% to 14.8%.

The MCP server's error reduction follows the same logic — a search structure that wires in only the needed symbols, instead of the full document, functions as a small semantic layer. When you attach an agent to a large industrial project, how tightly that search layer meshes with the enterprise asset ontology sets the real boundary of what can be automated.

From Rollout to Operations: An Automation-Boundary Checklist for Scheduling Agents

Declare the automation approval bar in code first. For small problems (a few hundred constraints or fewer), set Acc_F+C 60% or higher as the pass line, and budget for the role-split multi-agent's cost — about 730K tokens and 370 seconds, versus 440K tokens and 258 seconds for a single agent (66% and 43% higher). Since the easiest four problems reach 80.6%, keep separate target lines by difficulty tier.

The first failure pattern is stuffing the entire document into the prompt. Handing over 77,000 tokens at once drives up both hallucination and token waste, so switch to the two-stage "search then detail" retrieval that loads only the symbols actually needed.

The second pattern is assigning both formalization and implementation to one agent. A single agent tops out at 33.3%, while a role-split multi-agent reaches 59.3%. The 26-point accuracy gain outweighs the added cost.

Problems mixing 2,000-plus constraints with integer assignment are the third pattern, and the one that deserves the most caution. No workflow change moves formalization accuracy here — it still falls from 44.4% to 14.8% — so the abort condition should hand the formalization step to a human rather than trigger an automatic retry.

Keep error type (hallucination vs. API misuse), failure stage (formalization vs. implementation), constraint count, and model ID as mandatory fields in the execution error log. Since misuse outnumbers hallucination 1.8 to 1 (78 vs. 43), gate a single automatic re-patch attempt — pointed at the correct symbol location — ahead of anything else.

Measure token consumption on the same problem set before swapping models. Token usage can differ up to sevenfold between models on the identical workflow, so choosing on accuracy alone lets cost creep up unnoticed. Run a golden set of varied-difficulty problems like PC1 through PC6 before every deployment.

Tally weekly execution errors by hallucination, misuse, and formalization failure, and adjust the constraint-count threshold and automation pass line against that mix. Log the gap between a clean benchmark score — prior work like CP-Agent reported 101/101 — and this paper's 59.3% on six industrial problems in the same change log, so benchmark numbers alone never justify widening the automation scope.

Checklist to Use Right Away

Attaching an agent to production scheduling is a decision about the implementation stage, not the model. Cut hallucination and misuse with an MCP search-then-detail server, raise accuracy by splitting formalization from implementation, and draw the line in code: anything past 2,000 constraints goes to expert review, not automation.

References

Agentic AI-Assisted Modeling for Production Scheduling: Assessment in Constraint Programming — arXiv

Full HTML text of the same paper

Formalization Works, the Code Breaks: A 4x Execution-Rate Review of a Scheduling Agent Behind an MCP Document Server — sunny34.com Research