Why Individual Rules Hold While the Combination Collapses
EnterpriseRAG, a benchmark from Jiutian Research at China Mobile, measured compound instruction adherence across 13 LLMs on noisy retrieval results (447 cases). Loose IAS — the share of individual constraints satisfied, covering format, persona, and citation grounding — reaches 83.8% for the top model, Qwen3-235B-Thinking. Strict IAS, whether every constraint is met simultaneously in a single response, drops to 26.8%: a 57-point gap. Scaling within the same family from 8B to 235B barely moved factual fidelity (64.8% to 67.1%) while Strict IAS more than doubled (12.3% to 26.8%) — evidence that compound instruction adherence is a separate capability from factual accuracy.
The Gap Widens on Knowledge Gaps and Factual Conflicts
When retrieved documents lack an answer and the model should say so, even the best reasoning model, Claude-Opus-4.5, hit only 42.7% refusal accuracy — below half — while Qwen3-30B-Instruct managed just 6.6%. Detecting contradictory documents ranged from 44.3% (DeepSeek-R1, best) to 18.5% (GPT-4.1, worst). The correlation pattern is the more striking finding: reasoning models showed a positive correlation (+0.90) between conflict detection and answer coverage, while standard models showed a negative one (-0.50) — answering more often meant missing more conflicts.
From Design to Operations: A Compound-Instruction Gate Checklist for Production RAG
Planning should promote Strict IAS, not Loose IAS, to the deployment gate metric. A workable floor: Strict IAS at 60%+, knowledge-gap refusal accuracy at 70%+, and conflict-detection accuracy at 50%+ before shipping. Gating on individual-constraint pass rates alone misses the risk of a response that's correctly formatted but wrong on substance.
Four failure patterns recur. Rule-based format checks run without ever verifying compound adherence; low refusal accuracy goes unaddressed while the model fabricates answers in knowledge-gap situations; the negative correlation between conflict detection and coverage in standard models — answer more, miss more — goes unmonitored; and teams assume scaling model size alone lifts compound adherence.
Recovery branching starts by auto-rejecting responses that fail Strict IAS, routing them to regeneration or a human-review queue. Models that score low on refusal accuracy get a cross-check pass from a second model on knowledge-gap scenarios, and detected conflicts surface the contradicting documents side by side instead of a confident single answer.
An operational checklist needs a regression set that synthetically augments noise, knowledge-gap, and factual-conflict scenarios — the source paper found gaps and conflicts occur naturally in only 5.3% and 8.7% of production logs respectively, so an unaugmented eval set under-samples them. Scoring should combine rule-based format checks with an LLM-judge for behavioral protocol, periodically validated against a human-labeled sample.
The improvement loop tracks how often gaps and conflicts actually appear in post-deployment logs and rebalances the regression set accordingly. Every model swap should feed its Strict IAS delta into the release gate, and because reasoning and standard models show opposite correlation signs, each model family needs its own threshold rather than one shared bar.
The Takeaway
Individual instructions hold near 84% on noisy retrieval while full compound compliance collapses to around 27% — a signal to move production RAG's deployment gate from Loose to Strict metrics. Setting a floor of 60% Strict IAS, 70% refusal accuracy, and 50% conflict detection, backed by an augmented gap/conflict regression set and dual rule-plus-judge scoring, catches the compound failures that individual pass rates hide.
References
84% Alone, 27% Together: Reviewing the Enterprise RAG Instruction Gap — sunny34.com Research Review