Source Document
Huiqi Miao, Xinbao Sun, Bo Wang, Fanyu Meng, Lijun Mei, Na Wu, Di Jin, Chao Deng, Junlan Feng, "EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval", arXiv:2608.11584 [cs.AI], submitted 2026-08-12. Affiliation: Jiutian Research, China Mobile, Beijing, China. License CC BY 4.0. The benchmark and evaluation framework are to be released alongside publication.
This is a preprint without peer review. Every author works for Jiutian Research, China Mobile's internal research arm, and the 491 source queries behind the benchmark come from the company's own production RAG logs. That grounds the failure modes in a real deployment, but it also means the raw logs are not public, so outsiders cannot independently verify how the sample was drawn. One of the six domains, "Party Building," covers internal policy interpretation specific to Chinese state-owned enterprises — unlike the other five (Energy, Medical, Legal, Financial, Web Search), it has no direct analogue in most other markets.
Study Overview
Two questions drive the paper. First: when retrieved context is noisy, incomplete, or contradictory, how well do LLMs follow complex, multi-part instructions (formatting, persona, citation, and so on)? Second: does satisfying individual constraints move in the same direction as satisfying an entire instruction set at once? The authors argue that prior RAG benchmarks evaluate retrieval quality and instruction adherence separately, and introduce EnterpriseRAG to test both together.
The data comes from 491 authentic queries across six domains (Energy, Medical, Legal, Financial, Party Building, Web Search), each paired with documents from hybrid retrieval (BM25 + dense) and one of three non-ideal conditions: noise (topically similar but irrelevant documents), knowledge gaps (insufficient evidence), and factual conflicts (contradictory documents) — yielding 983 instances. Gaps and conflicts are rare in natural logs (occurring naturally in only 5.3% and 8.7% of cases respectively), so the authors state they augmented them through a controlled procedure. Evaluation runs on two axes: Loose IAS, the share of individual constraints satisfied, and Strict IAS, a binary score for whether all constraints were satisfied at once. Knowledge gaps are scored separately with rejection accuracy, and factual conflicts with conflict-recognition accuracy. Scoring combines rule-based checks (for formatting) with an LLM judge (for behavioral protocols), validated against 150 human-annotated samples (κ = 0.77, 88% agreement).
Key Results
The table below covers 13 models on the noisy-retrieval subset (447 cases). Loose IAS (per-constraint satisfaction) and Strict IAS (all constraints satisfied at once) use different bases and should be read side by side rather than substituted for each other.
| Model | Paradigm | Faithfulness | Loose IAS | Strict IAS |
|---|---|---|---|---|
| Qwen3-8b | Reasoning | 64.8% | 75.5% | 12.3% |
| Qwen3-235B-Thinking | Reasoning | 67.1% | 83.8% | 26.8% |
| DeepSeek-R1 | Reasoning | 68.9% | 83.1% | 21.9% |
| Gemini-2.5-Pro | Reasoning | 73.5% | 83.7% | 26.5% |
| GPT-4.1 | Standard | 69.8% | 80.0% | 19.5% |
| Claude-Opus-4.5 | Reasoning | 76.4% | 83.3% | 25.3% |
| Claude-Sonnet-4 | Standard | 76.8% | 79.8% | 19.5% |
Even the top model, Qwen3-235B-Thinking, drops 57 points from an 83.8% Loose IAS to a 26.8% Strict IAS: it satisfies nearly every individual rule, yet meets the full instruction set at once in only about one case in four. The scale effect within a single family is stark, too — Qwen3-Thinking's Strict IAS more than doubles from 12.3% at 8B to 26.8% at 235B, while Faithfulness barely moves, from 64.8% to 67.1%. That suggests compositional instruction-following demands far more scale than factual accuracy alone.
Knowledge gaps and factual conflicts expose a different weakness. On the 227 knowledge-gap cases, where the correct move is admitting the answer isn't available, Qwen3-30B-Instruct's rejection rate was just 6.6% (meaning it fabricated an answer 93.4% of the time), and even the best reasoning model, Claude-Opus-4.5, reached only 42.7%. On the 309 factual-conflict cases, conflict-recognition accuracy topped out at 44.3% (DeepSeek-R1), with GPT-4.1 at just 18.5%. The three non-ideal subsets — 447 noise cases (45.5%), 227 knowledge-gap cases (23.1%), 309 factual-conflict cases (31.4%) — sum to the full 983 instances.
Credibility Assessment
The methodology validation is reasonably thorough. Cross-checking three LLM judges (Kimi-k2-thinking, Qwen3-235B-Thinking, GPT-4o) yielded ICC reliability of 0.809 for Loose IAS and above 0.99 for rejection/conflict accuracy, and agreement with 150 human-annotated samples reached κ = 0.77 (88% agreement). The conflict of interest is nonetheless clear: every author is employed by China Mobile, and the data originates from the company's own operational logs. Citing privacy, the authors do not release the raw logs — only a desensitized sample and the generation pipeline — which limits outside researchers' ability to independently verify sampling bias. It also matters that 91–95% of the knowledge-gap and factual-conflict cases are augmented rather than naturally occurring; the appendix reports that synthetic and natural cases show equivalent difficulty on core metrics, but that check was performed by the same author group. No contradicting prior work was found — if anything, the paper's own framing is that prior benchmarks (CRAG, RAGBench, GaRaGe, and others) each cover only one or two of noise, rejection, and conflict, so few earlier studies surface the individual-versus-holistic gap this paper documents. The source data being Chinese-language operational logs is a further consideration before generalizing to deployments in other languages.
Reviewer's Judgement
First, this reviewer's take is that the paper's core value lies less in any single number than in reporting Loose and Strict IAS side by side. A RAG pipeline QA'd only against a checklist of individual constraints will show a different number than the "satisfy everything at once" bar that production actually demands. Internal evaluation criteria should include at least one Strict-style metric.
Second, Qwen3-30B-Instruct's 6.6% rejection rate is, in this reviewer's judgement, the most alarming figure in the paper: it fabricates an answer more than nine times out of ten when the correct response is "I don't know." That even the best reasoning model tops out at 42.7% shows "add reasoning" is not a simple substitute for safety. Handling knowledge gaps well appears to require a dedicated rejection mechanism and verification step, not model choice alone.
Third, the opposite-signed correlation between conflict recognition and answer coverage — ρ = +0.90 for reasoning models versus ρ = −0.50 for standard models — carries real operational weight. Reasoning models catch more conflicts as they cite more evidence, while standard models risk missing conflicts precisely when their answers grow longer. Applying the same coverage target uniformly across both model families could quietly raise risk in standard-model deployments.
Putting It to Work
- Gate on a Strict-style metric — don't clear a pipeline on Loose IAS (per-constraint satisfaction) alone; make simultaneous satisfaction of the full instruction set the final gate.
- Test rejection behavior separately — maintain a standing test set for knowledge-gap cases and check whether the model actually says "no information available"; standard models degrade the most here.
- Make conflict-handling protocols explicit — put an explicit instruction in the prompt for contradictory sources (e.g., cite both). The paper reports explicit protocols sharply raise conflict recognition but only modestly improve rejection accuracy.
- Split coverage strategy by model family — avoid pushing standard models toward longer, higher-coverage answers, and reserve coverage expansion for reasoning models.
- Check constraint density by domain — domains with heavy citation and conditional-statement requirements (legal, medical) are likely to show a larger Strict IAS drop, so set domain-specific bars.
Conclusion
EnterpriseRAG measures complex instruction adherence under three non-ideal retrieval conditions and finds a 57-point gap: up to 83.8% per-constraint satisfaction against only 26.8% holistic compliance. Rejection accuracy under knowledge gaps (42.7% at best) and conflict-recognition accuracy (44.3% at best) both fall short of production-grade reliability as well. The all-China-Mobile authorship and undisclosed raw data are real limitations, but the cross-judge reliability work is thorough enough that the methodology itself holds up. The broader case for evaluation criteria beyond raw accuracy continues in RAG Agent Evaluation Basics: Metrics That Matter More Than Accuracy.