Structure Didn't Help Accuracy, Evidence Compression Cut Tokens 38% — A Review of RAGFlow's Agentic Retrieval Study

The paper

Zhichang Yu, Baining Cao, Sbaaoui Idriss, Qi Wang, Gangqiang Xu, Ling Qin, Xiaoling Jin, An Liu, Shaohao Cai, Xiaoyang Bao, Shengle Chen, Changchang Ding, Pengfei Zhu, Jingxiang Xu, Yifeng Mao, Lu Liu, Hai Jin, Yingfeng Zhang, "Context as Infrastructure: Knowledge Compilation for Retrieval in Agentic Systems with Harnesses", arXiv:2610.41172 [cs.IR], submitted 2026-10-30. Affiliations: InfiniFlow Inc. (Shanghai), East China Normal University (ECNU). License CC BY-NC-SA 4.0. The system described is released as the open-source RAGFlow project (github.com/infiniflow/ragflow).

This is an unreviewed preprint. Sixteen of the eighteen authors, including the corresponding authors, work at InfiniFlow, the company that builds RAGFlow, and the default serving mode the paper measures (E2: evidence substitution) is the same default that company ships in production — a clear conflict of interest in which a vendor benchmarks the design choice it has already made. No external funding is disclosed.

Study design

The paper asks two questions. If a search harness — an LLM that writes its own queries, reads what it finds, and reformulates — has already solved most of the accuracy problem, does pre-compiling documents into structured artifacts (summary trees, knowledge graphs, ontologies, wikis) still add value, and if so, on which axis? The experiments fix a harness with no vector index, only BM25 and grep, with MiniMax M3 as the answering model, measured on FRAMES (multi-hop QA requiring facts from several Wikipedia articles) and BrowseComp-Plus (single-question, token-heavy search). Dense retrieval is excluded from every condition: the authors report that adding a dense leg lowered accuracy in their runs, because the harness's own query reformulation already does the semantic work. Two families are compared: structural artifacts (summary trees, entity-relation graphs, LLM wikis) and the one mechanism the authors say they can actually measure — claim-evidence pairs, where a verbatim evidence span is stored alongside a normalized claim.

Key results

First, on structure itself. Under a competent harness, RAPTOR-style summary trees, PageIndex-style outline trees, entity-relation graphs, and LLM wikis all failed to produce a detectable accuracy gain on FRAMES. The PageIndex-style outline actually turned negative at scale.

Second, the token economics. Serving the same claim-evidence tree via "evidence substitution" (E2) — swapping a located chunk for its claim-evidence pairs before the content enters the window — cuts per-query tokens by 91% when the evidence suffices, and raises them by 1.9% when it doesn't, for a 38% average reduction (N=200 questions). The asymmetry implies a 2.0% break-even sufficiency rate, and the measured 38% average implies an actual sufficiency rate near 43% (paper's Eq. 5-6; re-verified by arithmetic: 0.019/0.929=2.0%, (0.38+0.019)/0.929=43%).

A caution: the −52%/140→30s figures above (decision-gate arm mean) and the −38% figure (conditional split average) are on different bases and shouldn't be read side by side. The "FRAMES average" row is a weighted average over the hit/miss split; the "decision gate" row is the mean of a separate arm that includes the gate. Separately, on a 7-question paired subset where substitution ran without the gate, query tokens roughly halved but end-to-end latency roughly tripled — substitution trades a single raw-text read for more retrieval rounds. In short: the token saving is bought with latency, and the decision gate is a separate mechanism that buys the latency back.

Third, the gate's reliability. The gate's 43% "acceptance rate" is not the same thing as how often the accepted evidence was actually sufficient (accuracy). The authors give an estimator to separate the two (their Eq. 10-11) but state explicitly that the labeled audit (100-400 samples) needed to calibrate it is designed but not yet run — the "no accuracy loss" conclusion should be read as provisional until that audit lands.

Credibility assessment

This is a pre-review preprint with a clear conflict of interest, as noted above: most authors work at the company that builds the exact system under evaluation. The authors themselves flag real limitations in §8.8 — measurements come from a single run with no confidence intervals and no repeated seeds, and they explicitly call the 91%/1.9% figures "measured operating points," not population estimates. They also note FRAMES is English, Wikipedia-based text; enterprise corpora with tables, scans, and jargon may see lower extraction quality and smaller savings. As counter-evidence, the original RAPTOR paper reported roughly a 20-point absolute accuracy gain pairing its tree-based retrieval with GPT-4 on the QuALITY benchmark — this paper reports the same style of tree as "neutral" under a different benchmark (FRAMES) and a stronger, iterative-search harness. The two results are not a direct contradiction so much as a conditional one: structure appears to help more when the harness is weaker, and to help less once the harness is already strong.

Related research

Session egress was fully blocked, so related literature was independently identified and confirmed via WebSearch rather than by re-reading the source directly. Any numbers from these papers are used only to describe the relationship, not as part of this review's "key results."

• Sarthi et al. (2024, ICLR). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval — prior work and the structural baseline this paper tests. RAPTOR reported roughly a 20-point absolute accuracy gain pairing its tree with GPT-4 on QuALITY. This paper reports the same style of tree as accuracy-neutral on FRAMES, suggesting the benefit of structure depends on benchmark and harness strength.
• Sen et al. (2026). Is Grep All You Need? How Agent Harnesses Reshape Agentic Search — prior work. On a LongMemEval-based comparison, inline grep scored 93.1%, but with file-based delivery vector search won in five of ten pairs; the authors explicitly do not claim lexical search is universally superior. This qualifies this paper's "BM25/grep alone reaches 91%" claim as directionally consistent but not universal.
• Li et al. (2026, NeurIPS). Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction — prior work directly cited in §1 and §2.1 as the premise for excluding dense retrieval. It reports that grep-and-file-read access, with no embeddings or index, is competitive with or beats sparse, dense, and reranking baselines on BRIGHT, BEIR, BrowseComp-Plus, and multi-hop QA.

Together, these three papers show a consistent trend — as harnesses get stronger, the marginal value of retrieval structure shrinks — but RAPTOR's own original result, under a weaker harness, shows structure can still move accuracy. This paper's conclusion should be read as conditional on a strong harness, not a general law.

Reviewer's take

First, I'd argue the paper's most useful practical contribution isn't overturning the belief that "structure improves accuracy" — it's insisting that structure's value be judged on two separate ledgers, accuracy and cost/governance, with different metrics for each. Table 2's "two ledgers, one instrument" design is a safeguard against the common internal mistake of concluding a RAG investment "failed" just because accuracy didn't move.

Second, while the 91% token cut is the headline, I think the latency trade-off is the more operationally important number. On the same seven-question comparison, substitution alone roughly tripled latency, and the decision gate is what brings it back down from 140s to 30s. A team that adopts evidence substitution without also shipping the gate could cut token cost while making the product feel slower.

Third, given this is a vendor's own evaluation, I think the "no accuracy loss" claim should be treated as a hypothesis, not a settled fact, until the 100-400-sample audit the authors themselves specify but have not yet run is completed. The authors are transparent that acceptance rate and true sufficiency can diverge — but that transparency also means nobody yet knows how large that gap is.

Practical takeaways

• Check harness strength before investing in structure — if a BM25/grep harness with an iterative search loop is already close to your accuracy target, don't pre-build summary trees or graphs on spec.
• Separate token savings from latency — evidence substitution (E2) alone can slow responses. If latency matters to your product, ship the decision gate (E3) first and treat token cost as a secondary optimization.
• Re-derive the break-even point for your own corpus — the finite-horizon break-even (the paper's Eq. 9) can run well above 2% depending on your query volume and sufficiency rate; estimate your own sufficiency rate before committing to a compilation pipeline.
• Audit gate acceptance separately from correctness — "the gate accepted it" and "the evidence was actually sufficient" are different claims. Periodically sample and label at least 100 accepted queries to estimate the false-accept rate.
• Score ontologies, graphs, and wikis on operational metrics, not accuracy — vocabulary control, onboarding time, and audit traceability are the metrics a benchmark can't see.

Conclusion

The core finding, read conditionally, is this: once a search harness is already strong, pre-compiled structure stops helping it find more and starts only helping it read less. Compiling evidence alone cut average tokens by 38% with no measurable accuracy loss, but that saving was bought with added latency and has to be bought back with a separate decision gate. Given that most authors work at the company whose product is under evaluation, and given the single-run, audit-not-yet-run limitations the paper itself discloses, the numbers are best treated as directional signals to verify on your own corpus rather than settled facts. The operational playbook for moving RAG into a compiled knowledge layer is covered next in RAG를 컴파일형 지식 계층으로 옮기는 운영 기준 (Korean-language post).

References

• Context as Infrastructure: Knowledge Compilation for Retrieval in Agentic Systems with Harnesses — arXiv abstract
• Same paper, full HTML text — used for table cross-checks
• RAPTOR (prior work) — arXiv
• Is Grep All You Need? (prior work) — arXiv
• Beyond Semantic Similarity (prior work) — arXiv
• RAG를 컴파일형 지식 계층으로 옮기는 운영 기준 — sunny34.com blog (Korean)