22.6 Points More Accurate, 2.7x the Cost: Reviewing a Multi-Agent BI Platform

Source Document

Manoj N M, Vijayakrishna S, Manjunath Srinivas, Rohit Pahan, "A Multi-Agent Platform for Automated Enterprise Analytics and Insight Generation", arXiv:2608.18740 [cs.AI], submitted 2026-08-19, DOI 10.48550/arXiv.2608.18740. Affiliation: Rakuten India Enterprise Private Limited (Bengaluru, India).

This is a preprint without peer review, and an internal study — all four authors work for Rakuten India. One of the two evaluation datasets is production competitive-pricing intelligence data spanning two e-commerce platforms (with Japanese-language content) — the same company evaluating its own system on its own data, not an independent test. Funding is not separately disclosed. The evaluation design does layer automated functional testing, LLM-as-a-Judge, and human expert cross-checks, and adds a four-backend evaluation, partially offsetting the limits of a vendor self-evaluation. The full text was checked against a snapshot (a copy of the arXiv HTML full text) taken 2026-08-22T21:41:08Z — live source access was unavailable in this session, so the site's automated collection pipeline's copy was used.

Study Overview

Two questions drive the paper: does splitting conversational BI — turning a natural-language query into SQL, analysis and insight — across five specialized agents actually beat a single agent, and what does that improvement cost? The CrewAI-based sequential pipeline has five roles: Data Retrieval (schema-aware SQL generation), Data Analysis (pattern/anomaly detection plus MCP-driven chart generation), Report Aggregation (multilingual report synthesis), Follow-up Questions (suggested next questions), and Chart Configuration (visualization metadata).

Evaluation ran on BigQuery against a synthetic e-commerce dataset (4 tables, 12,500 records) and a production pricing-intelligence dataset (roughly 569,000 overlapping products, 1.17 million price-history records, 8.8 billion cumulative observations). 300 test cases spanned eight categories, and 100 of the 260 analytical cases were independently scored by three human domain experts. Three comparison axes: (1) versus a single-agent baseline, (2) per-agent removal ablation, (3) cross-model evaluation across GPT-4.1, Claude Sonnet 4, Gemini 2.5 Flash, and Llama 3.1 70B.

Key Results

Functional accuracy favored the multi-agent pipeline across every category, with the gap widening as complexity rose.

Quality, hallucination rate, latency and cost use different bases and should be read side by side, not conflated. Quality is a 1–5 LLM-as-a-Judge score (evaluator: Claude Sonnet 4.5); hallucination-free rate cross-checks factual claims against raw query results; tokens and cost are per-query averages.

The token multiple (~3.0x) and the paper's stated cost multiple (2.7x) diverge because GPT-4.1 prices input and output tokens differently ($2.00/M input, $8.00/M output) — the two should not be quoted interchangeably. Across the four backends, GPT-4.1 led on both accuracy and quality (95.3%, 4.52); open-source, self-hosted Llama 3.1 70B reached 79.7% inside the multi-agent pipeline, above the single-agent GPT-4.1 baseline (72.7%). Gemini 2.5 Flash trailed on accuracy (86.3%) but was fastest (17.4s) and cheapest (~$0.01 per query).

The five agents did not contribute equally. Removing Data Analysis dropped quality from 4.52 to 3.38 (−1.14), the largest single hit; removing Report Aggregation cost 0.66 points (3.86). Removing Follow-up Questions and Chart Configuration together cost only 0.02 points (4.50) — running them in parallel instead cut latency from 23.8s to 20.1s (−15.5%) while quality (4.52) and accuracy (95.3%) held. Security guardrails blocked all 40 adversarial inputs (static rules under 0.01s, LLM-based checks 6–7.5s), and three human experts scoring 100 responses correlated with the LLM judge at r=0.89 (p<0.001), κ=0.82, with the judge running 0.14 points generous on average (+0.24 on actionability).

Credibility Assessment

Three things earn trust: the core comparison holds the model (GPT-4.1) constant across both architectures, so architecture and model effects do not mix; the four-backend cross-model evaluation retests this control and holds in the same direction on an open-source model; and the correlation with 100 human-scored responses (r=0.89, κ=0.82) supports the LLM-as-a-Judge results.

The caveats are just as clear. Every author works for Rakuten India, and the production evaluation data is the company's own — closer to a vendor scoring its own system on its own data. The LLM-as-a-Judge methodology paper the authors cite (Zheng et al. 2023) itself flags judge-model bias, and this paper's own +0.14-point generosity runs in that same direction. Human evaluation covers only 100 of 260 analytical cases (38%), and the security-guardrail test set is 40 items — both small samples. The production dataset is confined to a single domain, e-commerce pricing intelligence, and generalization to other verticals is a limitation the authors themselves note. Against that, the paper's own comparison to SiriusBI (93–96% SQL accuracy, Jiang et al. 2025) puts this system's 93.5% SQL accuracy within the range of prior published results, rather than an outlier claim standing alone. No run-to-run variance or significance testing is reported, so it is hard to say how far the tabulated gaps exceed ordinary run noise.

Reviewer's Judgement

First, the most practically valuable result here is not the headline 95.3% but the cross-model table's Llama 3.1 70B figure (79.7%). An open-source, self-hosted model beating a top-tier single agent (GPT-4.1, 72.7%) once placed inside the multi-agent pipeline reads as evidence that adoption priority should go to decomposing the architecture first, not to upgrading the model.

Second, the asymmetry in the ablation bears directly on how to design an adoption. Data Analysis and Report Aggregation drive nearly all of the quality gain, while Follow-up Questions and Chart Configuration contribute essentially nothing to quality and serve engagement/UX instead. Rather than running all five sequentially, assigning the two high-value agents to a capable model and the rest to a lighter model or parallel execution looks like a sensible starting point.

Third, the divergence between the token multiple (~3.0x) and the cost multiple (2.7x) is itself an operational signal. That the gap comes from input/output pricing means pipelines with more output-heavy stages (report writing, translation) can see actual bills diverge sharply depending on the pricing plan chosen. The paper does not model the cumulative cost at millions of queries; that arithmetic needs to be run in-house before adoption.

Putting It to Work

• Decompose before you upgrade — if accuracy is short, try splitting roles before reaching for a bigger model. The open-source 70B-in-multi-agent-beats-single-agent-GPT-4.1 result is the evidence.
• Use ablation to prune low-value agents — run your own per-agent removal test before deployment, and move low-quality-contribution roles (here, follow-up and chart config) to parallel execution or a lighter model.
• Track token and cost multiples separately — identify why they diverge (input/output pricing) and project actual billing at your own query volume before committing.
• Two-tier security guardrails — pair sub-millisecond static rules with slower LLM-based judgment to get both instant blocking and nuanced coverage.
• Audit LLM-judge bias against human samples — periodically re-check correlation, kappa, and per-dimension bias direction with a small human sample so the judge does not drift generous over time.

Conclusion

This paper shows, through a controlled same-model comparison and a four-backend cross-check, that splitting conversational BI across five specialized agents lifts both accuracy (+22.6 points) and quality (+20.2%) over a single agent. The price is roughly 3x the tokens and 2.7x the cost, and the ablation shows most of that gain comes from just two of the five agents (Data Analysis, Report Aggregation). The limits of a vendor evaluating its own system on its own data should be kept in view, but the fact that the direction reproduces on an open-source model, plus the correlation with human scoring, is enough reason to test "how far does architectural decomposition pay off" on your own data rather than taking the numbers at face value. The risk of deciding deployment on accuracy alone is covered further in Beyond Accuracy: Designing Performance Checks.

References

• A Multi-Agent Platform for Automated Enterprise Analytics and Insight Generation — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables, tokens and cost figures (via snapshot)
• Beyond Accuracy: Designing Performance Checks — sunny34.com blog