Accuracy Lies: TriCalRAG's On-Premise Benchmark for LLM-Based Root Cause Analysis

Source Document

Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary, "TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps", arXiv:2609.14762 [cs.DC], submitted 2026-09-13. Affiliation: Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur. Code, dataset splits, and evaluation harness released publicly (GitHub: SPriTLab-iitj/TriCalRAG).

This is a preprint without peer review, classed cs.DC (distributed/parallel computing) even though the content is an LLM-based root-cause-analysis (RCA) benchmark with a calibration analysis. Conflicts of interest: the work was funded by IIT Jodhpur's internal Research Initiation Grant (RIG, Grant No. I/I/RIG/SKM/20250216); no commercial funder is disclosed. All three authors share the same institution, and the paper self-cites the authors' own prior work (TriShieldRAG) as the basis for its retrieval design. The paper explicitly discloses generative-AI use: "ChatGPT-5.6 was used only for grammatical revision." The full text was verified against a snapshot collected today (2026-09-16 KST, generated 2026-09-15T22:02:35Z) — this session's WebFetch was blocked even to an unrelated control domain (example.com), so we could not open the arXiv HTML directly and instead used the primary-source copy GitHub Actions collected automatically the same day.

Study Overview

The paper asks two questions. First, can a single workstation-class GPU deliver practical on-premise LLM-based root cause analysis? Second, can F1 alone be trusted to reflect a model's actual judgment, or does it hide other failure modes? The starting motivation is that cloud-API-based RCA carries three costs: the risk of leaking sensitive infrastructure details (hostnames, credentials) embedded in logs, per-query cost that scales linearly with log volume, and network latency that erodes incident-response speed.

The data draws on four real LogHub log datasets (BGL, HDFS, Thunderbird, OpenStack), each built into 150 incidents balanced 50/50 between normal and anomalous, for 600 incidents total. A single NVIDIA RTX PRO 6000 (96GB VRAM) workstation serves Qwen2.5-14B-Instruct and Mistral-Small-Instruct (22B) via vLLM, evaluated under three prompting strategies — zero-shot, few-shot (two fixed examples), and RAG (retrieving the top-3 most similar past incidents via FAISS, excluding the query itself) — across model x dataset x 3 random seeds, with bootstrap 95% confidence intervals. A classical LSTM-based anomaly detector, DeepLog, is evaluated on the same data as a baseline.

Key Results

Macro-averaged across the four datasets, the two models trade places depending on the metric — Mistral-Small leads on F1, while Qwen2.5-14B leads on throughput and calibration stability.

Predicted-positive rate is the share of "anomaly" labels a model gives on this balanced (50/50) data; the authors flag anything above 0.85 or below 0.15 as "degenerate" calibration failure. Under zero-shot prompting, both models crossed that threshold on 3 of 4 datasets (BGL, HDFS, OpenStack) — Mistral-Small reached 0.993 on BGL, 0.987 on HDFS, and 1.000 on OpenStack, at which point accuracy was 0.500-0.507 (barely above chance) while F1 still looked reasonable at 0.67-0.72 — a textbook case of class-imbalance gaming inflating F1. RAG corrected this: 7 of 8 configurations (2 models x 4 datasets) stayed within the calibrated range, versus only 2 of 8 under zero-shot. By dataset, RAG was strongest on BGL and Thunderbird (F1 0.88-0.94 versus 0.51-0.73 under zero-shot), while HDFS never exceeded F1 0.687 in any configuration because its per-block-ID labels mismatch the paper's fixed 5-line windowing.

Two deployment-relevant ablations follow. These use different bases (batch size, precision), so they are kept separate below.

Scaling the batch from 1 to 128 lifts throughput 41x (48.3 to 1,991.0 tok/s) while elapsed time grows only from 1.39s to 4.14s, showing the 96GB VRAM has ample headroom. The quantization comparison (same incident subset, bfloat16 vs. AWQ 4-bit) found essentially no change in F1 (0.730 to 0.739, which the paper calls "within run-to-run variation") while latency dropped 20% (5.06s to 4.04s) — on this task, 4-bit quantization cut latency with no measurable accuracy cost. DeepLog's initial evaluation was inflated by data leakage (F1 0.913); once corrected with a proper 80/20 split on normal sequences, it landed at F1 0.698 (precision 0.584, recall 0.867), between Qwen (0.560) and Mistral (0.644). That comparison, however, used a much smaller held-out set (120 incidents versus 600 for the LLM benchmark), which weakens the strictness of a direct comparison.

Credibility Assessment

What earns trust: bootstrap 95% confidence intervals and 3-seed repetition applied to every metric, and directionally consistent results across four operationally distinct real-world log datasets. The authors themselves caught and transparently corrected two inflation artifacts — the leaked DeepLog F1 (0.913 down to a corrected 0.698) and a parser bug that misclassified 57.1% of Mistral-Small's early responses as parse failures before a fix dropped it to 0.2% (of 10,800 total responses). Code, dataset splits, and the evaluation harness are open-sourced, enabling external replication.

Caveats are just as clear. This is a pre-peer-review preprint, and the comparison the paper is framed around — on-premise versus cloud API — is never actually run; the cloud baseline is listed only as a "planned addition," alongside a third open-weight model (Llama-3.1-8B, pending gated-repo access) and a 70B-class quantized model. In other words, the paper shows on-premise is feasible, not that it beats or matches the cloud. The quality of the generated natural-language explanations (the root_cause and remediation fields) is never automatically evaluated, leaving half of the paper's core value proposition — that LLMs explain, not just flag — outside the verified scope. Being three days old, it has no independent replication or contradicting literature yet.

Related Work (Academic Cross-Check)

• Mohanty, Patel, Yuvaraj, Chaudhary, Singhania (2026). TriShieldRAG: A Three-Ring Defense-in-Depth Framework against Knowledge Corruption in Retrieval-Augmented Generation — Extension (same authors' prior work). Addresses knowledge-corruption defenses in RAG pipelines; TriCalRAG's retrieval design builds on it, though that paper targets adversarial robustness while this one targets RCA accuracy and calibration.
• Du, Li, Zheng, Srikumar (2017). DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning — Benchmark origin. The classical LSTM-based anomaly detector TriCalRAG directly benchmarks against. After correcting for data leakage, TriCalRAG reports DeepLog landing at F1 0.698, between the two LLMs — a check on any easy claim that LLMs are simply better.
• Ahmed, Ghosh, Bansal, Zimmermann, Zhang, Rajmohan (2023). Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models — Prior work. A large-scale study applying LLM-based root-cause and mitigation recommendations across more than 40,000 production cloud incidents. It's an industrial-scale precedent that contrasts with TriCalRAG's 600-incident, single-GPU scope, marking this paper as an early attempt at the on-premise, small-scale end of the same question.

None of the three contradicts this paper; each situates it. TriShieldRAG shows the retrieval design didn't appear out of nowhere but builds on accumulated RAG research; DeepLog checks any easy claim that LLMs are unconditionally superior to classical methods; and Ahmed et al.'s large-scale cloud study confirms the gap this paper is trying to fill — on-premise, small-scale validation — genuinely exists.

Reviewer's Judgement

First, in this reviewer's reading, the paper's most practically useful contribution isn't F1 but its proposal to standardize "predicted-positive rate" as a calibration diagnostic alongside it. The zero-shot case — 0.500 accuracy (coin-flip) alongside a reasonable-looking F1 of 0.67-0.72 on balanced data — makes concrete just how risky it is to report F1 alone on a task like RCA, which turns on distinguishing anomalous from normal rather than complete from incomplete.

Second, what the title and abstract imply — an on-premise-versus-cloud comparison — is, in fact, missing. Without a cloud API baseline, the on-premise numbers alone provide no basis for concluding on-premise is competitive with cloud, and the authors themselves list this as future work. Readers should treat this benchmark as evidence that on-premise works, not evidence that it wins.

Third, the DeepLog comparison places F1 figures side by side even though the sample size shrinks to a fifth (120 versus 600) — a caveat the authors flag explicitly in their limitations section, but one a practitioner skimming the table alone could easily miss. When reading the three figures together, it's worth citing the sample-size gap in the same breath.

Putting It to Work

• Always report predicted-positive rate alongside F1 — on balanced data, if it falls outside 0.15-0.85, withhold production trust regardless of how good F1 looks.
• Pick the model for cold-start scenarios by zero-shot calibration — for novel incident types with no history to retrieve, choose based on zero-shot predicted-positive rate rather than RAG performance (Qwen2.5-14B was the safer choice here).
• Batch to max out a single workstation GPU — processing incidents in batches instead of one at a time yields a 41x throughput gain.
• Try 4-bit quantization first — on this task it cut latency 20% with no measurable accuracy loss, making it a good first lever when on-premise GPU capacity is tight.
• Validate the output parser per model — a parser that only strips markdown code fences produced a 57% false-failure rate on one model. Extract the brace-delimited substring instead, and check parsing robustness per model before trusting the numbers.

Conclusion

TriCalRAG's central finding is that the bottleneck for single-GPU, on-premise LLM-based RCA is prompt design, not hardware. RAG's bigger contribution wasn't the F1 gain (0.10-0.27) but stabilizing calibration (7 of 8 configurations versus 2 of 8), and deployment levers like batching and quantization are already practical. But a direct cloud-API comparison, a third model, and a 70B-class model all remain "planned," so it's too early to conclude on-premise is the answer.

Teams weighing their own infrastructure investment should read this benchmark's calibration diagnostic alongside a build-versus-buy framework — see Renting the Harness for Free: Rethinking Build-vs-Buy Against OpenAI's Agents API for the decision criteria this connects to.

References

• TriCalRAG — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and figures (via snapshot)
• TriShieldRAG — related-work source
• DeepLog — related-work source (ACM DOI)
• Recommending Root-Cause and Mitigation Steps for Cloud Incidents — related-work source
• Renting the Harness for Free — sunny34.com blog