Top Overall, 5th on Safety: Reviewing EGI-Bench's 14-Model Enterprise Agent Evaluation

Source Document

Zhiwei Liu, Zhujun Lan, Jielin Qiu, Zixiang Chen, Liangwei Yang, Ming Zhu, Akshara Prabhakar, Wenting Zhao, John Emmons, Jason Wu, Shafiq Rayhan Joty, Junnan Li, Shelby Heinecke, Silvio Savarese, Huan Wang, "EGI-Bench: An Enterprise General Intelligence Benchmark Suite", arXiv:2610.27234 [cs.AI], submitted 2026-10-19, Salesforce AI Research. Code and benchmark CLI released (github.com/SalesforceAIResearch/egi-bench), license CC BY-NC-SA 4.0. The full text was checked against a primary-source snapshot captured 2026-10-21T22:08:47Z (used in place of WebFetch due to a session-wide egress block).

This is a technical report without peer review. Every author works for Salesforce AI Research, but the models under evaluation are not Salesforce's own — all fourteen are third-party GPT, Claude, and Gemini models. That said, several of the eight contributing benchmarks (CRMArena, HERB, Benchforce, among others) were previously published by the same organization, so the group that designed and weighted the overall framework overlaps with the group that built the benchmarks filling it.

Study Overview

The question is simple: when deploying an LLM agent into enterprise workflows, how do you read a single model's report card at a glance? Until now, CRM handling, document retrieval, private-data QA, multi-turn planning, tool calling, long-horizon research, and long-context coding each had to be measured separately across eight benchmarks (CRMArena, HERB, PersonaBench, UserBench, MCP-Universe, MCPEval, Benchforce, LoCoBench) with their own repositories, runners, and scoring conventions.

The authors consolidate these behind a single CLI and re-route sub-metrics — rather than averaging benchmark-level scores — into six dimensions: M1 Task Outcome, M2 Tool Actioning, M3 Planning, M4 Retrieval & Grounding, M5 Safety & Abstention, and M6 Engineering Quality. Weights (28/22/16/14/14/6) are set by expert judgment, and two runtime-budgeted modes exist: quick (180-minute budget) and deep (about 12 hours). This report compares fourteen GPT, Claude, and Gemini-family models under quick mode.

Key Results

Overall EGI (0-100) is led by gpt-5.5 at 65.61, with Claude Sonnet 4.5 close behind at 65.25. The ranking shifts once you look at individual axes.

gpt-5.5 leads on task outcome, planning, and retrieval, but its safety/abstention score (M5, 63.47) sits about 17 points below Claude Sonnet 4.5's 80.28 — only 5th out of fourteen. Claude Haiku 4.5, sixth overall, posts the single best engineering-quality score (M6, 61.28) in the entire roster — a reversal the overall ranking alone never reveals.

M3 (Planning) is the uniformly weak axis, clustering between 15.68 and 30.89 across all fourteen models. The authors trace this to the scoring method of its contributing benchmark, UserBench, which averages per-turn reward rather than scoring only the final answer — so a model that wanders through small talk before eventually landing the right itinerary scores lower than one that asks a single clarifying question and resolves the task in three turns. M2 (Tool Actioning) is the most stable axis by contrast, with most frontier models clearing 70. Recomputing the weighted sum with the stated weights (0.28/0.22/0.16/0.14/0.14/0.06) reproduces both headline scores: gpt-5.5's 82.95×0.28+79.56×0.22+30.89×0.16+54.86×0.14+63.47×0.14+56.23×0.06=65.61, and Claude Sonnet 4.5's equivalent sum likewise reproduces 65.25.

Credibility Assessment

This is a pre-peer-review technical report with a clear conflict of interest. Every author works for Salesforce AI Research, and several of the eight contributing benchmarks (CRMArena, HERB, Benchforce, among others) were previously published by the same organization. The group that designed and weighted the overall framework overlapping with the group that built much of what fills it leaves room for its own benchmarks to be over-represented in defining what counts as "enterprise general intelligence." That the evaluated models are third-party (GPT, Claude, Gemini) rather than Salesforce's own does reduce the direct incentive to skew results.

The authors' own stated limitations are explicit. Quick mode subsets several benchmarks to fit its time budget, shrinking sample sizes and widening confidence intervals. The M1-M6 weights (28/22/16/14/14/6) reflect expert judgment, not empirical calibration. Part of MCP-Universe's scoring uses an LLM judge, so absolute scores can shift if the judge model or version changes. And EGI remains a single scalar, compressing genuinely distinct capabilities into one number.

Related Research

• Huang et al. (2025). CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks — one of the eight benchmarks EGI-Bench directly draws M1/M2 sub-metrics from (benchmark-origin). Its NAACL 2025 numbers show state-of-the-art agents succeeding below 40% with ReAct prompting and below 55% even with function-calling — the floor that explains why EGI's M1/M2 come in low.
• Patil et al. (2025). The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of LLMs — a prior benchmark with a finer-grained taxonomy for single, parallel, and multi-turn function calling scored on AST and execution criteria (prior). EGI-Bench's own authors acknowledge that "BFCL's taxonomy on the tool-calling axis is stronger than what EGI-Bench's MCP contributors currently surface."
• Barres et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment — evaluates a dual-control setting where both agent and user can act on the environment, reporting a significant performance drop moving from single- to dual-control (extension). EGI-Bench has no multi-trial reliability (pass@k) signal of this kind, which the authors flag as "concrete future work."

All three fill gaps EGI-Bench itself admits to — CRMArena is literal raw material for the EGI score, while BFCL and τ²-Bench each expose a depth of taxonomy or a reliability signal EGI hasn't absorbed yet. Low M2/M3 numbers in this report aren't evidence of underestimation; they reflect a mix of a genuinely hard axis (M3) and an axis (M2's MCP component) that isn't yet as finely decomposed as it could be.

Reviewer's Take

First, I judge that treating overall EGI rank as an adoption criterion can cost you on safety. gpt-5.5 and Claude Sonnet 4.5 differ by only 0.36 EGI points overall, yet their M5 (Safety/Abstention) scores are nearly 17 points apart. For workloads where refusal and jailbreak resistance matter — support, internal-data access — picking by M5 rank rather than overall rank is the more defensible choice.

Second, I judge that Claude Haiku 4.5's M6 (Engineering Quality) lead is the most operationally useful signal in this report. A model ranked sixth overall topping every other model on code quality is a concrete reason to trial a cost-efficient model first for code-generation support roles. That said, the authors' own caveat should be taken at face value: because M6 scoring runs on an LLM-judge rubric, it can reward plausibility over strict correctness.

Practical Applications

• Never pick by the single scalar — select models by the M1-M6 axis your workload actually needs, not by overall EGI rank.
• Lead with M5 for safety-sensitive work — for support or internal-data-access workloads where refusal and jailbreak resistance matter, treat M5 as the primary filter ahead of overall rank.
• Read the planning axis as a benchmark artifact — don't read a low M3 score as a pure model deficiency before separating out the effect of per-turn reward averaging.
• Treat quick-mode numbers as directional only — small subsets widen confidence intervals, so re-verify with deep mode or your own tasks before a deployment decision.
• Check for organizational overlap in benchmark sourcing — when the framework designer and the contributing-benchmark authors are the same organization, separately verify that the weighting doesn't over-represent a particular capability.

Conclusion

EGI-Bench's contribution isn't a new benchmark but a consolidation: eight benchmarks re-routed into one CLI and six axes. That consolidation reveals a reversal that overall rank alone hides, in both safety and engineering quality. But the conflict of interest between framework designer and contributing-benchmark author, plus quick mode's small sample sizes, are reasons to read these numbers as a starting point for axis-by-axis comparison rather than a final leaderboard. The same completion-versus-safety tension shows up again in our piece on surface-by-surface promotion gates for self-evolving agent harnesses.

References

• EGI-Bench: An Enterprise General Intelligence Benchmark Suite — arXiv abstract (source)
• Full HTML text of the same paper — used for table and figure verification
• CRMArena — arXiv:2411.02305
• τ²-Bench — arXiv:2506.07982
• Surface-by-surface promotion gates for self-evolving agent harnesses — sunny34.com blog