The Most Accurate Agent Was the Least Economical: Reviewing EcoAgent-Bench

Source

Jie Wu, Ming Gong, Feixiang Cheng, Qinqin Zhao, "EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents", arXiv:2608.05519 [cs.AI, cs.CL, cs.LG], submitted 2026-08-06, DOI 10.48550/arXiv.2608.05519. No author affiliations or contact details are printed in the paper.

This is a preprint that has not been peer reviewed. Within the scope this review could verify, the paper carries no author affiliations, no funding statement, no conflict-of-interest disclosure, and no code or data availability statement. For a benchmark that ranks commercial products (Claude Code, Codex, Gemini), undisclosed affiliation and conflict of interest is itself a mark against it, and the credibility section returns to this.

What the paper asks

The premise is simple. Agent benchmarks mostly ask whether the task was solved. In production, agents solve tasks while spending money. The authors' question follows: does an accuracy-only metric hand its top score to the strategy that always buys the most expensive option?

To find out, they attach priced actions and an explicit budget to every task. Four kinds of economic decision are in scope: (1) not escalating when cheap evidence already suffices; (2) escalating when a single cheap search returns partial or misleading evidence and the answer lives behind iterative deep research; (3) choosing a model tier when evidence is already complete and only reasoning difficulty remains; and (4) stopping on an unsupported premise — gathering evidence and abstaining rather than fabricating.

The suite holds 304 tasks derived from GAIA, HotpotQA and MuSiQue across five families. Seven commercial agents were evaluated alongside four scripted control policies. Because the controls act on fixed rules rather than judgement, they work as a litmus test for what the metric actually rewards.

The pivotal construct is the economic-consistency score. With Up as strict accuracy on the 222 upgrade-oriented tasks and Save as strict accuracy on the 82 save-oriented tasks, Econ = min(Up, Save). Not an average — a minimum. A one-sided strategy is recorded at its weaker side no matter how high the other side climbs.

Results

All four columns below are accuracy measures on a common basis. Micro is the conventional task-averaged strict success rate across all 304 tasks; Econ is the minimum defined above. Cost units are deliberately excluded from this table because the accounting rules differ by track — explained immediately after.

Two control rows carry the argument. CopilotFirst and RetryLoop escalate unconditionally, so their upgrade accuracy is 100% and their micro success rates — 76.3% and 73.0% — are the highest in the entire table. On the save side they score 12% and 0%, putting their Econ at 12.2% and 0.0%, near the bottom. The first and second place finishers under a conventional success metric are last under an economic one.

The three tool-API models reached 3.9-24.0% micro and at most 7.3% Econ. The gap to the workspace CLI track (44.6-53.6% Econ) looks dramatic, but the two tracks do not share cost accounting. Tool-API agents and the scripted controls price actions from a shared action ledger; workspace Units are a post-hoc execution proxy (10+0.5T+10M+5N) computed after the run from wall time, modified files and new files. It neither implements the shared 260-unit research price nor reveals which priced action was chosen. The authors draw the line themselves: "Comparisons should also remain within interfaces that expose the same actions and accounting rules; the workspace results are informative about completion across decision regimes, but not about adherence to the shared tool ledger." Workspace Econ is a regime-balanced completion diagnostic, not a cost measure.

Credibility

The design is the strong part. Running scripted oracles alongside real agents is the methodological contribution. A claim that "this metric overvalues a particular behaviour" is hard to demonstrate with model scores alone, but a fixed-behaviour oracle exposes the metric's shape directly. Putting budgets and price lists into the task definition — an ex-ante constraint rather than an ex-post estimate — and flagging cross-track incomparability before anyone else could are both careful choices.

The concerns are substantial. With no affiliation, funding source or conflict-of-interest disclosure, the neutrality of a commercial product ranking cannot be checked from outside, and no code or data availability statement was found, so the reproduction path is closed. The paper's own limitations are wide: human realism review covers 45 of 304 tasks; the main runs are single-shot at temperature zero, leaving run-to-run variance unknown; judge calibration used one human reviewer, estimating human-judge but not human-human agreement; and in that sample the judge showed a +12.5-point higher positive rate than the human on Sonnet and GPT-5.4 answers. For several QA families "grounded" means the episode accessed evidence, not that a cited span entails the answer, and stop-loss scoring accepts evidence-backed abstention.

Reviewer's take

First, summarising this paper as "agents are bad with budgets" misses the target. The subject is the metric, not the agent. RetryLoop lands near the top at 73.0% micro and at 0.0% economic consistency. A team whose release gate is a single success rate is, by construction, running a device that selects for the most expensive agent available. Simply dropping your current pass criterion into this table earns its keep.

Second, Econ = min(Up, Save) is deliberately punishing and, in my judgement, correctly shaped. Escalating well and saving well are not properties that trade off against each other; both are required. The weakness of a minimum, though, is that the whole score rides on the sample size of the weaker group. The save group holds 82 tasks and stop-loss only 10. A single stop-loss item moves that family by 10 points, so any agent whose Econ is set by the save side deserves a wide interval.

Third, the cross-track gap (44.6-53.6% against 3.6-7.3%) is the number most likely to be misquoted from this paper. "CLI agents make better economic decisions" is not a conclusion this data supports. The price lists differ, workspace cost is a post-hoc proxy, and the authors explicitly decline to make any cross-track cost, Pareto or tool-economy claim. Quote it within a track or not at all.

Fourth, the asymmetry among workspace agents is an operational signal in its own right. All four score far higher on Save than on Up — Claude Code (Opus 4.8) sits at 96% against 54%. The ability to avoid overspending when cheap evidence suffices is already strong; the ability to notice that cheap evidence is insufficient and escalate hovers near half. Operationally the second failure is the expensive one: over-calling costs money, but failing to escalate ships a wrong answer to a user. The 54% ceiling deserves attention before the 96% floor.

Applying it

• Split the pass criterion in two — stop gating releases on a single success rate. Measure the upgrade direction (did it escalate when it had to?) and the save direction (did it hold back when it could?) separately, then adopt the lower of the two. Averaging lets a biased strategy through.
• Fix the action price list in code — declare unit prices for search, premium tool calls and each model tier in configuration so that different interfaces still roll up under one accounting rule. Treat post-hoc cost estimates as reference values, never as the metric.
• Book under-escalation separately — do not merge over-calling (cost leakage) and under-calling (wrong answers shipped) into one failure rate. Pricing the latter in currency shortens tier-policy arguments considerably.
• Implement an explicit stop-loss path — for requests built on a false premise, abstain with evidence instead of generating an answer, and keep stop-loss items in the regression set. Without that family, fabrication scores as success.
• Ban cross-track comparison — before putting numbers from different vendors or interfaces in one table, confirm the accounting rules match. If they do not, split the table.

Conclusion

EcoAgent-Bench contributes less a new leaderboard than a demonstration of the old leaderboard's defect. One rule that always escalates climbed to the top of the table at 76.3% micro success while scoring 12.2% on economic consistency, and that contrast is the evidence. Undisclosed affiliation and funding plus single-shot runs are real weaknesses, so rather than quoting the absolute figures, the higher-value move is to port the measurement design — score the two directions apart and judge on the minimum — into your own evaluation. Splitting model tiers by intent continues in model routing by intent, and forecasting the spend itself in LLM cost forecasting.

References

• EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents — arXiv abstract (primary source)
• Full HTML text of the same paper — used to cross-check the results table, task composition and limitations
• Model routing by intent — sunny34.com blog