What happens when the pass criterion cannot see cost
Agent performance checks usually come down to one line: run the regression set, and if the success rate clears the bar, ship it. The problem is that nobody checks what behaviour that bar rewards. An agent that calls the premium tool on every question — including the ones a cheap search would answer — gets more answers right. A gate that looks only at success rate crowns that agent.
EcoAgent-Bench, released on 6 August 2026, demonstrated the mechanism experimentally. Across 304 tasks with priced actions and an explicit budget attached to each one, two fixed rules that always escalate took first and second place in task-averaged strict success at 76.3% and 73.0%. The economic-consistency scores for those same two rules were 12.2% and 0.0% — the bottom of the table. The important detail is that a rule, not a real agent, won: the metric itself prefers that behaviour.
Split it in two directions and use the lower one
The remedy is straightforward. Divide the check set along two directions: did it escalate when it had to, and did it hold back when it did not need to? EcoAgent-Bench split its 304 tasks into 222 upgrade-oriented and 82 save-oriented, then defined the economic score as the minimum, not the average, of the two accuracies. A one-sided strategy loses its strong score and is recorded at its weak one.
The minimum is right because the two abilities do not offset each other. An agent that saves money but never escalates when the evidence is thin simply ships wrong answers cheaply. An average lets these two failures hide each other.
From planning to iteration: a performance-check loop with a cost axis
Planning fixes three targets, not one: an upgrade target, a save target, and a pass line computed from the lower of the two. For a first rollout, 80% on each direction and 75% economic consistency is a workable starting point, adjusted quarterly. Fix the regression-set composition at the same time. The source benchmark runs roughly 73:27 upgrade to save, and once the save family drops below a quarter of the set it loses the power to catch over-calling.
Next comes the price list. Declare unit prices for search calls, premium tool calls and each model tier in configuration so execution logs roll up under one accounting rule. EcoAgent-Bench priced only its workspace-CLI track differently — a post-hoc proxy computed from wall time and file changes rather than the shared ledger — and the paper explicitly declines to compare costs across tracks as a result. Numbers on different accounting bases belong in separate tables.
The first failure pattern is the always-escalate habit above. The symptom is a success rate and an invoice climbing together; the cause is that no save-oriented tasks exist in the regression set. Recovery means assembling a batch of questions that cheap evidence fully answers, and scoring a premium tool call on those as a failure. Judged on correctness alone, this pattern is never caught.
The second is a missing stop-loss path. When an agent fabricates an answer to a question built on a false premise, the grader marks it wrong — but with no route to gather evidence and abstain, the team misreads that failure as a model-capability problem. Recovery means implementing abstention as a normal exit path and adding stop-loss items to the regression set. Watch the sample size, though: this family tends to stay small. The source paper's stop-loss family holds just 10 items, so a single task moves that family's score by 10 points.
The third is merging over-calling and under-calling into a single failure rate. The losses differ in kind. Over-calling costs money; failing to escalate ships a wrong answer to a user. EcoAgent-Bench's workspace agents scored 74-96% on the save direction against just 45-54% on the upgrade direction — the thrift is there, the judgement about when to escalate sits near half, and operationally the latter is the more urgent gap.
Four lines cover the operating checklist. Collect per-tier call prices on one dashboard. Book over-calling and under-calling separately. Run checks repeatedly rather than single-shot, and record the variance. If an LLM grades the runs, reconcile it against a human sample on a schedule — not a formality, given that the source paper found a stretch where the judge returned a 12.5-point higher positive rate than the human reviewer.
The improvement loop starts by tracking the economic score weekly. Because it is a minimum, it always points at the weaker direction, so the metric names next sprint's regression work by itself. A low upgrade score calls for more tasks that hinge on spotting insufficient evidence; a low save score calls for more that cheap evidence already answers. Once that cycle turns, the performance check stops being a pass stamp and starts producing a work list.
The checklist
Run a performance check on a single success rate and the most expensive agent passes it. Split the check set into upgrade and save directions, score each separately, and judge on the lower of the two. Fix action prices in configuration so accounting is uniform, and keep numbers from different accounting bases out of the same table. Always include save-oriented and stop-loss families in the regression set while watching their sample sizes, and keep over-calling and under-calling in separate books all the way through.
References
Full HTML text of the same paper — used to confirm the results table and task composition