What a Single Overall Score Can Hide
A benchmark's Overall score is an item-weighted average across task types, so it blends together gains that differ wildly by combination. In HealthLoopQA, switching Gemini from a prompting-only setup to an agentic one that executes code lifted the simulation Overall from 43.7 to 65.8, and the real-world Overall from 64.6 to 78.5.
Isolate the safety-critical anomaly-detection (AD) combination alone, though, and the gap widens further: 21.2 versus 52.4. Read only the headline number and conclude "switching to an agent helps," and you miss how unevenly that improvement is actually distributed across combinations.
In-Context Laziness: Plausible Guessing Instead of Computing
The researchers named a failure mode "in-context laziness": instead of actually scanning 30 days of time-series data, the model anchors on a rough median and wraps it in a plausible-sounding narrative to deliver a confident answer. This was confirmed by manually reviewing the model's incorrect reasoning traces; the paper does not quantify how often it occurs across the full benchmark.
Items requiring a single calculation (QC) held steady at 68.8-95.6 across all four conditions, but combinations requiring an event lookup followed by pattern correlation (ER→PC) collapsed to 0.8-23.5. The gap isn't a lack of arithmetic ability — it's a failure to actually chain multiple steps together.
From Design to Audit: Running an Eval Set Aggregate Scores Can't Fool
At the planning stage, declare a minimum pass line per combination alongside the item-weighted average. Build a matrix crossing task type by cognitive level, and set gates independent of Overall — for example, "hold the release if any combination scores below 50" — so an aggregate-score bump alone can't get mistaken for a green light.
Catching laziness means grading the reasoning trace, not just the final answer: check whether intermediate calculated values actually show up in it. When a skipped calculation, a misaligned timestamp, and an unsupported assumption appear together, treat it as a laziness signal and route it into a retry branch that forces a recomputation.
Unit mismatches belong to the same family of trap. As in the case where misreading an insulin infusion rate's unit scale threw predictions off by multiples, verify that output units match the required spec as a check kept separate from calculation accuracy.
Your operational checklist needs a step that filters out asymmetric comparisons. In this same study, only one model was measured under both prompting-only and agentic conditions, so there's no basis for extending "the agent beat the prompt" to the other models. For human review sampling, pull the items that scored lowest per combination in the previous run instead of sampling randomly — it raises audit efficiency.
Fix your log fields in advance so cross-validation stays possible later. Recording combination ID, normalized score, execution mode, and whether the data is simulated or real lets you cross-check the 44 tasks shared between the simulation and real-world sets and confirm the scores move in the same direction.
Every release, lay each combination's table next to the previous version's and track which one newly collapsed. Fold a combination that has shown collapse, like this one, into a fixed regression suite re-checked at every deploy, and rather than leaving laziness cases as qualitative notes the way this paper did, grow the eval set by adding samples to that combination until the incidence rate itself becomes a number.
The combination table pays off on cost, too. Adding recompute logic only to collapsed combinations saves tokens compared with blanket safeguards applied even to single-calculation combinations that already score above 68.
Takeaway
An agent eval set needs its combination table — task type crossed with cognitive level — locked in as a grading criterion alongside Overall, not in place of it. In this case, single-step calculation held at 68.8-95.6 while multi-step pattern combinations collapsed to 0.8-23.5, and that gap never surfaces from the headline number alone without trace review, unit cross-checks, and a regression suite for collapsed combinations.
References
Guessing Instead of Computing: Reviewing HealthLoopQA — sunny34.com Research