What Only Shows Up Past Year 20

Agent evals usually score single-task completion rates. But real operations are shaped by decisions that accumulate over years, not weeks. FM-Bench, a benchmark we reviewed recently, exposes that gap by having agents manage one club in a 16-club league for 20 years — roughly 340 to 400 decision points. Scoring runs on a deterministic engine, not an LLM judge or human rater, which is itself an argument for treating long-horizon management as a separate evaluation axis.

The Ranking That Compute Couldn't Move

The most directly applicable finding here is the null correlation between token spend and score. Per-model token usage ranged from 28M to 194M — roughly a 7x spread — yet the Spearman correlation with score was -0.19 (p>0.38 across all three seeds), effectively no relationship. Thinking longer or computing more didn't move the ranking; what the agent did, did. That's a signal to manage agent quality by behavioral metrics, not compute budget.

Running a Long-Horizon Agent on Behavioral Metrics

Before deploying a long-horizon agent, declare behavioral targets instead of a completion rate. The correlations FM-Bench verified put endgame investment discipline (Spearman r_s=-0.58) and idle-resource minimization (r_s=-0.50) as the strongest predictors, with early contract renewal (r_s=+0.45) close behind. Translated into team KPIs, that becomes: cut low-yield investment allocation by half starting four weeks before any deadline, and keep idle resources — unused credit, cash — under 10%.

Failure clustered at the two extremes of memory policy. Agents that kept appending to their notebook without compression (similarity 0.91) and agents that rewrote it from scratch every turn (0.20) both missed the top tier; the top-ranked model held the middle at 0.39. Logging similarity on every memory update and alerting when it drifts below 0.2 or above 0.8 turns this pattern into an actual recovery branch.

The second failure axis is price discovery. The median number of offers needed to close a deal was 30, against 1 for the oracle. Negotiation-style tool calls need an offer-reject cycle cap — around 5 is a reasonable start — with a fallback path that recalculates terms automatically or escalates to a human once the cap is hit, so runaway retries don't quietly burn budget.

The third axis is rank reversal under competition. Models that held the league title for all 20 years in the solo track, played against a fixed scripted opponent, defended it only twice across 19 title transitions once other agents could adapt in the arena. Locking a deployment decision to a static benchmark score risks collapsing once real competitors start reacting, so pre-deployment scenarios need an adaptive-competitor case.

The operations checklist comes down to four items: confirm from logs that investment actually contracts once the endgame window opens; audit idle-resource ratios on a schedule; track memory similarity as a standard log field and flag drift outside 0.3–0.5; and put offer-count caps and contract-term masking into pre-deployment scenario tests for any negotiation or transaction tool.

Four of six first-time human players in this benchmark were eliminated early, and even the better-performing human who finished ranked near the bottom of the 15-model field. Rather than reading this as models beating people, it's more useful operationally to treat long-horizon organizational management as a distinct, hard capability for both language models and humans. An improvement loop that reviews weekly trends in these four behavioral metrics — ahead of completion rate — works better than one anchored on completion alone, and separating memory-policy tweaks from negotiation-cap tweaks in the changelog makes it far easier to trace which adjustment moved the outcome.

Checklist to Use Now

Judge a long-horizon agent's quality by endgame response, idle-resource management, memory curation, and negotiation caps — not by token spend. Put those four metrics into pre-deployment checks and weekly reviews, and add an adaptive-competitor scenario to the test suite, and a static benchmark score is far more likely to hold up once the agent is actually running in production.

References

FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents — arXiv

Best Human Still Bottom of the Board: Reviewing FM-Bench — sunny34.com Research