Best Human Still Bottom of the Board: Reviewing FM-Bench

Source Document

Tianyou Wang, Chongyang Gao, Kezhen Chen, Chen Dong, Yinghao He, Donghan Li, Wangcheng Xu, Hongjiu Zhang, Chi Li, "FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents", arXiv:2608.18423 [cs.AI], submitted 2026-08-19, DOI 10.48550/arXiv.2608.18423. Affiliation: AnalogyAI. Code and engine released (github.com/Analogy-AI/fm-bench).

This is a preprint that has not been peer-reviewed. All authors are affiliated with AnalogyAI, the startup that built this benchmark. The 15 evaluated models (Anthropic, OpenAI, Google, xAI, Meta, and five open-weight systems) are all third-party, which limits the incentive to favor any single vendor, but a commercial interest in driving adoption of the benchmark itself cannot be ruled out. No funding source is disclosed in the paper. Live access to the source was blocked in this session, so the full text was cross-checked against an arXiv HTML snapshot the site's collection pipeline fetched at 2026-08-23T21:40:45Z.

Study Overview

The paper asks two questions: can a language-model agent sustain effective management of an organization over a long horizon rather than just completing bounded tasks, and what behaviors make up that capability. FM-Bench measures this by having an agent run one club in a 16-club league for 20 in-game years, roughly 340-400 decision stops. Through 26 tools, the agent trades players, negotiates contracts, invests in facilities and youth, sets lineups, and answers to a board that can fire it, with a deterministic engine computing the score — no LLM judge, no human rater. Evaluation runs in two modes: the solo track pits each of 15 frontier models against a frozen scripted world, while the Arena places the same 15 models plus one scripted anchor in one shared world. Every decision stop opens as a fresh conversation with no chat history, so the agent's self-written notebook is its only memory across stops, and curating it is itself part of what's measured.

Key Results

On the solo track, scores are absolute against the same scripted world. claude-fable-5 reached about 95% of the oracle ceiling (90.94 of 95.54, where the oracle is a scripted policy with privileged access to hidden state) to top the board, and the open-weight kimi-k2.6 (88.49) beat both gpt-5.6-terra (86.66) and gpt-5.6-sol (86.40) — scale, price, and vendor do not order the ranking.

The Arena, a competitive shared world, flips the picture. The scripted anchor was the first to be eliminated, in year 3, and the two weakest models (gemini-3.5-flash, claude-haiku-4.5) followed after exhausting their revivals. On the solo track, the top 4 models each held the league title continuously for all 20 years — a "dynasty." In the Arena, the title rotated: it changed hands in all but 2 of 19 season transitions, with ten different models winning it at least once — the clearest sign that competing against fixed opponents and adaptive rivals are different games.

4 of the 6 first-play humans were fired or went insolvent early. Of the two who completed the run, the stronger score (74.64) still landed between claude-opus-4.8 (75.02) and gemini-3.5-flash (74.59) on the 15-model board — near the bottom. The paper decomposes final score into six behavioral capabilities: cutting slow-payoff investment as the horizon nears its end tracked score the most strongly (Spearman r_s=-0.58, negative on all three seeds), followed by keeping idle cash low (r_s=-0.50) and opening contract renewals early (r_s=+0.45). Token spend, by contrast, spanned 28M to 194M (about 7x) with no correlation to score (r_s=-0.19, p>0.38 on every seed). Price discovery was weak across the board — the median number of offers needed to close a deal was 30 (versus 1 for the oracle) — and memory curation split into two opposite failure modes on notebook similarity: 0.91 for gpt-5.6-sol (an ever-growing archive) versus 0.20 for claude-sonnet-5 (rewritten from scratch each time), with the winner holding a middle ground at 0.39.

Credibility Assessment

Three things support trust here. First, a deterministic engine computes the score with no LLM judge or human rater, limiting grading bias. Second, four scripted anchors (oracle, heuristic, idle, random) were used to calibrate the scale's ordering and floor/ceiling before any scored run, so the scoring constants weren't fit to the results after the fact. Third, every result file records the engine commit, parameter hash, and score version, verified through HMAC signing and server-side replay — reproducibility is strong.

The caveats are just as clear. As the paper itself notes, the solo board is a mean over only three seeds, not enough to make statistical claims about adjacent models, and the Arena is a single seeded world with no error bars, so close scores should be read as ties. The six behavioral metrics are correlational, not causal, and the authors acknowledge that at least one of them can be confounded by other traits. Every author is affiliated with the company that built the benchmark — a real conflict of interest — this is a pre-peer-review draft, and no funding source is disclosed. The scripted opponents are, by design, non-adaptive, so generalization of the absolute scores to the real world remains untested. Prior long-horizon benchmarks the paper cites — YC-Bench, CoffeeBench, StockBench — used fixed reference agents or markets that don't react to the agent, and FM-Bench's Arena result (title rotation under adaptive competition) shows that this static design misses real dynamics, filling a gap those studies left open.

Reviewer's Take

First, I judge the most practically valuable finding here is not the 90.94 top score but the null correlation between token spend and score (r_s=-0.19). A 7x compute gap failing to sort the ranking suggests long-horizon agent quality should be judged by what an agent did, not how much it "thought."

Second, I read the two opposite memory-curation failures — ever-growing archive versus wholesale rewrite — as a direct warning for practical agent design. Both extremes failed in this benchmark; the middle ground the winner held (0.39: preserve the strategy skeleton, update the details) looks like a reasonable target.

Third, I think it would be premature to compress "4 of 6 humans were fired early" into "models beat humans." The sample is six first-play, untrained participants — this is not a comparison against skilled managers. Still, the overlap between the humans' failure pattern (holding cash only to be billed for it later) and the models' credit-assignment weakness suggests the capability this benchmark measures is genuinely hard.

Practical Applications

• Audit endgame awareness — check whether your long-horizon agent cuts slow-payoff investment on its own as a deadline nears. This was the strongest signal in the paper (r_s=-0.58).
• Audit idle resources — periodically check what share of cash, credits, or quota sits unused. This was the second-strongest predictor (r_s=-0.50).
• Land memory policy between archive and rewrite — don't let conversation state accumulate without bound, and don't rewrite it wholesale each round; keep the core strategy while updating only the details.
• Manage quality by behavior, not token budget — since compute spend doesn't guarantee performance, add endgame handling, credit assignment, and price discovery as explicit evaluation axes.
• Re-test under competitive, adaptive conditions — an agent that scores well on a static benchmark can still be outranked once rivals adapt, so run an Arena-style competitive scenario before deployment.

Conclusion

FM-Bench shows, through deterministic scoring and a six-axis behavioral decomposition, that "completing a task" and "managing an organization over the long run" are different capabilities for language-model agents. claude-fable-5 topped both tracks at about 95% of the oracle ceiling, yet in the competitive Arena the title rotated to a different model nearly every season, and most first-play humans were fired early. The conflict of interest — every author works at the company that built the benchmark — and the limited seed count are worth keeping in mind, but the null correlation between token spend and score, and the two memory-curation failure modes, are signals worth applying to your own agent designs right away. The same question — that completion rate itself is the deployment-critical number for long-horizon benchmarks — continues in Only Six of Ten Finish: Deployment Criteria for Long-Horizon Browser Agents.

References

• FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents — arXiv abstract (source)
• Full HTML text of the same paper — used to check tables and behavioral metrics (via snapshot)
• Only Six of Ten Finish: Deployment Criteria for Long-Horizon Browser Agents — sunny34.com blog