Source Document
Austin Tudor David Andrews, Liam Wilkinson (joint first authors), Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster, Rui Ponte Costa, "CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI", arXiv:2609.02459 [cs.AI], submitted 2026-09-02. Affiliations: University of Oxford, Tony Blair Institute for Global Change, Google DeepMind, UK AI Security Institute (AISI), Imperial College London. Submitted to the NeurIPS 2026 Evaluations and Datasets Track, currently under review.
This is a preprint without peer review, submitted but not yet accepted. The author list mixes a commercial lab (Google DeepMind) with a government body (UK AISI) — notably, the DeepMind-affiliated author works for the developer of one of the four evaluated model families, Gemini. The paper explicitly states it is not a model ranking, and in practice Gemini does not come out ahead: on the reflection-action gap metric, Claude scored lowest of the three models with enough data for comparison. No dedicated funding-disclosure statement was found.
Study Overview
The paper asks two questions: in a long-horizon environment spanning 300+ turns and thousands of tool calls, do agents (1) proactively query globally relevant state, and (2) actually execute the plans they set for themselves? The authors connect Civilization VI to language model agents via the Model Context Protocol (MCP), exposing 76 tools and a 29-function narration layer. The central design choice separates availability of information from whether it is queried — state a human player absorbs passively from the screen (victory progress, diplomatic standing) is exposed to the agent only through an explicit tool call.
Four model families — Claude Opus 4.6, Gemini 3.1 Pro, GPT-5.4, and Kimi-K2.5 — played 23 admissible games (19 Ground Control, 4 Snowflake). The authors are explicit that the sample is a pilot, not a model ranking. Instead, they define two behavioral metrics: Proactive Monitoring Rate (PMR, the share of tool calls spent on strategic state queries) and the Reflection-Action Gap (RAG@10, the share of diary-stated commitments executed within 10 turns).
Key Results
Aggregate win/loss does not discriminate between the four models. Only 3 of 23 games were won, all technology victories on Ground Control, and Fisher's exact test is not significant (p=0.488).
| Model | Scenario | Games | Wins | Losses |
|---|---|---|---|---|
| Claude Opus 4.6 | Ground Control | 6 | 2 | 4 |
| Claude Opus 4.6 | Snowflake | 2 | 0 | 2 |
| Gemini 3.1 Pro | Ground Control | 6 | 1 | 5 |
| GPT-5.4 | Ground Control | 6 | 0 | 6 |
| GPT-5.4 | Snowflake | 2 | 0 | 2 |
| Kimi-K2.5 | Ground Control | 1 | 0 | 1 |
| Total | — | 23 | 3 | 20 |
An ICC (intraclass correlation) analysis across 13 candidate metrics found most at noise level; only exploration at turn 100 showed clear discriminative power (ICC=0.717, Kruskal-Wallis H=12.28, p=0.0065). Final score, city counts, and economic yields all sat near zero or negative ICC.
Where aggregate outcomes fall silent, the behavioral metrics speak. The figures below use different bases (rates, counts, means) and should not be compared directly against one another.
| Metric | Value | Basis / note |
|---|---|---|
| Proactive Monitoring Rate (PMR) | 0.96-2.13% | Strategic monitoring share of non-infrastructure tool calls |
| Victory-progress query share | 0.05-0.29% (3.7-10.0 per game) | Playbook recommends every 20 turns; observed every 30-75 |
| Missed-warning defeats | 7 / 20 | Stated directly in the text (per-model subtotals do not sum to this total in the source, so those are not cited) |
| Reflection-Action Gap (RAG@10) | 48.2-65.8% | Claude, Gemini, GPT — share of commitments executed within 10 turns |
| City count (T50 / T100) | 2.00-2.83 / 3.00-4.67 | Playbook target is 3 cities by T50 — every model falls short |
Commitment labeling was performed by Claude Haiku 4.5, validated against a human-labeled sample of 50 commitments at 92.0% agreement (Cohen's kappa=0.879).
Credibility Assessment
What earns trust: PMR does not rise even late in the game (Appendix Fig. 11), suggesting a persistent allocation habit rather than a temporary context shortage. The RAG labeling pipeline was validated against human labels, and no same-family bias in favor of Claude was observed. Presenting outcome scores alongside behavioral metrics is itself the paper's demonstration of its central claim — that aggregate metrics are insufficient.
What to weigh: with 23 games (1-8 per model family), the authors themselves state the study is "powered only for large effects." Results were obtained under a shared, explicit playbook, so they should be read as deviations under guidance rather than a lack of capability — only 21% of pre-harness runs (without the shared playbook) reached completion at all. There is no random or scripted baseline, and the missed-warning table has an internal arithmetic mismatch: the per-model subtotals do not sum to the reported total. These findings converge with prior work such as BALROG (2025), which reports agents that can explain strategies without executing them; no contradicting prior evidence was found.
Reviewer's Judgement
The most practically useful contribution here, in this reviewer's judgement, is the demonstration that aggregate outcome metrics fail to discriminate at all. Ranking benchmark leaderboards by win rate or normalized score may be an unreliable proxy for production reliability, at least for long-horizon, tool-mediated tasks.
I read the "sensorium effect" — information being available but not queried — as a structural issue likely to recur across tool-mediated agents generally, not a quirk of Civilization VI specifically. The basis for that reading is that monitoring rate did not rise even as the game progressed, which points to an allocation habit rather than context exhaustion.
Putting It to Work
- Enforce a query schedule — don't leave global state checks to agent discretion; wrap the harness with a forced call every N turns or N tool calls.
- Add a commitment-tracking queue — log "next actions" stated in planning reflections into a separate queue and automatically check execution within a defined window.
- Never rank on aggregate outcomes alone — pair success rate or normalized score with behavioral metrics that have demonstrated statistical discriminative power.
- Audit same-family labeler bias — when an evaluation pipeline uses a model from the same family as one being evaluated, validate against a human-labeled sample.
- Stay humble at pilot scale — read a dataset of ~20 runs as a pattern, not a ranking.
Conclusion
CivBench's contribution is a measurement method, not a new model or algorithm. Two failure modes — not querying available information, and not executing stated plans — are invisible in win/loss tallies but surface once behavioral metrics are introduced. The sample is small and the source has a minor arithmetic inconsistency, so the numbers are better read as a directional signal than a definitive ranking: aggregate outcomes alone cannot establish the reliability of a long-horizon agent. For the operational side of behavioral metrics, see Behavioral Metrics for Long-Horizon Agents.