Source

Koutian Wu, Junjie Zhou, Ergan Shang, Jiayu Wang, Pengqian Han, Junkai Wang, Wanghan Xu, "Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation", arXiv:2609.11115 [cs.AI, cs.IR], submitted 2026-09-10, DOI 10.48550/arXiv.2609.11115. Corresponding author Koutian Wu lists affiliations with Earth-Space-AI and Tacite AI (contact [email protected]). The full text was cross-checked against a snapshot captured 2026-09-11T23:22:12Z (a copy of the primary source taken the day before this review).

This is a preprint that has not been peer reviewed. The source-code repository owner account matches the corresponding author's email domain, which reads as the authors auditing and presenting a tool built by their own company. No separate funding statement or conflict-of-interest disclosure was found in the text.

What the paper asks

Two questions drive the paper. First, when records scattered across several benchmark catalogs (LLM Stats, OpenCompass Hub, Artificial Analysis, model reports) are merged into one searchable database, how many actually accumulate, and how many of those are genuinely comparable? Second, what changes when that database is put to practical use?

The method normalizes records from four sources into a shared schema while retaining source identity, then runs a "census" script that counts records, models, documents and scores across the full population with no date or score filter. Records without a score, date or citation are kept rather than dropped. Section 5 walks through a case where a coauthor used the project's own CLI client to conduct a prior-art search on August 2026 credit-assignment work (Table 4), showing the catalog in actual use.

Results

At the 2026-09-07 collection cutoff, the v0.11.0 release held 1,283 records across four sources. The per-source totals below match the paper's own table exactly (records, scored, unscored and numeric-score columns all check out).

Source	Records	Scored	Unscored	Numeric scores
LLM Stats	687	679	8	5,544
OpenCompass Hub	461	0	461	0
Artificial Analysis	25	25	0	7,050
Model reports	110	86	24	322
Total	1,283	790	493	12,916

The most important number in the paper is not in that table. Of the 790 scored records, only 82 (10.4%) satisfy all three conditions needed for a direct comparison (e.g., headroom = 100 minus the top score): a declared percentage unit, a confirmed direction, and values within 0-100. The remaining 708 use other or unverified scales, and the authors explicitly warn not to assume a 100-point ceiling.

Measurement state	Records
Percentage scale + confirmed direction + 0-100 range	82
Other or unverified scale	708
No numeric score	493
Full population	1,283

The taxonomy has a crack of its own. Of 345 records with an agentic character, the paper reports only 128 carry the top-level label "Agentic & Tool Use," while 117 sit under "Coding & Software Engineering" instead — a benchmark that resolves repository issues stays classified as coding even when it runs as an agent. On documentation, 1,171 of the 1,208 cited documents (97%) carry no named organization, and only 615 of the 1,283 records (48%) have a release date (668, or 52%, do not). One concrete case: the Artificial Analysis record for GPQA Diamond cites a single document yet carries 586 scored models, while the corresponding model-reports record cites 27 documents but holds only 19 models and 21 scores — citation count and coverage do not scale together.

Credibility

The rigor of the method is the strongest reason to trust it. Records, models, documents and scores are kept as distinct counting units throughout, avoiding a common source of confusion, and the authors publish checksum-based verification scripts (a `--check` mode) for reproducibility. Re-summing all six of the paper's audit tables (catalog composition, score coverage, scale eligibility, documentation, score distribution, date evidence) against their stated totals (1,283 / 790 / 493 / 12,916) checks out in every case.

The concerns are real. Because the paper audits the coverage of the authors' own product, its conclusion about how large and well-organized the catalog is carries an inherent promotional risk. The authors themselves acknowledge that retrieval precision, task suitability and time saved remain unmeasured, and that the Section 5 use case is a single demonstration with no controlled baseline. They also disclose that the arXiv discovery route does not backfill papers first posted before its collection window, causing two relevant studies to be missed until a coauthor found them later.

Reviewer's take

The most practically important number in this paper is the 82/790 figure the authors do not headline. It means that before quoting a benchmark leaderboard or a "how many points behind SOTA" comparison, one first has to check whether the score was ever defined on a 0-100 percentage scale with a confirmed direction. Trusting a single aggregate score for a decision can rest on the far more basic trap of scale mismatch.

Second, the 128-versus-117 classification gap is a practical signal that counting "how many agent benchmarks exist" requires multi-facet queries, not a single top-level label. Filtering on one domain label alone misses roughly a third of evaluations that actually run as agents. A competitive-landscape report or internal coverage comparison that falls into this trap ends up making decisions on an undercounted figure.

Applying it

Make scale-checking a precondition for citation — before citing a benchmark score, confirm the percentage unit, direction and 0-100 range are all declared. In this audit, about 90% of scored records failed to meet that bar.
Count coverage with multi-facet queries — when counting how many "agent benchmarks" your evaluation suite covers, filter on interaction paradigm or input modality facets, not just a single top-level domain label.
Keep document counts and model counts separate — when designing an internal evaluation dashboard, do not conflate "cited documents" with "scored models" as one metric; a single citation can carry hundreds of model scores, as with GPQA Diamond.
Preserve unscored records — records without a score (roughly 38% of the population here) are still worth keeping as reference if they carry paper, repository or dataset links.
Verify self-reported audit numbers separately — treat a tool or catalog provider's own "coverage figures" as something to spot-check or cross-cite before adopting.

Conclusion

Benchmark Radar's real contribution is not a new leaderboard but an audit that lays bare the actual state of existing benchmark data. The fact that only 82 of 790 scored records were honestly comparable shows how fragile the common practice of comparing models or policies on a single aggregate score really is. Because this audit targets the authors' own product, the absolute figures should be read with that in mind; the more transferable value is the checking method itself — confirm the scale, query on multiple facets, and separate documents from models — ported into your own evaluation pipeline. The aggregate-score trap is examined from another angle in Designing an Agent Eval Set: A Checklist.

Only 82 of 790 Scored Records Were Comparable: Reviewing Benchmark Radar's Catalog Audit