Once the aggregate saturates, the leaderboard stops being a reason to choose
As of April 2026, GPT-5.5, Gemini 3 Deep Think, Claude Opus 4.7, and Qwen 3.5 Omni have all cleared 80% on MMMU-Pro. The gap between leader and trailer, once more than 10 points, has narrowed to under 3. When the top of the aggregate clusters this tightly, picking the number-one model is little different from a coin flip inside statistical noise. In a benchmark-convergence phase, you should choose a model not by the blended score but by the specific axis that maps to your own task.
The deciding axis split into video, OCR, and charts
As the aggregate scores converged, the gaps on individual axes actually widened. Video understanding favors Gemini 3, long-form document OCR favors Claude Opus 4.7, and chart and diagram reading favors GPT-5.5. So "which model is best" is no longer a well-formed question; only "which model has the highest pass rate on this task" remains valid. Operationally, building an axis-by-axis evaluation set from your pipeline's real inputs matters more than reading a leaderboard.
Full guide: from planning to operations
In planning, drop the aggregate score and fix a numeric target per task axis. For example, set video summarization at a scene-misrecognition rate of 2% or less, long-form OCR at a per-page character error rate of 0.5% or less with p95 latency of 8 seconds or less, and chart reading at a value-extraction accuracy of 98% or more as the pass line. Build a gold set of 200 or more real inputs per axis, run every candidate through the same set, and compare pass rates axis by axis.
Failure patterns differ by axis. Typical ones are dropped scenes late in long clips for video, collapsed table and formula structure for OCR, and hallucinations that misread axis labels as actual values for charts. Define recovery branches in advance. On low confidence, retry once; if value-extraction accuracy is still below the pass line after the retry, send the item to a human-review queue. When input exceeds the model context, split pages via safe truncation; if the structure still breaks after splitting, stop that item and return only the source link.
On the operations checklist, log the task axis, selected model, pass rate, p95 latency, retry count, and recovery-branch outcome in standard fields for every request. Because OCR and video inputs easily carry personal data, mask faces, national IDs, and contact numbers before writing logs, and purge originals after a set retention period. For quality control, track per-axis pass rate and violation count on a dashboard, and hold PII-masking misses to a standard of zero violations.
The continuous improvement loop reviews per-axis pass rate and human-review queue inflow weekly. When one axis drops below its pass line, swap only that axis to a different model — do model swaps per axis, not wholesale. When a new model ships, run it against the existing gold set first, promote it only when its per-axis pass rate beats the incumbent, and hold it otherwise.
Key takeaways
In short, after the 80% clearance on MMMU-Pro and a sub-3-point gap, the aggregate score is no longer a basis for choice. Fix the axes that map to your task — video, long-form OCR, charts — set numeric targets like pass rate and p95 latency per axis, and compare candidates on a real-input gold set. Branch failures into retry, human review, safe truncation, and stop; keep standard logs and zero PII-masking violations; and swap models per axis rather than wholesale. That is evaluation design for the benchmark-convergence phase.