Top Score Just 50.6%: Reviewing PLE, the AI-Agent Benchmark for Physical Engineering

Source Document

Rujun Han, Yufan Zhuang, Arijit Ray, Han Yu, Zoey CuiZhu, George Lee, Yifei Ming, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, Tomas Pfister, Chen-Yu Lee (all affiliated with Google Cloud AI Research or Google), "PLE: Physical-Engineering's Last Exam", arXiv:2610.30537 [cs.AI], submitted 2026-10-21, CC BY 4.0. The full text was cross-checked against a primary-source snapshot taken 2026-10-24T22:08:22Z (used in place of WebFetch because this session's egress was fully blocked).

This is a non-peer-reviewed preprint. All 13 authors are affiliated with Google Cloud AI Research or Google, and task creation and validation were paid through the data vendor Mercor (the paper self-discloses an average of $4,663 per expert-authored task and $3,157 per co-authored task for validation and difficulty intensification). Notably, though, the top-scoring model was not one of Google's own Gemini models but Anthropic's Claude Opus 5.5 — and the LLM judge used for rubric grading is that same Claude Opus 5.5, a point addressed separately in the credibility assessment below.

Study Overview

The question is simple: frontier language models already score above 90% on popular knowledge benchmarks, so can they also perform the real, economically valuable work of a professional physical engineer — electrical, civil, structural, mechanical design? Existing engineering benchmarks (EngiBench, Frontier-Eng, etc.) are either textbook-style questions or toy design problems that fit entirely in a prompt, and are already close to saturated. PLE instead hands the agent a realistic project file system (61 files on average, spanning PDFs, spreadsheets, CAD/BIM drawings) and demands a signable deliverable — a memo, calculation workbook, equipment schedule, or 2D/3D drawing.

Tasks come from two routes. The 44 Expert-Authored tasks are written end-to-end by practising engineers and graded with a weighted rubric of binary checks. The 41 Co-Authored tasks start from an engineer's brief; an adversarial Creator-Solver agent pipeline then builds the environment and grader, before practising engineers validate and intensify the difficulty. Co-authored tasks use a hybrid score blending constraint-compliance (p) and a cost-optimization objective (O′) at a 0.4:0.6 weight, with a hard safety/feasibility gate (G) that zeroes out any design that fails it, however cheap it is.

Key Results

14 frontier models were run under 7 publicly available agent harnesses (Claude Code, Antigravity, Codex, Hermes, OpenCode, OpenHands, Pi) across all 85 tasks, three runs each. GPT 6 Astra was tested only in Codex and Grok 4.7 only in its own harness (Grok Build), making 86 model-harness pairs in total (12 models × 7 harnesses + 2 models × 1 harness each).

The best configuration, Claude Opus 5.5 in Claude Code, averaged 50.6% at $17.18 per task. OpenHands came close at 49.9% ($14.82), and GPT 6 Astra in Codex reached 48.3% at roughly half the cost ($9.15). The spread when the same model switches harnesses is only a median of 3.0 points (Claude Opus 5.5 spans just 47.3-50.6% across all seven harnesses) — far smaller than the gap between models shown in the chart above (50.6% down to 14.2%, with four weaker models all under 7%).

By discipline, civil engineering was easiest (69.9%), while mechanical (40.5%) and structural (46.6%) were hardest (the paper does not give a single aggregate figure for electrical). Co-authored tasks were harder than expert-authored ones — 48.0% versus 53.1% in Claude Code. Solving a task took 27.0M tokens and 63 minutes on average, splitting into 39.3M tokens / 97 minutes for co-authored tasks and 15.6M tokens / 31 minutes for expert-authored ones — the weighted average matches the overall figure exactly.

Failure patterns stand out too. Across all runs of the 86 model-harness pairs, 23.2% scored 0%; of those, 70.4% delivered files that the grader scored 0% (often failing the safety/feasibility gate), 25.3% ended without a gradable deliverable, and 4.3% hit the time budget or an agent-side error (summing to 100%). The top configuration (Claude Opus 5.5 in Claude Code) rarely failed outright, scoring 0% in only 11 of 255 runs — but on co-authored tasks, its deliverables satisfied 91% of constraints on average while only 41.3% satisfied every hard safety/feasibility constraint needed for objective credit. The gap between a "mostly correct" design and a "signable" one is this benchmark's central challenge.

Credibility Assessment

This is a non-peer-reviewed preprint, and Google-affiliated authors ran the evaluation design, grading, and writing. The most notable conflict-of-interest point is that the LLM judge used to grade rubric items is the same model that topped the leaderboard, Claude Opus 5.5. The authors disclose and re-check this themselves: re-grading the same 1,159 LLM-judged runs with a second Claude Opus 5.5 pass, Claude Opus 5, and Gemini 4 Argon, the second Opus 5.5 pass reproduced 98.7% of verdicts (Cohen's κ=0.97, essentially unchanged), while Opus 5 and Gemini 4 Argon agreed on 95.8% and 95.5% of items (κ=0.90), raising overall scores by 2.1 and 0.9 points respectively. Narrowed to the 44 rubric-graded tasks, switching the judge to Gemini 4 Argon raised the score 2.3 points more than a repeat self-pass by Opus 5.5 — a measurable but, next to the roughly 50-point gap between models, modest self-preference. Harness rankings held up almost unchanged under different judges too (Kendall's τ=0.90-1.0).

Funding is transparent — the paper discloses per-task payment rates through Mercor. As contrary evidence, the same authors cite existing engineering benchmarks like EngiBench (2025) and Frontier-Eng (2026), where leading models already score 80-97.9%. That doesn't make PLE wrong; it shows that "textbook, single-task" benchmarks and "complete a real project deliverable" benchmarks paint very different pictures, and that benchmark design itself drives the conclusions drawn about agent capability. Per-task variance is also substantial — of the 595 task-harness cells (85 tasks × 7 harnesses), 105 swung by 30 points or more across three runs.

Related Research

• Patwardhan et al. (2025). GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — benchmark-origin. GDPval first established the paradigm of grading real deliverable-producing tasks from 44 occupations via blinded expert comparison; PLE applies that same paradigm in depth to the single high-value domain of physical engineering.
• Vidgen et al. (2026). APEX-Agents — extension. PLE's sandboxed environments run directly on APEX-Agents' published Archipelago harness (MCP-based multi-application containers). APEX-Agents' own best score was only 24.0% (Gemini 3 Flash), echoing the same pattern of low scores on long-horizon professional tasks in both finance/consulting/legal (APEX-Agents) and physical engineering (PLE).
• Zhou et al. (2025, accepted to ACL 2026). EngiBench — contradiction. Covering 143 of 1,760 textbook-style engineering problems, this benchmark finds leading models already exceeding 80% at its Level 2 difficulty. PLE showing the opposite picture (50.6%) in the same broad "engineering" space illustrates how large the gap is between textbook problem-solving and completing a real project deliverable.

Reviewer's Take

First, I judge the most important practical signal here is not the headline 50.6%, but the gap between "91% of constraints satisfied" and "41.3% of hard constraints fully satisfied." In engineering, a mostly-correct design is close to worthless — one violated safety requirement and the design doesn't get signed off. It's safer to read current agents as close to "mostly right" but still far from "fully right."

Second, the small harness effect (median 3.0 points) is a warning to teams over-investing in harness engineering. Based on this paper, swapping in a better model buys far more than refining the harness around it — though that conclusion only holds within the seven public harnesses the authors tested, not necessarily for custom in-house harnesses.

Third, I credit the authors for surfacing and re-checking that their judge model shares a lineage with the top-scoring model, but their framing of the resulting 2.3-point self-preference as "small" is their own interpretation, not an absolute standard. In a benchmark where models are separated by 2-3 points in places, that's enough to reshuffle rankings — so the fine-grained order among the top three or four models is probably best read as indicative rather than definitive.

Practical Applications

• Keep full human verification before sign-off — with only 41.3% full hard-constraint satisfaction, don't auto-publish engineering deliverables without a human's final approval.
• Invest in model selection first — the gap between models (tens of points) dwarfs the gap between harnesses (median 3.0 points), so prioritize benchmarking newer models over refining your harness.
• Pick a harness off the cost-score frontier — the same model's per-task cost varies 2-3x by harness (Claude Opus 5.5 ranged $12.53-$35.15), so check for the cheapest configuration on the frontier first.
• Audit your own LLM-judge bias — if your evaluation pipeline's judge model shares a lineage with the model it grades, re-grade with a different model first to measure the size of the bias.
• Read benchmark type before trusting a score — a high score on a "textbook-style" benchmark can coexist with a low score on a "complete real work" benchmark, so weight the benchmark closest to your actual use case.

Conclusion

PLE's contribution is judging agents by "signable engineering deliverables" rather than "engineering knowledge questions." Only one of 86 model-harness pairs crossed 50%, and even that one fully satisfied hard constraints only 41.3% of the time. The model mattered far more than the harness, and the authors' own disclosed judge self-preference (2.3 points) is modest next to the roughly 50-point gap between models, though large enough to reshuffle top rankings. The pattern of scores dropping as benchmarks get closer to real work recurs in other industrial tasks too, such as where a single formula decides the automation boundary in production scheduling — related discussion continues in Designing the Automation-Boundary Gate for Agentic Production Scheduling Models.

References

• PLE: Physical-Engineering's Last Exam — arXiv abstract (primary source)
• Full HTML text of the same paper — used for table/figure verification (via a primary-source snapshot copy, since this session's egress was blocked)
• GDPval — arXiv:2510.04374
• APEX-Agents — arXiv:2601.14242
• EngiBench — arXiv:2509.17677
• Designing the Automation-Boundary Gate for Agentic Production Scheduling Models — sunny34.com blog