Same Budget, Different Control: ProgramBench 63.7%→71.5% in Meta's Meta-Reasoning Review

Source Document

Paras Dahal, Anton Bakhtin, Taco Cohen, and 9 others, "Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning", arXiv:2609.38147 [cs.AI], submitted 2026-09-29. Affiliation: Meta Superintelligence Labs.

This is a preprint without peer review, and every author is affiliated with Meta Superintelligence Labs. No separate funding disclosure is given; this reads as the lab's own internal research. The clearest conflict of interest sits in the comparisons: the authors benchmark their own method (the meta-reasoning controller) against competitors' production agents — OpenAI's Codex and Anthropic's Claude Code — but ran those competing systems themselves, inside their own container and call-budget accounting. This is not an independent, third-party replication. The full text could not be fetched directly due to a session-wide egress block (confirmed against both example.com and arxiv.org as unrelated controls), so tables and figures were cross-checked against a primary-source HTML snapshot collected by GitHub Actions (collected 2026-10-01T22:11:46Z).

Study Overview

The starting observation is that deciding "what to do next" during a long agent run is itself a cost and a capability, separate from doing the work. The authors split that decision (meta-reasoning) from task execution (the workers) and make it its own agentic process. At each cycle, a controller runs four stages — assess, propose, evaluate, dispatch — to decide which worker gets which context, and between turns it keeps only a compact state text rather than the full conversation history. Every output is stored as an "artifact," and reference relationships are recorded as a graph, so a run can be diagnosed by its structure rather than only its final score.

The central comparison matches a "Direct Control" agent that uses the same workers, same model, and same call budget — isolating the effect of how the run is controlled. The paper also compares against external systems: the Recursive Language Model (RLM), mini-SWE Agent, Codex, and Claude Code. Evaluation covers four benchmarks — IMO ProofBench-Advanced (30 proof problems), ARC-AGI-2 (120 abstract-reasoning tasks), LongCoT-mini (507 long-horizon problems), and ProgramBench (200 program-reconstruction tasks) — across three models: Gemini 3.1 Pro, GPT-5.5, and Opus 4.8.

Key Results

At the main budget settings (100 calls for reasoning benchmarks, 1200 for ProgramBench), meta-reasoning beat direct control in every model-benchmark pairing. Averaged across the three models, the gains on the three reasoning benchmarks were:

ProgramBench is where the paper's headline figures come from. It was compared not only against matched direct control but against external production coding agents (external systems were evaluated only on their native model, so some cells are not applicable).

The gap widens as the budget grows. With GPT-5.5, raising the ProgramBench allowance from 400 to 1200 calls lifts meta-reasoning from 64.1% to 71.5%, while direct control stays near 64% throughout. But at low budgets the ranking flips: with Opus 4.8 at 400 calls, meta-reasoning scores 56.6% against direct control's 62.7%, then overtakes it at 1200 calls, 67.2% versus 65.3%. The extra deliberation needs enough budget to pay for itself.

Meta-reasoning also produces more worker outputs and more reuse between them. On IMO ProofBench-Advanced with Gemini 3.1 Pro, worker artifacts roughly double while recorded dependencies rise by an order of magnitude (Figure 4). Coverage — the share of runs where a correct candidate appears at all — improves in most settings too: +20 points on IMO ProofBench-Advanced and +10 on LongCoT-mini for Gemini 3.1 Pro (Figure 6). The controller's own quality judgments frequently outperformed worker self-confidence: in the same model-benchmark pairing, worker-confidence discriminability (Type-2 AUC) was 0.55 — close to chance — versus 0.88 for the controller's verdict.

Credibility Assessment

Three things earn trust. The core comparison holds workers, model, and budget fixed, so the effect of control is not confounded with model capability. The direction is consistent — meta-reasoning never loses at the main budget across four benchmarks and three models. And the authors devote an appendix to the "scope of the evidence," explicitly stating that there is no component-level ablation and that call-count comparisons are not token- or latency-matched — a credit to the paper's own candor.

The caveats are just as real. This is a pre-peer-review preprint, every author is both designer and evaluator of the proposed method, and the competitor comparisons are not independent replications. The proof benchmark has only 30 problems, leaving wide uncertainty. Contradicting evidence sits inside the paper itself: the authors report that meta-reasoning underperformed direct control at low budgets (56.6% vs 62.7% for Opus 4.8 at 400 calls) and on the LongCoT-mini chess subset. Any simplification to "meta-reasoning wins everywhere" is contradicted by the paper's own data.

Related Academic Work

These related works were identified from the paper's own bibliography.

• De Sabbata et al. (2024). Rational metareasoning for large language models — prior work. Formalizes metacognition as estimating the value of remaining computation to pick the next action. This paper extends that frame from a single model call to a full multi-turn agent run, instantiating the controller as a four-stage agentic process.
• Cao et al. (2026). LLMs Know When They Know, But Do Not Act On It: A Metacognitive Harness for Test-Time Scaling — prior work. Studies a single-call metacognitive trigger: LLMs often know internally whether an answer is right but fail to turn that signal into action (retry, tool call). This paper widens the same problem to an entire multi-turn agent run, making the signal-to-action mechanism itself a separate agent (the controller).
• Zhang et al. (2025a). Recursive Language Models — comparison baseline. RLM, which holds context in a code-editable variable rather than a transcript, is also an external baseline in this paper. Per this paper's own Table 1, on ARC-AGI-2 with Gemini 3.1 Pro, RLM scored 76.7% versus meta-reasoning's 84.2%; on LongCoT-mini with GPT-5.5, RLM scored 63.3% versus meta-reasoning's 65.1% (figures are from this paper's reproduction of RLM; RLM's own paper was not independently checked).

All three are works this paper cites directly, either building on their framing or testing against them empirically. For the operational side of the build-vs-rent decision on agent harnesses, see No Extra Fee to Rent a Harness.

Reviewer's Judgement

First, the number that matters most operationally here is not 71.5% but the low-budget reversal (56.6% vs 62.7%). Adopting meta-reasoning is not a free improvement — it is an investment whose payoff depends on budget size, and bolting it onto small-call-budget tasks can leave you worse off.

Second, the comparisons against Codex and Claude Code deserve caution as headline numbers. The matched direct-control comparison (+7.8 points) is the more trustworthy figure for the effect of control itself; the competitor comparisons are best treated as reference points from an environment the same author team reconstructed, not independent benchmarks.

Third, the finding that the controller's judgment (AUC 0.88) beats worker self-confidence (0.55) reads as a signal that many agent pipelines are under-investing in a separate judgment stage, relying instead on worker self-reported confidence alone. A single added verification step could buy a large reliability gain relatively cheaply.

Putting It to Work

• Gate adoption by budget size — skip meta-reasoning, or validate a lightweight variant separately, on tasks with small call budgets; it was a net loss at low budgets (Opus 4.8, 400 calls: 56.6% vs 62.7%).
• Split control into four stages — run assess, propose, evaluate, and dispatch as separate agentic steps rather than one call, letting each draw on its own share of the inference budget.
• Manage history as compact state — instead of carrying the full accumulated conversation as context, rewrite a state of a few thousand to tens of thousands of characters each turn to avoid context blowup on long runs.
• Add a judgment stage beyond worker confidence — don't accept results on worker self-confidence alone; an independent judgment stage can achieve better discriminability.
• Prefer matched internal A/B over external comparisons — before adopting, run your own same-worker, same-budget comparison of control strategies rather than leaning on comparisons against competitor systems.

Conclusion

The contribution here is not a new model but a design that separates "deciding what to do" from doing the work, and turns that decision into its own agentic process. The matched comparison is consistently favorable and the gap widens with budget, which is persuasive — but the low-budget reversal, the chess-subset exception, the missing ablations, and the lack of independent replication for the competitor comparisons all mean the numbers call for your own comparison before adoption. For the build-vs-rent decision on agent harnesses, see No Extra Fee to Rent a Harness.

References

• Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and figures (via snapshot)
• Rational metareasoning for large language models (De Sabbata et al., 2024) — related work
• LLMs Know When They Know, But Do Not Act On It (Cao et al., 2026) — related work
• Recursive Language Models (Zhang et al., 2025a) — related work
• No Extra Fee to Rent a Harness — sunny34.com blog