More Thinking Didn't Help on GAIA: Reviewing Adaptive Reasoning in Agentic AI

Source Document

Md Jueal Mia, M. Hadi Amini, "Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI", arXiv:2608.26442 [cs.AI], submitted 2026-08-26, DOI 10.48550/arXiv.2608.26442. Affiliation: Knight Foundation School of Computing and Information Sciences, SOLID Lab, Florida International University (USA).

This is a preprint without peer review, which the authors themselves label a "preliminary study." Funding comes from the U.S. DOT's National Center for Transportation Cybersecurity and Resiliency (TraCR), the NAIRR Pilot, and AWS CloudBank (NSF-backed) — all federal research funding. The three models tested (Qwen3.5-4B, Llama-3.1-8B-Instruct, Phi-4-reasoning) are not products of the authors' institution or a sponsor, so the conflict-of-interest risk of favoring a particular vendor's model is low. Because this session's egress was blocked (confirmed by three retries at 10-minute intervals), the full text was cross-checked against a snapshot taken 2026-08-30T22:03:27Z rather than a live WebFetch.

Study Overview

The question is direct: what actually changes when an agent reasons too much (over-reasoning) or too little (under-reasoning)? The authors fixed Qwen3.5-4B as the tool router in a LangGraph-based agent framework and swapped the final-response model across Qwen3.5-4B, Llama-3.1-8B-Instruct, and Phi-4-reasoning, evaluating each on MATH-500 (500 math problems) and the GAIA validation set (165 general-assistant tasks). Deterministic decoding (temperature 0) and a 4,096-token generation cap were applied uniformly, and GPT-4.1 independently classified every trajectory as over-, under-, or adequate-reasoning.

Key Results

Accuracy and cost told different stories on the two benchmarks. On the structured math problems (MATH-500), more reasoning bought real accuracy at a steep price; on the real-world tasks (GAIA), more reasoning bought almost nothing.

On MATH-500, Phi-4-reasoning edges out Qwen3.5-4B by 4.8 points (92.40% vs. 87.60%), but at 2.6× the final-answer time, 1.6× the output tokens, and a token-limit-hit count that jumps from 190 to 482. On GAIA, the relationship flips: Phi-4-reasoning's accuracy is actually lower than Qwen3.5-4B's (11.52% vs. 12.12%). More reasoning did not translate into more correct answers.

The dominant failure mode also swaps between benchmarks. The rates below are the paper's Table 2 counts divided by sample size, re-checked against the reported percentages.

Over-reasoning dominates on MATH-500; under-reasoning dominates on GAIA. Llama-3.1-8B-Instruct is classified as under-reasoning on 71.52% of GAIA trajectories (118 of 165) — it answered before gathering enough evidence. Tool-call frequency diverges too: an average of 0.39 calls per sample on MATH-500 versus 0.93 on GAIA, yet the extra tool use never lifted GAIA accuracy above 12%.

Credibility Assessment

Three things earn trust: the tool router is held fixed at Qwen3.5-4B while only the response model varies, giving a controlled comparison; reasoning classification (GPT-4.1) and correctness judging are performed independently, so the two metrics don't contaminate each other; and the authors commit to releasing code, prompts and per-sample evaluation outputs.

The caveats are equally clear. By the authors' own admission this is a preliminary study limited to one tool-routing model and three response models, and it does not generalize to other agent architectures. Monetary cost, GPU utilization and throughput were not measured. There is no ablation over the fixed 4,096-token generation cap and no repeated trials, so it is unclear how much of the reported averages reflects run-to-run variance. On the other side, prior work (Tran and Kiela, 2026) reported that a single agent can outperform multi-agent systems under an equal thinking-token budget, which means this paper's pushback against "more reasoning and coordination is always better" is not an isolated finding.

Reviewer's Judgement

First, reading MATH-500 and GAIA side by side, the payoff from "make it think more" depends heavily on task type. On structured math problems, extending reasoning genuinely raises accuracy but at sharply rising cost; on multi-step real-world tasks, nearly tripling reasoning time slightly lowered accuracy instead. Expanding the reasoning budget is not a prescription that applies regardless of task type.

Second, the relationship between hitting the token limit and accuracy differs by benchmark in a way the tables show more clearly than the prose does. On MATH-500, hit-accuracy (73.68% for Qwen) is well below no-hit-accuracy (96.13%), so "budget exhaustion equals failure" holds. On GAIA, Qwen3.5-4B's hit-accuracy (13.33%) is actually higher than its no-hit-accuracy (11.11%) — with comparably sized groups (75 vs. 90 samples). On complex tasks, running out of tokens looks more like a symptom of task difficulty than a cause of failure.

Third, Llama-3.1-8B-Instruct's 71.52% under-reasoning rate on GAIA is, in this reviewer's view, the most operationally useful signal in the paper. The fix for a weaker model on complex agentic tasks is probably not to let it think longer, but to add a separate verification step that catches premature termination before enough evidence has been gathered.

Putting It to Work

• Split reasoning budgets by task type — extend reasoning for structured, single-answer tasks, but for multi-step real-world tasks, check evidence-gathering and tool-call design before adding more thinking time.
• Treat the token-limit-hit rate as an alarm — when a model's hit rate crosses roughly 90% (Phi-4-reasoning hit 96.4%), read it as a signal to adjust the prompt or the max length, not to add more budget.
• Instrument the under-reasoning rate separately — track incomplete or prematurely-terminated outputs, not just accuracy, so that cost-cutting reasoning reductions don't quietly increase failures.
• Measure the accuracy-latency curve before swapping models — a model change that doubles latency for a few points of accuracy should be checked against your own SLA and cost targets first.
• Prefer a verification role over a bigger reasoning budget on complex tasks — since more reasoning shows clear diminishing (or negative) returns on multi-step real-world work, a dedicated audit step is worth testing before extending think time.

Conclusion

This preliminary study puts numbers behind the claim that "make it think more" does not work regardless of task type. On structured math problems, extending reasoning raises accuracy; on multi-step real-world tasks, the same prescription only raises cost while accuracy stays flat or drops. The lack of repeated trials and a cost ablation is a real limitation, but the pattern held consistently across three models and two benchmarks, which is enough to justify rethinking how reasoning budgets are allocated by task type in production. For how to measure cost against accuracy, see Performance Checks Beyond Accuracy.

References

• Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI — arXiv abstract (source)
• Full HTML text of the same paper — used to verify Tables 1 and 2
• Performance Checks Beyond Accuracy — sunny34.com blog