Source Document
Zhi Hong, Qianlong Wang, Hongchen Wei, Aidan Wu, Daohong Ying, Mingze Kong, Bei Liu, Chong Luo, "HarnessFlow: MCTS-Guided Harness Optimization for LLM Agents", arXiv:2611.06254 [cs.AI], submitted 2026-11-04, DOI 10.48550/arXiv.2611.06254. Affiliations: South China University of Technology, Fudan University, Wuhan University, Cornell University, UC San Diego, The Chinese University of Hong Kong (Shenzhen), and Microsoft Research Asia.
This is an arXiv preprint that has not been peer reviewed. It is an academia-industry collaboration: most of the eight authors are at Chinese universities (South China University of Technology, Fudan, Wuhan, CUHK Shenzhen), with the rest at U.S. universities (Cornell, UC San Diego) and Microsoft Research Asia. Two corresponding authors (Bei Liu, Chong Luo) are MSRA staff, and several other authors note the work was done during an MSRA internship — a commercial interest in agent-harness automation technology that can't be ruled out. That said, the comparison set includes competitors' products (Claude Code, Codex) under identical conditions rather than being designed to flatter an in-house tool. No separate funding or conflict-of-interest disclosure was found. The full text was checked against a session snapshot captured 2026-11-09T22:07:57Z (UTC) — direct WebFetch access stayed blocked by a session egress proxy fault across retries spaced more than ten minutes apart.
Study Overview
The question is direct: if agent performance depends on both the underlying model and the harness — the code that governs tool and environment interaction — can harness design be automatically searched the way model training is? The authors formalize this as optimizing a complete executable harness under a limited candidate-evaluation budget.
The method combines two pieces. A coding agent (Codex 0.154.0 running GPT-5.5 at the highest reasoning effort) rewrites the full harness code using the archived search history, execution feedback, and a reusable "Harness Component Library." These candidates are organized in a refinement tree navigated by Monte Carlo Tree Search (MCTS), with a selection policy (ε=0.35, mixing score-based exploitation with preservation of under-explored branches) choosing which candidate to refine next. Every harness runs on the backbone model GPT-5.4-mini at medium reasoning effort, with a budget of 15 candidate attempts per benchmark and 5 repeated runs (mean ± standard deviation reported) applied identically to Vanilla, Meta-Harness, and HarnessFlow.
Key Results
Across six benchmarks — TAU2-bench, AppWorld, WebShop, BrowseComp-Plus (BC+), SWE-bench Verified (SWE-V), and BigCodeBench-Hard (BCB-H) — HarnessFlow averaged 69.48%, the highest of every method compared (this review independently re-summed the six per-benchmark scores and confirmed the average). That's +10.89 points over Vanilla (the initial harness, 58.59%), +4.66 points over the production agent Codex (64.82%), and +3.95 points over Meta-Harness (65.53%), a competing automation method using the same 15-candidate budget. The largest gaps over Meta-Harness were on TAU2-bench (+7.21pp) and SWE-bench Verified (+6.60pp).
| Method | TAU2 | AppWorld | WebShop | BC+ | SWE-V | BCB-H | Avg. |
|---|---|---|---|---|---|---|---|
| Vanilla (initial harness) | 63.98 | 72.00 | 47.79 | 72.40 | 67.10 | 28.30 | 58.59 |
| Codex (production) | 75.91 | 73.24 | 66.15 | 74.70 | 68.70 | 30.20 | 64.82 |
| Meta-Harness | 74.62 | 76.76 | 66.33 | 76.60 | 68.80 | 30.10 | 65.53 |
| HarnessFlow | 81.83 | 80.76 | 69.49 | 76.90 | 75.40 | 32.50 | 69.48 |
Each column uses the benchmark's own official metric (success rate, score, resolution rate, Pass@1, or accuracy), so read within a column rather than across benchmarks (Table 1 in the source). The gains aren't limited to accuracy — average execution cost per task was $0.232 for HarnessFlow, below all four hand-engineered production harnesses ($0.398–$0.561) and below Meta-Harness ($0.351) too. HarnessFlow sat on the score-cost Pareto frontier on every one of the six benchmarks, meaning a more accurate harness here wasn't necessarily a more expensive one.
| Method | Avg. score (%, 6 benchmarks) | Avg. cost (USD/task) |
|---|---|---|
| Vanilla (initial harness) | 58.59 | 0.451 |
| Claude Code | 61.71 | 0.446 |
| OpenHands | 62.61 | 0.398 |
| Hermes | 64.28 | 0.555 |
| Codex | 64.82 | 0.561 |
| Meta-Harness | 65.53 | 0.351 |
| HarnessFlow | 69.48 | 0.232 |
Search itself was sample-efficient: just three candidate evaluations per benchmark reached a mean best validation score of 71.5%, rising to 74.2% by round 15. The ablations confirm both design pieces actually matter. Removing the reusable component library lowered the three-benchmark average (TAU2, AppWorld, BCB-H) from 65.03% to 63.39%; replacing the exploitation-exploration mix (ε=0.35, averaging 79.33%) with pure exploitation (ε=0, 73.93%) or pure exploration (ε=1, 73.65%) dropped the average by 5.40 and 5.68 points respectively. These figures were cross-checked twice against Tables 1, 2, 3, and 8 in the source, with every average independently recomputed and matched.
Credibility Assessment
Grounds for trust: every method was evaluated under the same backbone model, tool access, per-task resource limits, and 5-run repetition (mean ± SD), controlling for factors other than the harness design itself. The authors included four hand-engineered production agents (Claude Code, Codex, OpenHands, Hermes) in the comparison set — a harder bar than comparing only against other automation methods. They also showed the advantage holds with a stronger executor model (GPT-5.4: three-benchmark average 83.58%, +2.73pp over Claude Code) and when starting search from a different agent architecture (OpenHands), supporting that the result isn't tied to one model or one initial harness.
Caveats: this is a pre-peer-review preprint, and two corresponding authors are MSRA staff — a disclosed-but-real interest. The search only applies to tasks with an automatic grader, and the evaluation budget itself (15 attempts × 5 repetitions × 6 benchmarks) is a substantial one-time cost that must be amortized post-deployment to justify the lower per-task cost claim. The harness is organized as a single-parent refinement tree; combining mechanisms discovered on separate branches via a graph search is left as future work by the authors themselves. No prior work reporting results that directly contradict HarnessFlow's core claim was found, though a contemporaneous study solving the same problem with an opposite design is discussed below.
Related Research
- Lee et al. (2026). Meta-Harness: End-to-End Optimization of Model Harnesses — the direct baseline in this paper (Tables 1 and 8 show HarnessFlow ahead by +3.95pp average score and about 34% lower average cost). It refines harnesses iteratively from execution feedback but does not maintain a persistent search tree — the key structural difference. Prior-work relation.
- Zhang et al. (2025, ICLR). AFlow: Automating Agentic Workflow Generation — the closest search-based precedent, explicitly named as such in HarnessFlow's own text. AFlow applies MCTS to code-represented candidates but only rewires a fixed vocabulary of workflow operators, generating each candidate with a single model call from a textual summary; HarnessFlow's coding-agent proposer instead edits the full harness — execution loop, state handling, and tool interfaces included — across files. Different layer of the search space. Prior-work relation.
- Zhang et al. (2026). Self-Harness: Harnesses That Improve Themselves — tackles the same "automatic harness improvement" problem with the opposite design: no external coding agent or search tree, just the agent itself mining its own failure traces and validating small edits through regression testing. It reportedly raised held-out pass rates on Terminal-Bench-2.0 across three base models (confirmed independently via WebSearch only, not direct access to the source, so that figure is not carried into this review's Key Results). It was not benchmarked head-to-head against HarnessFlow. Alternative-design extension relation.
Meta-Harness and AFlow are both directly cited and discussed in HarnessFlow's own text; their bibliographic details were cross-checked via WebSearch. Self-Harness was confirmed via WebSearch alone, so its numbers are used only to describe the relationship, not folded into this review's verified results. No prior work was found that reports results directly contradicting HarnessFlow's core claim.
Reviewer's Judgement
First, the number worth remembering here isn't 69.48% but $0.232. The claim that automated search finds more accurate harnesses than humans isn't new, but a result where the same search also lowers cost cuts against the usual assumption that automation trades cost for quality. That said, this comparison excludes the one-time search evaluation cost (15 attempts × 5 repetitions × 6 benchmarks is not a trivial amount of model usage), and the paper should have foregrounded that distinction more clearly.
Second, the fairness of the Meta-Harness comparison is what this review trusts most. Same coding agent, same initial harness, same 15-candidate budget — only the search structure (persistent tree vs. iterative refinement) differs, so the +3.95pp gap can reasonably be attributed to that design choice. One reservation: both methods rely on this paper's own reimplementation, so results could differ from an evaluation run by Meta-Harness's original authors.
Putting It to Work
- Treat the harness as versioned code, not config — managing the full execution harness as a swappable unit, rather than a settings file, lets both automated search and manual improvement reuse the same version-compare-and-rollback infrastructure.
- Make the evaluation budget an explicit design variable — a search budget like 15 × 5 repetitions is one-time but not free. Estimate post-deployment call volume first to check whether the search cost actually amortizes.
- Build a reusable component library before you search — the library alone was worth 1.64pp on the three-benchmark average. Collecting proven fragments from past harnesses (retry logic, compaction strategies) into reusable form is worth investing in before the search algorithm itself.
- Benchmark automation against real hand-built production harnesses — dropping tools people actually use (Claude Code, Codex) from the comparison set makes an "optimized" claim hollow. Apply the same bar to internal tool selection.
- Match search structure to task shape — a persistent tree that revisits and builds on earlier branches favored tasks with non-linear, cumulative improvement (SWE-V-style stepwise refinement), a cue worth weighing when choosing a search strategy.
Conclusion
HarnessFlow's contribution isn't the single claim that "harness design is better treated as a search problem than a manual one" — it's validating that claim against four hand-engineered production agents and a competing automation method under identical conditions, while reporting cost alongside accuracy. The headline numbers (69.48% average, $0.232 average cost) are solid evidence that search-based harness optimization improves both performance and cost on these six benchmarks, but this is a pre-peer-review preprint with a substantial evaluation budget and MSRA-affiliated corresponding authors, both worth weighing. What a self-evolving harness costs in safety is examined further in our surface-by-surface promotion gate for self-evolving agent harnesses.
References
- HarnessFlow: MCTS-Guided Harness Optimization for LLM Agents — arXiv abstract (source)
- Full HTML text of the same paper — used for table/figure verification
- Surface-by-Surface Promotion Gates for Self-Evolving Agent Harnesses — sunny34.com blog
Ask AI about this review
The assistant has read this review and the numbers it verified. Ask anything — it answers from the text and says so when something isn't in it.
Loading the chat…