Source Document
Ziyu Ma, Hailang Huang, Shun Zou, Yong Wang, Shidong Yang, Yiming Hu, Fei Wei, XiangXiang Chu, "LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks", arXiv:2608.01964 [cs.CV], submitted 2026-08-03, 29 pages, DOI 10.48550/arXiv.2608.01964. Affiliation: DreamX Team, Alibaba Group. Code and project page released (github.com/AMAP-ML/LongHorizon-Harness).
This is a preprint without peer review. Its primary arXiv class is Computer Vision (cs.CV), which sits oddly with the content — this is agent-harness engineering, and the classification most likely follows from the GUI benchmark (OSWorld) in the evaluation set rather than from the contribution itself. The conflict of interest is unambiguous: every author works for Alibaba Group, and the main experimental model, Qwen 3.7-Plus, is that company's own. Releasing code and a project page counts in the other direction, since it leaves a path to external replication.
Study Overview
The problem will be familiar to anyone who has operated a long-running agent. Conventional harnesses hold execution history, task state and completion judgement inside a single growing context. State becomes hard to track, and — more damagingly — an incorrect self-assessment ("that step is done") propagates into every later decision.
The authors' answer is to recast execution as task-state management. The Manage-Execute-Audit (MEA) loop splits the work across three roles. The manager reads the current task state and defines one subtask with its dependencies, constraints and acceptance criteria. The executor runs each round as a fresh, budget-bounded episode containing only the information supplied for that round. The auditor decides completion by independently comparing the resulting environment state, rather than trusting the executor's report.
The most contestable design decision is what happens between rounds: the executor's raw interaction trajectory is discarded, and only the task state and its supporting audit reports persist. That runs against the usual instinct to preserve memory by growing the context. Evaluation covers WeaveBench (114 tasks), Terminal-Bench 2.1, and OSWorld 2.0 (the full 108-task benchmark plus a 34-task Opus subset).
Key Results
The headline comparisons hold the underlying model fixed (Qwen 3.7-Plus) and swap the harness. Metric definitions differ by benchmark and should be read with their bases: WeaveBench PassRate is full completion, Overall is a composite that credits partial progress, and OSWorld's Binary and Partial are full-success and partial-success rates respectively.
| Benchmark (metric) | Baseline harness | LongHorizon-Harness | Difference |
|---|---|---|---|
| WeaveBench, 114 tasks — PassRate | 51.8% | 80.7% | +28.9 pts |
| WeaveBench — Overall score | 0.702 | 0.835 | +0.133 |
| Terminal-Bench 2.1 — success rate | 69.7% | 77.2% | +7.5 pts |
| OSWorld 2.0, 108 tasks — Binary | 2.8% | 8.3% | +5.5 pts |
| OSWorld 2.0, 108 tasks — Partial | 21.5% | 35.2% | +13.7 pts |
On the 34-task Opus subset of OSWorld the same comparison was repeated with Claude Opus 4.7 as the underlying model, giving 20.6% → 35.3% on Binary and 55.8% → 66.9% on Partial. That cross-model check — showing the effect is not confined to the authors' own model — is, in this reviewer's reading, the most persuasive part of the paper.
Cost moved in different directions across benchmarks, and the bases differ enough that the numbers must not be lined up naively.
| Benchmark | Token consumption change (basis) | Token share by role (manager/executor/auditor) |
|---|---|---|
| WeaveBench | 2.3× increase (total tokens) | 2.8% / 77.8% / 19.4% |
| OSWorld 2.0 | 3.6× increase (output tokens only) | 2.0% / 73.2% / 24.8% |
| Terminal-Bench 2.1 | 24% decrease | 8.1% / 53.8% / 38.1% |
The OSWorld 3.6× is stated on output tokens because of limitations in the official data, so it is not comparable side by side with WeaveBench's 2.3× on total tokens. The paper is also explicit about where the method does not help: gains are small when a task is dominated by a single-model capability such as visual perception, mathematical reasoning, coding or algorithm design, and concentrate on tasks that require preserving, inspecting and revising multiple dependent environment states.
Credibility Assessment
Three things earn trust. The core comparison holds the model constant across harnesses, so harness effect and model effect do not blur together. The direction is consistent across three benchmarks of quite different character — multi-domain tasks, terminal work and GUI desktop control. And the code and project page are public, leaving a route to external verification.
The caveats are just as clear. Every author is employed by the company whose model anchors the experiments, and the cross-model check rests on 34 tasks. There is no ablation removing individual components, so the improvement cannot be attributed to the manager's decomposition, the executor's fresh context or the auditor's independent verification. The Terminal-Bench results are presented alongside external leaderboard entries for other models (GPT-5.6, Claude Opus 4.8), which run under different conditions and are not a like-for-like comparison with this system. There is no dedicated limitations section, and no run counts or variance are reported, so it is impossible to say how far the tabulated gaps exceed run-to-run noise. The work has not been peer reviewed.
Reviewer's Judgement
First, the most counterintuitive — and therefore most valuable — result is not 80.7% but the Terminal-Bench pairing: success up 7.5 points while token consumption fell 24%. That is a counterexample to the assumption that adding a verification stage necessarily adds cost, and the likely cause is not the audit itself but the discard rule. If verbose logs of failed attempts never carry into the next round, you can verify one more time and still spend less overall.
Second, the OSWorld figures invite misreading when quoted as multiples. Going from 2.8% to 8.3% is roughly a threefold gain, and still means nine runs in ten fall short of full success. The parallel rise in partial success to 35.2% shows real progress was made, but this is not evidence for handing desktop GUI automation to an unattended agent. Separating relative improvement from absolute level matters more here than in most tables.
Third, the per-role token split is an operational metric the paper supplies almost incidentally. Knowing that the auditor consumes between 19.4% and 38.1% of all tokens puts a number on the "verification tax" for the first time. Teams debating whether to add a verification stage to their own pipeline can start from that band instead of guessing.
Fourth, the missing ablation bears directly on adoption. Implementing all three roles is not cheap, and without knowing where the gain comes from there is no way to predict what a reduced version — say, adding only an auditor — would deliver. Anyone adopting this would be wise to run their own auditor-on/auditor-off comparison on their own tasks first.
Putting It to Work
- Move state out of the context — keep task state as a structured object rather than as conversation history, and pass only state plus verification results between rounds.
- Adopt a trajectory discard policy — stop auto-including raw logs of failed attempts in the next round's prompt. Terminal-Bench's 24% token reduction is what that rule is worth.
- Make completion judgement independent — verify against environment state (files, database rows, screen state) in a separate role instead of accepting the agent's self-report.
- Account for tokens by role — track planning, execution and verification spend separately, and revisit the design if the verification tax drifts outside the 20–40% band.
- Choose by task type — apply the harness to stateful multi-step work only; leave it off tasks decided by a single capability such as perception, mathematics or coding.
Conclusion
The contribution here is not a new model but an inversion in how context is handled: keep the state and the verification record, throw the rest away. That choice produced consistent improvement across three benchmarks and, on one of them, cost savings as well. Given the absent ablation, the missing variance reporting and a clear conflict of interest, the numbers are best treated as a direction rather than a specification — run your own comparison before adopting. For the operational side of state handover, see Agent Handoff Playbook.