GPT-5.5 Still Misses One in Three: Reviewing Wuying-Browser-Agent

Source Document

AIMAE Team (Alibaba Cloud AI Model Application & Engineering Team, End-User Intelligent Computing BU), Tianxiang Chen, Yan Cheng, Zhangye Han, Xiaowei Li, Chang Liu, Cheng Liu, Zhongqiang Ma, et al., "Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agents", arXiv:2608.17319 [cs.AI], submitted 2026-08-18, license CC BY-NC-SA 4.0, DOI 10.48550/arXiv.2608.17319. Affiliation: Alibaba Cloud.

This is a preprint without peer review. The full text was checked against a source snapshot collected 2026-08-20T21:48:51Z (via GitHub Actions) — this session's own WebFetch access was blocked outright (EGRESS_BLOCKED) across three retries at 10-minute intervals, so we followed the snapshot fallback procedure. The conflict of interest is unambiguous: every author works for Alibaba Cloud, and the evaluated model (Wuying-Browser-Agent) is the company's own product. On the other side of the ledger, the authors release model weights at 4B, 9B, and 27B scales, which leaves a path to external replication.

Study Overview

The research question is what has to change together to bring browser agents up to real deployment conditions. The authors argue that agents trained on short, clean demonstrations mismatch real deployment, where an agent must sustain dozens of decisions including error recovery. They address this with a pipeline that co-designs execution, supervised training, reinforcement learning, and evaluation.

A structured browser harness standardizes the execution layer with 24 atomic actions across navigation, interaction, extraction, file handling, and flow control. Reflection and UI-specialized Curriculum SFT (RUIC-SFT) trains not only on successful trajectories but on recovery trajectories and complex-UI interaction data through a three-phase curriculum. Divergence-Aware Online GRPO (DAO-GRPO) eases long-horizon credit assignment through potential-based reward shaping and divergence-aware step weighting. For evaluation, the authors introduce BrowserBench, a benchmark they built themselves: 350 real Chinese and English web tasks across 254 sites, averaging 37.9 steps — filling a gap in long-horizon, bilingual coverage that existing benchmarks underrepresent.

Key Results

Success rates on WebVoyager, Online-Mind2Web, and BrowserBench (all under a 100-step budget) for the key models:

Wuying-Browser-Agent-27B tops the open-source models at a 70.8% average, edging out the previous best open model Qwen3.8-Max (70.3%) and coming close to closed-source GPT-5 (69.6%), though it still trails the strongest closed model, GPT-5.5 (75.7%). On BrowserBench — the benchmark with the longest average step count of the three — even GPT-5.5 manages only 67.4%. Subtracting from 100 puts the miss rate at 32.6 points, roughly one task in three left unfinished, matching the paper's own statement that "even the strongest proprietary agent we evaluate fails on roughly one-third of these tasks."

Online RL (DAO-GRPO) contributes more as tasks get harder and longer. Adding the paper's stated deltas to the RUIC-SFT baseline, on the 9B backbone and BrowserBench:

The figures in parentheses are DAO-GRPO's added contribution over RUIC-SFT, stated directly in the paper. The same pattern holds by trajectory length: DAO-GRPO adds only +1.1 points on 15–25 step tasks (where RUIC-SFT alone already gained +22.2 points), while on tasks exceeding 50 steps it adds +13.4 points (13.3% → 26.7%) — the largest gain of any bin.

Credibility Assessment

Three things earn trust. First, the ranking direction is consistent across three benchmarks of quite different character — the English-centric WebVoyager, the temporally-shifting Online-Mind2Web, and the bilingual real-web BrowserBench. Second, the paper breaks results down by difficulty and trajectory length, exposing variance that an aggregate score would hide. Third, the LLM-based judges used for reward shaping (the divergence estimator, the progress estimator) and the evaluation judge were each validated against human-labeled samples (100–500 cases), with the numbers published (61% exact match, 78% within one step, Spearman correlation of 0.81 between human and estimator progress ratings, and more).

The caveats are just as clear. Every author works for Alibaba Cloud, and the evaluated model is the company's own product — a plain commercial interest. The same authors designed and scored the new headline benchmark, BrowserBench, so the possibility that it was constructed to favor their own model cannot be fully ruled out (though a similar top-tier ranking reproduces on the externally built WebVoyager and Online-Mind2Web). All three reliability checks above are self-conducted; there is no external replication yet. As counter-evidence, the paper itself shows the closed-source GPT-5.5 still averaging 75.7%, ahead of the 27B model's 70.8% — the "open-source state of the art" claim does not hold once closed models are included. No run counts or variance are reported, so it is hard to judge how far the tabulated gaps exceed run-to-run noise.

Reviewer's Judgement

First, the more operationally important number is not the "open-source #1" headline but BrowserBench's absolute success rate of 65.1%. That means roughly six tasks in ten complete — not yet a basis for unattended, fully delegated long-horizon browser work. Separating relative rank (first) from absolute level (65.1%) matters for any real deployment decision.

Second, the fact that DAO-GRPO's contribution concentrates on hard and long tasks (+12.4 points on hard, +13.4 points beyond 50 steps) reads as a meaningful signal that online RL here is actually patching weaknesses rather than sharpening what the model already does well. The near-flat +1.0-point gain on easy tasks is partly a ceiling effect, but it also shows where the method is spending its budget.

Third, having the same organization build the benchmark and top its leaderboard is a structural risk worth flagging, but this paper mitigates a good part of it: the ranking largely holds on third-party benchmarks (WebVoyager, Online-Mind2Web), and the judge-reliability numbers against human labels are disclosed. That mitigation is a reduction in suspicion, not an acquittal — the fact that the benchmark's designer and the top model's author are the same team still warrants a careful read.

Putting It to Work

• Evaluate by length bin, not just the average — when adopting or comparing browser agents, split success rate by trajectory-length bin (e.g., under 25 steps vs. over 50). Looking only at short-task scores hides long-task weaknesses.
• Schedule recovery data late in the curriculum — the ablation shows that mixing recovery/reflection data in aggressively from the start hurts performance (35.1% vs. 38.0% for the curriculum). Treat data ordering as an experimental variable in your own fine-tuning pipeline.
• Audit LLM judges separately — if you use an LLM to judge reward or progress, validate it against a small human-labeled sample the way this paper does, and report agreement rates and correlations as numbers, not "it looks plausible."
• Count budget overruns as failures explicitly — the evaluation rule of treating incompletion within a 100-step budget as failure reflects real deployment interaction cost. Set an explicit step budget in your own protocol and count overruns as failures.
• Pair a self-built benchmark with third-party ones — if you construct a new evaluation metric, don't claim superiority on it alone; report whether the same ranking reproduces on externally built benchmarks.

Conclusion

Wuying-Browser-Agent's contribution is chaining harness, SFT, online RL, and evaluation into one pipeline that lifts the open-source browser-agent average. But the improvement still falls short in absolute terms: even the top model misses one task in three on the authors' own long-horizon benchmark, and still trails closed-source GPT-5.5. Given the conflict of interest — every author works for the company behind the evaluated model — and the self-designed benchmark, these numbers are best read as a directional signal ("the gap widens with task length") rather than a specification, worth re-checking on your own tasks before adoption. For the operational side of running browser agents in production, see The OpenAI Atlas Shutdown Browser-Agent Migration Checklist.

References

• Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agents — arXiv abstract (source)
• Full HTML text of the same paper — used to verify the results tables
• The OpenAI Atlas Shutdown Browser-Agent Migration Checklist — sunny34.com blog