35% of Finished Products Still Fail: A ViBench 2.0 Benchmark Review

The Paper

Daniel Furman, Preeya Kirani, Ibrahim Sheikh, Zhen Li, Peter Zhong, Michele Catasta, "ViBench 2.0: A Benchmark for Long-Horizon Software Building", arXiv:2611.05620 [cs.AI], submitted 2026-11-03. Affiliation: Replit (all authors). No peer-reviewed venue listed (arXiv preprint). Code and data: github.com/ViBench/vibench-public.

This is a preprint that has not gone through peer review, and all six authors are Replit employees. ViBench 2.0 is the internal tool Replit uses to decide which models power its own agentic platform, and the paper itself states that the inference provider Fireworks AI has folded this benchmark into its paid "Specialized Intelligence Index" product. In other words, the benchmark's creator uses the results to make its own model-selection decisions, and a third-party commercial service also sells access to the same results. That said, the authors disclose their partnerships with some model providers and separately audited the grading model for self-bias, both covered under Credibility Assessment below.

Study Overview

The research question: can a coding agent build an entire product from an empty project (greenfield), across dozens of feature requests, and have it still work at the end? Existing coding benchmarks typically score a single, fully specified task against an existing codebase (the SWE-bench family), or, even when multi-turn, grade each step as it completes. ViBench 2.0 instead has an agent grow 17 full-stack products (8 public: modeled on Amazon, Asana, Discord, Figma, GitHub, Google Docs, Jira, and Uber; 9 held out) from an empty project, delivering 9 to 18 feature requests per product (241 total) in a single conversation. Grading happens only after the last request, when a browser-operating grading agent runs 544 test plans (85% of which exercise multiple features together) against the finished product — one failed plan fails the whole build. Across all builders, the same seeding and grading agent (Opus 5.5, medium reasoning effort) was used to keep conditions constant.

Key Results

All figures below are read directly from the paper's tables. "Build pass@1" is an all-or-nothing metric that requires the whole product to pass every plan, while "test-plan pass rate" is the pass rate on individual plans — these use different bases and are kept separate in the table and chart.

Source: paper Table 1 (± are 95% intervals; near-0% rows use the exact binomial range). Figures are from 68 independent builds per model (17 products x 4 builds each). The top three models (Sol, Astra, Opus) pass 93.7-98.0% of test plans, yet their build pass@1 lands at only 48.5-64.7% — of their 84 seeded failed builds, 42 missed just one plan and 62 missed at most two, meaning they got nearly everything right and still failed the all-or-nothing bar. All three open-weight models (DeepSeek, GLM, Kimi) reached test-plan pass rates as high as 77.4% (DeepSeek) yet failed every single one of their 204 builds.

Source: paper Table 2 (8 public products only, 4 builds per model). Looking only at the three "Sol" releases, build pass@1 climbs 3.1% → 31.2% → 53.1% while cost per build actually falls, $26.5 → $19.4 → $11.8. The authors state that build pass@1 only starts moving once the test-plan pass rate clears roughly 70% — consistent with the two Luna releases in the table (pass rates 6.6% and 59.8%, both 0% pass@1) contrasted against the Sol/Terra releases (pass rates 84-94.9%, pass@1 31.2-53.1%).

Credibility Assessment

Three things support trust. First, the authors audited their own grading reliability: planting 34 bugs (24 subtle, such as a one-cent rounding error) produced zero false passes across 96 graded instances, the wrong-fail rate on working plans was 0.20% (3 of 1,521), and the wrong-pass rate (settled by hand after re-grading) was 0.8% (4 of 477). Second, the correlation between the 8 public and 9 held-out products is high on both ranking (Spearman ρ=0.99) and test-plan pass rate (ρ=0.97), supporting the claim that the public subset is a faithful, if harder, proxy. Third, an external team at Fireworks AI independently reproduced all nine models' scores by running the released benchmark with a single command — genuine third-party replication, not just the authors' own re-run.

The caveats are just as real. This is a pre-peer-review preprint, and all authors are Replit employees who use this benchmark both to pick models for their own platform and as an input to Fireworks' commercial index. The grading and seeding model, Opus 5.5, was simultaneously one of the nine evaluated builders — the authors self-audited for same-family bias (4 of 237 wrong passes on Claude-built apps versus 0 of 240 on GPT-built apps, called "within chance") but did not separate grader and gradee by design, which remains a structural limitation. The 8 public products carry canary strings to filter them from training data, but that filtering is not guaranteed to be complete, while the 9 held-out products rely only on an NDA with the external team. Scope is also limited to English-specified, PostgreSQL-backed full-stack web products, and every builder runs on a single harness built on the OpenHands SDK — the paper itself notes results may not transfer to a different production agent setup, including Replit's own.

Related Work

• Jimenez et al. (2024, ICLR). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Prior work. A single-shot task of resolving one GitHub issue across 2,294 cases in 12 existing Python repositories; the best model at release (Claude 2) solved 1.96%. ViBench 2.0 moves the axis from "fix existing code" to "build a whole product from nothing, end to end."
• Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Benchmark-origin. Measures tool-using agent reliability in retail and airline domains via pass^k (probability of succeeding on all of k repeated trials); GPT-4o scored roughly 35-61% on a single trial but fell below 25% at pass^8. ViBench 2.0 explicitly cites this paper for the insight that "the headroom lies in reliability," reframing it as all-or-nothing scoring within a single build rather than repeated trials.
• Zhong et al. (2026, CAIS). ViBench: A Benchmark on Vibe Coding — Direct prior work (the original benchmark). A single-shot task of building an MVP or adding one feature, with a 51% pass@1 for the best model at release. This paper's own authors rerun that exact "Vibe-on-Vibe" task with current models (Sol, Astra, Opus) and get 82-89% pass@1 (paper Appendix H) — a 30-plus point jump purely from model generations, on an unchanged single-shot task. ViBench 2.0 extends this benchmark's grader to a long horizon of 9-18 cumulative requests instead of one.

Taken together, coding-agent benchmarks have scaled from "single-shot fixes on existing codebases" (SWE-bench) to "repeated-trial reliability on tool use" (τ-bench) to "single-shot greenfield product building" (ViBench 1.0). ViBench 2.0 adds the missing piece — long-horizon greenfield building — and shows the same reliability gap (near-perfect test-plan scores, much lower whole-build scores) persists even as the task horizon stretches out. Nothing here contradicts the prior work; it reinforces that the gap doesn't disappear as tasks scale up.

Reviewer's Take

First, I judge the gap between the two metrics is the more important finding than either number alone. A 30-45 point gap between test-plan pass rate (93.7-98.0%) and build pass@1 (48.5-64.7%) among the top three models means the kind of single "feature completion rate" metric vendors commonly publish answers a different question than the one practitioners actually care about: does the whole product work. Relying on the partial metric risks badly understating real-world failure rates.

Second, while the cost-and-performance trend across GPT releases (Table 2) is encouraging, I judge it should not be generalized beyond the one model family it was measured on. All three open-weight models sit at 56-77% test-plan pass rate yet 0% build pass@1 — consistent with the authors' own ~70% threshold observation, suggesting open-weight models simply haven't crossed that inflection point yet. Whether and when they do looks like the next real milestone for closing this gap.

Third, the fact that the grading model (Opus 5.5) was also one of the evaluated builders is partially addressed by the same-family bias check in Appendix D (0/240 vs. 4/237), but I judge the underlying design — not separating grader from gradee from the start — still deserves a follow-up with an independent third-party grading model, especially given how small those sample counts (237, 240) are for ruling out bias conclusively.

Practical Takeaways

• Track both metrics together — don't rely on feature-completion rate alone; put whole-build pass rate in the same deployment dashboard, and use the ~70% test-plan threshold as a rough signal for when build pass@1 starts moving.
• Re-run wide-interval comparisons — with 95% intervals of ±9-11 points, models as close as Sol and Astra need more than 4 builds each to separate confidently.
• Pressure-test context handoff explicitly — since summarization kicks in past 200k tokens here, test what survives compaction in your own long-horizon agents rather than assuming nothing is lost.
• Favor benchmarks that others can reproduce in one command — Fireworks' one-command, no-modification reproduction is a model worth demanding from any benchmark whose numbers you plan to rely on.
• Separate your grading model's family from the models it grades — if you must reuse a model as both grader and gradee, cross-check with an independently-sourced grading model before trusting the result.

Conclusion

ViBench 2.0 puts a number on "nearly every feature works, but the whole product doesn't": across 17 products and 544 test plans, the top three models pass 93.7-98.0% of test plans yet deliver a working build only 48.5-64.7% of the time, and all three open-weight models fail every one of 204 builds despite test-plan pass rates as high as 77.4%. At the same time, looking only at three GPT-family releases, build pass@1 rises from 3.1% to 31.2% to 53.1% while cost per build actually falls — a genuinely encouraging signal that the gap is closing fast.

That said, every author is a Replit employee who uses this benchmark both for the company's own model selection and as an input to Fireworks' commercial index, and the grading model doubled as one of the evaluated builders — reasons to read the numbers with some caution. This "nearly passes every test, still fails as a whole" gap connects directly to our checklist on coding-agent merge-gate design (sunny34.com blog): that piece covers how to design the gate itself, while this review explains why the pass rate through that gate runs lower than raw test coverage would suggest.

References

• ViBench 2.0: A Benchmark for Long-Horizon Software Building — arXiv abstract (primary source)
• Full HTML text of the paper — used for table/number verification
• SWE-bench (Jimenez et al., 2024) — related work, primary source
• τ-bench (Yao et al., 2024) — related work, primary source
• ViBench 1.0 (Zhong et al., 2026, CAIS) — related work, primary source
• Coding-agent merge-gate design checklist — sunny34.com blog