35% of Finished Builds Still Fail Tests: A Checkpoint and Early-Stop Gate for Long-Horizon Coding Agents

Reliability Doesn't Decay Linearly Over a Long Horizon

Replit's ViBench 2.0 evaluates coding agents on long-horizon software-building tasks — multi-step work that produces a finished product, not a single short task. Even the top-ranked model shipped finished products where 35% failed the final test suite, and three open-source models failed to complete a product in all 204 build attempts. A single-task benchmark's pass rate and a long multi-step build's completion rate turn out to be entirely different metrics.

A Full-Pass-Rate Gate Alone Doesn't Stop This

The Co-Design Bench-based merge gate covered earlier was a post-hoc check — it decides whether to merge based on full-pass rate once the build is already done. What ViBench 2.0 exposes is a problem upstream of that: if a build is already on a failure trajectory before it finishes, a post-hoc gate only confirms the failure after every resource has already been spent on it. A long-horizon build needs a separate mechanism that inspects the trajectory mid-build and stops or rolls it back early.

From Design to Operations: A Checkpoint and Early-Stop Checklist for Long-Horizon Builds

Start by pinning a number to how many checkpoints a long build should be split into. For a ViBench 2.0-scale task, split the full build into 4–6 checkpoints, and at each one, run regression tests confirming the intermediate output hasn't introduced new defects relative to the prior stage. Set the targets alongside this: an automatic halt if the checkpoint pass rate falls below 50% for two consecutive checkpoints, and a final test pass rate of 80% or higher once the full build completes.

Four failure patterns recur. First, running the build to completion with no checkpoints and only measuring full-pass rate at the end — exactly the pattern behind ViBench 2.0's open-source total-failure result, where a build already on a failure trajectory still burns its entire resource budget before anyone notices. Second, applying the same checkpoint bar regardless of model tier (frontier vs. open-source), so a model that's historically weak over long horizons still gets allocated a long budget by default. Third, letting the build keep running past a falling checkpoint pass rate until a human manually checks in. Fourth, pinning rollback points only to code commits, so when the actual failure is a bad design decision, rolling back still reproduces the same failure.

Build two recovery branches. If the checkpoint pass rate drops below the threshold at one stage, continue to the next checkpoint in an observation mode with a lowered budget cap; if it drops for two consecutive checkpoints, automatically halt the build and roll back to the last passing checkpoint, without waiting for human sign-off. For model tiers with historically low long-horizon completion rates — open-source and lightweight models — design in more checkpoints from the start, so an early-stop decision can fire sooner.

Lock four items into the operations checklist: ① split the full build into 4–6 checkpoints with a fixed regression test at each; ② log checkpoint pass rate per model tier on a separate dashboard to track the completion-rate gap between tiers; ③ code the automatic halt-and-rollback rule for two consecutive below-threshold checkpoints; ④ keep the last checkpoint log from any halted build as the budgeting basis for the next task.

Run the improvement loop weekly. Aggregate the checkpoint distribution of halted builds to see where failures cluster, and when a specific checkpoint keeps absorbing halts, split that stage's scope further into its own checkpoint. Recalibrate the checkpoint thresholds every time the benchmark itself moves to a new version.

Takeaways at a Glance

A full-pass-rate gate alone doesn't stop a long-horizon coding agent from burning resources on a build that was never going to finish. Split the build into checkpoints, track completion rate by model tier, and code an automatic halt-and-rollback for two consecutive below-threshold checkpoints — and an open-source model's failure trajectory shows up mid-build instead of only at the very end.

References

35% of Finished Products Failed: A Review of Replit's ViBench 2.0 Benchmark — sunny34.com Research Review

ViBench 2.0: A Benchmark for Long-Horizon Software Building — arXiv