Requirements Filled, Merge Still Blocked: A Full-Pass-Rate Gate Checklist for Coding Agents

Coverage and full-pass are different numbers

The Co-Design Bench review found a simple gap: the top model, gpt-6-astra, fills 69.9% of a task's requirements on average, but only 26.4% of the 273 tasks pass every hidden criterion (the full-pass rate, F). That 43.5-point gap means that if you ship on the average score alone, 73.6% of tasks still carry at least one unresolved requirement. The same trap waits when you wire an in-house coding agent into a CI/CD merge pipeline — a commit log that says "requirements mostly met" is not the same signal as a task being actually done.

Gate on zero-pass, not on the average

The review also reports that as edit scope widens from 1-2 edit types to 3-4, the full-pass rate barely moves (9.1% to 9.7%), while the zero-pass rate — tasks that satisfy not a single criterion — jumps from 0.9% to 5.8%. Harder tasks don't quietly lose "partial credit"; they polarize into outright failures. A merge gate built on one average completion number misses this. You need a separate alert on the zero-pass count to catch the tail.

From design to operations: a coding-agent merge-gate checklist

Declare three thresholds in code before anything merges: a full-pass rate (F) floor of 30%, zero zero-pass tasks, and a separate review track for any request spanning 3+ edit types before it can be promoted. Using Co-Design Bench's own ceiling — the best model still lands at F 26.4% — as your baseline lets you say "not deployment-grade yet" in numbers when your in-house model falls short of it.

Set modality-specific thresholds too. The paper finds that document-attachment tasks have the highest requirement coverage (R̄ 65.2%) of any modality, yet the lowest full-pass rate (F 18.2%) — below text-only tasks at F 37.0%. A single promotion bar across modalities makes document-attachment tasks look safer than they are.

The most common failure is mistaking partial credit for completion. If an auto-merge bot checks only the graded completion rate (which counts partial credit) before promoting, the unresolved criteria hiding between strict completion and full-pass leak straight into production. The second failure is re-running a wide-scope request wholesale instead of retrying by edit type — retrying a 3-4 edit-type task as one block drives the zero-pass rate even higher.

Both failures need a recovery branch defined ahead of time. When the full-pass rate misses the gate floor, split the task by edit type and retry each piece; if a split piece still fails, route it to human review. If the same task zero-passes twice in a row, flag the harness+model combination itself as suspect and stop auto-retrying — a circuit breaker.

Log the model ID, harness version, per-task strict completion rate, full-pass flag, edit-type count, and attachment modality as required fields. Appendix H of the paper shows that swapping harnesses reshuffles model rankings, so track pass rates by harness+model combination, not by model alone — that is what lets you compare fairly after the next harness upgrade. Keep multimodal editing (ME) and repository editing (RE) as separate pre-merge test suites so you can see which track is actually blocking the gate.

Weekly, roll up zero-pass tasks by edit-type combination. When one combination keeps producing zero-pass tasks, add it to your task-decomposition rules so requests in that combination get split smaller from the start. Revisit the gate's promotion floor itself every quarter — as the field's best full-pass rate rises, your internal floor should rise with it, or the gate quietly loosens.

What to do this week

A coding agent reporting "requirements mostly met" is not a merge signal. Put a full-pass-rate floor and a zero-pass-task count into your promotion criteria, split wide-scope requests before retrying them, and roll up pass rates by harness+model combination — that is how you keep Co-Design Bench's 26.4% trap out of your own pipeline.

References

• Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models — arXiv
• 69.9% of requirements filled, only 26.4% fully pass — Co-Design Bench coding-agent review (sunny34.com Research)