Requirements 69.9% Met, Full Pass Only 26.4%: A Co-Design Bench Review of Agentic Coding Models

The paper

Xilin Wei, Shengyuan Ding, Xinyu Fang, Tiancheng Gu, Yuchi Wang, Zhixun Li, Ziyu Liu, Yuhang Zang, Weixiang Yu, Cheng Lu, Chonggao Sun, Tong Zhou, Dahua Lin, Hang Zhang, Shuai Bai, "Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models", arXiv:2610.35138 [cs.AI], submitted 2026-10-26. Affiliations: Fudan University, Alibaba Group (Alibaba Token Hub), Zhejiang University, CUHK, Shanghai Jiao Tong University, NTU. Project page public (codesignbench.github.io), license CC BY-NC-SA 4.0.

This is an unreviewed preprint. All three corresponding authors, and a majority of the author list, are affiliated with Alibaba Group — a clear conflict of interest. However, the overall top-ranked model was not an Alibaba model but OpenAI's gpt-6-astra; Alibaba's own models (Qwen3.8-max-0902, Qwen3.8-27B) ranked third and fourth overall, so the incentive to favor an in-house model does not appear to have shaped the result.

Study design

Two questions drive the paper: how completely can an agent finish a compound revision request pulled from a real production coding session (a localized multimodal edit, or a coordinated repository-wide feature change), and how different is "meeting most requirements" from "fully passing the task"? The data comes from 39,325 production queries, deduplicated to 37,463, drawn from 26,314 real coding sessions. After filtering for recoverable, buildable, Playwright-renderable starting states, the authors compressed this into 273 final tasks (100 Multimodal Editing/ME + 173 Repository Editing/RE) with 2,331 hidden, expert-authored acceptance criteria (553 for ME, 1,778 for RE).

Evaluation follows a "judge-then-refute" protocol. A first pass grades each criterion Pass/Partial/Fail using source code, rendered output and recorded Playwright traces. A separate LLM verifier then challenges every positive verdict, checking for stubs, hardcoded outputs, unreachable functionality and requirement mismatches. Visual claims require a screenshot and interaction claims require a recorded browser action; missing evidence aborts scoring before aggregation. The rationale: code that implements a feature doesn't by itself prove a user can reach and correctly operate that feature through the rendered interface. Two frontend experts additionally re-reviewed 200 task-model outputs. Ten model endpoints were evaluated under a shared Claude Code harness (3-hour, 120-turn cap), producing 2,730 model-task pairs (1,000 ME + 1,730 RE).

Key results

Three metrics need to be kept separate. Strict Completion (R̄) averages, per task, the fraction of criteria that fully passed. Graded Completion (Ḇ) adds partial credit (Partial = 0.5). Full-Pass Rate (F) is the share of tasks where every single criterion passed. By construction, F ≤ R̄ ≤ Ḇ.

Split the top model (gpt-6-astra) by track and ME reaches R̄ 75.4%/F 41.0%, RE reaches R̄ 66.7%/F 17.9% — meaning 73.6% of all 273 tasks retain at least one unresolved criterion. Combined and per-track rankings also diverge: Qwen3.8-27B ranks fourth overall (R̄ 64.4%) but second on the RE track alone, where its R̄ of 64.5% edges out opus-5's 64.2%.

Going from 1-2 edit types to 3-4 drops R̄ by 6.3 points (65.4% to 59.1%), but F barely moves (9.1% to 9.7%). What spikes instead is the zero-pass rate — tasks that fail every single criterion — from 0.9% to 5.8%: harder tasks don't erode the share of complete successes so much as they inflate the share of complete failures. Input modality shows a similar split: document-attachment tasks have the highest pooled requirement coverage of any modality (R̄ 65.2%) but a lower full-pass rate (18.2%) than text-only tasks (37.0%).

The authors also ran a separate ablation (Appendix A) — not on Co-Design Bench itself, but on 150 tasks from an existing benchmark, WebCompass-edit, comparing the same model (qwen3.7-plus) under single-shot versus agentic execution. Because the basis differs, this isn't placed alongside the table above.

Running opus-4-8 agentically on the same 150 tasks reportedly reaches 99.4%, but the authors are explicit that this is a different model's separate data point, not a controlled paired comparison — it shouldn't be read alongside qwen3.7-plus's 73.6%-to-95.1% jump.

Credibility assessment

This is a pre-review preprint with a clear conflict of interest: all three corresponding authors, and a majority of the author list, work at Alibaba Group (Alibaba Token Hub). Two of the ten evaluated models (qwen3.8-max-0902, qwen3.8-27b) and the model used in the single-shot/agentic ablation (qwen3.7-plus) are also Alibaba's own. The paper states no separate funding or sponsorship disclosure. Offsetting this: the in-house models did not win (gpt-6-astra led every track), two frontend experts independently re-reviewed 200 task-model outputs to check verdict validity (Appendix J), and the results held direction consistently across three separate robustness checks — swapping the judge model, swapping the harness, and the single-shot/agentic comparison (Appendices C, H, A). Stated limits are explicit too: each configuration ran once, so run-to-run variance is unreported, and the harness-sensitivity analysis (Appendix H) itself shows that model rankings shift with the harness, meaning model ability and harness interaction aren't fully separable. The headline claim (69.9%/26.4%) repeats identically across the abstract, body and appendix, which is a good internal-consistency signal.

Related research

With this session's network access fully blocked, related literature was identified and checked independently via WebSearch rather than by re-reading the original papers. Any numbers from these papers below are used only to describe relationships, not folded into this review's "key results."

• Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — benchmark-origin. Established repository-level code-fix evaluation using 2,294 real GitHub issues; the best model at release solved just 1.96% of them. Co-Design Bench's Repository Editing track extends this framing to compound, multimodal requests drawn from real coding sessions.
• Lei et al. (2026). WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models — a direct prior work. Co-Design Bench's own authors reused WebCompass's 150-task editing subset to verify the single-shot-versus-agentic effect shown above. Where WebCompass constructs tasks from predefined edit operations, Co-Design Bench extracts them from real user sessions — the key distinction between the two.
• Xiao et al. (2025). DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation — prior work. Evaluates generation, editing and repair across 900 webpage samples limited to React, Vue and Angular. Co-Design Bench's Multimodal Editing track broadens this scope beyond any single framework, covering image, audio, video and document attachments pulled from real sessions.

All three predecessors trace a widening arc — static reconstruction, then editing, then repository-scale tasks — and Co-Design Bench sits at the end of that arc, the first to quantify just how far "covering most requirements" diverges from "finishing the task."

Reviewer's take

First, I judge the most practically important finding here isn't the model ranking but the split itself between requirement coverage and full task success. The top model covering 69.9% of requirements while fully passing only 26.4% of tasks means that trusting an average score for an automation decision leaves unresolved requirements in 73.6% of tasks. It's easy to mistake a "mostly done" report for a deployable one.

Second, the pattern where full-pass rate barely moves (9.1% to 9.7%) while zero-pass rate spikes (0.9% to 5.8%) as edit breadth grows reads as: harder tasks don't meaningfully shrink the share of complete successes, they inflate the share of complete failures. That's an argument for decomposing requests into smaller pieces rather than expecting graceful partial credit at higher difficulty.

Third, the single-shot-versus-agentic comparison (73.6% to 95.1%) is striking, but it's an ancillary result borrowed from a different benchmark, not Co-Design Bench's own finding. Quoting that jump in the same breath as Co-Design Bench's own headline numbers (69.9%/26.4%) mixes two benchmarks and two models. The paper itself flags this distinction, but placing both figures in the same Figure 1 makes the mix-up easy for a casual reader.

Practical takeaways

• Track completion and full-pass rate separately — relying on average requirement coverage alone to approve a deployment misses the unresolved items left in a large share of tasks. Put both metrics on the same dashboard.
• Keep edit requests to 1-2 change types — zero-pass rate spikes (0.9% to 5.8%) once a single request spans 3-4 kinds of changes; decompose compound requests instead.
• Set separate thresholds per input modality — document-attachment tasks show coverage (R̄ 65.2%) and full-pass rate (F 18.2%) moving in opposite directions versus text-only (F 37.0%).
• Attribute results to the harness-plus-model pair — Appendix H shows rankings shift with the harness, so compare and attribute at the harness-plus-model level, not model alone.
• Don't mix cross-benchmark numbers — ancillary figures like the agentic-execution effect (73.6% to 95.1%) come from a different benchmark and model; keep them out of the same comparison table as this benchmark's own headline metrics.

Conclusion

Co-Design Bench's conclusion is that "how much of the requirement does a coding agent meet" and "how completely does it finish the task" are not the same question. Across 273 tasks and 2,331 hidden acceptance criteria evaluated over 10 models, even the top model covered only 69.9% of requirements and fully passed only 26.4% of tasks. The more notable finding is a polarization pattern: as edit breadth grows, complete failures rise faster than partial successes shrink. Given each configuration ran only once and a majority of the authors share an Alibaba Group conflict of interest, the numbers are best read as a directional signal to re-verify against your own tasks before adoption. On bringing agent evaluation in-house, see our self-hosted coding agent evaluation checklist.

References

• Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models — arXiv abstract
• Full HTML text — used for table and figure cross-checking
• SWE-bench (benchmark-origin) — arXiv
• WebCompass (direct prior work) — arXiv
• DesignBench (prior work) — arXiv
• Self-hosted coding agent evaluation checklist — sunny34.com blog