43.6% Task Completion, 19.0% Full Compliance: Coding Agents' Hidden Non-Compliance — ANCHOR Review

Source Details

Wenqiang Zhu, Xinping Lei, Zhiyi Lai, Zhengshuo Gong, Tianzhuang He, Songwei Yu, Weihao Li, Jingxuan Xu, "ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding", arXiv:2610.16859v2 [cs.CL], submitted 2026-10-12 (v2 revised 2026-10-14), DOI 10.48550/arXiv.2610.16859. Dataset on Hugging Face (Yone-01/ANCHOR); code on GitHub (bupterlxp/ANCHOR).

This is an unreviewed preprint (primary category cs.CL). The HTML full text carries no author-affiliation footnote, and a WebSearch pass could not confirm the authors' institution either (likely because secondary indexing is still thin four days after submission) — funding disclosure is unconfirmed for the same reason. One author (Jingxuan Xu) also co-authored the coding-agent benchmark SWE-Compass (2025), suggesting this is a follow-up from the same research lineage.

Study Overview

Two questions drive the paper. (1) Is "functional completion" of a repository-grounded coding task the same as following the engineering constraints a developer specifies — scope, workflow order, validation, tool use, reporting? (2) Does adding explicit constraints change completion rates themselves?

The data: 358 tasks across 23 repositories and eight language groups. A 10-category, 78-leaf taxonomy — built bottom-up from real developer-agent interactions — yields 2,836 atomic constraints and 600 relation items (AND/THEN/PRIORITY). Evaluation is a rule+LLM hybrid: scope, file-count, and import checks that reduce to a diff-level rule go to rule-based judges; order and reporting-fidelity items that require reading the trajectory go to an LLM judge. Because the checklist itself is auto-generated, the authors separately human-audited both constraint-generation quality and judge reliability (see Credibility below).

Key Results

Picking representative rows from the 19 agent configurations (10 foundation models × OpenCode/Cline, with Claude Opus 4.8 evaluated under Cline only) shows that similar completion rates can hide very different compliance rates. ACR/CCR/IPR pool atomic, relation, and all checklist items respectively; FCCR is the strictest metric — a task passes only if every scored item passes.

Even the best configuration, Claude Opus 4.8 under Cline, completes 43.6% of tasks but passes every checklist item (FCCR) on only 19.0%. The two are different marginal rates, not subtractable directly, but their 24.6-point gap lower-bounds the share of "completed" tasks that fail at least one item — 81.0% fail something. Holding the model fixed and swapping only the scaffold (GPT-5.3-Codex, OpenCode vs. Cline) moves completion by just 0.3 points (37.4%→37.7%) while IPR drops 5.0 points (67.9%→62.9%) and FCCR drops 3.1 points (12.3%→9.2%). A completion-only dashboard would never surface that gap.

The category breakdown shows which rules survive and which don't: local prohibitions close to "don't do X" (dependency, security, scope) score high; trajectory-grounded rules that require watching the whole process (ordering, tool choice, reporting) score low.

Scaffold sensitivity also flips sign by model. GPT-5.4-Mini's IPR falls 27.7 points from OpenCode (58.5%) to Cline (30.8%), while DeepSeek-V3.2 and Qwen-3.5-Flash score 10.5 and 14.1 points higher under Cline (paper Table 7). Which scaffold is "the safer pick" depends entirely on the model.

Credibility Assessment

Because the benchmark runs on an auto-generated checklist, the authors ran two separate human audits. Constraint-generation quality: five engineers independently rated atomic constraints from 50 task instances at 94.0% feasibility validity (92.0% raw agreement, Gwet AC1 0.91), 94.0% repository consistency (96.4%, 0.96), and 96.0% realism (97.6%, 0.97). Judge reliability: a 480-item human-labeled consensus (five evaluators averaging 8.4 years of experience; inter-evaluator κ=0.83) agrees with the LLM judge at Cohen's κ=0.78 (89.4% raw agreement). Rule-verifiable items reach κ=0.86, but THEN/PRIORITY relation items are lowest at κ=0.69; re-ranking configurations on the 480-item human subset still correlates 0.96 (Spearman) with the full judge-based ranking.

The caveats are just as concrete. The paper itself states it ran "no cross-provider judge comparison or controlled prompt-sensitivity study," so model-family bias in the judge cannot be ruled out — and since the judge model's identity is not disclosed in the available text, that caveat matters when the top scorer is Claude Opus 4.8. Opus 4.8 is also the only configuration evaluated under Cline alone, with no OpenCode counterpart among the 19 rows. Category sample sizes are uneven (49–530 items; C6/C7/C8 under 70). No prior work reaches a contradicting conclusion — this is a newly proposed axis, best read as exposing a risk that completion-centric benchmarks (the SWE-bench family) were missing rather than refuting one.

Related Work

• Ding et al. (2026). OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding — a concurrent benchmark (217 tasks, 7,098 checklist items, ACL 2026) that measures scaffold- and platform-level policy compliance. ANCHOR is complementary: it targets developer-specified, task-level engineering constraints and their hierarchy/ordering rather than scaffold policy.
• Zhang et al. (2025). Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions? — accepted at ICLR 2026. Prior work showing LLMs exhibit "cognitive inertia" that keeps them from overriding trained conventions to follow explicit instructions in general settings. ANCHOR's low FCCR (19.0%) shows the same failure mode recurring in multi-step agentic coding trajectories.
• Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — ICLR 2024 oral, the origin benchmark for repository-grounded coding-agent evaluation, which measures only whether the final patch is correct. ANCHOR extends this lineage by adding a separate axis: whether the process that produced the patch followed the developer's rules.

Reviewer's Take

First, I judge the most practically valuable finding here is not the 19.0% headline but the paired scaffold comparison for the same model. A 0.3-point gap in completion alongside 5.0- and 3.1-point gaps in IPR and FCCR means a completion-only dashboard would completely miss the risk of a scaffold swap.

Second, I judge that Claude Opus 4.8 being evaluated under Cline alone is a reason to treat the "best model wins" conclusion with caution. The one unpaired configuration happens to rank first, and without an OpenCode counterpart there is no way to separate model quality from a model-scaffold interaction effect.

Third, I judge that withholding the LLM judge's model identity is the first thing to check for reproducibility. The authors admit they cannot rule out model-family bias, and without knowing the judge's lineage, readers cannot independently verify any affinity between the top-scoring model and the judge.

Practical Takeaways

• Track completion and compliance separately — keep ACR/IPR/FCCR-style compliance metrics alongside task completion on operational dashboards; assume they can move in opposite directions.
• Regression-test scaffold swaps — a harness/toolchain change can hold completion steady while compliance drops up to 27.7 points for the same model, so never sign off on a swap using completion alone.
• Dedicated checks for reporting — the weakest category (41.0%), so verify "disclose what you did" with an automated check rather than trusting free-form summaries.
• Favor rule-based checks for ordering constraints — THEN/PRIORITY relation items have the lowest LLM-judge reliability (κ=0.69); verify critical ordering rules (e.g., reproduce-before-fix) from logs, not judge output.
• Demand judge transparency — when adopting a third-party or in-house LLM-eval pipeline, always ask which model serves as judge and whether it is in the same family as the models being scored.

Conclusion

ANCHOR's core message is that "done" and "done right" are different axes. Across 358 tasks and 19 configurations, completion and compliance correlate loosely, and even the best model fully passes every checklist item on only 19.0% of tasks. The dual human audit (constraint-generation quality plus judge reliability) gives the benchmark real credibility, but the undisclosed judge model and Opus 4.8's unpaired evaluation are gaps to flag before repeating the headline number. For a decision like a harness swap that looks safe simply because "it's the same model," this paper shows that skipping the compliance measurement can make that judgment wrong. Structural guardrails inside agent harnesses continue, on a different axis (credential scoping), in our agent-harness credential brokering design.

References

• ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding — arXiv abstract (source)
• Same paper, HTML full text — used to cross-check Tables 5, 6, 7, 8
• OctoBench (related work) — arXiv abstract
• Agent harness credential brokering design — sunny34.com blog