Source Document

Vahid Haratian, Atakan Akar, Eray Tüzün, "ADAPT: Agentic Diff-Aware Patch Testing for Pull Requests", arXiv:2611.16114 [cs.SE], submitted 2026-11-11, DOI 10.48550/arXiv.2611.16114. Affiliation: Bilkent University, Turkey. Replication package (run artifacts and full trajectories) released.

This is a preprint without peer review. All three authors are university researchers, so there is no obvious conflict around promoting a vendor's model, but the acknowledgments section ends with a boilerplate line — "our major funders, member institutions" — with no institution actually named, so the funding source cannot be pinned down. A more direct caveat: two of the four annotators who scored test quality for RQ3 are the paper's own authors. The scoring was blind to which configuration produced a given test, but it is not a fully independent evaluation.

Study Overview

The motivation is simple: as code changes move through pull requests, the PR frequently ships without a test that guards the new behavior. A prior survey the authors cite (Amore et al., 160 repositories, 2026) reports a mean Test Engagement Ratio of 0.37 across merged PRs — 63% of PRs that touched production code did not also touch a test file. Four research questions follow: RQ1 compares the F2P (fail-to-pass) generation rate against BLAST; RQ2 asks whether the verification tool is an active refinement signal rather than a final check; RQ3 asks how a human rates the generated tests' quality; RQ4 compares patch coverage against ChaCo.

The method builds a dedicated F2P verification tool on top of an OpenHands-based agent. The agent sees only the PR's code diff, with no issue description, and writes candidate tests. Those tests are run in a separate workspace against both the base and head versions of the repository to check that each one fails on base and passes on head. Once a test is F2P-valid, the tool additionally reports how many changed executable lines remain uncovered, driving further refinement. The evaluation runs on the same 426-PR SWE-bench Verified subset (12 Python repositories) that BLAST used.

Key Results

Under the same model, GPT-4o, ADAPT outperforms BLAST, and swapping in a lighter model lifts the rate again. All three rows' counts and rates reconcile against the shared 426-PR denominator.

ApproachF2PNo F2PErrorF2P rate
BLAST (GPT-4o)151275–35.4%
ADAPT (GPT-4o)2641332962.0%
ADAPT (GPT-5 mini)38343989.9%

The cost gap is large too. Across all 426 PRs, GPT-4o totals $478.95 versus GPT-5 mini's $16.62 — about a 29x difference (Table III) — and GPT-5 mini also uses fewer tokens per PR (413.6K versus 698.8K). The verification tool's call frequency splits by model as well: an average of 5.4 calls under GPT-4o versus 1.8 under GPT-5 mini (Table IV) — the weaker model needed more rounds of feedback to reach the same destination.

On the 137 PRs shared with ChaCo (44 pandas, 49 SciPy, 44 Qiskit), ADAPT reaches full patch coverage on 83.9% (115) versus ChaCo's 79.6% (109). By repository, ADAPT leads on pandas and SciPy, while ChaCo edges ahead by one PR on Qiskit.

ApproachOverall (137)pandas (44)SciPy (49)Qiskit (44)
ADAPT83.9% (115)90.9% (40)81.6% (40)79.5% (35)
ChaCo79.6% (109)79.5% (35)77.6% (38)81.8% (36)

On that same 137-PR set, ADAPT consumes more tokens per PR (1.98M versus 0.60M) but costs and runs faster (about $0.145 and 17.4 minutes per PR versus roughly $0.44 and 56.7 minutes for ChaCo, Table VIII). In the manual evaluation (70 sampled tests per configuration, four annotators), GPT-5 mini's tests scored 4.54/5 on importance, 4.35/5 on assertion strength, and 4.49/5 on readability, each above GPT-4o's 4.01, 3.79, and 4.16 (Table VI).

Credibility Assessment

The study earns trust on several counts: both comparisons (against BLAST and against ChaCo) hold the dataset and the success criterion fixed, and the reported counts and rates reconcile arithmetically. Quality is not judged by the automatic F2P check alone — four blind human annotators separately scored a sample of tests — and a replication package with run artifacts and trajectories is public.

The caveats are just as clear. This is a pre-peer-review draft, and half of the manual annotators are paper authors. Only two models were tested, GPT-4o and GPT-5 mini — not Claude, Gemini, or open-weight models, which the authors defer to future work — and the benchmark covers a single language (Python) across 12 repositories. Because SWE-bench Verified is built from public repositories, the authors themselves acknowledge that pretraining exposure (data leakage) cannot be fully ruled out. The ChaCo comparison needs particular care: the 79.6% figure is ADAPT's own measurement on the 137 PRs the two approaches share, and it is not directly comparable to whatever ChaCo's own paper reports on its full evaluation, since the sample and measurement scope differ.

Related Work

Read together, these three papers suggest agentic approaches have a consistent edge over non-agentic ones (SBST, augmentation) on PR- and issue-based F2P test generation. But the ChaCo case is also a caution: even a metric with the same name — "patch coverage" — can diverge sharply once the sample, model, and measurement scope change.

Reviewer's Judgement

First, any citation of the headline success rate should carry the model name with it. The jump from 62.0% to 89.9% (27.9 points) conflates ADAPT's own contribution with the effect of swapping GPT-4o for GPT-5 mini. The method's own effect is better read as the same-model comparison against BLAST: 26.6 points.

Second, RQ2's average of 5.4 verification calls under GPT-4o versus 1.8 under GPT-5 mini is, in this reviewer's reading, an operational signal the paper did not set out to deliver. Attaching an F2P verification loop to a weaker model means more rounds and more cost to reach the same outcome, so it is worth measuring that trade-off before adopting the pattern on a weaker in-house model.

Third, the roughly 50-point gap between ChaCo's self-reported figure (about 30%) and the number ADAPT's paper attributes to ChaCo (79.6%) illustrates how hard benchmark reproduction can be. A baseline number reproduced by a third party, rather than by the original authors, is best not carried forward as-is.

Putting It to Work

  • F2P first, coverage second — secure a test that fails on base and passes on head before chasing line coverage of the diff; treat patch coverage as a secondary signal measured only after F2P succeeds.
  • Make the verifier a feedback loop, not a final gate — give the generation agent a dedicated step that runs candidates against both base and head, and use the average number of rounds as a signal of model fit.
  • Pick a model on success rate and cost together — a 29x cost gap can hide behind an attractive headline success rate, so never choose the model from a single number.
  • Pair automatic checks with a small human sample — F2P alone does not capture importance, assertion strength, or readability; spot-check a small sample by hand.
  • Break results out by repository or language — ADAPT itself trailed ChaCo on Qiskit, so do not decide on adoption from an overall average alone.

Conclusion

ADAPT is not a new model but a combination — an agent that reads only the PR diff, paired with a dedicated F2P verifier that runs candidates against both base and head — and under the same model it beats BLAST's success rate by 26.6 points while reaching broader patch coverage than ChaCo at lower cost. The limits remain real: a pre-peer-review draft, half the manual annotators being authors, and validation on only two models and one language. Read the numbers as "for this benchmark, this model pairing" and re-measure F2P rate and cost on a small pilot before adopting it on your own codebase. For how generated tests might feed into a merge gate, see The Coding-Agent Full-Pass-Rate Merge Gate Checklist.

References

Ask AI about this review

The assistant has read this review and the numbers it verified. Ask anything — it answers from the text and says so when something isn't in it.

Loading the chat…