SAST 82% vs AI Security Agents 79%: What VulnBench 2.0 Shows About Combining Them

The paper

Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark, "How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0", arXiv:2610.38617 [cs.CR], submitted 2026-10-28. Affiliation: Snyk. The harness, scorer and full run archive are public (github.com/snyk-labs/snyk-vulnbench), Apache License 2.0.

This is an unreviewed preprint. All authors are Snyk employees and one of the compared engines (Snyk Code) is Snyk's own product — a clear conflict of interest. The authors offset that by publishing the complete harness, scorer and run archive so anyone can re-score the results.

Study design

The question is simple: now that AI coding agents are used for security review, what does deterministic static analysis (SAST) miss that an AI agent catches, and vice versa? Building on a predecessor limited to 10 JavaScript apps, the authors seeded 108 attacker-reachable vulnerabilities across 20 open-source web applications in four language families — Go, JS/TS, Python, Java (62 flows cross a file boundary). Nine productized configurations were evaluated: Snyk Code CLI (SAST), Claude Code's built-in /security-review (Opus 5 xhigh, Sonnet 5 xhigh, Opus 5 medium + Snyk MCP), Codex Security (GPT-5.6 Luna/Terra/Sol), and DeepSec (Opus 5 and GPT-5.6 Sol backends). Scoring uses an "endpoint rule": a finding counts only if it locates both the attacker-controlled source and the dangerous sink within two lines of the curated flow. Crucially, every result is attributed to the full harness-plus-model-plus-version combination, never to a model alone.

Key results

The primary metric is macro recall (the 20 per-app recalls averaged equally, which can diverge slightly from the pooled fraction since vulnerability counts per app range from 2 to 10) — a divergence the authors designed in, not an error.

*DeepSec doesn't report source/sink roles, so it's scored separately under a looser rule (class match plus any location within two lines) and isn't comparable to the seven endpoint-aware configurations above.

The headline is the union, not the ranking. Snyk Code plus Opus 5 xhigh together reach 101 of 108 — the same as the union of all seven comparable configurations; no other configuration added anything beyond that pair. They disagree on 28 vulnerabilities (16 Snyk-only, 12 Opus-only), and McNemar's test (p=0.57) says this corpus is too small to separate them. Snyk Code alone caught 10 vulnerabilities (open redirect, prototype pollution, path traversal, and others) that no AI configuration matched; Opus 5 xhigh alone caught 12 (7 cross-file, including authorization classes). The AI configurations also surfaced 7 real vulnerabilities absent from the reference set entirely, verified by the authors through live reproduction.

Harness policy effects are just as concrete. Claude Code's /security-review excludes ReDoS outright and reports prototype pollution and open redirect only at extremely high confidence. On the 20 reference vulnerabilities in those three classes, Snyk Code matched 18 and Opus 5 xhigh matched 9 — but on the other 81, the two were close (71 vs 67). That policy alone explains most of the headline gap. A prompt-swap control (same model, the benchmark's default audit prompt instead of /security-review, 19 of 20 apps, run 2026-10-01) matched 16 of those 20 policy-class vulnerabilities with the policy removed, versus 9 with it. Overall it rose from 83/105 to 93/105, at 0.63x the cost and 0.52x the wall time — the model didn't change, the policy did.

Spend and coverage weren't reliably linked. Within Codex Security, GPT-5.6 Sol cost 35.0x as much as Luna and matched 11 fewer vulnerabilities. Within Claude Code, Opus 5 cost 4.2x as much as Sonnet 5 and matched 12 more. Codex Security's other weakness is localization: GPT-5.6 Sol located 80 of 108 at file level but only 34 under the two-line rule — in 53 of 57 misses it pointed at a location it had itself flagged as the sink, typically six lines above the real one. It finds the function; it misses the line.

Credibility assessment

This is a pre-review preprint with a clear conflict of interest: all authors work at Snyk, and Snyk Code is Snyk's own product. Snyk Code's product owner co-authored the paper, and after the authors privately shared seven newly-found vulnerabilities, Snyk shipped a rule update for one of them on 2026-10-28, before publication (the reported numbers predate that update). The authors address bias in both directions: seeding vulnerabilities with GPT-5.6 models could have favored Codex Security, which ran the same models — yet it scored lowest of the seven. Conversely, aligning class names to Snyk's vocabulary and including five apps previously used in Snyk demos could favor Snyk Code — but dropping those five apps left the ranking and union relationship unchanged (Snyk 74/87, Opus 72/87, union 83). The predecessor study reported LLM recall trailing SAST reproducibility (100%) by 24.6 points; this paper's narrower gap (3.1 points) counts as mildly contrary evidence, though it may simply reflect a different benchmark design. Stated limits: the sample (108 vulnerabilities, 20 apps) can't separate close rankings statistically, each AI configuration ran only once so run-to-run variance is unmeasured, and Snyk Code is the only SAST engine tested — the conclusions are about Snyk Code specifically, not SAST in general.

Related research

• Tal et al. (2026). Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice? — direct predecessor (prior) by the same team. Measured only the repeatability of LLM security review against a Snyk Code-derived reference set on 10 JS apps; the best LLM configuration trailed deterministic SAST reproduction (100%) by 24.6 points, reaching 75.4%. VulnBench 2.0 fixes that study's main gaps (single language, derived ground truth).
• Priyanshu et al. (2026). Vulnerability Localization Benchmark — a concurrent extension. Tests 500 real vulnerabilities across 290 repositories with 27 LLMs and 4 SAST tools, scoring file-level localization; the strongest system reaches only 0.229 File F1. Where VulnBench 2.0 scores 108 vulnerabilities strictly at line level, VLoc Bench scales up and loosens the bar to file level, approaching the same question from the opposite direction.

Both studies agree with this paper's direction — AI review alone doesn't replace SAST — but VulnBench 2.0 is the first to isolate, via a controlled experiment, that harness policy can matter more than the model itself.

Reviewer's take

First, I judge the most practically valuable result here isn't the 82.4%-vs-79.3% horse race but the harness-policy control. Teams will be tempted to read "Claude Code missed this vulnerability class" as a model limitation; this paper shows the same model, same effort, catching 16 instead of 9 just by swapping the prompt. Before adopting a vendor harness, read what its built-in review policy excludes — this paper's appendix is a ready comparison point.

Second, DeepSec's result is a warning that the pipeline stage before the model can set a hard ceiling on recall. Its file-selection stage queued candidate files for only 56 of 108 vulnerable flows before any model ran, and neither backend matched a single vulnerability whose files weren't selected. When evaluating an agentic security scanner, "which stage narrows the candidates" matters as much as "which model runs."

Third, the way this paper handled its own conflict of interest is itself a notable precedent for this kind of benchmark. It disclosed choices that could favor its own product (reference-class naming, demo-app inclusion) and then showed the conclusions survive removing them. The existence of a conflict of interest matters less than how transparently it's handled — that should be the standard other vendor benchmarks are held to.

Practical takeaways

• Run SAST and AI review together — neither alone reaches the 101/108 union; budget triage time for the overlap.
• Read the built-in review policy first — know which classes a vendor harness excludes or restricts (ReDoS, prototype pollution, open redirect here) and cover those with a separate rule-based tool.
• Distinguish "found it" from "pointed at the right line" — file-level hit rates alone overstate configurations like Codex Security that find the function but miss the line.
• Compare cost and coverage only within the same harness — a pricier model tier can raise coverage (Opus vs Sonnet) or just raise cost (Sol vs Luna); hold the harness fixed when testing model tiers.
• Audit any pre-model filtering stage — a file-selection step that misses a vulnerable file can't be rescued by a better model downstream.

Conclusion

VulnBench 2.0's conclusion is that "does AI replace SAST" is the wrong question. Of 108 vulnerabilities, 73 were caught by both, and the rest split by type — SAST led on single-file, pattern-matched classes, AI led on cross-file logic flows and authorization issues. The more consequential finding is that harness policy can move results more than model choice, made more credible by the fact that conflicted authors proved it themselves with a controlled experiment. Still, the sample (108 vulnerabilities, 20 apps, one run per configuration) means the numbers are a directional signal, not a verdict — worth re-checking against your own benchmark before adoption. On where to draw trust boundaries for coding agents more broadly, see our trust-boundary playbook for coding agents.

References

• How Deterministic SAST and AI Security Agents Divide the Work — arXiv abstract
• Full HTML text — used for table and figure cross-checking
• Snyk VulnBench JS 1.0 (predecessor) — arXiv
• Vulnerability Localization Benchmark (VLoc Bench) — arXiv
• Trust-boundary design for coding agents — sunny34.com blog