How Real Are AI Coding Productivity Gains? A Review of NBER w35275

The Paper

Mert Demirer (MIT), Leon Musolff (Wharton), Liyuan Yang (MIT), "Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools", NBER Working Paper No. 35275, May 2026. DOI: 10.3386/w35275

The NBER (National Bureau of Economic Research), founded in 1920, is the nonpartisan nonprofit that officially dates US recessions. Working papers, however, are circulated for discussion and have not been peer-reviewed. Read this as the fastest look at fresh data — not as settled literature.

What the Study Asks

Two questions: how have productivity effects changed across three generations of AI coding tools — autocomplete, synchronous agents, and asynchronous agents — and how much of the gain at the code-writing level actually reaches shipped software? The data combines public GitHub activity from over 100,000 developers, Microsoft's internal Copilot telemetry, and launch-and-consumption records from four app marketplaces (App Store, Google Play, Chrome Web Store, SourceForge).

The method is a matched event study: adopters are compared not to contemporaneous non-adopters, but to developers with similar activity exactly one year earlier in the same calendar week. Contemporaneous controls were rejected because of contamination (non-adopters may use AI privately) and selection bias. Outcomes are normalized by each developer's 11-week pre-adoption average. As observational designs go, this one is careful.

Core Results: Bigger Each Generation, Smaller Each Layer

Using the authors' preferred 21–30 week post-adoption window, the paper's backbone is how effects decay across six production layers: lines of code → files → commits → PRs → repositories → releases.

The abstract's "commits +180% → repos +50% → releases +30%" is the cumulative effect of adopting all three generations, while the table above shows marginal per-generation effects. The two reconcile exactly — cumulative multiplier = 1 + sum of marginal effects — across all six layers, so the seemingly contradictory numbers in the abstract simply use different baselines. Async agents' PR effect (+72%) exceeding their commit effect (+34%) is no anomaly: these tools create PRs directly, intervening at that layer rather than letting upstream gains trickle down.

Heterogeneity points the same way. For sync agents, developers in the lowest pre-adoption activity quartile gained +217% in commits versus +62% for the most active quartile — all three generations help low-activity developers most. The market-side data is sobering: new iOS apps roughly doubled from 30–50k to about 100k per month, yet cohort-level consumption stayed flat, and the share of new iOS apps failing to attract even 10 ratings within three months rose from about 79% to 86%.

Reliability Assessment

The credibility case is reasonably strong. Placebo runs with non-AI tools show GitHub Pro at essentially 0% long-run effect; pre-trends are flat; the autocomplete estimate (+36%) is consistent with a prior field experiment on the same tool (~+26%). A dynamic analysis ties Claude Code's rising effect to model release timing rather than user learning, which is persuasive.

Three caveats carry real weight. First, conflicts of interest: two of the three authors are paid Microsoft research consultants, and the key telemetry is Microsoft-internal (disclosed in the paper). Second, contradicting experimental evidence: the Becker et al. (2025) RCT found agentic coding tools reduced experienced developers' productivity by 19% — an unresolved sign flip against this observational study. Third, coverage: only public repositories are observed, identifying just 24.2% of async-agent users, and enterprise-internal software is entirely absent. Whether commit counts measure productivity at all remains a fundamental objection.

Reviewer's Verdict: Only One Tool Actually Shipped

The number this review found most important is one the paper does not headline — per-tool release effects.

All three tools significantly increased commits; only one translated into shipped releases. The authors note that Claude Code's CLI-first usage may select for more experienced adopters, and concede their design cannot separate tool effects from adopter composition. That concession, in our judgment, is the paper's real message: tools don't break bottlenecks — organizations and individuals with an already-functioning release pipeline profit from tools. If the pipeline is blocked, switching tools inflates commits and leaves releases unchanged.

The structural estimate points the same way. The calibrated elasticity of substitution σ ≈ 0.25 means AI-produced upstream output and human review effort are strong complements: automate upstream all you like, and final output still hits a finite ceiling as long as downstream judgment stays human-bound. You cannot buy your way past that ceiling with more coding tools. The scarce resource has shifted from writing code to deciding whether it may ship.

What to Do About It

Within the paper's model, only a few levers raise final output. The actionable ones:

• Evidence-bundled PRs — require AI to submit impact scope, tests, coverage diffs, and rollback steps alongside code, turning reviewers from readers into judges of evidence. This needs only PR templates and merge rules, not new tools.
• Deploy AI downstream — regression-test selection, release notes, change-risk scoring. Empirically, async agents' PR-layer effect (+72%) beat their commit-layer effect (+34%): intervening closer to the output wins.
• Instrument the decay curve — measure monthly pass-through across the six layers (lines → files → commits → PRs → repos → releases) to find your bottleneck before investing. Buying more coding tools without knowing the bottleneck is waste by definition.
• Shrink release batches — releases in the sample occur just 0.37–1.17 times per developer-week. Trunk-based development and feature flags widen the exit today, no AI required.

Conclusion

The surface reading is pessimistic — AI productivity gains don't reach final output well. But where autocomplete entered only one production layer, agent generations have already climbed to the PR layer. The bottleneck hasn't disappeared; it is moving up, and today it sits at verification and judgment. The value of people standing at that layer will rise — but only for those who design judgment into an automatable form, not for those who merely guard the gate.

References

• NBER Working Paper 35275 — paper page
• Full PDF (43 pages + online appendix)