← Back to BlogAI Coding

Study: AI Agents Raise Code Output, Not Resolved Issues

Two Harvard researchers studied 718 firms: agents raised code output, resolved issues showed no significant rise, and review slowed. What to measure.

If your team ships more AI-written code but nothing finishes faster, a study of 718 firms by two Harvard researchers found the same pattern at scale. After firms adopted AI coding agents, lines of code, commits and pull requests per worker rose, but resolved Jira issues did not rise by a statistically significant amount, and code review got slower, with more change requests and comments.

Sources: Chen and Stratton, Artificial Intelligence in the Firm: Bottlenecks in Software Production (PDF, Aug 4, 2026 version), InfoQ: Study Finds AI-generated Code Fails to Boost Productivity Due to Downstream Bottlenecks (Oct 22, 2026)

The authors read this as a code review bottleneck: writing code is one stage of several, and review limits how much of the extra code becomes finished work. Our practical takeaway, which the study did not test, is to measure completed work instead of output, shrink what you hand to reviewers, agree on the design before the code exists, and verify before a pull request is opened. This post covers what the paper measured, what it did not, and a table of metrics to swap.

Insight

Disclosure: we build TLM Forge, a process layer for Claude Code that moves checking earlier, so weigh the product section near the end accordingly.

What the study actually is

The paper is a job market paper by Fiona Chen and James Stratton of Harvard, whose current version is dated August 4, 2026 and which InfoQ covered on October 22, 2026. It uses data on roughly 300 million work events and 700,000 workers across 718 firms, from January 2021 to March 2026, drawn from HR systems, version control such as GitHub, issue trackers such as Jira, calendars, and usage of GitHub Copilot, Cursor and Claude Code.

Sources: Chen and Stratton (PDF), InfoQ coverage

The method matters more than the headline. The authors use a staggered difference-in-differences design: they compare firms that adopted AI tools earlier with firms that adopted later or not at all, relying on the fact that procurement, security review and piloting take different amounts of time at different firms. That makes it stronger evidence than a survey of how developers feel, though it is still observational and rests on an assumption that early and late adopters would have followed similar trends.

Sources: Chen and Stratton (PDF), sections 1 and 3

Two disclosures belong here. The data partner, Jellyfish, sells an analytics platform that engineering teams use to understand their own activity. The authors state that their conclusions do not represent Jellyfish's views and that the company reviewed the manuscript only to check that it disclosed no proprietary information, not for scientific content. The research was funded by the Stripe Economics of AI Fellowship, a Harvard Kennedy School program and the Schmidt Sciences AI at Work Program.

Sources: Chen and Stratton (PDF), abstract and acknowledgments

What changed, and what did not

The paper separates AI assistants (autocomplete and chat) from AI agents, and it estimates the agent effect as the added effect of adopting agents on top of assistants. That distinction matters when you compare the two sets of numbers, because the agent figures are the extra effect on top of assistant adoption. The coding and issue figures below are per-worker, per-month effects measured at the firm level, so each blends heavy users with people who barely touched the tools.

Sources: Chen and Stratton (PDF), section 3.1

Coding output went up

  • Agents: lines of code added up 30% (1,494 more lines per worker-month on a baseline of 5,017), commits up 20% (baseline 22.58), pull requests created up 23% (1.22 more per worker-month on a baseline of 5.34). All three were statistically significant.
  • Assistants: lines up 12%, commits up 9%, pull requests up 5%. Only the commit effect was statistically significant.

Resolved issues did not significantly rise

The paper counts a Jira issue as resolved once the related code is deployed to production, and it also tracks resolved epics, which are whole features or projects. For agents the estimate was 0.12 more issues per worker-month on a baseline of 3.67, with a standard error of 0.17, which is not statistically significant. The 95% confidence interval rules out an effect above 12% of the baseline, well below the 30% coding gain. Epics also showed no significant effect.

Sources: Chen and Stratton (PDF), section 3.3

InfoQ's summary differs from the paper on two numbers. It says agents brought a 12% increase in resolved issues and that comments per pull request rose 53%. In the August 4 version we read, the 12% is the upper bound that the confidence interval rules out, not a measured gain (the point estimate is 0.12 on a baseline of 3.67, roughly 3%, and not significant), and the comment figure is 35%. InfoQ's coding, merge-time and reviewer-share figures match the paper. Use the paper's numbers when you cite this study.

Sources: Chen and Stratton (PDF), sections 1, 3.3 and 4.1, InfoQ coverage

The authors checked whether bigger issues explain the flat count, and found no evidence of a shift in predicted issue size. They also checked employment: the agent effect on overall employment was a precise zero, which rules out a decline greater than 2.9%. Neither finished work nor overall headcount showed a significant change, so the coding gain did not visibly show up as either.

Sources: Chen and Stratton (PDF), sections 1, 3.3 and 3.4

Where the time went: review

The authors interpret the results as a bottleneck. Writing code is only one stage, and the authors' estimates say engineers spend 42% of their time writing code by an imputation from Jellyfish data and 30% in a survey of 100 Prolific engineers. When one stage speeds up, the stages around it set the pace. They test review specifically, and for agents the review measures they tracked all got heavier.

Sources: Chen and Stratton (PDF), section 4.1

  • Time from pull request submission to merge rose 49%: 3.45 more days (standard error 1.60) on a baseline of 7.03 days.
  • The share of pull requests with at least one formal change request rose by 0.12 from a baseline of 0.13, which the authors describe as nearly doubling the rate.
  • Review comments per pull request rose by 0.58 (baseline 1.66), a 35% increase.
  • The share of workers who review code rose 14% (0.04 on a baseline of 0.29). The authors found no significant change in the share of workers who write code, so firms appear to have pulled more people into review rather than moving people out of coding.

For assistants, none of the three review-process measures (time to merge, change requests, comments per pull request) moved by a statistically significant amount. The authors also looked at whether pull requests were simply getting bigger and found no significant change in lines added per pull request, so larger diffs do not appear to explain the extra change requests and comments. They offer two possible reasons for those: worse code entering review, or changed review standards. Their model reads the increase as a code-quality effect, but the data do not include code content, so that is an interpretation rather than a direct measurement. Our inference is that a smaller pull request alone is not a complete answer.

Sources: Chen and Stratton (PDF), section 4.1

AI review tools did not close the gap in this data. By March 2026 nearly 80% of firms had used AI in code review, yet in the paper's data only 23.3% of review comments were AI-generated and only 10.8% of pull requests received at least one AI comment. Humans stayed heavily involved. For related evidence on the same bottleneck, see our coverage of Qodo's 2026 Report: Review Is the AI Coding Bottleneck.

Sources: Chen and Stratton (PDF), section 4.3

What this study cannot tell you

  • It is a working paper from one dataset, drawn from Jellyfish clients who consented to research use, and the Jira and GitHub fields are proxies. A team with unusual Jira habits will read differently.
  • Adoption is measured at the firm level, so the effects are averages over mixed usage. A team that uses agents heavily may see more, or less.
  • Agent adoption is inferred partly from GitHub bot and signature signals, and the tools evolved during the window.
  • It shows that review gets slower and draws more change requests and comments. It does not measure code quality directly, though the authors model worse code as one channel alongside higher volume.
  • The authors say they would generally expect any remaining confounders to bias the estimates upward, toward overstating effects.

What to change on your team

The paper's conclusion says that measuring the returns to AI requires evaluating the entire production process, that intermediate metrics like lines of code or commits may not capture the true gains, and that realizing the benefits requires complementary organizational investments. The five changes below follow from that. They are our reading, not a result the study tested.

Sources: Chen and Stratton (PDF), conclusion

1. Measure completed work, not output

Count issues or features that reached production per person and per month, and the days from first commit to merge. Treat commit and line counts with caution, since for agents the study found them up 20% to 30% while resolved issues showed no significant change. The wider method is in How to Actually Measure the ROI of AI Coding Tools.

2. Shrink the batch each reviewer sees

Cap what an agent can deliver in one pull request by task, not by line count: one behavior change, one reviewable story. Reviewer attention is the scarce resource, and a single senior approver for every change becomes a queue. One junior engineer quoted in the paper described a project where one expert could approve changes and reviewed once a week.

Sources: Chen and Stratton (PDF), section 4.1

3. Agree on the design before the code exists

A change request on a finished diff often means the agent built something the reviewer did not want, and every rework loop restarts the clock. Approving a short design first, covering files touched, behavior, and what is out of scope, moves the disagreement to the cheapest moment. The reviewer then checks the code against an agreed plan instead of reconstructing intent, a problem we cover in I Don't Understand the Code My Agent Wrote.

4. Verify before the pull request opens

Tests, the build, a linter and a fresh-context review pass should finish before a human sees the diff, so reviewers spend their time on judgment, not on catching failing builds. Anthropic's account of reviewing code that is mostly written by Claude takes a similar approach, with layers of automated checks and human approval tiered by risk, described in our write-up How Anthropic Does Code Review on 90% AI-Written Claude Code. If the agent loops on one failure, stop it early: see Why Claude Code Keeps Fixing the Same Bug.

Sources: How the Claude Code team does code review (claude.dev, Oct 15, 2026)

5. Track rework, not just speed

The share of pull requests that come back with requested changes is one of the clearest signals in the paper, because the paper found it nearly doubled for agents. Track it per team, per author type, and per task size, and review the trend monthly. If it climbs while commits climb, that may mean you are generating work for reviewers rather than finishing it.

Common metricWhat the study foundWhat to track instead
Lines of code addedAgents: up 30%. Assistants: up 12%, not significantIssues or features in production per person per month
CommitsAgents: up 20%. Assistants: up 9%Days from first commit to merge, by task size
Pull requests openedAgents: up 23%. Assistants: up 5%, not significantPull requests merged without a change request
Time to reviewAgents: pull request submit-to-merge time up 49% (7.03 to about 10.5 days)Review queue age and pull requests waiting more than two days
Review commentsAgents: comments per pull request up 35%Comments that led to code changes, grouped by cause
Change-request rateAgents: nearly doubled from a 13% baselineRework rate per team, tracked monthly with the plan it was built from
Resolved issuesAgents: no significant change; 95% interval excludes gains above 12%Same, plus features shipped (epics) and incidents after release

Where TLM Forge fits

TLM Forge sizes each task Light, Medium or Deep, and Light tasks get no review agents. For Medium and Deep it shows a design for your approval before building. It tells Claude to build tests first, then runs a final review wave of independent reviewer agents: a code-reviewer and phase-auditor on Medium, more on Deep. Details are on the final audit and risk classification pages.

The limits are real. The phase-auditor is an LLM agent and can miss things, and the test-first rule is an instruction rather than something a hook checks. We have no evidence that it shortens human review time, lowers your change-request rate, or reproduces any number in this study, and we make no savings claim. It moves some checking earlier, which is our own response to the review bottleneck, not something the paper tested. Your own completed-work metrics will show whether it helps.

Frequently asked questions

01Do AI coding agents actually make teams ship faster?

In this firm-level study, agents raised code output (pull requests up 23%) but did not produce a statistically significant rise in resolved Jira issues, and the interval rules out gains above 12%. Individual tasks can still go faster. The finding is about resolved issues across whole firms.

02Why is code review slower after adopting AI agents?

For agents, the study found more pull requests, more change requests and more comments per review. Merge time rose 49%. The authors name two possible channels: more code to review, and a changed review process, from worse draft code or different standards.

03Is the Chen and Stratton Harvard study reliable?

It uses a large dataset and a staggered adoption design, which is stronger than a survey. It is still a working paper using proxies such as Jira issue counts, with firm-level adoption. Jellyfish, which sells engineering analytics, supplied the data but did not review the science.

04What should I measure instead of lines of code?

Track work that reaches production: issues or features completed per person, days from first commit to merge, review queue age, and the share of pull requests needing changes. Lines, commits and pull requests opened all rose with agents while resolved issues showed no significant rise, so they may overstate progress.

05Can AI code review fix the review bottleneck?

The study did not find that it had. By March 2026 about 80% of firms had used AI in review, yet AI wrote only 23.3% of review comments and 10.8% of pull requests got any AI comment. Humans stayed heavily involved, and the authors say the bottleneck persisted.

Ship AI-written code you can trust

TLM Forge is the missing process layer for Claude Code: a spec audit, independent multi-agent review, enforced TDD, and an adversarial red-team gate.

Get TLM Forge