Lines of code and acceptance rate do not measure AI coding ROI. Here is what to track instead: rework rate, cycle time, and total review cost.
Every team that adopts an AI coding assistant eventually gets asked the same question by a VP or a finance partner: is this thing actually paying for itself? The instinctive answer is to point at acceptance rate in the Copilot dashboard, or lines of code generated per week, or the number of pull requests merged. All three numbers go up. All three numbers are close to useless for answering the actual question, which is whether the team ships more working software per dollar and per engineer-hour than it did before.
The gap between "feels faster" and "is faster" is not a hunch. A randomized controlled trial from METR in 2025 found that experienced open-source developers who were allowed to use AI tools on real issues in their own repositories took 19% longer to complete those issues than when they worked without AI, even though the same developers predicted beforehand that AI would speed them up by 24%, and still believed afterward that it had sped them up by 20%. That is not a story about bad tools. It is a story about how bad humans are at estimating their own productivity when the friction of AI use (prompting, reviewing, correcting) is invisible in the moment and the friction of typing is not.
Sources: METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Why the obvious metrics lie
Lines of code, suggestion acceptance rate, and commits per engineer share a common flaw: they measure output volume, not outcome. An AI assistant can inflate all three while making the codebase worse. Verbose, over-abstracted, duplicated code accepts more suggestions and produces more lines than a tight, well-factored diff that solves the same problem in a third of the space. None of that shows up in a dashboard that only counts keystrokes saved.
Acceptance rate is especially misleading because it measures a moment, not an outcome. A developer can accept a suggestion, ship it, and then spend the next two days debugging a subtle issue it introduced. The acceptance event is recorded as a productivity win. The two days of rework are recorded nowhere, or worse, they show up as unrelated bug-fix commits that make the team look busy rather than slow.
- Lines of code and commit counts measure typing, not problem solving.
- Acceptance rate measures a single accept/reject decision, not what happens after.
- PRs merged per week rewards small, easy tickets and penalizes teams tackling hard ones.
- None of these metrics account for the reviewer time spent catching what the model got wrong.
Perceived speedup versus actual speedup
The METR result matters beyond its headline number because of what it implies about self-reported metrics generally. If experienced developers, working on code they know well, systematically overestimate how much AI is helping them by roughly 40 percentage points, then any ROI measurement built on developer surveys or subjective confidence ratings is measuring sentiment, not throughput. Sentiment is worth tracking, since it affects retention and adoption, but it should never be the number a team reports to finance as "productivity gain."
The fix is to measure the same unit of work (a ticket, an issue, a well-scoped PR) from open to merged-and-stable, and compare that cycle time across a period with AI assistance against a comparable period or comparable team without it. This is harder than reading a dashboard. It also happens to be the only version of the number that survives contact with a skeptical CFO.
Rework and defect rate: the cost AI tools hide
The single most important number missing from most AI-coding ROI conversations is rework rate: the percentage of AI-assisted changes that require a follow-up commit within a short window (say, two weeks) to fix a bug, revert a regression, or address review feedback that a human-written equivalent would not have needed. If an AI assistant helps a developer produce a PR in half the time, but that PR needs three rounds of review and a hotfix a week later, the true cycle time (open to genuinely done) may be no better than the manual baseline, or worse.
Defect rate matters even more once you account for where defects are caught. A bug caught in code review costs a review comment and a re-push. A bug caught in staging costs a deploy cycle. A bug caught in production costs an incident, and possibly a security disclosure if the defect is a vulnerability an AI assistant introduced with no awareness of your threat model. Teams that only look at "time to green CI" are blind to this cost gradient entirely; a multi-agent code review pass or an explicit AI code review checklist is what actually catches this before it reaches production, and its cost has to be counted as part of the AI workflow, not treated as free.
AI coding ROI is not "time to write the code" divided by cost. It is "time to a change that stays fixed" divided by cost, including every hour spent reviewing, correcting, and re-reviewing what the model produced.
Cycle time is the metric that cannot lie to you
Cycle time (the elapsed time from when work starts on a ticket to when it is merged, deployed, and has not been reverted or hotfixed) is harder to game than any single-stage metric because it forces every hidden cost through one door. If AI assistance speeds up the drafting phase but slows down the review phase because reviewers now have to read denser, less familiar diffs, cycle time captures the net effect. If it speeds up both drafting and review but produces more rollbacks two weeks later, extending the measurement window to include stability catches that too.
The practical version of this: track cycle time per ticket size (small, medium, large, using whatever sizing convention the team already has) before and after AI adoption, segmented by whether AI was used heavily, lightly, or not at all on that ticket. Do not average across ticket sizes; a team that starts using AI heavily on small, boilerplate tickets and lightly on complex ones will show an average speedup that has nothing to do with how AI performs on the hard problems that actually matter to the roadmap.
Total cost of ownership, not just time saved
A full ROI calculation has to include the license cost of the tooling itself, the token or subscription spend for the models in use (which can vary meaningfully depending on which model you choose for coding, since model performance for coding is not uniform across tasks), the reviewer time spent on AI-authored diffs, and the incident cost of anything that reached production and had to be rolled back or patched. Most teams track the first two (they show up on an invoice) and ignore the last two (they show up scattered across engineering time and incident postmortems, never itemized as "AI review overhead").
This is also where a process layer earns its keep rather than adding friction. A workflow that runs an independent spec check and a red-team pass before code ships costs some wall-clock time up front, but it converts an unbounded, unpredictable production-incident cost into a bounded, predictable review-time cost. That trade is close to always worth making; the question worth asking is whether your current setup makes that trade deliberately or by accident. TLM Forge is built around forcing that trade to be explicit: how it works walks through the spec audit, independent review, and convergence gate that sit between "AI wrote a diff" and "the diff merges."
Before adopting a new AI-coding metric, ask: "could this number go up while the codebase gets objectively worse?" If yes, pair it with a defect-rate or rework-rate check, or drop it.
What to measure and what to distrust
A useful ROI dashboard is short. It tracks cycle time by ticket size and AI-usage tier, rework rate within a fixed window post-merge, defect escape rate by stage caught (review, staging, production), and reviewer hours spent per AI-assisted PR versus per manually written PR. It deliberately excludes lines of code, acceptance rate, and raw commit counts as headline numbers, though those can still be useful as secondary diagnostics for adoption and tool usage patterns.
| Metric | What it actually tells you | Better alternative |
|---|---|---|
| Acceptance rate | How often a suggestion was clicked, not whether it was correct | Defect escape rate on accepted suggestions |
| Lines of code generated | Typing volume, not problem-solving progress | Cycle time per ticket, normalized by size |
| PRs merged per week | Ticket throughput, biased toward easy work | Cycle time segmented by ticket complexity |
| Developer self-reported speedup | Sentiment; shown by METR to diverge ~40pp from reality | Measured cycle time, same team, before/after |
| Time to first green CI | Whether tests passed once, not whether the change held | Rework rate within 2 weeks of merge |
None of this requires exotic tooling. Most of it can be pulled from existing issue-tracker timestamps, git blame on revert and hotfix commits, and review-tool time-in-stage data. The barrier is rarely instrumentation; it is that "lines of code went up" is a more comfortable story to tell in a quarterly review than "we are still measuring whether this is actually working." Teams serious about the answer treat AI coding ROI the same way they would treat ROI on any other engineering investment: with a control group, a defined unit of work, and a willingness to find out the number is disappointing.
It is also worth separating "does AI coding help at all" from "is our current workflow capturing the help it provides." Some of the productivity myths around AI coding tools come from measuring the wrong thing rather than from the tools being ineffective; see AI coding productivity myths for a broader look at where those myths come from. Context loss is a related, underrated tax on real ROI: an assistant that has to re-derive the same architectural context every session burns tokens and developer patience without producing a corresponding output gain, which is why context management for AI coding is worth treating as a cost-control lever, not a nice-to-have. Persisting that context outside the coding session itself, the way MemX keeps a private, persistent memory layer across tools rather than forcing every session to start cold, is one way teams reduce that specific overhead, though it addresses context cost rather than review cost or defect rate.
Frequently asked questions
01Is the METR 19% slowdown finding still representative of current AI coding tools?
METR itself has flagged the result as reflecting a specific snapshot: experienced developers using early-2025 models (primarily Cursor Pro with Claude 3.5/3.7 Sonnet) on mature repositories they already knew well. Models and tooling have moved since then, and METR has since revised its experiment design for future studies. The finding is best read as a durable warning about the size of the perception-versus-reality gap, not as a fixed, permanent number for all AI coding tools going forward.
02What is the single best metric to start with if we track nothing today?
Cycle time per ticket, segmented by ticket size and by how heavily AI was used on that ticket. It is derivable from data most teams already have in their issue tracker and git history, it is hard to game by simply generating more code, and it naturally absorbs review and rework overhead instead of hiding it.
03How long should the rework-rate measurement window be?
Two weeks post-merge is a reasonable default for most web and backend work; it is long enough to catch regressions that surface under real traffic or a follow-up sprint of related changes, but short enough to still causally link the fix to the original change. Teams with slower release cadences or higher-risk domains may need a longer window.
04Does a heavier review process for AI-generated code cancel out the speed gains?
It reduces raw drafting speed gains, but that is the point: a review step that catches defects before merge converts an unbounded, unpredictable production-incident cost into a bounded, predictable review-time cost. When measured on total cost including rework and incidents rather than time-to-first-draft alone, disciplined review is typically the difference between a real ROI gain and a net loss that only looked like a gain.