Four common claims about AI coding speed do not hold up: universal speedups, optional review, code volume as progress, and tests becoming unnecessary.
Ask a team how much faster AI coding assistants have made them, and you will get a confident number. Ask them to show the math behind that number, and the confidence usually thins out. A handful of claims about AI coding speed get repeated in standups and blog posts often enough that they have started to sound like settled facts, even though none of them survive close inspection.
This is not an argument against using AI to write code. It is an argument against measuring productivity by the wrong signal: how fast a first draft appears. Teams that ship faster over a quarter, not just a sprint, are the ones that pair a fast draft with a process that catches rework before it compounds. Everyone else is trading a small daily win for a slower quarter.
Myth 1: AI Makes Every Developer Instantly Faster
The claim sounds obvious: a tool that writes code for you should make you faster at writing code. The trouble is what happens after the draft appears. A 2025 randomized controlled trial by METR gave experienced open source developers real tasks in codebases they already knew, then let them use AI coding tools for a random half of those tasks and not the other half. Developers were, on average, about 19 percent slower when using AI, even though beforehand they predicted a large speedup and afterward still believed AI had made them faster. The gap between how fast a task felt and how fast it actually was is the whole problem with this myth. Some of that slowdown is fixable and some of it is not: a meaningful share comes from context loss between sessions, where a model re-explains itself, re-derives your conventions, and repeats a mistake it already made yesterday because nothing carried forward. A private, persistent memory layer like MemX is aimed at exactly that gap, so decisions your team already made do not have to be re-litigated every session. That reduces one source of rework. It does nothing for the other three myths below, which are process problems rather than memory problems.
Sources: METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
- Task familiarity: unfamiliar territory rewards AI drafting more than code you already know cold.
- Review overhead: every generated diff has to be read, understood, and checked against intent, and that time rarely gets counted as part of the task.
- Rework cost: code that looks right and compiles clean can still encode the wrong assumption, and fixing it later costs more than writing it correctly the first time.
- Codebase size and convention debt: large, opinionated codebases punish generic output more than a greenfield script does.
Myth 2: Review Is Optional Now
If a model can produce a passing test and a clean diff, it is tempting to treat review as a formality. This is where the biggest hidden cost lives. Generated code fails in ways that look nothing like typical human bugs: it invents a plausible-sounding function that does not exist in your dependencies, quietly drops an edge case the ticket implied but never stated, or solves last week's version of the problem because the context it was given did not include a file that changed yesterday. None of those failure modes show up from a quick skim and a gut check that the diff "looks reasonable." They show up when someone deliberately checks the change against the actual spec and the actual codebase, which is the core idea behind an AI code review checklist rather than an informal glance.
Skipping review does not remove the review step. It just moves it later, into production, where the same defect costs an incident report instead of a comment thread.
Myth 3: More Code Shipped Equals More Progress
Lines of code, pull requests merged, and commits per day are easy to count and easy to inflate. AI tools make all three trivially easy to pad: ask for a feature, get five files and three hundred lines back in ten seconds. None of that tells you whether the change moved the product forward, whether it introduced a dependency nobody asked for, or whether it will need to be substantially rewritten the next time someone touches the same file. A ten-line fix that closes the actual bug beats a two-hundred-line refactor that papers over it. Shipped is not the same as done, and done is not the same as durable. Managers who track output volume as a proxy for progress end up rewarding whoever generates the most text, not whoever solved the problem with the least surface area, and that incentive gets stronger, not weaker, once AI makes large diffs cheap to produce. The real question is not how much code landed this week, but how much of it will still be there, unchanged, in a month.
Myth 4: AI Removes the Need for Tests
A model that can generate a function can generate a test for that function, so the argument goes: testing becomes automatic. In practice, a model asked to test its own output tends to test what the code does, not what the code was supposed to do. If the implementation misunderstood the requirement, a self-generated test built on the same misunderstanding will pass every time and prove nothing. Tests written independently, ideally against the actual specification the way spec-driven development describes, are what catch that class of failure. AI can help write test cases fast; it cannot replace the discipline of writing them against the requirement instead of against the code that was just produced.
When a model proposes both the implementation and the test in the same pass, read the test first. If the test would still pass given a subtly wrong implementation, it is not testing the right thing.
The Real Throughline: Fast Draft Plus Fast Catch
Every one of these myths shares the same failure pattern: they measure the visible, fast part of the loop, which is draft generation, and ignore the invisible, slower part, which is review, rework, and validation. A faster first draft is a real and valuable thing. It is just not the same thing as faster delivery. Teams that get a genuine productivity gain from AI are the ones that shorten the distance between a mistake being introduced and a mistake being caught, not the ones that generate the most code the fastest.
| Myth | What people assume | What actually determines the outcome |
|---|---|---|
| AI makes everyone faster | Generation speed equals delivery speed | Task familiarity, review overhead, and rework decide the real number |
| Review is optional | A clean diff means a correct diff | Generated code fails in unfamiliar ways that only deliberate review catches |
| More code means more progress | Volume shipped tracks value delivered | Durability matters more than volume; rewritten code was never really done |
| Tests are unnecessary | The model can validate its own output | Independent tests against the spec catch what self-generated tests miss |
None of this means slowing down. It means moving the checks that used to happen informally, inside one senior engineer's head, into a process that runs on every change: a spec audit before code gets written, independent review of both the plan and the output, and a gate that will not let known issues ship quietly. TLM Forge is built around that loop, and you can see the mechanics on the how it works page: a spec audit before any code, multi-agent review of the plan and the diff, mechanical test-driven development, and an adversarial red-team gate that has to hit zero critical issues before anything ships. None of those stages ask a developer to slow down and second-guess a fast draft by hand; they run the same way on every change, so the catch happens whether or not anyone remembers to look. The draft can still be fast. What changes is how quickly a wrong assumption gets caught, instead of how long it survives in the codebase.
Frequently asked questions
01Does this mean AI does not help with coding productivity at all?
No. It means the benefit shows up in draft speed, not automatically in delivery speed. A faster draft is real value if the review and validation steps that follow it are fast and rigorous too. If those steps stay ad hoc, the draft speedup gets eaten by rework later.
02How do you measure whether AI is actually saving time, not just producing more code?
Track outcomes that survive contact with review: pull requests that merge without a second review round, defects caught in review versus found in production, and how much of a change gets rewritten within the next few weeks. Lines shipped and commits per day do not tell you any of that.
03Is skipping tests ever reasonable when AI writes the implementation?
Only for throwaway scripts with no lasting consequence. For anything that ships to users, an independently written test against the actual requirement is what catches the gap between what the code does and what it was supposed to do, a gap that self-generated tests systematically miss.