AI-generated tests often pass without verifying anything. How to spot tautological assertions and mock-heavy tests, and use mutation testing to check.
Ask an AI coding assistant to build a feature and write tests for it, and you will get a green checkmark within minutes. Run the suite, watch every test pass, and it is easy to treat that as proof the code works. Usually it is not. The fastest way for a model to produce a passing test is to trace through the code it just wrote, compute what that code currently returns for a given input, and assert on that exact value. The test never encodes what the function was supposed to do, only what it happens to do right now.
This is the dominant failure mode in AI-generated test suites: tests that are syntactically clean, fully passing, and structurally blind to regressions. A tautological test passes today and will keep passing after a bug changes the behavior tomorrow, because the expected value was derived from the implementation instead of from the specification. Treating a green test run as verification, when the same process wrote both the code and the test, is circular. Getting real signal out of AI-written tests takes a different discipline: writing behavior first, forcing edge cases explicitly, reading coverage numbers skeptically, and periodically checking whether the tests can catch anything at all.
Why passing tests do not mean working code
When a model generates tests for code it just wrote, it has complete visibility into the implementation. The path of least resistance is to read the logic and assert on whatever it currently produces, which yields a test that looks like a regression test but carries almost no information. It will pass for the correct implementation and for nearly any variant that preserves the same code path, including a version with the exact bug you are trying to prevent.
The same shortcut shows up with dependencies. A 2026 study that analyzed more than 1.2 million commits across 2,168 TypeScript, JavaScript, and Python repositories found that coding agents added or changed test files in 23% of their commits, compared with 13% for human-authored commits, and added mocks to those tests in 36% of cases versus 26% for humans. A mock is easier to generate than a real fixture, but a test that mocks the unit under test's own dependency and then asserts the mock returned what it was told to return has verified nothing about the real code path. These are exactly the blind spots a guardrails for AI-generated code checklist is meant to catch before merge.
Sources: Hora & Robbes, "Are Coding Agents Generating Over-Mocked Tests? An Empirical Study" (2026)
Three patterns that produce hollow tests
- Tautological assertions: the expected value is computed by calling the function under test, so any change that alters both the implementation and the "expected" output at once goes undetected.
- Mock theater: the test mocks the very dependency it should be exercising, then asserts the mock was called with the arguments the code passed it, verifying wiring instead of behavior.
- Implementation mirroring: the test reimplements the algorithm instead of hardcoding expected values from the spec, so a bug in the logic gets copied into the test and cancels itself out.
- Coverage padding: every branch executes once with a weak or missing assertion, so the coverage percentage climbs while the number of catchable defects does not move.
| Trait | Weak AI-Generated Test | Strong Test |
|---|---|---|
| Where the expected value comes from | Computed by calling the function under test | Derived from the spec or ticket before the code was written |
| What makes it fail | Almost nothing; it was derived from current behavior | Any change that alters the documented behavior |
| Mock usage | Mocks the dependency being tested, then asserts the mock fired | Mocks only true external boundaries: network, disk, clock |
| Edge cases covered | Only the happy path implied by the prompt | Empty, null, boundary, negative, and failure-path inputs, stated explicitly |
| Coverage vs mutation score | High line coverage, low mutation score | Moderate coverage, high mutation score |
Behavior-first tests: assert against the spec, not the implementation
The fix is procedural, not aspirational. Before code exists, or before an assistant is allowed near it, write down a handful of input/output pairs from the ticket, the API contract, or the domain rules, not from running anything. Those pairs become the test's expected values. When the assistant generates the implementation afterward, its output has to match a target that was fixed independently, so it cannot quietly launder a wrong implementation into a matching test. This is close to why mechanical TDD enforcement requires a test to exist and fail before any implementation is written: it pins the expected value down first, which rules out the tautological case by construction.
Before asking an AI assistant to write tests for a function, write 3-5 input/output pairs yourself, including at least one edge case, without looking at the implementation. Hand those to the model as fixed fixtures it must assert against. It can still write the boilerplate around them, but it cannot quietly derive the expected values from the code it is testing.
Force boundary and edge cases into scope
Left to its own defaults, a model tends to generate one happy-path test per function, because that is the example it can infer most confidently from the prompt and the code. Boundary conditions have to be requested explicitly, every time, because the model has no way to know which edge cases have actually broken this codebase in the past unless the context is supplied again.
- Empty and null inputs: empty strings, empty arrays, missing optional fields.
- Boundary values: the first and last valid index, zero, negative numbers, the exact max or min allowed.
- Duplicate and out-of-order input: repeated IDs, out-of-order timestamps, concurrent writes to the same key.
- Malformed or partial data: truncated payloads, wrong types, unicode and encoding edge cases.
- Failure paths: timeouts, partial writes, a dependency that returns an error instead of a value.
Part of the reason boundary cases go missing is that every AI session starts cold. The model has no memory of the null-pointer bug this same module produced three weeks ago, so nothing prompts it to test for that case again unless the context is resupplied. Keeping a running list of known edge cases and past incidents, whether in a spec file, a changelog, or a persistent memory layer like MemX, turns a one-off catch into a standing requirement instead of knowledge that quietly evaporates between sessions.
Coverage measures what ran, not what was verified
Line and branch coverage answer one question: did this code execute during the test run. They say nothing about whether the assertions that ran alongside it would notice if the logic were wrong. A try/except block can reach 100% coverage from a test that triggers the exception and asserts nothing about it, or asserts only that some exception propagated, missing the fact that the wrong exception type was raised. AI-generated suites are especially prone to this gap because the model optimizes for a passing, executed test, and a coverage number rewards execution, not verification.
Coverage percentage answers "did this code run during testing." It never answers "would this test catch a bug." Those are different questions, and only a test that fails on a broken implementation answers the second one.
Mutation testing: testing the tests
Mutation testing answers the question coverage cannot. Tools such as PIT for Java or Stryker for JavaScript, TypeScript, and C# automatically introduce small deliberate bugs, called mutants, into the code: flipping a comparison operator, changing a constant, negating a condition. The existing test suite then runs against each mutated version. A mutant that causes a test to fail is "killed"; a mutant that slips through with every test still passing has "survived." The mutation score, the percentage of mutants killed, measures whether the suite would actually catch a behavioral change, which is precisely the property tautological and mock-heavy tests lack.
Sources: Stryker Mutator, "What is mutation testing?"
A mutation testing framework is not required to get a version of this signal today. Pick a conditional or a boundary check in a function an assistant just wrote, invert it by hand, and rerun the suite. If nothing fails, that test would not have caught the bug you were worried about, which is the same gap a full red-teaming pass is designed to surface before code ships. This check belongs in the same review pass as the rest of an AI code review checklist: a fully green suite that cannot detect an inverted condition is not a safety net, it is a false sense of one.
Frequently asked questions
01Should AI write tests before or after the implementation?
Before, if the test is meant to mean anything. Writing the test first, against expected values that come from the spec rather than from running the code, forces the assertion to be fixed independently of the implementation. Writing tests after the code, from the same context window that just produced the code, is what generates tautological assertions in the first place.
02Is high coverage on an AI-generated test suite a reliable signal?
No. Coverage counts executed lines and branches, not whether the assertions attached to them would fail on a wrong implementation. A suite can reach 100% coverage with assertions that check almost nothing. Treat coverage as a floor that flags untested code, not a ceiling that proves tested code is correct.
03How do you spot a mock-heavy test in review?
Check what the assertion actually checks. If a test mocks a function's own internal dependency and then asserts that the mock was called with certain arguments, it is verifying that the code called the mock, not that the code behaves correctly. A strong test asserts on a return value, a persisted state change, or a real side effect, and mocks only genuine external boundaries like the network, the filesystem, or the clock.
04Do you need a mutation testing tool to apply this?
No, though tools like PIT or Stryker give a repeatable score over time. A manual substitute works for spot checks: invert a conditional or change a constant in a function an assistant just wrote and rerun the suite. If every test still passes, that gap is one the coverage report would never have shown.