← Back to BlogEngineering

When AI Tests Mock Everything, Production Breaks

AI-written tests that mock the database, API and queue pass while the real integration fails. What mocks hide, and the checks that catch it.

When AI-written tests pass and the real integration still fails, a likely cause is that the tests never touched the integration. A mock replaces a database, an API or a queue with a belief about how it behaves, and when the same agent wrote the code and the mock, both carry the same belief. The fix is to choose, per boundary, between a real dependency, a fake that is checked against the real thing, and a mock kept for what you cannot or should not run, then give the agent a real-dependency check it has to run before it says done.

Disclosure: we build TLM Forge, a process layer for Claude Code, so weigh the product section accordingly. Most of this post is about testing and applies with or without us.

What the research shows, and what it does not

A mining study of commits counted how often they add test doubles. Across 1,254,878 commits made in 2025 in 2,168 TypeScript, JavaScript and Python repositories, 36 percent of the test commits attributed to coding agents added mocks (3,934 of 11,035), against 26 percent of other test commits (40,966 of 158,326). The authors use the word mock for the whole family of test doubles, fakes included, and the mix differed too. In the 496 repositories with at least one agent mock commit, 95 percent had an agent commit using the mock type and 32 percent one using a fake. For non-agent mock commits in the same set, the figures were 91 percent for mocks and 57 percent for fakes.

Sources: Hora and Robbes, Are Coding Agents Generating Over-Mocked Tests? (arXiv), Full text (HTML)

Read that for what it is. The study detects test doubles from code identifiers in commits. It does not measure whether any of those mocks hid a bug, and the authors leave the effect of such mocks on test quality, maintainability and coverage to future research. They describe tests with mocks as potentially easier to generate automatically but "less effective at validating real interactions", and they recommend putting guidance on mocking practices in agent configuration files such as CLAUDE.md.

Sources: Hora and Robbes, Are Coding Agents Generating Over-Mocked Tests? (arXiv)

Individual reports point the same way without being data. In a Hacker News thread about Claude Code, one commenter advises that "one should tell CC to not overmock". In another thread, a commenter says "AI loves to jest.mock". One user filed a GitHub issue reporting that Claude Code wrote hard-coded mock responses where a real protocol call belonged and called the work complete without running it, despite CLAUDE.md rules requiring verification. That is one user, so read it as a symptom, not a rate. Why such rules get ignored is covered in Why Your CLAUDE.md Gets Ignored.

Sources: Hacker News comment by danielbln, Hacker News comment by howToTestFE, anthropics/claude-code issue 8945

Why a mock cannot catch what the agent got wrong

A mock or a stub is told what to return. A Google Testing Blog post on overusing mocks states the cost: when you tell a mock how to behave, "the only assurance you get with your tests is that your code will work if your mocks behave exactly like your real implementations." That assurance gets weaker when one author writes both sides. Suppose the agent misreads an API and assumes a field is called user_id when the service sends userId. It writes the client for user_id, then writes a mock that returns user_id, and the test goes green. Nothing in that loop ever consulted the real service. This is reasoning about how the pieces fit, not a measured failure rate.

Sources: Google Testing Blog: Testing on the Toilet, Don't Overuse Mocks

Two more gaps follow. First, mocks drift. The mining study notes that mocks only guarantee success "if the mocks match the real implementations", which is hard to ensure as the real code evolves. Second, a mock models only what its author thought to write down. Software Engineering at Google calls this fidelity and warns that with a fake that does not match the real thing, "a test might pass when the fake is used, but this same code path might not work properly in the real implementation." Constraints, ordering, pagination and error shapes are easy to leave out, and the next section takes them one boundary at a time.

Sources: Hora and Robbes, Are Coding Agents Generating Over-Mocked Tests? (arXiv), Software Engineering at Google, Test Doubles

None of this makes mocks wrong. The same chapter lists real reasons to substitute: a real dependency can be slow, nondeterministic or hard to construct, and when the system clock is the problem a test can use a double that hardcodes a specific time. It also says unit tests that use doubles often need to be supplemented by larger-scope tests that exercise the real implementation. Tautological assertions are a related failure, covered in How to Test AI-Generated Code Properly. This post is about the boundary where the test stops.

Sources: Software Engineering at Google, Test Doubles

What each mock hides, boundary by boundary

Database: the SQL never runs

A mocked repository or ORM never executes the query it stands in for. A misspelled column, a wrong join and a missing migration all pass, because nothing parses the SQL. Real engines also push back. PostgreSQL reports unique_violation (23505), foreign_key_violation (23503), not_null_violation (23502) and check_violation (23514) as distinct error codes, and a mock returns success for a duplicate insert unless someone wrote that case. The check is to run the real queries against the production engine, with a duplicate insert and a missing parent row in the test data.

Sources: PostgreSQL docs: error codes

Drivers add a second layer. The pg-types README for node-postgres says the driver returns int8 (64-bit integer) results as strings and leaves the parsing to you, and it notes that COUNT(*) returns int8. A mock that returns { count: 5 } hides the fact that the real driver hands back "5" by default, which is how count + 1 turns into "51".

Sources: node-pg-types README

An in-memory SQLite database standing in for Postgres swaps one gap for another. The SQLite documentation says foreign key constraints are disabled by default and must be enabled separately for each connection, that numeric arguments such as the 255 in VARCHAR(255) are ignored, and that any column except an INTEGER PRIMARY KEY can store a value of any storage class unless the table is declared STRICT. PostgreSQL raises an error when a longer string is stored in a varchar(n) column, unless the excess characters are all spaces. A quick local run through the sqlite3 module in Python, which reported SQLite 3.54.1, matched the docs: a child row pointing at a nonexistent parent, a 21-character string in a VARCHAR(5) column and the text "not a number" in an INTEGER column were all accepted.

Sources: SQLite: foreign key support, SQLite: datatypes, SQLite: STRICT tables, PostgreSQL docs: character types

Third-party HTTP APIs: one tidy page

A mocked API client usually returns one clean response. Real APIs paginate: GitHub states that when a response would include many results it will "paginate the results and return a subset of the results", and its own example returns 30 issues from a repository with far more. They also rate-limit. The HTTP 429 status means the client has sent too many requests in a given amount of time, and a Retry-After header may say how long to wait. Code that reads only the first page, or never honors a retry delay, passes against a mock and misbehaves against the service. Capture one real response, scrub tokens and personal data from it, save it as a fixture, and add cases for a second page, a 429, a 5xx and an empty body. Re-capture the fixture when the vendor changes the API, or it becomes another mock that drifts.

Sources: GitHub docs: Using pagination in the REST API, MDN: 429 Too Many Requests

Queues: delivered once, in order

A mocked queue hands the consumer each message once, in order. Amazon SQS standard queues document something else: they ensure at-least-once delivery, but "more than one copy of a message might be delivered, and messages may occasionally arrive out of order." A consumer tested only against the mock has never seen a duplicate. The check is a test that delivers the same message twice and a pair in swapped order, then asserts the final state is correct. Other brokers make other promises, so read the delivery guarantees for yours before deciding what to simulate.

Sources: Amazon SQS standard queues

Your own services: the contract

When both sides of an HTTP or message boundary are yours, a contract test replaces guessing. Pact describes itself as "a code-first tool for testing HTTP and message integrations using contract tests." The consumer tests run against a mock provider and write a pact file describing each interaction. Provider verification then sends each recorded request to the provider and compares the actual response with the minimal response the consumer expected. A consumer mock that disagrees with the provider now fails at verification time instead of in production, for the interactions the consumer recorded. Contracts are not a full substitute for integration tests: the Pact FAQ says contract tests replace a certain class of integration test, the ones that check you use an API correctly and that it responds the way you expect, and not the tests that check the core business logic of your services.

Sources: Pact docs, Pact docs: How Pact works, Pact docs: FAQ, Do I still need end-to-end tests?

The short version

BoundaryWhat a mock lets throughTest that catches itWhere a mock is fine
DatabaseBad SQL, constraint errors, driver type conversions, missing migrationsRun the queries against the production engine, with a duplicate insert and a missing parent rowBusiness logic above the repository layer
Third-party HTTP APIOne clean page, no 429, no error bodyFixture captured from a real response, plus second page, 429, 5xx and empty casesCalls with side effects you must not fire in tests, such as charges and emails
Message queueExactly-once, in-order deliveryConsumer test with a duplicate and a swapped pair, and a real broker for the wiringHandler logic, once duplicates and reordering are covered
Your own serviceThe response shape the consumer assumedContract test that the provider is verified againstOnly a consumer mock that the provider verifies, as in Pact. Otherwise rarely
Clock and randomnessLittle. The real clock and real random numbers make tests nondeterministicFreeze the clock and test boundaries such as midnight and month endUsually, for the clock your own code reads

Pick the cheapest double that keeps the contract

Treat the choice as a ladder and start at the top. Drop a rung only when you can write down why: speed, determinism or a side effect you must not fire. The first two are among the reasons the Google chapter gives for substituting.

Sources: Software Engineering at Google, Test Doubles

  • Real dependency. Testcontainers describes itself as "an open source library for providing throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container." You need a Docker-API compatible container runtime wherever the tests run, and starting containers adds to the run time.
  • Fake plus contract tests. A fake is a lightweight implementation that behaves like the real one but is not fit for production, such as an in-memory database. Software Engineering at Google describes running the same tests against both the real implementation and the fake, which it calls contract tests, because without tests a fake can diverge as the real implementation evolves.
  • Consumer-driven contracts. Pact means something different by contract test: a pact file records what the consumer expects, and the provider is verified against it, as described above. It covers HTTP and message boundaries.
  • Mock. Use it for the clock, randomness, side effects, and for forcing failures that are hard to trigger for real, such as a timeout. Prefer assertions on returned values and state over assertions that a mock was called. The Google chapter likewise says to prefer state testing over interaction testing.

Make the agent run the real check

Start with guidance, since the study authors recommend it. They note that the CLAUDE.md of one repository tells agents never to mock anything except the LLM, and that they found only five agent mock commits there, three related to LLMs and two that seem to fall outside that exception. That is a single repository with no comparison group, so it shows a rule can coexist with low mock counts and nothing more. Here are rules to paste into CLAUDE.md, adjusted to your stack.

Sources: Hora and Robbes, Are Coding Agents Generating Over-Mocked Tests? (arXiv)

  • Tests of code that reads or writes the database run against a real instance of the production engine. Do not substitute SQLite for Postgres. Mock the repository only in tests of the logic above it.
  • Mock only the clock, randomness and third-party services with side effects. Build mock HTTP responses from a captured real response saved under tests/fixtures, not from memory.
  • Every mock or fake needs failure-path tests: error response, timeout, duplicate, empty result, second page.
  • Put a one-line comment above each test double saying whether it is a fake or a mock, and why the real dependency is not used.
  • Before saying a task is done, run the integration test command and paste the command and its output.

The last rule follows Anthropic advice to ask Claude for evidence: the command it ran and its output. It is still guidance. Anthropic documents CLAUDE.md as instructions the model may decide not to apply, and recommends a hook, a deny rule or a managed setting for anything that has to happen every time.

Sources: Claude Code docs: Best practices

A Stop hook is the enforced version. The Claude Code docs show one that runs a typecheck when Claude tries to finish and exits with code 2 to send it back to work, with stderr returned to Claude as the reason. Put your real-dependency command in .claude/settings.json (the script name here is a placeholder): {"hooks":{"Stop":[{"hooks":[{"type":"command","command":"npm run test:integration 1>&2 || exit 2"}]}]}}. The hook runs whenever Claude finishes responding, so keep that suite fast or scoped. After eight blocked stops in a row with no tool call in between, Claude Code overrides the next block and ends the turn.

Sources: Claude Code docs: Best practices, enforce the check with a Stop hook, Claude Code docs: Hooks reference, Stop

Insight

The hook checks only that the suite passes. An agent can still edit the integration test or mock inside it, which is the problem in How to Stop Claude Code Changing Tests to Pass. Read the diff of the test files, not just the green result.

Review the doubles, then try to break them

In review, treat each mock as a claim about a dependency and ask where the claim came from.

  • Where did the return value come from: a captured real response and the vendor docs, or the model?
  • What does the real dependency do that the mock cannot, such as reject a duplicate, paginate, reorder, time out or change a type, and is there a failure-path test for it?
  • Does the assertion check a returned value or stored state, or only that the mock was called?
  • Does any test in the suite run the real thing end to end?

A rough count tells you where to look first. This counts mock lines added in test files across the commits on a branch: git diff main...HEAD -- "*test*" "*spec*" | grep "^+" | grep -cE "jest.mock|vi.mock|mocker.patch|@patch|MagicMock|sinon.stub". The number is a prompt for the questions above, not a threshold, and the patterns cover only a few common libraries.

The most direct check is sabotage. Start from a clean working tree, create a throwaway branch with git switch -c scratch/sabotage, and break the real thing on purpose: rename a column in the schema, change a field name in a provider response, make a seed row violate a constraint. Run the full suite. At least one test should fail for each break. A break that nobody notices marks a boundary no test covers, and a mock may be what is insulating it. Undo it with git restore ., git switch - and git branch -D scratch/sabotage, and edit existing files only, because git restore . does not remove new ones. Git reverts files, not data, so run the suite against a throwaway database, not a shared one. It is a boundary-level cousin of the check in Mutation Testing for AI Code: Why Coverage Lies.

Where TLM Forge fits

TLM Forge tells Claude to build tests first. That is an instruction, and no hook checks test results. Deep tasks are verified on a real environment, so on those a green suite of mocks is not the only evidence.

The limit is plain. Test-first stays an instruction, so the CLAUDE.md rules, the Stop hook and the review questions above are still yours to set up. A Light task also gets no review agents, so no reviewer agent looks at its test doubles.

Frequently asked questions

01Why do my AI-generated tests pass but the app fails in production?

One likely cause is that the tests replace the database, API or queue with mocks written by the same agent that wrote the code. The mock encodes the agent assumptions, so a wrong assumption passes in both places. Real constraints, pagination, duplicate deliveries and driver type conversions never run. Add tests against a real dependency for each boundary.

02Should unit tests mock the database?

Not the code that talks to it. Mock the repository only when testing business logic above it, and test the repository itself against the same engine you run in production. Substitutes such as in-memory SQLite differ on foreign keys, type enforcement and length limits, so they can accept data that production rejects.

03What is the difference between a mock, a stub and a fake?

A stub returns values you specify. A mock is a test double whose behavior is set inline in the test, often to check how it was called. A fake is a lightweight working implementation, such as an in-memory database, that behaves like the real one but is not fit for production.

04Does a contract test replace an integration test?

Only one class of them. The Pact docs say contract tests replace integration tests that check you use an API correctly and that it responds as expected, but not tests of core business logic. Contract tests also run without the services together, so keep a few real-dependency integration tests for the wiring and configuration.

05How do I stop Claude Code from mocking everything in tests?

Write the mocking policy in CLAUDE.md, ask Claude to paste the command it ran and its output, and add a Stop hook that runs your real-dependency suite. The first two are guidance the model may skip. The hook is enforced by Claude Code, though it only checks that the suite passes.

Ship AI-written code you can trust

TLM Forge is the missing process layer for Claude Code: a spec audit, independent multi-agent review, enforced TDD, and an adversarial red-team gate.

Get TLM Forge