← Back to BlogEngineering

AI Code Edge Cases: Five Failures to Test For

AI-written code can cover only the case you described. Five failure modes to test: timeouts, retries, partial failure, empty input and races.

When AI-written code works in the demo and breaks on a hung request, a repeated payment or an empty list, one likely gap is that nobody said what should happen on failure. Name five failure modes in the prompt (timeouts, retries, partial failure, empty input, concurrency), turn each one into a test that forces the failure, and review the diff for the code shape that goes with each.

Disclosure: we build TLM Forge, a process layer for Claude Code, so weigh the product section accordingly. The checklist below works with any agent and any test runner.

What the reports and the research say

One Hacker News commenter describes the pattern this way: "I've found they optimize largely for the happy path and don't consider any (or enough) edge cases", and gives a timeout in one specific HTTP request as the example. That is one engineer's report, not a measurement. Claude Code's documentation describes a related tendency: Claude tends to read a request conservatively and "build the smallest thing that satisfies it", and the remedy is to say what you want, naming the edge cases and the empty and error states.

Sources: Hacker News comment by ventsys, Claude Code docs: Best practices

Measured data is thinner than the anecdotes. An arXiv study that scored generated code on input validation and error handling, not only on passing tests, ran four small open-source code models (1.3B to 7B parameters) on the CoderEval benchmark. It reports that 35.2% of the outputs handled those cases worse than human-written code, and that over 90% of the deficiencies were missing conditional checks. In a separate 2023 paper, EvalPlus extended the HumanEval test cases by 80x, and pass@k across 26 models dropped by up to 19.3% at pass@1 and up to 28.9% at pass@100, which shows how much wrong code a thin test suite lets through.

Sources: arXiv 2503.20197: robustness of LLM-generated code (CoderEval), arXiv 2305.01210: EvalPlus

Neither of those studies measures timeouts or retries in generated service code, and we found no study of how often generated code omits timeouts. For retries there is one small measurement: a single-author arXiv preprint in which five models wrote Python backend code for 12 idempotency and retry-safety tasks. It reports 32 findings across the 60 plain-prompt outputs for those tasks, and 2 when the prompt was preceded by a fixed specification frame. Its idempotency checker flags a retryable, side-effecting handler with no deduplication. Treat it as a preprint, not a settled number. The five failure modes below come mainly from distributed-systems practice and vendor documentation, so use them as a review checklist for any diff, whether a model or a person wrote it.

Sources: arXiv 2609.23270: specification frame for LLM-generated code (preprint)

Concurrency has dedicated benchmarks. The CONCUR benchmark has 115 Java concurrency problems and checks generated programs with a model checker that explores thread interleavings. Its authors report that many of the 23 models they evaluated produce compilable code but frequently fail to produce programs that are correct under all thread interleavings. Spaghetti Bench, a benchmark by Vasu Vikram, Ao Li and Rohan Padhye covering 39 Java concurrency bugs, reports that agents fix race conditions less successfully than conventional software engineering tasks.

Sources: arXiv 2603.03683: CONCUR benchmark, CONCUR paper (HTML), Spaghetti Bench: Evaluating AI Agents on Concurrency Bug Fixes

A prompt that names the failure modes

The cheapest change is to put the failure modes in the request. The best-practices page shows the shape with a settings page: ask Claude to be ambitious and list the empty and error states instead of leaving them implied. A version for backend work follows.

Sources: Claude Code docs: Best practices

Pro Tip

Prompt to copy: Before writing code, write down what this change does in each failure case: (1) a call to another service never answers, (2) a call fails after it already took effect and is retried, (3) step two fails after step one succeeded, (4) the input is empty, missing or null, (5) two requests run this at the same moment. For each case, state the behavior you will implement and write one failing test first. If the spec does not say what the behavior should be, ask me. If a case cannot happen here, say why instead of adding code for it.

Two details matter. The failing test per case turns a described behavior, which is a claim, into evidence. The permission to skip cases that cannot occur matters because the best-practices page warns that a reviewer asked to find problems will always find something, and that fixing every finding leaves extra abstraction layers and tests for cases that cannot happen. The same page points a review subagent at the spec, asks it to check that the listed edge cases have tests, and tells it to skip style. It also suggests raising the effort level when getting it right depends on cases that are quick to overlook.

Sources: Claude Code docs: Best practices

A prompt is advice. The memory documentation says Claude treats CLAUDE.md as context, not enforced configuration, so a failure-mode rule saved there is advice too. Put the five cases in the spec as well (How to Write a Spec an AI Coding Agent Can Follow covers where edge cases belong), then let the tests below do the enforcing.

Sources: Claude Code docs: How Claude remembers your project

The five failure modes

1. Timeouts: some defaults wait forever

Python's requests library does not time out unless you pass a timeout. Its documentation says nearly all production code should set one, and that without it a program can hang indefinitely. Go's net/http documents that a Client Timeout of zero means no timeout, and DefaultClient is a zero-value Client. In JavaScript, MDN's example passes AbortSignal.timeout(5000) as the signal option of fetch.

Sources: Requests docs: Quickstart (timeouts), Go docs: net/http, MDN: AbortSignal.timeout()

The same word means different things. In requests, one value applies to both the connect and read phases, and the read timeout counts the wait between bytes, so it does not cap total download time. In Go, the Client timeout includes connection time, redirects and reading the response body. The Amazon Builders' Library recommends a timeout on any remote call, covering both the connection and the request.

Sources: Requests docs: Quickstart (timeouts), Requests docs: Advanced usage (timeouts), Go docs: net/http, Amazon Builders' Library: Timeouts, retries, and backoff with jitter

Test it with a stub dependency that accepts the call and never answers, a short timeout set through config, and an assertion that the call fails inside the budget with an error the caller handles.

2. Retries: a timeout does not mean the work failed

Insight

A timeout is the caller giving up. It says nothing about whether the server finished the work.

The Amazon Builders' Library says a timeout or failure does not necessarily mean the side effects did not happen, and recommends designing APIs to be idempotent so they can be retried safely. RFC 9110 lists PUT, DELETE and the safe methods as idempotent, which leaves POST out. It says a client should not automatically retry a non-idempotent request unless it knows the request is actually idempotent or can detect that the original was never applied.

Sources: Amazon Builders' Library: Timeouts, retries, and backoff with jitter, RFC 9110: HTTP Semantics, 9.2.2 Idempotent Methods

The usual fix is an idempotency key: the client makes one key per logical operation and sends the same key on every attempt. For API v1, Stripe saves the status code and body of the first request made with a key once endpoint execution begins, returns that saved response on retries, including 500 errors, and errors if the retry's parameters differ from the original. Amazon's guidance describes a similar design, with a client request identifier and a validation error on a parameter mismatch.

Sources: Stripe API reference: Idempotent requests, Amazon Builders' Library: Making retries safe with idempotent APIs

Then bound the retries. Google's SRE book advises limiting retries per request, considering a server-wide retry budget, and not retrying permanent errors or malformed requests. It also warns against retrying at several layers: with three retries (four attempts) at each of three layers, one user action can produce 64 attempts on the database. Amazon adds jitter to backoff so clients do not all retry at the same moment, and notes that a retry spends more of the server's time to raise one caller's chance of success.

Sources: Google SRE book: Addressing Cascading Failures, Amazon Builders' Library: Timeouts, retries, and backoff with jitter

To test it, build a fake that applies the write and then drops the response. A retry loop with no key shows two writes. We ran a toy fake of this shape in Python: with no key, the retry produced two charges, and with one key reused across attempts it produced one. A loop that retries every exception also retries errors that can never succeed, so add a case where the fake returns a validation error and assert there is no second attempt.

3. Partial failure: what state is left behind

Any change that touches two systems can stop between them. Chris Richardson's transactional outbox page describes the problem of updating a database and sending a message together: sending a message in the middle of a database transaction is unreliable because the transaction may not commit, and sending it after the commit is unreliable because the service may crash first. The pattern stores the message in the database inside the same transaction, and a separate relay publishes it. The page notes that the relay can publish a message more than once, so the consumer has to be idempotent, which brings the retry rules back in.

Sources: microservices.io: Transactional outbox

Inside one database the question is the same: if step three fails, is the data still valid? Shapes to look for in the diff are two writes with no transaction around them, a catch block that logs and carries on, and a default value returned after a failure, because each can leave data half changed or hide that a step failed. The test makes step two raise, then asserts that nothing is half applied or that a rerun converges to the finished state.

4. Empty input: the language decides for you

Empty input is where the language makes a choice on your behalf, and each language chooses differently. Python's max() raises a ValueError on an empty iterable unless you pass default, and statistics.mean() raises a StatisticsError on empty data. In JavaScript, Math.max() with no arguments returns -Infinity, and reduce on an empty array with no initial value throws a TypeError. None of those is a business rule.

Sources: Python docs: built-in functions (max), Python docs: statistics, MDN: Math.max(), MDN: Array.prototype.reduce()

The code has to say whether empty means an error, zero or skip, and that choice belongs in the spec. Empty also covers None, an empty string, a missing key, a page with no results and a zero count that ends up as a divisor. The test is a table of cases: empty, one item, None and a missing key, run on every function that takes a collection or an optional value.

5. Concurrency: the check and the write must be one step

A classic shape is check-then-act: read a value, test a condition, write based on it, as three separate steps. With one unit of stock left, two requests can both read 1, both pass the test and both reserve it. The safer shape collapses the steps into one statement, UPDATE items SET stock = stock - 1 WHERE id = 1 AND stock >= 1, and then reads the affected-row count to learn whether the reservation happened. PostgreSQL's documentation explains why this holds in Read Committed mode: a second updater waits for the first, then re-evaluates the WHERE clause against the updated row. For insert-or-update, it documents that with INSERT ... ON CONFLICT DO UPDATE, each proposed row either inserts or updates, barring unrelated errors. Those are PostgreSQL's guarantees at that isolation level, so check your own database before copying the pattern.

Sources: PostgreSQL docs: Transaction Isolation (Read Committed)

Ordinary tests can miss this. The Spaghetti Bench authors note that a concurrency test can pass because the bad interleaving did not occur in that run. Go's race detector, as documented, finds only races that happen at runtime, and it reports data races: two goroutines accessing one variable concurrently with at least one write. A stock check against a database row is a race in your application logic, not a memory race, so that tool is not aimed at it.

Sources: Spaghetti Bench: Evaluating AI Agents on Concurrency Bug Fixes, Go docs: Data Race Detector

The fix is to force the interleaving. Give the function a callback that runs between the check and the write, and have the test hold two threads at a threading.Barrier(2) there, so both finish the check before either writes. We ran this in Python 3.14.10 with the standard sqlite3 module and a table holding one unit of stock. The check-then-write version let both threads reserve it and left stock at -1. The single conditional UPDATE let one thread succeed and left stock at 0. The result was the same on each of five runs.

A barrier test checks one bad ordering. Tools that explore many orderings check more: the Spaghetti Bench post gives agents a tool called Fray that checks fixes across thousands of interleavings, and CONCUR uses a model checker. Users can also send two requests at once on purpose, and Red-Teaming Your AI-Generated Code covers that adversarial pass.

Sources: Spaghetti Bench: Evaluating AI Agents on Concurrency Bug Fixes, CONCUR paper (HTML)

The checklist in one table

Failure modeShape to look for in the diffTest that forces itSafer shape
TimeoutA network, database or queue call with no timeout argumentStub that accepts the call and never answers; assert failure inside the budgetExplicit timeout on every remote call, set through config
RetryA loop that re-sends a write after any exception, with no cap and no keyFake that applies the write, then drops the response; assert one write after the retryOne idempotency key per operation, capped attempts, backoff with jitter, no retry of permanent errors
Partial failureTwo writes to different systems, or a catch block that logs and moves onMake step two raise; assert the data is valid and a rerun convergesOne transaction where possible; outbox plus an idempotent consumer across systems
Empty inputIndexing, max, min, mean or division over input nobody checkedTable of cases: empty, one item, None, missing keyA stated rule for empty (error, zero or skip), written in the spec
ConcurrencyA read, a condition and a write as separate stepsBarrier between the check and the write; run two threadsOne conditional UPDATE, a unique constraint, or a lock

Where this advice runs out

  • Prevalence is poorly measured. We found no study of how often generated code omits timeouts, and the one retry and idempotency measurement we found is a small preprint, so this is a review checklist, not a finding about any model.
  • Tests written by the same agent that wrote the code can be hollow. Confirm each failure test fails when you remove the guard it protects. How to Test AI-Generated Code Properly covers mutation testing, which automates that check.
  • A barrier test checks one bad ordering, and some races only appear across services or under load.
  • Defaults differ by library, database and isolation level. Confirm them in the documentation for the version you run.
  • Failures that slip through still need to be visible. When AI-Written Code Breaks in Production covers logging the catch blocks that swallow errors.

Where TLM Forge fits, and where it does not

On Deep work, TLM Forge adds a tester agent to the final review wave, and its job is to ask "what happens when" questions about the change (the tester agent). It also tells Claude to build tests first, so a failure case can exist as a failing test before the code that handles it. That is an instruction, and no hook checks test results. The tester is an LLM agent, so it can miss a case, and it does not replace the test that forces a race or a lost response. Medium work has no tester in the final wave, which holds a code-reviewer and a phase-auditor, and Light work gets no review agents, so there the prompt and table above carry the load.

Frequently asked questions

01Why does AI-generated code miss edge cases?

One likely reason is a request that describes only the happy path: Claude Code's documentation says Claude tends to build the smallest thing that satisfies a request. A study of four small open-source code models found missing conditional checks caused over 90% of their deficiencies. Naming failure cases in the prompt or spec, then testing them, is the practical fix.

02How do I get Claude Code to handle edge cases?

Name them. Ask for empty states, error states and each failure case in the prompt or spec, require one failing test per case before the code, and say which cases to skip so Claude does not add code for situations that cannot happen. Anthropic's best-practices page shows naming the empty and error states in the request.

03How do I test for a race condition in AI-written code?

Force the bad ordering instead of hoping for it. Add a callback between the check and the write, hold two threads at a barrier so both finish the check first, release them, and assert the invariant, such as stock never going below zero. Repeated normal runs can pass because the bad ordering never occurred.

04Is it safe to retry a failed API call?

Only if the operation is idempotent or carries an idempotency key. A timeout does not prove the work failed, so a blind retry of a write can apply it twice. Cap the attempts, add backoff with jitter, and do not retry permanent errors such as malformed requests.

05Does Python requests have a default timeout?

No. The requests documentation says that if no timeout is specified, requests do not time out, and that nearly all production code should set one. Go treats a Client Timeout of zero as no timeout too. Checking every network call in generated code for an explicit timeout is a cheap review step.

Ship AI-written code you can trust

TLM Forge is the missing process layer for Claude Code: a spec audit, independent multi-agent review, enforced TDD, and an adversarial red-team gate.

Get TLM Forge