How to Debug With AI Without Chasing Ghosts

A protocol for debugging with AI: demand a stack trace and a failing test instead of a guess, force a falsifiable hypothesis, and prove every fix before it ships.

Paste an error into a chat window and the model answers. It almost always answers, whether or not it has enough information to know what actually went wrong. Without a stack trace, a failing test, or a way to reproduce the failure, an AI debugging session runs on pattern-matching against the most common cause of similar-looking errors in its training data, not on what is happening in your code. The result reads like debugging. It is closer to a confident guess.

The fix is not a better prompt. It is a different discipline: treat the model as a hypothesis generator that has to earn its fix, not an oracle that hands one down. Give it evidence instead of a description. Make it commit to a single, falsifiable hypothesis before it touches code. Require proof, a failing test that turns green, before anything counts as fixed. Skip any of those three steps and you get the pattern this post is named for: a patch that quiets one symptom, moves the underlying bug somewhere else, and sends you chasing it again.

Why the model guesses when you let it

Language models are trained to produce a plausible continuation of the text in front of them, not to say there is not enough information to answer. Handed a one-line error message with no context, the model completes it with whatever explanation best matches similar error messages in its training data, a reasonable bet in aggregate and often a poor one for the specific bug in front of you. Confident phrasing is a property of how the model writes, not a signal of how much evidence backs the claim, so a wrong answer reads exactly like a right one.

This is worse than a human guessing wrong, because a person who is unsure usually signals it: a hedge, a clarifying question, a request to see the logs first. A model that was not given enough evidence rarely does the equivalent, and an engineer under deadline pressure rarely stops to ask why a fix actually worked before shipping it. The gap between the error message going away and the underlying defect being addressed is exactly where debugging sessions go wrong.

Feed it evidence, not a symptom description

A description like the feature breaks sometimes on submit gives the model almost nothing to reason from. A full stack trace with line numbers, the exact error text, and steps that reproduce the failure on demand give it something to test a theory against instead of something to guess at. The gap in output quality between those two inputs is larger than the gap between any two prompting techniques.

• The complete stack trace or error output, not a paraphrase or a description of what the screen showed
• A failing test, or reproduction steps that fail reliably on every attempt
• The relevant diff or recent commits touching that code path, so the model can see what actually changed
• Exact versions: language runtime, framework, key dependencies, and OS when the failure looks environment-sensitive
• What has already been ruled out, so the model does not re-propose it

Make it commit to one hypothesis before it edits anything

Before any code changes, ask for one specific, falsifiable hypothesis: which line or condition is responsible, and what observation would prove or disprove it. A model that has to name a mechanism, for example that the retry logic does not reset its counter on a 429, is doing something different from a model that jumps straight to a diff. A named mechanism is checkable against the actual trace and the actual code. A diff produced without that step is a bet that happens to compile.

If the model cannot state a hypothesis that fits the evidence provided, that is useful information: either it has not been given enough evidence, or the failure lives in a part of the system it cannot see. Both are reasons to gather more evidence, not reasons to accept a plausible-looking patch and move on. TLM Forge applies the same sequence to code changes generally, state the plan, have it checked independently, then implement, and you can read how that audit step works in how it works.

Prove the fix, do not just trust the explanation

A fix is not finished when the model says it is finished. It is finished when the failing test used to define the bug goes from red to green, and everything that was passing before is still passing after. If there was no failing test at the start, write one from the reproduction steps before accepting any change; a fix with nothing to fail against has nothing to prove it worked.

Watch for the model editing the test instead of the code: loosening an assertion, adding a special case for the input that was failing, or catching and silencing the exception the test was checking for. This is the debugging equivalent of moving the goalposts, and it is common enough to check for on every AI-proposed fix, not only the ones that look suspicious. See testing AI-generated code for how to tell a fix that actually holds from a test that was quietly rewritten to pass.

The confident-wrong-fix loop, and how it starts

The loop looks like this. An error gets pasted in, the model proposes a fix, the original symptom disappears, and the change ships. A few days later a related failure shows up somewhere else, because the patch addressed the point where the bug became visible rather than the point where it originated. The new error gets pasted in. The model, with no memory of the first fix and no evidence connecting the two failures, proposes a second patch that treats the second symptom exactly like the first one was treated. Each round trip looks like progress. The underlying defect is still there, now with two patches sitting on top of it.

The pattern also shows up at the level of overall developer speed. A 2025 randomized controlled trial by METR gave sixteen experienced open source developers a set of real tasks in repositories they already knew well, randomly allowing AI tools on some tasks and not others. Developers using AI were about 19 percent slower on average, even though beforehand they had predicted a large speedup and afterward still believed the tools had made them faster. Researchers traced part of the gap to time spent evaluating and correcting output that looked right and was not, the same dynamic that turns a single debugging session into a chase once a plausible fix gets accepted without proof.

Sources: METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

The pattern shows up in industry code data too, not only as an anecdote. GitClear's analysis of more than 150 million lines of code across thousands of repositories found a strong correlation, a coefficient of 0.98, between the prevalence of GitHub Copilot and a rise in code churn, the share of newly written code reverted or substantially rewritten within two weeks of being committed. Churn held around 3 to 4 percent annually before Copilot shipped, then climbed toward 7 percent as adoption grew. Correlation is not proof that AI wrote the code being thrown away, but it lines up with what the confident-wrong-fix loop predicts: more first drafts accepted on confidence, more of them undone soon after.

Sources: GitClear code churn research, via Arc.dev

Isolate before you hand off the fix

Once there is a reliable reproduction, shrink it. Cut the input, the request payload, or the sequence of steps down to the smallest version that still triggers the failure. A smaller repro does two things at once: it removes the noise that was letting the model latch onto irrelevant details, and it usually does most of the work of finding the root cause, because reducing a failure to its minimal form is most of what isolating a bug means. Ask the model to do this reduction explicitly instead of jumping straight to a fix; a model that can shrink a repro from fifty lines to five has demonstrated it understands the failure rather than pattern-matched to it. The same scoping discipline applies to structural changes; see refactoring with AI safely for the equivalent protocol when the goal is restructuring code instead of fixing a defect.

A debugging session rarely ends where it started. The hypothesis that got ruled out, the fix that turned out to be a symptom patch, and the actual root cause once it was found are worth keeping somewhere the next session can read, because a fresh chat has no memory of any of it and will happily re-propose the fix that was already rejected last week. A private, persistent memory layer like MemX is built for carrying exactly that kind of context, prior decisions and ruled-out hypotheses included, across sessions instead of starting from zero every time the bug gets reopened. None of this replaces the discipline itself; it just stops the same investigation from happening twice. TLM Forge applies the equivalent idea to code review: findings should not have to be rediscovered every session either.