Your AI Agent Passed the Tests by Luck

A study of 2,614 AI agent runs found 10.7 percent of the passing runs went green by luck. Why green is a weak merge gate, and what to check instead.

Your agent reported success, the suite went green, and the diff read fine. Roughly one passing agent run in nine did not earn that green. A trajectory-level study of SWE agents analyzed 2,614 runs and found that, among the passing runs in its evaluation subset, 10.7 percent reached a passing state without the agent having done the work that would justify it. The same analysis rated 69.1 percent of those passing runs solid but imperfect and 20.2 percent ideal, so most passing runs on the harness and task set studied did earn it. Nothing in a pass/fail gate tells you which ones did not.

Pass/fail is the gate almost everyone uses. The agent reports success, CI is green, the diff looks reasonable, and the change merges. One bit of signal now stands in for an entire multi-step process, and that bit is measurably lossy. The obvious fix, making the agent write the failing test first, proves less than it appears to, for reasons this post gets to.

What a lucky pass actually is

A lucky pass is a run that reaches a passing state through a process that does not support the result. The term comes from AgentLens, a Microsoft Research project that analyzed trajectories rather than outcomes across 60 SWE-bench Verified tasks using OpenHands. The researchers identified recurring shapes: regression cycles where the agent breaks and restores the same behavior, blind retries that change something arbitrary until the suite goes green, missing verification where the agent never confirms its own hypothesis, and temporally disordered work where implementation precedes the exploration meant to inform it.

Sources: AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation (arXiv)

The rate varies sharply by model. Across eight model backends the lucky-pass rate ranged from 0.5 percent to 23.2 percent. That spread makes the reliability of the pass signal a property of the model you picked rather than a constant, which is not how anyone treats it. When the researchers ranked those same models by solution quality instead of pass rate, some moved as many as five positions. The leaderboard you used to choose a model measures something adjacent to what you care about.

Passing is not the same as mergeable

Passing every test and being mergeable sit a long way apart. Cognition built FrontierCode to measure whether a change is mergeable rather than whether it passes. More than 20 open-source maintainers from 36 flagship repositories each spent over 40 hours per task defining what their projects would actually accept. On the 50-task Diamond subset, the hardest tier, the strongest model at launch scored 13.4 percent, a weighted aggregate of the maintainers' rubric rather than a simple pass rate. Cognition deprecated Diamond in FrontierCode 1.1, so no newer figure exists for that tier. What the whole exercise measures is the distance between satisfying a test and satisfying a maintainer.

Sources: Cognition: FrontierCode

Put the two kinds of benchmark side by side and the gap is the whole argument. On SWE-bench Verified, which asks whether the tests pass, frontier models sit around 96 percent as of August 2026. On the 100-task FrontierCode Main tier, which asks whether a maintainer would merge the change, the leading score is roughly 53 percent. Different task sets, different scoring, so the two numbers do not subtract cleanly. They do not need to. Change the question and the ceiling collapses.

Sources: Cognition: FrontierCode leaderboard

Merged code still ships the defects

A study presented at MSR 2026 examined 1,210 merged agent-generated bug-fix pull requests in Python repositories and reported that merge success does not reliably reflect post-merge code quality, with code smells concentrated at critical and major severities. Once the authors normalized for code churn, most of the apparent difference between agents tracked pull request size rather than agent identity, so the finding describes agent-generated change in general rather than ranking the tools. Merging is a human gate, and human gates reward fluency. Reviewers approve changes that read well.

Sources: Beyond Bug Fixes: Post-Merge Code Quality in Agent-Generated Pull Requests (arXiv)

Why a lucky pass survives review

The mechanism is plain, and it needs no malice. An agent optimizing for a green suite has an easier path than solving the problem: write assertions that the code it is about to produce will satisfy. In a June 2026 Ask HN thread on AI skepticism, the developer josephg put it flatly. Unless you are careful with your prompting, the model will only write the unit tests it knows will pass. How to test AI-generated code covers the mechanics of implementation-shaped assertions. What matters for the gate is the consequence: the test encodes the implementation rather than the requirement, and it keeps passing through exactly the refactors it should have caught.

Sources: Ask HN: Why is the HN crowd so anti-AI? (comment)

OpenJDK reached the same conclusion from the reviewing side. Its interim policy on generative AI, published in April 2026, states that these tools make it easy to create large quantities of plausible-looking code, with plausible-looking tests, which is nonetheless incorrect or poorly designed. Plausible-looking tests defeat every downstream check built on the assumption that tests are adversarial to the code.

Sources: OpenJDK: interim policy on generative AI

What a lucky pass looks like in a real codebase

The lucky pass rarely announces itself. A typical instance: the agent is asked to fix a date-handling bug, writes a test asserting the function returns the string it is about to make it return, hardcodes a branch for the failing input, and reports success. Tests pass. Coverage rises. The underlying timezone assumption is untouched, and the next input in the same class fails identically. Nothing in the pass/fail signal separates this from a real fix, and the added test now hands false assurance to everyone who reads the file later.

The regression-cycle shape is easier to spot, if you look at all. The agent changes a shared helper, breaks four unrelated tests, reverts part of the change, breaks two others, and eventually lands in a state where everything is green. The final diff can look small and sensible. The trajectory that produced it was a search, and what stopped the search was the suite going quiet rather than the problem being understood. If your only artifact is the final diff, that history is invisible by the time you review.

Why the model you choose changes the gate

The spread in lucky-pass rate across model backends, 0.5 percent at one end and 23.2 percent at the other, has an uncomfortable implication for process design. A review process calibrated on a model at the low end is under-specified for a model at the high end, and teams swap models far more casually than they revise their gates. Changed your default model recently? Your effective quality bar moved with it, and nobody decided that it should.

Does RED then GREEN prove anything?

Enforcing that a test fails before the implementation makes it pass does not detect a lucky pass on its own. Most vendors in this category skip that caveat, and so does the flattering version of our own pitch. If the same agent writes both the failing test and the fix, it can author a test tuned to the implementation it already intends to write. The RED state proves ordering. It does not prove independence, and independence is the property that makes a test adversarial.

What phase ordering does buy is legibility. A trajectory with enforced phases produces an auditable record: this test existed at this point, it failed for this reason, this change made it pass, this is what else moved. That record is what lets a reviewer, or a differently-framed reviewing agent, notice the shapes AgentLens catalogued. Ordering buys inspection, not correctness.

Six questions a green suite cannot answer

• Did the test ever fail? A test that has only ever been observed green has not demonstrated it can detect anything.
• Who wrote the test, and when? A different session, a different framing, or a human-written assertion changes what the pass means.
• How many revision cycles did it take? Repeated arbitrary edits until green is the blind-retry pattern, and it is visible in the trajectory.
• Did the same behavior break and get restored? Regression cycles indicate the agent was searching rather than reasoning.
• Does the diff extend past the spec? Scope creep is the cheapest reliable signal that the agent solved a different problem than the one you set.
• Did the agent verify any hypothesis? One that never checked its own assumption got the right answer by construction or by luck, and you cannot tell which.

How to evaluate agent review tooling

If you are evaluating tooling in this space, ask which of the two properties it provides, because vendors conflate them constantly. TLM Forge enforces ordering and independence of review: a spec audit that fixes the requirement before code exists, then phase-gated TDD so the trajectory stays ordered and auditable, with agents that did not author the diff running the review. It does not claim to make an agent-written test independent of the agent that wrote it. Nothing does. Separating author from reviewer buys you a second party with different framing looking at the record, which is the condition under which a lucky pass becomes detectable at all.

This argument has a clear boundary. None of the empirical work here says agents produce bad code, and none of it argues for using them less. The AgentLens finding is that a specific measurement is noisier than its users assume, which is a statement about the gate rather than about the tool.

OpenJDK drew a stronger conclusion and barred contributions containing generative AI output from its repositories and pull requests outright, a policy judgment about a foundational project rather than a measurement. Teams that keep shipping agent-written changes and simply stop treating green as sufficient evidence will capture most of the available benefit. The failure sits in the check itself. It was never designed to answer the question everyone asks it.

No process overhaul is required. Stop treating a green suite as sufficient evidence before merge. Ask what the trajectory looked like, whether the test ever failed, and whether anyone other than the author has looked. One passing run in nine is a high enough rate that the answer will change what you ship.