Build an Eval Harness for Your Coding Agent

Coding agents regress silently when you swap a prompt, tool, or model. Build a private, tiered eval harness from your own PRs to catch it.

If you cannot tell whether last week's prompt tweak made your coding agent worse, you do not have an evaluation setup, you have hope. The fix is a private, versioned regression suite built from your own recent pull requests and bug fixes, run in tiers so a bad change gets caught before it merges instead of after a customer finds it. Public leaderboards will not do this for you, because they answer a different question.

This is a different problem from picking a model, and conflating the two is the common mistake. A public benchmark like SWE-bench Verified answers a purchase question: which model resolves the most real GitHub issues on a fixed, human-filtered set of 500 tasks. A private eval harness answers an operations question no leaderboard can reach: did my workflow regress after I changed a prompt, added a tool, or upgraded a model. The first is a one-time comparison you make when you adopt a tool. The second runs for as long as the agent is in your pipeline.

Sources: SWE-bench Verified (500-instance human-filtered subset, built with OpenAI)

One disclosure up front, because the thesis points somewhere commercial. We build TLM Forge, a process layer for AI coding, and an eval harness is close kin to what it does. So read the product mention near the end with that interest in mind. The recipe here is vendor-neutral and works whether or not you ever touch our tool.

Why a green agent gets quietly worse

A coding agent can hold a steady public benchmark score while getting measurably worse on your codebase, and nothing in a pass/fail signal will tell you. The reason is that outcome-only measurement is lossy. AgentLens, a Microsoft Research study, analyzed 2,614 OpenHands trajectories across eight model backends on 60 SWE-bench Verified tasks and found that among passing runs, 10.7 percent reached green through what it calls a Lucky Pass: regression cycles, blind retries, missing verification, or steps taken in the wrong order. The pass looked identical to a real solution. The process behind it did not support the result.

Sources: AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation (arXiv)

The rate is not a constant. Across the eight model backends AgentLens measured, Lucky Pass rates ranged from 0.5 percent to 23.2 percent, and some models moved by as many as five rank positions when ranked by solution quality instead of raw pass rate. That spread has a direct operational consequence: the reliability of your green signal is a property of the model you happened to pick. Swap a default model on a Friday afternoon, which teams do casually, and the evidential weight of every passing run shifts underneath you. Without a private harness that reruns your own tasks, you have no instrument that would show the drift.

Sources: AgentLens (arXiv)

Seed the suite from your own PRs and bug fixes

The best regression corpus is already in your git history. Take recent merged pull requests and bug fixes, and turn each one into a reproducible eval task. Freeze the repository at the commit just before the change, capture the issue or spec text that drove it, and record the accepted diff and its tests as the reference answer. To run the eval, hand the agent the same frozen starting state and the same prompt, then compare what it produces against the solution your team actually accepted. You have just built a private analog of SWE-bench, except every task is drawn from work your reviewers already approved.

Bug fixes deserve special weight in the corpus. A merged fix encodes a concrete failure your product hit once, plus the behavior that resolved it, plus a test that pins the behavior in place. That is exactly the shape of a high-value regression task: a starting state where a naive agent fails, and a checkable definition of success. Mining incidents and fixes gives you tasks that no public benchmark contains, because they are specific to your domain, your framework versions, and the mistakes your codebase invites.

• The frozen commit or repository snapshot the agent starts from, so the task is reproducible months later.
• The prompt or issue text, verbatim, so you are testing the agent under the instruction it will really receive.
• The accepted diff and the tests that shipped with it, as the reference the agent output is scored against.
• The tool calls the task legitimately requires (file reads, test runs, searches), so you can score whether the agent used them correctly.
• A short rubric of what "correct" means beyond green: files that should change, files that must not, and any rollback the agent should perform if a step fails.

Run it in tiers, not all at once

A single monolithic suite is too slow to gate a pull request and too shallow to trust overnight, so run the harness in tiers matched to how much latency each moment can absorb. The principle is the same one CI already uses for tests: fast and narrow on every change, broad and slow on a schedule. Four tiers cover most teams.

• Smoke evals on every PR to the agent config: a handful of representative tasks that run in seconds and catch gross breakage from a prompt or tool edit.
• A golden set before merge: a curated, hand-reviewed set that spans your main task types (a refactor, a multi-file feature, a tricky bug fix, a migration), run as a required gate on changes to the agent.
• The full suite nightly: every task you have accumulated, run against the current config so slow drift shows up as a trend rather than a surprise.
• Incident evals after failures: every production incident the agent caused or missed becomes a new frozen task, so the same class of failure can never regress silently twice.

Score behavior, not just pass@k

Scoring only whether tests pass, or reporting pass@k, hides exactly the failures that matter for an agent. Pass@k is the chance that at least one of k independent attempts passes, which makes it a ceiling of a parallel thread pool rather than the reliability of the single answer your pipeline actually ships. It is binary and oriented to one-shot generation, so it says nothing about how an agent behaves across a multi-turn debugging loop, where most real degradation lives. Practitioners have made the point plainly: pass@k treats all k attempts as equivalent and cannot capture partial correctness, code quality, or how a solution holds up in a real workflow.

Sources: Runloop: I have Opinions on Pass@K

Multi-turn behavior is measurably different from single-shot output, which is why a one-number pass rate misses it. Work on multi-turn code generation shows that the way an agent reasons, reprompts, and reuses earlier results across turns changes outcomes substantially, and that model performance falls as cross-step dependency grows. An agent that aced your single-file tasks can fall apart the moment a change spans four files and requires it to keep an earlier decision straight. If your harness only records the final pass, that collapse is invisible until it reaches production.

Sources: What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (arXiv), CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation (arXiv)

Score three things a pass/fail gate leaves out. First, multi-file coherence: did the change touch the files it should, leave untouched the files it should not, and stay internally consistent across all of them, rather than fixing one call site and forgetting three. Second, tool-call correctness: did the agent run the tests it claimed to run, read the files it edited, and use its tools in an order that supports the result, or did it assert success without verifying. Third, rollback discipline: when a step failed mid-task, did the agent cleanly revert the partial work, or did it leave the repository in a broken intermediate state and press on. Each of these is checkable from the trajectory, and each is a place agents regress without moving the pass rate at all.

Capability evals graduate into regression tests

Eval tasks come in two kinds, and the harness should move tasks between them on purpose. A capability eval asks an open question: can the agent do this thing at all, such as handle a new framework or a harder class of refactor. You expect these to fail early and improve as you tune prompts and tools. A regression test asks a closed question: does the agent still do the thing it already learned to do. The move that keeps a harness honest is graduation. Once a capability stabilizes across several runs, freeze that task as a regression test so the behavior can never silently break later.

Without graduation, capability evals rot into noise. They stay red for reasons nobody tracks, people learn to ignore them, and the one that quietly starts passing gets no credit while the one that quietly starts failing gets no alarm. Promoting a stabilized capability into the required regression tier turns a research question into a standing guarantee, which is the only form of the result that protects you during next month's model upgrade.

Treat the eval set like production code

An eval suite is production code, and the moment you treat it as a throwaway script it starts lying to you. Version it in the same repo or a dedicated one, review changes to it in pull requests, and hold it to the same bar as the code it guards. Tasks go stale as the codebase moves: a frozen commit references a file that no longer exists, an accepted solution is superseded by a better pattern, a reference test now conflicts with a real API change. Left unmaintained, a rotting eval set produces false alarms, teams mute it, and the harness dies the same death as an ignored flaky test.

Persist the run history too, because a single night's score means little without the trend behind it. You want to see that tool-call correctness slid three points over two weeks after a prompt change, not just that tonight failed. A private, persistent memory layer such as MemX can hold that history and the frozen task context across runs, so the harness reasons over how your agent behaved last month rather than only how it behaved last night. Store the trajectory, the scores, and the config fingerprint for every run, and regressions become a line on a chart instead of a surprise in an incident channel.

Where this connects to the convergence gate

An eval harness is the standing, automated form of a convergence gate. A convergence gate blocks a single change from shipping until its critical issues reach zero; a regression harness does the same thing continuously across every change to the agent itself, so the process cannot degrade without someone deciding it should. TLM Forge runs that gate inside Claude Code today: a spec audit before code, independent multi-agent review of plan and diff, and phase-gated TDD, all of which produce exactly the artifacts a harness wants to score. The tie-in is direct. If you already practice phase-gated TDD, your RED-then-GREEN phase tests are the natural seed corpus for the harness, since each one already pins a behavior with a checkable definition of done.

The same logic extends to review. Independent multi-agent review gives you a second party with different framing looking at each trajectory, which is the condition under which a Lucky Pass becomes detectable at all, and the harness is where you keep score on whether that review is still catching what it used to. If you want the argument for why a green suite is a weak gate in the first place, the convergence gate explained covers the reasoning that this harness operationalizes.

None of this requires a large program. Pull ten recent PRs, freeze them into tasks, wire a smoke tier into the pull request that changes your agent config, and grow the set one incident at a time. The payoff is narrow and real: when someone swaps a model or rewrites a prompt, you will know within a run whether the agent got better, got worse, or got lucky, instead of finding out from a customer three weeks later.