A coding-agent benchmark graded every criterion. The top endpoint passed 69.9% of criteria per task on average but fully finished 26.4% of tasks.
A benchmark submitted to arXiv on October 26, 2026 measures the gap between "mostly done" and "done." On Co-Design Bench, the strongest of ten model endpoints passed an average of 69.9% of the acceptance criteria per task but fully satisfied only 26.4% of tasks. Score an agent by how much of each request it covers and it looks good. Score it by whether the request is complete and the picture changes.
Sources: arXiv 2610.35138: Co-Design Bench (submitted 26 Oct 2026)
This post covers what the benchmark measured, what its numbers cannot tell you, and a two-number scoring method you can apply to your own agent runs. It is one paper's result on one kind of work, so use it to think about completeness, not as a verdict on any tool.
Disclosure: we build TLM Forge, a process layer for Claude Code, so weigh the product section accordingly. Almost all of this post is about the benchmark and applies with or without us.
The pattern developers report
Developers describe this failure in their own words. On the Claude Code issue tracker, one user's issue from February 2026 is titled "Claude minimizes limitations and presents incomplete work as complete" and another, from September 2026, is titled "Broad requirements are silently narrowed to an easier one, implemented partially, and reported as complete". Each is one user's report of their own sessions, not a measurement. A benchmark is the closer thing to a measurement, which is why the October result is worth reading.
Sources: anthropics/claude-code issue 25703, anthropics/claude-code issue 96447
What Co-Design Bench measured
The paper, titled Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models, has 16 authors from Fudan University, Alibaba Token Hub (Alibaba Group) and four other universities. It asks a coding agent to revise a runnable application based on a request taken from a real coding session, then grades the result against criteria the agent never sees.
Sources: arXiv 2610.35138 (abstract page), arXiv 2610.35138 (full text, HTML)
- 273 tasks, curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Of those sessions, 25,373 were frontend application sessions.
- Two tracks. Multimodal Editing has 100 tasks: localized edits to a runnable page, some with image, audio, video or document attachments. Repository Editing has 173 tasks: coordinated changes across components, routes, state and data flow in multi-file frontend repositories.
- 2,331 hidden acceptance criteria written by experts, averaging 5.5 per task on the first track and 10.3 on the second. The criteria state the requested outcomes and any explicit architectural constraints, and they allow any implementation consistent with them.
- Ten model endpoints from several vendors. The paper says the main comparison holds the Claude Code harness, container manifests, system prompts and tool interfaces fixed across endpoints. Runs were capped at three hours and 120 turns.
- Grading by evidence. Each criterion gets Pass, Partial or Fail based on source diffs, rendered pages and recorded browser actions.
Why a 70% average can hide a 26% finish rate
The paper reports three numbers for each model. Strict completion is the share of a task's criteria graded Pass, averaged over tasks. Graded completion gives half credit to Partial. Full-pass rate is the share of tasks where every criterion passed. Here are the top endpoint's scores across all 273 tasks.
Sources: arXiv 2610.35138 (full text, HTML)
| Metric | How a task is scored | Top endpoint, all 273 tasks |
|---|---|---|
| Strict completion | Share of criteria graded Pass, averaged over tasks | 69.9% |
| Graded completion | Pass counts 1, Partial counts 0.5, averaged over tasks | 75.3% |
| Full-pass rate | Share of tasks where every criterion is Pass | 26.4% |
The gap comes from where the misses land. One missed criterion barely moves a task's average, but it removes that task from the full-pass count. The authors note that a high strict score can coexist with a low full-pass rate when unresolved requirements are spread across many tasks. For the top endpoint, 73.6% of tasks kept at least one criterion that was not fully satisfied.
Sources: arXiv 2610.35138 (full text, HTML)
Simple arithmetic shows how fast this compounds. This is an illustration, not the paper's model: if a task has 8 criteria and an agent meets each one 90% of the time, independently, it finishes about 43% of tasks (0.9 to the power of 8). Real criteria are correlated, so the actual numbers differ. Treat it only as a reason to expect the two numbers to diverge.
The authors ran a related check on 150 tasks from a different benchmark, WebCompass-edit, holding the rubric, judge and rendering pipeline fixed. For one model endpoint, moving from single-shot execution (edits returned in one response) to agentic execution (the model can inspect the app, edit files and run code) lifted mean graded completion from 73.6% to 95.1%. All-pass tasks rose from 49 to 104 of 150, or 32.7% to 69.3%. Even at 95.1% mean completion, the remaining 46 tasks each had at least one feature below Pass.
Sources: arXiv 2610.35138 (Appendix A)
How the gap varies by task group
The top endpoint fully passed 41.0% of tasks on the multimodal track and 17.9% on the repository track, while its strict score went from 0.754 to 0.667. The other nine endpoints fully passed between 6.6% and 18.3% of all 273 tasks. The paper also slices results by attachment type, edit breadth and primary capability. The table pools all ten endpoints, and the authors stress that these slices describe the task groups, not controlled effects.
Sources: arXiv 2610.35138 (full text, HTML)
| Task group | Pooled strict completion | Pooled full-pass rate |
|---|---|---|
| Edits with no attachment (20 tasks) | 0.637 | 37.0% |
| Edits with document attachments (17 tasks) | 0.652 | 18.2% |
| Repository edits spanning 1 or 2 change types | 0.654 | 9.1% |
| Repository edits spanning 3 or 4 change types | 0.591 | 9.7% |
| Grounded Understanding (92 tasks) | 0.604 | 20.2% |
| UI Transformation (55 tasks) | 0.615 | 13.6% |
| Functional Implementation (126 tasks) | 0.619 | 10.3% |
- Document tasks had the highest pooled strict score of any attachment type, yet only 18.2% of their model-task pairs fully passed, against 37.0% for tasks with no attachment.
- Functional Implementation, the largest group, had the highest strict score of the three capability groups and the lowest full-pass rate. Treat that with care: 123 of its 126 tasks are repository-track tasks, and the paper says these groups overlap strongly with track membership, so the breakdown describes performance rather than isolating independent abilities.
- On repository edits, every endpoint scored lower on strict completion for edits spanning three or four change types than for one or two. Pooled full-pass rates stayed near 9% to 10% in both groups, while the share of model-task pairs where no criterion passed rose from 0.9% to 5.8%.
The breadth result is a good example of why to report both numbers. Tasks with broader edits scored lower on coverage and had more model-task pairs where no criterion passed, but the full-pass rate was already low in both groups and stayed similar.
What the 26.4% cannot tell you
The headline number is easy to over-read. Several features of the study limit what it says about your work.
- It is not a base rate. The task set is curated, and candidate requests were ranked by structural difficulty, verifiability and historical failure signals such as reformulations and rollbacks. It is not a random sample of everyday requests.
- It covers web application work. Most source sessions were frontend sessions and the repository track uses frontend repositories, so it says little about backend, data or infrastructure tasks.
- It scores one fixed request. The tasks preserve goals from real sessions but, in the paper's words, "they do not simulate further user feedback during execution." In real use you reply, correct and re-run, so where a session ends up can differ from what a single request produces.
- An LLM did most of the grading. A fixed agentic evaluator assigned Pass, Partial or Fail, and a separate LLM verifier then challenged the positive verdicts. The authors write that this pass "does not by itself establish the correctness of every verdict." Two frontend experts checked 200 task-model outputs out of a grid of 2,730 pairs, and the paper's text reports no agreement rate for that check.
- Swapping the grader moved scores a little. Rescoring two endpoints with three alternative judge models shifted absolute graded scores by at most 0.053 and kept the order of the two endpoints on each track, which the authors describe as stable "without establishing evaluator invariance."
- The harness matters alongside the model. For two solvers run under three harnesses (Codex, opencode and qwen-code, not Claude Code), the gap between the two solvers varied by 10.0 percentage points on the first track and 16.2 on the second. The authors write: "These results motivate reporting the model and harness together and holding the harness fixed for a direct model comparison."
- It scores requested outcomes. The rubrics cover what was asked and any explicit architectural constraints. The paper does not report code quality, security or maintainability results.
A second October paper: what a test-graded score hides
A different arXiv paper, submitted on 22 October 2026 by Fei Ding, looks at the other common way to grade agents: with tests. It audits SWE-Milestone, a benchmark with 98 graded milestones and 35 published runs, and finds that 25 milestones contain a test that depends on a name or path the task description never states and an agent cannot infer. Units with such a dependency failed in 80.6% of runs, against 48.4% for comparable units without one. The author describes the association as weaker than a naive comparison suggests. The leaderboard's ranking survives removing the affected tests, though scores shift within repositories, and the tests at risk are few overall (0.6% of the required tests) but concentrated in particular repositories.
Sources: arXiv 2610.30635: What the Specification Did Not Say (submitted 22 Oct 2026), arXiv 2610.30635 (full text, HTML)
Read the two papers together. A rubric-graded benchmark like Co-Design Bench asks whether the requested outcomes were met. A test-graded one can also ask for details nobody stated. Either way, a score is only as good as the written definition of done behind it, and that holds for your own tasks. If the requirements live in your head, neither you nor the agent can tell when the work is finished. We cover how to write that definition in How to Write a Spec an AI Coding Agent Can Follow and Definition of Done for AI-Generated Code.
Score your own agent runs on two numbers
You can borrow the paper's idea without running a benchmark. Write the criteria before the run, grade each one after it, and report the average and the share of fully finished tasks. The steps below use the paper's formulas, applied to your tasks.
- Before the run, write 3 to 10 yes/no criteria per task. Each should be checkable with a command, a page load or a diff. Include one that protects existing behavior. Template:
C1: <observable outcome> | evidence: <command, test or screenshot> | verdict: pass/partial/fail. - After the run, ask the agent for evidence per criterion: the command it ran and its output, or a screenshot. Claude Code's best-practices page recommends asking for evidence, since a bare claim that tests pass tells you little.
- Verify the criteria that matter. The same page suggests asking a subagent to review the diff against the written spec, checking that every requirement is implemented and the listed edge cases have tests.
- Grade each criterion
pass,partialorfail. Decide what partial means before you start, for example implemented but broken on some input. For the strict score, partial counts as zero. - Per task:
strict = passes / criteria. Across tasks, average it, then computefull-pass = tasks where every criterion passed / all tasks. If you also want partial credit, addgraded = (passes + 0.5 * partials) / criteriaper task and average that too. - Track strict and full-pass over a rolling set of tasks. Keep the sample size in mind: with ten tasks, one task moves the full-pass rate by ten points. When strict is high and full-pass is low, the agent covers most of each request and leaves something out. Look at which criteria fail most often and state those more explicitly in the next brief.
A worked example, with invented numbers, shows how the three scores behave across five tasks.
| Task | Criteria | Pass / partial / fail | Share passed |
|---|---|---|---|
| Task 1 | 5 | 5 / 0 / 0 | 100% |
| Task 2 | 6 | 5 / 0 / 1 | 83% |
| Task 3 | 4 | 3 / 1 / 0 | 75% |
| Task 4 | 8 | 7 / 0 / 1 | 88% |
| Task 5 | 5 | 4 / 0 / 1 | 80% |
Strict completion is 85.2% and full-pass is 20% (1 of 5 tasks). Graded completion, with partial counted as half, is 87.7%. The agent looks about 85% right and finished one task in five. Neither number is wrong. They answer different questions, and the full-pass rate is closer to the question you ask when you decide whether to merge. A single failed criterion might be a one-line fix or the heart of the request, and the full-pass rate treats both the same, so read the failing criteria before you judge the gap.
If a criterion is a command, such as a typecheck or a test file, a Stop hook can hold Claude until it passes. The Claude Code docs show a Stop hook in .claude/settings.json that runs a typecheck and exits with code 2 on failure, and they document that after eight consecutive blocks with no tool call in between, Claude Code overrides the next block and ends the turn. A command hook only checks what its command can check, so criteria such as layout on a phone still need a person or a browser check.
Two related posts help here. Claude Code Said It's Done. It Wasn't. covers the false-completion side of this problem, and Build an Eval Harness for Your Coding Agent covers repeating a measurement like this on a fixed set of tasks.
Where TLM Forge fits
This benchmark is about one gap: partial delivery that looks fine on average. Two parts of TLM Forge touch on it. On Medium and Deep tasks it shows you a design for your OK before building, which gives you a chance to correct a misread request first. After the build, a final review wave includes a phase-auditor that checks whether what the spec promised was delivered and whether out-of-scope items were respected. See Shows you the design and waits for your OK.
The limits are plain. The phase-auditor is an LLM agent, so like the grader in the paper it can be wrong and can miss an unmet requirement. Light tasks get no review agents and no wait for your design OK. Nothing in Co-Design Bench involved TLM Forge, so the paper tells you nothing about how it would score.
Frequently asked questions
01What is Co-Design Bench?
Co-Design Bench is a coding-agent benchmark submitted to arXiv on October 26, 2026. It has 273 tasks drawn from real user coding sessions, mostly frontend web applications, and 2,331 hidden expert-written acceptance criteria. Ten model endpoints were graded on whether their edits to a runnable application met each criterion.
02What does 69.9% of criteria but only 26.4% of tasks mean?
For the strongest of ten endpoints, the average task had 69.9% of its criteria passed. But only 26.4% of tasks had every criterion passed. Most tasks therefore had at least one requirement that was not fully met, even though average coverage looked high.
03Do AI coding agents fail 74% of the time?
Not as a general rate. On that curated set of web edits, even the best of ten endpoints left at least one criterion not fully satisfied in 73.6% of tasks, and the other nine did worse. Each task was one fixed request with no follow-up feedback, so it is not a failure rate for your own work.
04How do I measure whether my coding agent finished the task?
Write three to ten yes/no criteria before the run, ask the agent for evidence on each, verify them, and mark pass, partial or fail. Then track the share of tasks where every criterion passed alongside the average. The gap between the two shows unfinished work.
05Can an AI model grade whether another AI finished the job?
It can help, but it is not proof. The benchmark used an LLM evaluator and a second LLM verifier, and two experts checked 200 outputs. The authors say the verifier does not by itself establish every verdict, so spot-check results with your own tests and eyes.