An AI reviewer left 30 comments. Sort them into four piles by evidence and blast radius, then tune the reviewer so the next pull request gets fewer.
When an AI reviewer leaves 30 comments, do not work through them top to bottom. Put four questions to each one: is it in code this pull request changed, can I reproduce it, what breaks if it is right, and is the fix small? The answers sort the comments into four piles: block the merge, fix in this pull request, list for later, and drop with a reason. The reviewer's severity label is a model's guess, so use it to pick what to read first and let the questions decide the pile.
Disclosure: we build TLM Forge, so weigh the product section accordingly. Most of this post is about review triage in general and applies with or without us.
Why 30 comments does damage
The failure is well documented by the people who build these tools. Greptile, which makes an AI reviewer, wrote that when it first launched, "the biggest complaint by far was that the bot left too many comments." On a pull request with 20 changes it could leave as many as 10, "at which point the PR author would simply start ignoring all of them." Its analysis of existing comments found "~19% were good, 2% were flat-out incorrect, and 79% were nits." That is one vendor describing its own product in late 2024, so treat it as an illustration of the failure mode rather than a benchmark. When most comments are noise, the one real finding gets the same shrug as the rest.
Sources: How to Make LLMs Shut Up (Greptile Blog)
Thirty is also far above the averages in two published deployments. Beko's case study of a bot built on the open-source Qodo PR Agent counted 3.65 bot comments per pull request on average, and Atlassian's year-long deployment of its RovoDev reviewer averaged 2.1 per pull request. So a 30-comment review sits well outside what those two deployments saw on average, and diff size and reviewer settings are the first things to check. Anthropic notes that most pull requests over 1,000 lines get findings from its reviewer and most small ones come back clean, so if your 30-comment reviews are 1,000-line pull requests, the first fix sits upstream: Stop Your AI Agent Writing 1,000-Line PRs.
Sources: Automated Code Review In Practice (arXiv 2412.18531), RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian (arXiv 2601.01129), How the Claude Code team does code review (claude.dev)
High volume comes with a tradeoff that Anthropic names in its write-up of its own reviewer: "It's easy to catch more bugs if you don't mind flagging a lot of non-bugs." The cost of that noise also depends on who receives it. The same write-up says: "A person can wave off a nit in seconds. An agent fixes it, which means another commit and another round of review." One Beko survey respondent described the same loop: "With each fix, a new review is generated." Pasting all 30 comments into a Claude Code session with "fix these" invites it, and so does /code-review --fix, which the docs say also asks Claude to apply the findings to your working tree. How Anthropic Does Code Review on 90% AI-Written Claude Code covers the rest of that process.
Sources: How the Claude Code team does code review (claude.dev), Automated Code Review In Practice (arXiv 2412.18531), Claude Code docs: Review locally with /code-review
Branch protection adds pressure from the other side. GitHub describes the "Require conversation resolution before merging" rule this way: "Requires all comments on the pull request to be resolved before it can be merged to a protected branch." With it on, every nit thread needs closing, so your policy needs a cheap, honest way to close one without a code change. In Claude Code Review, per the docs, replying alone does not dismiss a finding; resolving the thread does. The docs also say a review never approves or blocks the pull request, so "block the merge" below means a person holds back approval or your branch rules do.
Sources: GitHub docs: About protected branches, Claude Code docs: Read and request reviews on a pull request, Claude Code docs: Code Review
What published numbers say about how many comments get acted on
No healthy rate applies everywhere, and the published numbers measure different things. They still give you a range to compare your own reviewer against.
| Source | What it measured | Result | Read it carefully |
|---|---|---|---|
| Greptile blog (its own bot, late 2024) | Share of the bot's comments that developers addressed before merging, before and after a filter that blocks comments resembling ones developers downvoted | Address rate went from 19% to 55+% within two weeks for existing users | A vendor reporting on its own product; no independent check |
| Beko case study (ICSE 2025, Software Engineering in Practice) | 1,408 bot comments on merged pull requests | 73.8% labeled Resolved, 21.3% labeled Won't Fix (the comment was faulty or could not be implemented) | One company; Resolved was 55% in one project and 90% in another |
| Atlassian RovoDev Code Reviewer paper | Share of RovoDev comments whose commented lines were changed in later commits, in a deployment of 54,000+ comments across 1,900+ repositories | 38.7% in the paper's one-week window of July 10 to 15, 2025, against 44.45% for human-written comments | A changed line counts as resolved; that does not prove the change was right |
| Google, Tricorder (CACM, 2018) | Policy for static-analysis results shown during code review | Up to 10% effective false positives allowed; an analyzer is disabled until its authors improve it when its not-useful click ratio goes above 10% | Static analyzers, not LLM reviewers |
Two readings matter. The spread is wide, from roughly 4 in 10 AI comments leading to a code change at Atlassian to nearly 3 in 4 labeled resolved at Beko, and two Beko projects sat at 55% and 90%, so your number depends on your code, your settings and how your team closes threads. And Google's bar is a policy: it set the noise ceiling in advance, and its tooling team acted when a tool crossed it. The paper says the 10% figure was initially chosen somewhat arbitrarily, though it seems to be a sweet spot for developer satisfaction.
Sources: Automated Code Review In Practice (arXiv 2412.18531), RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian (arXiv 2601.01129), Sadowski et al., Lessons from Building Static Analysis Tools at Google (CACM, April 2018)
The Google paper also defines false positives by what developers do, which fits AI reviewers. It counts an issue as an "effective false positive" if developers "did not take positive action after seeing the issue," and a real fault the developer did not understand counts too. By that definition a correct but unclear comment is a false positive, so write findings that name the file, the line and the failing behavior.
Sources: Sadowski et al., Lessons from Building Static Analysis Tools at Google (CACM, April 2018)
A four-pile triage policy
Read the findings in severity order, but read all of them, because the label is only a hint. Then decide each one with the same questions in the same order.
- Is it about code this pull request changed? If not, it is pre-existing, unless this change makes the old bug newly reachable. Claude Code Review has a Pre-existing severity level for this, defined as a bug "that exists in the codebase but wasn't introduced by this PR." Fixing one here widens the diff, so list it. Why Claude Code Edits Files You Never Named covers keeping a change inside its approved scope.
- Can you reproduce it? Point to the line that proves it, or write a failing test. The Google paper lists findings that are theoretically possible but do not actually manifest in practice among the reasons engineers ignore static analysis warnings. A finding you cannot demonstrate is a question, not yet a defect. Races, timing bugs and problems that appear only with production data are hard to pin down in a quick test, so for those, trace the code path by hand and ask a person before you drop the finding.
- What breaks if it is right and you ignore it? Data loss, a bypassed permission check, a wrong charge or a crash on the main path means block. A slower query on a rarely used report does not.
- Is the fix small and inside lines you changed? Then fix it now, in this pull request. Otherwise list it.
| Pile | What goes in it | What you do | How to close the thread |
|---|---|---|---|
| Block the merge | A reproducible defect in changed code that breaks behavior, leaks data or cannot be rolled back. Claude Code Review defines Important as "A bug that should be fixed before merging." | Write the failing test, fix it, run the suite, and have a person read the fix | Resolve the thread after the fix lands |
| Fix in this PR | A real, small, local problem in lines you changed: an unhandled null, a wrong error message, a changed branch with no test | Fix it before merge, in the same pull request | Resolve with a one-line reply and the commit |
| List for later | Pre-existing bugs, refactor and design suggestions, and nits worth doing someday | One tracked list, one line per item: file, finding, why it was not done now | Resolve with a link to the list |
| Drop | Contradicts the code, could not be reproduced and is not a timing or data-dependent bug, already enforced by CI, in generated files, or about a style your team has no rule for | Reply with the reason in one line, then add a rule to the reviewer config so it stops | Resolve with the reason |
A word on nits. Greptile's account says "Nits are subjective - definitions and standards vary from team to team," so no tool can pick your threshold. Do not drop a style or readability finding just because its label is low: in Atlassian's research, developers most often resolved Rovo Dev comments about readability, bugs and maintainability, at rates between 36% and 43%. If your team keeps fixing the same class of nit, move it into a linter or rule file.
Sources: How to Make LLMs Shut Up (Greptile Blog), Atlassian Rovo Dev Research: What Types of Code Review Comments Do Developers Most Frequently Resolve? (Atlassian blog)
Escalate by file path, whatever the label
One override beats every label. A finding in authentication, payments, data deletion, migrations or secrets handling gets read by a person, even if the reviewer called it a nit. Risk follows where a change lands and what it can do if it is wrong, the argument in Approval Gates for High-Risk AI Code Changes. The path list is yours to write.
Why the severity label is only a hint
Severity labels are generated by a model, not measured. Greptile reported that it tried a filter where the model rated each comment from 1 to 10 and anything under 7 was dropped. In its words, "The LLMs judgment of its own output was nearly random," and its lessons included "LLMs are bad evaluators of severity." That is one vendor's account, and it does not name the model. An academic study looks at an adjacent question: two automated judging methods run on three models, compared with engineers' fixed or won't-fix labels on 2,604 Beko bot comments, reached agreement ratios of about 0.44 to 0.62. That measures usefulness rather than severity labels, and its authors warn that the engineers' labels reflect workflow pressure as well as comment quality, so neither side is clean ground truth.
Sources: How to Make LLMs Shut Up (Greptile Blog), Understanding the Limits of Automated Evaluation for Code Review Bots in Practice (arXiv 2604.24525)
The better systems add a second pass. Claude Code Review runs a verification step that checks candidate findings against the code and filters out false positives, and Cloudflare's coordinator drops speculative issues, nitpicks and false positives, reading the source when it is unsure. Both still point to a person: the Claude docs say a review with no findings does not mean your code is error-free or secure, and Cloudflare writes that its system is not a replacement for human code review.
Sources: Claude Code docs: Code Review, Orchestrating AI Code Review at scale (Cloudflare Blog)
Spot false positives with a failing test
For anything headed for the block pile, ask for evidence before a fix. If the finding describes a behavior bug, a test can usually be made to fail because of it. If none can, the finding may be wrong, the behavior may never have been covered, or the bug may be one a quick test cannot reach, and you want to know which. Read the test yourself, because a test that fails for a setup reason proves nothing. After the fix, the same test shows it worked, which beats a second opinion from the same reviewer.
Prompt to paste into Claude Code: "Here is a review finding: [paste it]. Do not edit production code yet. Write the smallest test that fails because of this finding, run it, and show me the failing output. Say whether it fails for the reason the finding gives. If you cannot make a test fail, say so and explain why the finding may be wrong or why a test cannot catch it. Then wait for me."
Then close the loop at the class level. Anthropic describes one pull request that went through more than fifty rounds of review and fixes overnight. The team's changes to re-review were a round cap (after three rounds, the reviewer asks the author to step back and reconsider the approach), diff-only re-review after the first few passes, flagging the whole class of bug so the author fixes every instance, and no repeats of findings already posted. Claude Code Review does part of this on its own: a later review that starts automatically focuses on what changed and posts fewer nits and pre-existing findings, and a @claude review comment runs a full pass again. By hand, review only the new diff, and after three rounds step back and reconsider the approach instead of starting a fourth.
Sources: How the Claude Code team does code review (claude.dev), Claude Code docs: Read and request reviews on a pull request
Make the reviewer quieter at the source
Triage lowers the cost of each comment. Configuration lowers the number of comments. These controls come from vendor docs, which can change, so check them before copying values.
- Claude Code Review (the managed GitHub service): put instructions in a root
REVIEW.md. The docs' example redefines what Important means, caps nits with "Report at most five Nits per review.", lists what not to report (anything CI already enforces, generated files, lockfiles) and adds team rules. The docs also suggest requiring a file and line citation for behavior claims to cut false positives. Severity lowering, exclusions and nit caps are honored when the file is read from a protected default branch. - Local
/code-review: it readsCLAUDE.md, notREVIEW.md. Atloweffort it reports the findings it is most confident in, so you see fewer false positives;mediumthroughmaxbroaden coverage. - CodeRabbit:
reviews.profiletakesquiet,chill(the default) orassertive. Its docs sayquietposts only the most important comments inline and groups the others in the review summary, andassertivegives more feedback that may feel nitpicky.reviews.path_filterscan exclude paths such as!src/generated/**. - GitHub Copilot code review: repository-wide rules go in
.github/copilot-instructions.mdand path-specific rules in.github/instructions/NAME.instructions.md. Copilot reads them from the pull request's head branch, so a pull request can change the rules that review it. Claude Code Review readsREVIEW.mdfrom the base or default branch and never from the pull request's own changes. - A reviewer you build yourself: write a short do-not-flag section. Cloudflare's security reviewer prompt has a What NOT to Flag list that includes theoretical risks needing unlikely preconditions and issues in unchanged code. Its team wrote that "telling an LLM what not to do is where the actual prompt engineering value resides," and reported about 1.2 findings per review across 131,246 review runs in its first 30 days, which it calls deliberately low.
The principle carries across these tools, though the knobs differ: say what Important means, cap the minor findings, exclude what CI already checks, and write down what the reviewer should not flag. For what a review should look for in the first place, see An AI Code Review Checklist That Actually Catches Bugs.
Measure whether the reviewer earns its comments
Triage rules drift without feedback. Anthropic describes sorting every comment its reviewer leaves by how the person on the pull request responded: they fixed it, said it was wrong, said the behavior was intentional, or decided it did not matter. You can run a smaller version by hand. Take your last ten reviewed pull requests, tag each finding with one of those four outcomes, and act on the counts.
Sources: How the Claude Code team does code review (claude.dev)
- Fixed it: keep this kind of finding, and consider moving the class into a pre-merge check.
- Said it was wrong: add a do-not-report rule, or require evidence such as a file and line citation.
- Said it was intentional: the convention lives in someone's head, so write it into
CLAUDE.mdorREVIEW.md. - Decided it did not matter: cap or exclude that category.
Two traps. A thread closed as won't fix is not proof the comment was wrong, because the authors of the study of 2,604 Beko bot comments warn that developer labels reflect workflow pressure as well as comment quality. And a real bug that nobody acted on because the author did not understand the fault still counts as an effective false positive under Google's definition, so fix the wording of the finding, not only the volume. Pick a ceiling in advance, the way Google did, and revisit the reviewer config when you cross it.
Sources: Understanding the Limits of Automated Evaluation for Code Review Bots in Practice (arXiv 2604.24525), Sadowski et al., Lessons from Building Static Analysis Tools at Google (CACM, April 2018)
Where TLM Forge fits
TLM Forge sizes each task Light, Medium or Deep, and Light work gets no review agents, so a typo fix is not sent through a review wave. On Medium and Deep, a final wave of independent reviewer agents reads the change: CRITICAL and HIGH findings are fixed test-first or brought to you, and MEDIUM and LOW findings go on a written list.
The limit is the one this post keeps returning to. Those severity labels come from LLM agents and can be wrong, so someone still has to read the written list and decide whether a MEDIUM is really medium.
Frequently asked questions
01Why does AI code review leave so many comments?
It is easy to catch more bugs if you accept flagging many non-bugs, a tradeoff Anthropic names for its own reviewer. One vendor reported that 79 percent of its bot's comments in one analysis were nits. Repeated review rounds can add more, as one Beko respondent described. Caps, exclusions and a do-not-flag list cut the volume.
02How do I tell a false positive from a real bug in AI code review?
Try to reproduce it. Point to the line that proves it, or write a test that fails because of the finding. If neither works, ask the reviewer for a file and line citation. If it still cannot be shown, reply with the reason and resolve it, unless it involves races, timing or production data. Those need a person.
03Should I fix every AI code review comment?
No. Block the merge on reproducible defects in changed code, fix small real problems in the same pull request, list pre-existing bugs and nice-to-haves for later, and drop wrong or already-enforced findings with a one-line reason. Anything touching auth, payments, deletion or migrations gets a human read whatever its label.
04What is a good acceptance rate for AI review comments?
No universal figure exists. Atlassian reported 38.7 percent of AI comments led to a code change, Beko 73.8 percent labeled resolved, and Greptile 55 percent or more after filtering. Google allowed up to 10 percent effective false positives for static analysis in review. Measure your own and set a ceiling.
05Can I trust the severity label an AI reviewer gives?
Treat it as a hint for reading order, not a decision. Greptile reported that its model rating its own comments was nearly random. A Beko study of LLM judges scoring comment usefulness found agreement ratios of about 0.44 to 0.62 with engineers' fixed or won't-fix labels. Read every comment and check reproduction and blast radius yourself.