How to reduce noise and false positives in AI code review
Most noise in AI code review comes from asking a model to comment on everything, with no repo rules and no view past the diff. Cut false positives by leaving linter and type-checker findings out, giving the reviewer your rules, letting deterministic rules route risk, and keeping only a few comments that quote a changed line. Measure the result by the share of comments people act on.
Walkthrough
Video
A short walkthrough of cutting noise and false positives in an AI code review will be added here. There is no video on this page yet.
1. Measure the acted-on rate
A comment can be wrong, or it can be true and still not worth interrupting someone. Count those apart before you change settings.
For about two weeks, on one repository, list every review comment. Acted on means it led to a code change or a reply that agrees. Divide by all comments in the window. That share is your acted-on rate. You tally it. MergeJudge does not calculate it.
Tag each dismissal in the reply. Wrong means a false positive: the code was already correct. Not worth saying means noise, such as a formatting nit. A thumbs-down does not record which tag you meant. Note the reviewer, the mode, and whether a linter runs, so the next two weeks match. There is no standard false positive rate to copy.
2. Take linter territory off the table
If the linter or the type checker already fails the pull request, the reviewer should not say it again. Formatting, import order, unused variables, and a missing type are CI. A second copy teaches people to skip the review, including the one real bug.
Put the skip list in a policy file on the default branch:
## Ignore
- Formatting, import order, and unused variables. The linter and the type checker already fail the pull request.
- Typo fixes in comments and docs that do not change behavior.
- Docs-only wording. Do not leave line comments on a docs-only change.The model already looks for bugs, security holes, and broken logic. MergeJudge reads that policy from the base branch. It does not run your linter. diff_exclude keeps lockfiles, build output, images, and generated files out of the model prompt. Rules still lane those paths. See AGENTS.md and policy files for AI review and .mergejudge.yml.
3. Give the reviewer your repo rules
Use rules you already wrote, in AGENTS.md, REVIEW.md, or REVIEW_POLICY.md. MergeJudge does not read CONTRIBUTING.md unless you list it in policy_files.
By default it reads REVIEW_POLICY.md, REVIEW.md, APPROVAL_POLICY.md, AGENTS.md, and CLAUDE.md from the repo root on the base branch tip, never from the pull request head. A pull request cannot swap in a looser copy. Title, body, and diff are untrusted and cannot lower the lane. Policy can only make the review stricter. Each file is capped at 8 KB, and 8 KB in total. An empty policy_files list turns this off. Keep the review section short so build notes do not crowd it out.
Rubrics are separate yes or no checks (rubric packs). Security and correctness are on by default. Only a failed high-severity criterion raises the lane, and only with a quote in the diff. Low and medium failures, including cor.tests, are reported and do not raise the lane. Custom criteria live in .mergejudge.yml on the base branch.
4. Match context to the change
A diff-only pass guesses when the bug is in a file the pull request did not touch: a renamed function, a caller elsewhere, a config key still read by its old name. That is a false positive when the break is not real, and a false negative when the caller is never opened.
review.mode changes context only. Lanes stay the same. Set it on the base branch (Review modes).
diff(the default): hunks, policy, and rubrics. Use it when the edited file is the whole story.context: also the changed files, their imports, and a few callers.deep: a temporary index on a runner or in the GitHub Action, then thrown away. Hosteddeepwith no runner runs ascontext. A hosted tool loop is not offered.
Do not set every pull request to deep. Extra files can add comments on code the change did not touch.
5. Route by risk with rules first
Let rules decide who must look. The model may only raise that decision. A docs typo and an auth change are not the same review.
A low-risk docs change should get no line comments. Protected paths (CI, deploy, auth, migrations), source files, and a huge diff need a person. No path rule catches a test edited so the suite passes. You still read that diff (Review pull requests from AI coding agents).
MergeJudge posts auto, review, or block. Rules set the minimum. The model can only raise it, apart from lanes.code_auto, off by default, which a pull request cannot turn on. Auto means low risk, not a guarantee. Docs, markdown, CSS, and public/** stay auto at or under 200 lines. Code and lockfiles are review unless that opt-in is on. Protected paths, including .github/**, are at least review, and a .mergejudge.yml edit is always review. Replacing lanes.protected replaces the default list. Over 5000 lines is review. Common secret shapes, and a removed safety guard on a protected path, are block (a coarse screen, not a secret scanner). The strictest file wins. See lanes and what can never be auto.
mergejudge/lane is success, neutral, or failure. A neutral check does not stop a merge, so require it for block and keep a human review on review (how to require the check, risk-based review). auto does not delete line comments. The ignore list in step 2 does. The summary and the check still post.
6. Post fewer verified comments
Post one summary. Post an inline comment only when it sits on a changed line, quotes that line, and names what breaks. Drop a "consider" note and any comment that would still be true if the line had not changed.
MergeJudge posts a line comment only when the quote is in that file's diff, at most 20 per review. Without the quote, the finding stays in the summary under What to check, marked "needs a human check", and it does not raise the lane. Only a failed high-severity rubric item with a matching quote raises the lane. Low and medium failures stay in the comment.
7. Close the loop
Record the dismissal or the same comment returns. On the hosted App, a writer dismisses a finding with "false positive", "not a bug", "by design", "won't fix", "noise", a thumbs-down, or by resolving the thread when that code did not change. A thumbs-up, or a resolve after the code changed, marks it useful. Bots do not count.
After two dismissals the repeat moves into the summary. After three it is hidden. Confirmations count against that. A file laned block is never hidden. The lane does not change.
MergeJudge does not edit .mergejudge.yml itself. A proposal pull request, which only a human can merge, can add a protected path (two reverts or quick fixes of auto-lane pull requests in one folder) or a rubric note (3 pull requests, 2 people). Security checks never get a note. The proposal is labelled do-not-merge and needs-human and is laned block. The hosted schedule is Mondays at 11:00 CEST. The GitHub Action records summary checkboxes and /mergejudge learn only. It does not hide repeats. See Learning loop.
Checklist you can copy
Copy this into the notes for the two-week sample. Empty boxes are not a measurement.
- [ ] Acted-on rate for about two weeks: comments that changed code or got an agreeing reply, divided by all comments.
- [ ] Each dismissal tagged wrong (false positive) or not worth saying (noise).
- [ ] Formatting, unused variables, import order, and type errors left to the linter.
- [ ] Rules live in a base-branch policy file (AGENTS.md, REVIEW.md, or REVIEW_POLICY.md), not in the pull request.
- [ ] CI, auth, migrations, and deploy are deterministic rules. The model can only raise that lane.
- [ ] diff for a self-contained edit. context or deep when a caller lives in another file.
- [ ] A docs-only pull request gets a summary and no line comments.
- [ ] Each inline comment quotes a changed line and names a defect. "Consider" notes are dropped.
- [ ] Dismissals are recorded. A rule change waits for a pull request a human merges.
- [ ] Known-bug pull requests are still flagged after the cut.Where the noise comes from
Wrong comments need better context or a quote rule. Not-worth-saying comments need an ignore list or fewer line comments.
| Source of noise | Example | Fix by config or rule | What MergeJudge does |
|---|---|---|---|
| Linter territory | Formatting, or an unused variable | Leave it to the linter. Say so in policy. | Policy comes from the base branch. No linter is bundled. diff_exclude keeps generated files out of the prompt. Rules still lane them. |
| Diff-only guess | A caller that is not in the diff | Use a mode that can see callers. | diff is the default. context adds callers. deep indexes the repo on a runner or in the Action. Hosted deep with no runner runs as context. |
| Same lane for every change | Docs and auth both get line comments | Rules first. Docs get no line comments. | auto, review, or block. Small docs stay auto. The model can only raise the lane, apart from lanes.code_auto, off by default. auto means low risk, not a guarantee. |
| Vague comments | "Consider renaming this" | Quote a changed line and name a defect. | A line comment needs a quote in the diff, at most 20. Otherwise it stays under What to check and does not raise the lane. |
| Pull request edits its own rules | A looser AGENTS.md in the diff | Read policy and config from the base branch. | Both come from the base branch tip. A .mergejudge.yml edit is always review. Policy cannot lower the lane. |
| Repeat false alarm | The same dismissed comment next week | Record it. Change rules only in a pull request a human merges. | Hosted: summary after two dismissals, hidden after three. The lane does not change. Only a human can merge the proposal. |
False positives and false negatives
Quieter is not the same as better. A false positive is a wrong comment. A false negative is a real bug with no comment. Telling the model to say less can hide the second kind.
Keep a few pull requests with a bug you already understand, such as a dropped auth check, a swallowed error, or a caller broken by a rename. After you change the policy or the mode, run them again. If they are no longer flagged, you cut too far. The acted-on rate will not show that miss, because a bug with no comment cannot be acted on. Ten small test pull requests are in How to evaluate an AI code review tool.
How MergeJudge helps
MergeJudge (mergejudge.si) is an AI pull request reviewer with an open-source engine for GitHub. Deterministic rules set auto, review, or block. The AI step can only raise the lane, apart from lanes.code_auto, off by default. Auto means low risk, not a guarantee. Branch protection still applies. MergeJudge does not merge pull requests, and approvals are off by default.
Policy and .mergejudge.yml are read from the base branch. The review posts one summary, at most 20 quoted inline findings, and mergejudge/lane. On the hosted App, repeated dismissals get quieter, and a rule change is a proposal a human merges. Learning does not lower a lane. Start in Getting started.
FAQ
What is a false positive in AI code review?
A false positive is a comment that is wrong: the code does what it should, and the reviewer flagged a bug that is not there. Noise is different. The comment may be true and still not worth a person's time, such as a formatting nit a linter already owns.
What is a normal false positive rate for an AI code reviewer?
There is no single standard, and this page does not give one. Track your own acted-on rate: the share of comments that led to a code change or a reply that agrees, over about two weeks. Tag what you dismiss as wrong or as not worth saying.
How do I make an AI code reviewer less noisy on GitHub?
Exclude linter findings. Put rules in policy files on the base branch. Deterministic rules set the lane. The model can only raise it, apart from opt-in lanes.code_auto, which is off by default. Keep a few verified line comments, and record dismissals so the same one does not return.
Is a quieter AI reviewer always better?
No. Quieter can mean the reviewer missed a real bug, which is a false negative. Watch both. Keep a few pull requests with known bugs and confirm the reviewer still flags them after you cut noise. A comment people act on is the one to keep.
Should AI code review comments block a merge?
Not by themselves. A comment is easy to ignore. Require a status check for changes that must stop, and keep a person on the rest. In MergeJudge, only a block lane fails mergejudge/lane. Review is neutral and does not stop a merge. Auto means low risk, not a guarantee.