How do you run risk-based code review for AI-written pull requests on GitHub?
Risk-based code review means you match the review to the risk of the change, not to the size of the diff alone. A documentation fix and a change to payment or auth code are not the same kind of pull request.
MergeJudge is an AI pull request reviewer with an open-source engine. It puts every pull request into one lane: auto, review or block. Auto means low risk. That pull request can go straight through once required checks pass. That is not a guarantee. Review means a person should look. Block means do not merge.
That is the short answer. The rest of this page is an AI code review framework in five steps, and how to set each step up so you can scale code review when AI writes more of the pull requests. Use it when you review AI generated code and you do not want every change to get the same human pass.
The five steps come from Vineet Arya's article How to Scale Code Review for the AI Era: A 5-Step Framework (Level Up Coding, 30 Oct 2026). The steps are his. The setup under each step is written here in our own words.
1. Match the review to the risk
Not every pull request needs the same review. If you treat a docs typo and a billing change the same way, you either spend too long on the quiet change or not long enough on the dangerous one.
Arya's first step is to put a real risk label on the change, and to let that label drive required reviews and checks. He also says a change that is both large and high risk should be split before anyone tries to read it. MergeJudge does not split a pull request for you. The review comment can name files that could have been their own pull request. A change of more than 5000 lines goes to review.
The label MergeJudge posts is the lane. Make the check mergejudge/lane required. How to require it: stop AI agents merging their own pull requests. What each lane means: sort pull requests by risk.
| Lane | What it means | mergejudge/lane |
|---|---|---|
auto | Low risk. The pull request can go straight through once required checks pass. This is not a guarantee. If branch protection also requires a human review, GitHub still waits for that review. | success |
review | A person should look. GitHub treats a neutral required check as a success, so this check does not stop a merge. Keep a required human review. | neutral |
block | Do not merge. When the check is required, GitHub stops the merge. | failure |
Rules set the lane first. The AI step can only make it stricter, apart from lanes.code_auto, which is off by default. By default, code files go to review. Docs, markdown, CSS and public/** can be auto when the change is at most 200 lines and nothing else is wrong. One stricter file sets the lane for the whole pull request.
2. Tell the reviewer what matters in this repo
A generic review treats every file the same. Your repo has its own rules: what a reviewer must check, and what the linter already handles.
Put the judgement in policy files on the default branch. With no extra config, MergeJudge reads these from the repo root:
AGENTS.mdCLAUDE.mdREVIEW_POLICY.mdAPPROVAL_POLICY.md
.mergejudge.yml is the config file. It is read from the base branch too. Use it for hard rules, such as paths that always need a person. Use the policy files for judgement, such as "new endpoints must check authorization on the resource".
Files are read from the tip of the base branch, never from the pull request head. A pull request cannot swap in a looser AGENTS.md and be reviewed under it. That edit applies from the next review after it is merged. A pull request that edits .mergejudge.yml goes to review.
Policy text can raise a lane, from auto to review to block. It cannot lower a lane below what the rules decided. The pull request title, body and diff are untrusted, so a note in the pull request does not replace these files. Each policy file is capped at 8 KB, and 8 KB in total, by default. How the files fit together: AGENTS.md and policy files.
3. Run special passes you do not have to remember
The third step is to run named passes, instead of hoping someone remembers a checklist. MergeJudge runs three of those as deterministic rules, before any model call. They use .mergejudge.yml from the base branch.
Protected paths. Files that match lanes.protected are at least review, even when no AI provider is configured. The defaults are .github/**, deploy/**, **/auth/**, **/migrations/**, wrangler.toml, _headers, **/_headers and **/.env*. _headers files stay at least review even if you drop them from the list. If you set lanes.protected, you replace the default list, so keep the defaults you still want:
# .mergejudge.yml on the default branch
lanes:
protected:
- ".github/**"
- "deploy/**"
- "**/auth/**"
- "**/migrations/**"
- "wrangler.toml"
- "_headers"
- "**/_headers"
- "**/.env*"Secret screen. Added lines that look like an AWS key, a private key block, a GitHub token (ghp_) or a hard-coded api_key assignment are block. This is a coarse screen, not a secret scanner. Obvious fakes, such as EXAMPLE or PLACEHOLDER, are noted and not blocked.
Removed safety guards. In a protected path, removing --match-head-commit, persist-credentials: false, set -e, or --frozen-lockfile / npm ci is block.
The full rule list is on sort pull requests by risk.
4. Review every pull request, and let low-risk ones go through
The fourth step is to make the review a system. A review should run on every pull request, and a low-risk pull request should be able to go through without someone remembering to wave it on.
MergeJudge reviews every pull request it is installed on. Each one gets a lane, a summary comment and the mergejudge/lane check.
Approvals are off by default. MergeJudge does not merge pull requests itself. See Approvals.
If you want the auto lane to go straight through once required checks pass, that is GitHub's native auto-merge. Set approval.mode to auto-merge-only. MergeJudge then asks GitHub to auto-merge an auto-lane pull request only after your branch protection and required checks are met. Turn on Allow auto-merge in the repository settings. Leave both off until you choose this. The auto lane still means low risk. It is not a guarantee.
People who look up how to auto approve low risk pull requests should read it that way. Approvals stay off. auto-merge-only does not submit an approval. It turns on GitHub auto-merge for the auto lane only.
AI-written pull requests are still pull requests. To keep agent-marked ones out of the auto lane, set lanes.trusted_authors with agents: untrusted and applies_to: all_auto. The agent marks the pull request with the mergejudge:agent label or a MergeJudge-Actor: agent line. That setting can only take the auto lane away. Setup: stop AI agents merging their own pull requests.
5. Learn from reverts and follow-up fixes
The last step is to watch what happens after merge, and to tighten review where a low-risk call was wrong. Do not invent a score for it. Record the outcomes you can actually see, and let a person merge any rule change.
On the hosted GitHub App, learning is on by default (learning.enabled: true). Reviewer feedback is recorded. Merges over a lane, reverts and follow-up fixes are recorded too. Learning does not change a lane by itself, and it does not edit your config until a human merges a proposal.
A scheduled run can open one proposal pull request. With enough evidence it edits .mergejudge.yml in two ways only:
- a stricter protected path, after auto-lane pull requests in one folder were reverted or quickly fixed at least twice
- a rubric note on an existing check that people keep marking as too strict or a false alarm, at least 3 pull requests and 2 different people. Security checks never get a note
The proposal is labelled do-not-merge and needs-human. MergeJudge lanes it block and does not approve or merge it. With too little evidence the proposal is a report only and changes no config.
The GitHub Action records checkbox feedback and /mergejudge learn notes, and uses those notes for proposal pull requests. The revert and follow-up fix records above are for the hosted App.
Checklist
- Require the
mergejudge/lanecheck. Also require a human review, because a review lane does not stop a merge on its own. - Put judgement in
AGENTS.md,CLAUDE.md,REVIEW_POLICY.mdorAPPROVAL_POLICY.mdon the default branch. - Keep
lanes.protectedon the base branch. If you set the list, copy the defaults you still want. - Leave approvals off until you want GitHub auto-merge for the auto lane. Then set
approval.modetoauto-merge-onlyand turn on Allow auto-merge. - If coding agents open pull requests, mark them, and keep them out of the auto lane until you accept that risk.
- When a proposal pull request appears after reverts or follow-up fixes, read it. Merge it yourself only if you agree.
Five steps and what to configure
| Step | MergeJudge feature | What you configure |
|---|---|---|
| 1. Match review to risk | Lanes auto, review and block, posted as the check mergejudge/lane. Auto means low risk and can go straight through once required checks pass. | Require mergejudge/lane. Keep a required human review. See lanes. |
| 2. Context for the reviewer | AGENTS.md, CLAUDE.md, REVIEW_POLICY.md, APPROVAL_POLICY.md and .mergejudge.yml, read from the base branch. | Write the files on the default branch. A pull request cannot use its own copy. See policy files. |
| 3. Special passes | Protected paths are at least review. The coarse secret screen is block. Removed safety guards in a protected path are block. | lanes.protected in .mergejudge.yml on the base branch. The secret screen and guard checks are already on. |
| 4. Review every pull request, and let low-risk ones through | A review on every installed pull request. Opt in if the auto lane should use GitHub auto-merge. | approval.mode: auto-merge-only, and Allow auto-merge in GitHub. Off by default. See approvals. |
| 5. Learn from reverts and follow-up fixes | The hosted App records reverts and follow-up fixes. A proposal can add a protected path. A human merges it. | learning.enabled is on by default. Do not merge the proposal until you have read it. |
FAQ
What is risk-based code review for AI-written pull requests?
Risk-based code review means the human work follows the risk of the change. A docs fix and a change to auth or billing should not get the same review. MergeJudge is an AI pull request reviewer with an open-source engine. It puts every pull request into auto, review or block. Auto means low risk. The pull request can go straight through once required checks pass. That is not a guarantee.
How do you scale code review when AI writes more of the pull requests?
Run one AI code review framework on every pull request, and spend human time where the lane is review or block. MergeJudge sorts the pull request before anyone has to read the whole diff. The auto lane is low risk and can go straight through once required checks pass. Branch protection still applies.
Can a pull request loosen AGENTS.md or .mergejudge.yml for its own review?
No. MergeJudge reads AGENTS.md, CLAUDE.md, REVIEW_POLICY.md, APPROVAL_POLICY.md and .mergejudge.yml from the base branch, not from the pull request head. A pull request that edits .mergejudge.yml goes to review. A policy file edit applies only after it is merged.
Which review passes run before the model?
Three deterministic passes run first. Protected paths are at least review. Added lines that look like common secrets are block. Removing a safety guard in a protected path is block. These do not need a model. The secret check is a coarse screen, not a secret scanner.
How do you auto approve low risk pull requests?
Approvals are off by default, and MergeJudge does not merge pull requests. Auto means low risk. The pull request can go straight through once required checks pass. For that lane, set approval.mode to auto-merge-only so GitHub's own auto-merge can run, and turn on Allow auto-merge in the repository. That setting does not submit an approval. The lane is still not a guarantee.