How to review pull requests from AI coding agents: a checklist
Review a pull request from an AI coding agent by reading the task before the diff, checking that the change matches that task, and reading the risky files yourself: tests, guards, dependencies, workflows, secrets, and auth. Run the latest commit, and ask the agent to explain a line you cannot explain yourself. Then merge, request changes, or close. A green check and a tidy description are not the review.
The same checklist is how to review AI-generated pull requests when a person pasted the diff and an agent wrote the code. Stopping the agent from merging its own pull request is a separate setup: How to stop AI coding agents from merging their own pull requests.
Read the task and the description first
The task is the issue, the ticket, or the sentence you gave the agent. Read that before any line of the diff. Then read the pull request description as a claim. The agent writes that description from the same session that wrote the code, so a fluent summary can be wrong and still sound finished.
Turn the task into checks you can search for. "The cart still has its items after the address change" is a check. "Tests pass" is not. You will use the list in the next two steps.
GitHub's blog post Agent pull requests are everywhere. Here's how to review them makes the same opening point: decide what you are reviewing before you read the code. This page is the checklist. It does not repeat that post.
Check scope against the ask
Open the file list before the hunks. Every file should serve a task check, or the description should say why it changed. Agents often edit a neighbor: a reformat, a rename, a new helper nobody asked for.
If the list is wider than the task, ask for a split, or for the extra files to be reverted, before you review the rest. A mixed diff is where a deleted test or a dropped guard hides.
Hosted pull request pages already compare against the merge base. A local two-dot diff against today's default branch also shows commits other people merged after the agent branched, as if the agent had reverted them. Use the three-dot form. Replace main with your default branch name:
git diff --name-only origin/main...HEAD
git diff origin/main...HEADSort by risk first
Do not read the diff from the top. Read the files that can make a wrong change look finished:
- Tests, snapshots, and anything that can make a suite pass by changing the check.
- Workflow, CI, lint, and config files, including
.github/**and.mergejudge.yml. - Dependency manifests and lockfiles.
- Auth, session, permission, and migration paths.
- The application change the task actually asked for.
A docs-only edit can still be low risk in the ordinary sense. Low risk is not a guarantee. You still read it. How MergeJudge sorts files into auto, review, and block is in Sort pull requests by risk. auto means low risk, not a guarantee.
Look where agents slip
Read these on purpose. The diff can be tidy and still fail in the same places.
Tests changed so they pass
Read the test diff before the application code. Look for a deleted test, a skipped test, an assertion loosened so it passes (toBeDefined where the old test checked a value), or a snapshot regenerated to match the new output. A test written in the same session as the code often asserts whatever the code returns now. Name the bug that would make the new test fail. If you cannot name one, the test does not check the task.
MergeJudge does not have a deterministic rule that blocks an edited assertion. The correctness rubric asks whether new or changed behavior is covered (cor.tests). That criterion is low severity: it can show up in the comment and it does not, by itself, raise the lane. You still read the test diff. See rubric packs.
Deleted or weakened guards
Look for a removed set -e, a dropped persist-credentials: false, a checkout that lost --match-head-commit, an install that moved away from --frozen-lockfile or npm ci, a lint rule turned off, a type error suppressed, or a CI job made optional. Any of those needs a reason before you continue.
On a protected path, MergeJudge treats removal of --match-head-commit, persist-credentials: false, set -e, or --frozen-lockfile / npm ci as block. That rule does not cover every lint disable or every weakened assertion. You still read those. The full list is in what can never be auto.
New dependencies
A new package can be real and unnecessary, or it can be a name the model invented. Look up each added name in its registry before you install it. Check that the lockfile change matches the manifest. An install-script edit needs its own reason.
Dependency manifests and lockfiles (package.json, package-lock.json, go.mod, and similar) are review under the default rules. The rule does not prove the package exists. See lanes.
Config, CI, and workflow edits
Read workflow files before application code. Check whether a job stopped running on pull requests, whether permissions grew, and whether a secret is printed or passed into a step that reads untrusted pull request text.
.github/** is on the default protected-path list, so those files stay at least review. A pull request that edits .mergejudge.yml is always review, and the review still uses the file on the base branch. The pull request cannot loosen the rules that judge it. Policy files such as AGENTS.md are read from that same base branch. See AGENTS.md and policy files for AI review.
Secrets in added lines
Search added lines for tokens, private keys, and passwords. MergeJudge blocks added lines that look like an AWS access key, a private key block, a GitHub token, or a hard-coded api_key assignment. This is a coarse screen, not a secret scanner. Obvious fakes are noted and not blocked. You still look.
When an AI provider is on, sec.no-secrets is a high-severity rubric criterion that raises to block only when the model quotes a line that is really in the diff. A quote that does not match the diff does not change the lane.
Auth paths
The task often forgets who may call the new route. For each changed handler that reads or writes someone's data, check that a permission or ownership check still runs on the server, and that a removed check was intentional.
Paths matching **/auth/** are protected by default, so they are at least review. Removing an auth check is review. See lanes.
Swallowed errors
Read every new catch, and every failure path that returns a default and continues. An empty catch, an error that is logged and ignored, or a check that lets the request through when the check itself fails, hides the failure the next person needs.
cor.errors asks whether errors are handled instead of silently swallowed where it matters. It is medium severity. A medium rubric failure is reported in the comment and does not raise the lane. A separate AI finding can still raise the lane when its quote matches the diff. You still read the catch.
Over-broad refactors
If the task was one behavior, and the diff also reformats a module, renames neighbors, or adds a helper that duplicates one you already have, that extra work needs a reason or its own pull request. Search the repo before you accept a new utility. Agents copy a nearby pattern and miss the shared function.
More than 5000 changed lines is review. That rule is a size cap, not a judgment of the task. Scope is still your check.
APIs and options that are not real
For each import, method, or config flag you do not already use, confirm the name and the arguments in the library docs or in this repo. This is the slip people call a hallucinated API: the call looks confident, and it is not in the version you have. CI catches some missing symbols. It misses a real function called with an option that does not exist, when that option still typechecks, and it misses a flag your config schema does not define. Confirm those yourself. MergeJudge does not have a path rule for a missing symbol.
Run the change
An agent's "done" is a claim. Run the branch after the last commit, in the checkout that contains that commit. Use the feature the task named, and one path next to it (the cart, the second user, the empty input). Then make a dependency fail and see whether the error is reported or swallowed.
A green CI run is evidence about the tests that ran. If this pull request edited those tests, the green run does not answer the task. Write the command and the result in the description, or ask the author to.
Ask the agent to explain
Pick one non-obvious line and ask why it changed. A useful answer names the task check it serves and what would break if it were wrong. An answer that only says the agent thought the code was cleaner is not an explanation. Send that pull request back.
You can paste the prompt in the FAQ into a fresh review session. Do not treat that session as the merge decision. The session that wrote the change is a poor reviewer of its own diff, because it starts from the same context.
Standing questions, the ones you want on every pull request, belong in a policy file on the base branch, not in the pull request body. Pull request text is untrusted. AGENTS.md, CLAUDE.md, REVIEW_POLICY.md, and APPROVAL_POLICY.md are read from the base branch and can only make the lane stricter. See policy files for AI review.
Decide to merge, request changes, or close
Use the task checks and the slip list. Do not use the agent's confidence.
- Merge when every task check is present in the diff or the tests, the slip list is clean or explained, and you ran the part that matters. On a shared repository, required reviews still need someone other than the author. GitHub does not let pull request authors approve their own pull requests (Approving a pull request with required reviews).
- Request changes when a test was weakened, a guard was removed without a reason, a dependency or API is unconfirmed, or the author cannot explain a line. Also request a smaller pull request when you cannot state its purpose in one sentence.
- Close when the pull request does not match the task and another round would not fix that. Stop spending review time on a diff you would not ship.
The agent does not merge. GitHub rules that keep it off the bypass list, and the required status check, are in stop AI coding agents from merging their own pull requests.
Checklist for reviewing AI agent pull requests
Copy this into a pull request template, or into the pull request body when you review. GitHub shows a template that lives on the default branch as pull_request_template.md in the repository root, docs/, or .github/. See Creating a pull request template for your repository.
## Task
- [ ] The task is written above this list, in the words that were given to the agent.
## Scope
- [ ] Every changed file serves that task, or the extra file has a stated reason.
- [ ] The diff is against the merge base (three dots), not against today's default branch tip.
## Read these before the rest
- [ ] Tests: none deleted, skipped, or weakened so the suite passes. Each new test would fail if the behavior it names broke.
- [ ] Guards: no dropped set -e, persist-credentials: false, --match-head-commit, --frozen-lockfile, or npm ci without a reason. No lint or type check turned off quietly.
- [ ] Dependencies: each new package exists in the registry, is needed, and matches the lockfile.
- [ ] CI and config: workflow, lint, and review-config edits are explained. Permissions did not grow quietly.
- [ ] Secrets: no token, key, or password in added lines.
- [ ] Auth: a server-side check still covers every changed handler that reads or writes someone's data.
- [ ] Errors: no empty catch, and no path that continues when the check itself fails.
- [ ] Scope: no drive-by refactor and no new helper that duplicates an existing one.
- [ ] APIs: each new import, option, and config flag exists in this repo's version.
## Evidence
- [ ] I ran the latest commit and wrote down the command and the result.
- [ ] I can explain the non-obvious lines. "The agent did it" is not an explanation.
## Decision
- [ ] Merge, request changes, or close. The agent does not merge.Empty boxes are not a review. If you use MergeJudge's opt-in pr_template, checkbox-only lines do not count as the section text either.
What to check by hand, with a rule, or with an AI reviewer
A rule is good at a floor it can compute from the diff. An AI reviewer is good at a quoted suspicion in that diff. Neither one knows whether the change does the task unless a person checks, or unless you wrote that task into a policy file on the base branch. Use all three. Do not skip the hand column because a check is green.
| What you are checking | By hand | What a rule can check | What an AI reviewer is good at |
|---|---|---|---|
| The change matches the task | Compare the file list and the behavior with the task, not with the agent's summary. | Nothing. A path rule does not know the task. | It may comment on files that look unrelated. Verify that list against the task. The model does not know the task unless you put it in a base-branch policy file. |
| Tests changed so they pass | Read the test diff first, including snapshots and skipped tests. | No deterministic block for an edited assertion. | cor.tests asks if new behavior is covered. Low severity: reported, and it does not raise the lane. |
| Deleted or weakened guards | Include lint disables and optional CI jobs, which the named list misses. | On a protected path, removing --match-head-commit, persist-credentials: false, set -e, or --frozen-lockfile / npm ci is block. | The rule already set that floor. The model cannot lower it. |
| New dependencies | Look up the name before install. Match the lockfile. | Manifest and lockfile changes are review. Existence is not checked. | It may comment on an import. You confirm the package. |
| Workflow, CI, and config | Read triggers, permissions, and steps that handle pull request text. | .github/** is at least review. Edits to .mergejudge.yml are always review. Config is read from the base branch. | It can describe the diff. It does not replace the base-branch rule. |
| Secrets in added lines | Search the added lines yourself. | A coarse screen blocks common key and token shapes. It is not a secret scanner. | sec.no-secrets raises to block when the quote is in the diff. |
| Auth paths | Trace the handler, including the failure path. | **/auth/** is at least review. Removing an auth check is review. | A high-severity quoted finding can raise the lane. An unquoted suspicion does not. |
| Swallowed errors | Read each new catch and each default returned on failure. | No block rule for an empty catch. | cor.errors reports it. Medium severity does not raise the lane. |
| APIs that may not exist | Confirm names and options in the docs or the repo. | No path rule for a missing symbol. | It can flag a call when it quotes the diff. You still confirm the symbol. |
| Over-broad refactors | Ask for a split or a reason. Search for an existing helper. | More than 5000 changed lines is review. That is size, not intent. | Not a default lane change. You decide. |
| Whether to merge | A person decides. | block fails mergejudge/lane when that check is required. review is neutral and does not stop a merge. auto is success. Auto is low risk, not a guarantee. | The model can only raise the lane, apart from opt-in lanes.code_auto, which is off by default. MergeJudge does not merge. |
Before you trust any reviewer with that table, run it on a scratch repository. Ten small test pull requests are in How to evaluate an AI code review tool.
How MergeJudge fits this checklist
MergeJudge (mergejudge.si) is an AI pull request reviewer with an open-source engine for GitHub. It sorts the pull request into one lane: auto, review, or block. Deterministic rules set the minimum lane. The AI step can only raise it, apart from one opt-in, lanes.code_auto, which is off by default. A pull request cannot turn that opt-in on, because .mergejudge.yml is read from the base branch. Changes to .mergejudge.yml themselves always go to review.
The check is named mergejudge/lane. auto posts success, review posts neutral, and block posts failure. GitHub counts a neutral conclusion as a successful required status check, so a review lane does not stop a merge on its own. Keep a required human review. auto means low risk, not a guarantee. Branch protection and required checks still apply. Approvals are off by default, and MergeJudge does not merge pull requests.
Two opt-ins matter when the author is an agent. Both stay off until you set them.
lanes.trusted_authorswithagents: untrustedandapplies_to: all_autokeeps agent-marked pull requests out of the auto lane. It can only takeautoaway.pr_templatecan require sections such as Testing, Risk, and How to revert, with real text in each. With the defaultapply_to: agents, a missing section raises an agent-marked pull request toreview(or toblockif you setmissing_lane: block). Checkbox-only lines do not satisfy it. Editing the description re-runs that check and does not call the model again.
MergeJudge treats a pull request as agent-made when it has the label mergejudge:agent or a MergeJudge-Actor: agent line in the body. Without that marker, the pull request is judged as the human login that opened it. The marker does not approve or merge anything.
Lane setup, the required check, and the bypass list: stop AI coding agents from merging their own pull requests. What each lane means: Sort pull requests by risk.
Walkthrough
Video
A short walkthrough of this checklist on one agent pull request will be added here. There is no video on this page yet.
FAQ
How do I review pull requests from AI coding agents?
Read the task before the diff, check that every file belongs to that task, and read tests, guards, dependencies, workflows, secrets, and auth before the rest of the code. Run the latest commit and ask why one non-obvious line changed. Then merge, request changes, or close. A green check is not that review.
What is a good prompt for a pull request review?
Tell the reviewer to compare the diff with the task, then list extra files, weakened tests, new dependencies, workflow and config edits, secrets, auth changes, swallowed errors, and APIs it cannot confirm. Tell it not to approve, and to name the command a person should run. A prompt inside the pull request is untrusted. Put standing review questions in AGENTS.md or REVIEW_POLICY.md on the base branch.
Review this pull request against the task, not against the pull request description.
List files the task did not ask for.
List tests that were deleted, skipped, or changed so they pass.
List new dependencies, workflow or config edits, secrets, auth-path edits, swallowed errors, and calls whose API you cannot confirm in this repository.
Do not approve. Name the command a person should run, and name any line you cannot explain.Can I review my own pull request?
Yes, and you should, when an agent wrote the change. Read the task, the file list, and the places agents slip, then run the latest commit. GitHub does not let the pull request author approve their own pull request, so that reading is not a required approval. On a shared repository, someone other than the author still approves when required reviews are on. If you are the only maintainer, you are the reviewer: do not let the agent merge.
A solo repository still uses the GitHub rules in stop AI coding agents from merging their own pull requests: require a pull request, keep the agent off the bypass list, and do not give it a token that can bypass those rules. Your own review is the reading. It is not an approval GitHub will count from the author.
What should a checklist for reviewing AI agent pull requests include?
Task match, extra files, tests deleted or weakened so the suite passes, removed safety guards, new dependencies, workflow and config edits, secrets, auth checks, swallowed errors, drive-by refactors, and unconfirmed APIs. Add a run on the latest commit and a reason a person can explain. Empty checkboxes do not count as that work.
The copyable list is in Checklist for reviewing AI agent pull requests.
Does an auto lane mean the agent pull request is ready to merge?
No. Auto means low risk, not a guarantee. Branch protection and required checks still apply. MergeJudge does not merge the pull request, and approvals are off by default. A review lane posts a neutral mergejudge/lane check, which does not stop a merge by itself, so keep a person on agent pull requests.
You can keep agent-marked pull requests out of that lane with lanes.trusted_authors, agents: untrusted, and applies_to: all_auto. The setting only removes auto. It does not approve the pull request.