How it works

MergeJudge answers one question for every pull request: can this merge on its own, does a human need to look, or should it stop? Those are the three lanes: 🟢 auto, 🟡 review and 🔴 block.

1. 1

   A pull request is opened or updated

   Nothing to set up per PR. MergeJudge reacts to the GitHub event.

2. 2

   GitHub sends a webhook to the Worker

   The MergeJudge GitHub App notifies the Cloudflare Worker. The Worker checks the signature, queues a job and answers right away.

3. 3

   The rules floor runs first

   Deterministic rules (protected paths, secrets, safety guards) set the minimum lane. No model is involved yet. Two opt-in checks, both off by default, can only raise it: a PR description check (sections such as Testing, Risk and How to revert) and trusted authors (only the GitHub users you list can get small code changes into auto).

4. 4

   AI review: hosted with your key, or your own runner

   The AI reads the change for real bugs, security holes and broken logic, and follows your repo's review guide (REVIEW_POLICY.md, AGENTS.md or CLAUDE.md). In context or deep mode it also reads the surrounding code: full files, imports and callers. It checks your rubric too, and a failed criterion only counts if it quotes a line from the diff. What your reviewers taught it, such as known false alarms, keeps repeat noise down. It runs on the hosted service with your own provider key, or in the open-source runner on your machine (connected to the hosted service with an enrollment code) with your own Claude Code, Codex or GLM login. In deep-local mode the runner reads your local clone. The AI can only raise the lane (one opt-in exception for small code changes).

   What a good REVIEW_POLICY.md looks like

5. 5

   One result, and MergeJudge never merges

   A lane check (auto, review or block), a sticky summary comment and a capped number of inline comments. Merging stays with you. In the default modes, the diff and code snippets sent to your runner are removed once the review is done (a daily sweep catches the rest), and job records are deleted after 90 days. In deep-local mode the AI step and code index run on your machine; our servers still read the PR diff from GitHub for the rule checks and don't store it.

6. 6

   Learning, with a human in charge

   Reviewer feedback, merges over a lane, reverts and follow-up fixes are recorded. Findings your reviewers dismiss stop coming back. A weekly proposal PR suggests stricter protected paths and clearer rubric notes for a human to merge. Learning never lowers a lane.

   How learning works

The same steps, in words

1. A pull request is opened or updated. Nothing to do on your side.
2. GitHub sends a webhook from the MergeJudge GitHub App to the Cloudflare Worker. The Worker checks the signature (X-Hub-Signature-256), puts a small job on a queue and answers right away.
3. The rules floor runs first. Deterministic rules (protected paths, secrets, safety guards, file types) set the minimum lane. No model is involved.
4. AI review. The AI reads the change for real bugs, security holes and broken logic, following your policy files (for example REVIEW_POLICY.md), and in context or deep mode the surrounding code. It also scores your rubric, and learned false alarms keep repeat noise down. It runs either on the hosted service with your own provider key, or on your own runner on your machine (it connects to the hosted service with an enrollment code): it polls the Worker and uses your own Claude Code, Codex or GLM login. The AI can only raise the lane, never lower it (one opt-in exception: lanes.code_auto). Where your code goes is described in Privacy and your code.
5. Result. One lane check (mergejudge/lane), a sticky summary comment and inline comments on diff lines (at most 20 findings per review). MergeJudge never merges.
6. Learning. Reviewer feedback, merges over a lane, reverts and follow-up fixes are recorded. A weekly proposal PR suggests config changes for a human to merge. See how learning works.

How learning works

MergeJudge learns from what your team does with its reviews. It uses that to cut repeat noise and to suggest stricter config. It never changes a lane, and it never changes your config on its own.

Reactions and repliesYour notesMerges, reverts, fixes

fewer repeat comments → weekly proposal PR → a human decides

1. Your reactions to findings are recorded. A 👎, a reply such as "false positive" or "by design", or resolving a thread without changing the code marks a finding as a false alarm. A 👍, or resolving it after the code there changed, marks it as useful. Only people with write access to the repo count. Merges over a lane, and reverts or quick fixes after an auto-lane merge, are recorded too. Bots and marked coding agents don't count.

2. Dismissed findings stop coming back. When a finding that was dismissed before shows up again, it moves out of the inline comments into the summary after two dismissals, and is hidden after three. Confirmations count against this, so a finding people found useful keeps showing. Findings on files laned block are never hidden.

3. Your notes guide the next reviews. Tick a feedback box on the summary comment (Correct, Too lenient, Too strict, Wrong reason, Missed something) or comment /mergejudge learn <note>. The latest notes go to the AI with the next reviews, as background it may not use to relax any rule. They are stored on the mergejudge-learning branch, never in your PR.

4. Once a week, a proposal PR. Every Monday MergeJudge looks at what it has recorded. If a pattern is strong enough, it opens one PR that edits only .mergejudge.yml, and only in two ways:

   • a stricter protected path, after auto-lane PRs in one folder were reverted or quickly fixed at least twice;
   • a clearer rubric note on an existing check that people keep marking as too strict or a false alarm (at least 3 PRs and 2 different people): "fail it only with a concrete defect quoted from the changed lines". Security checks never get one.

   The PR is labelled do-not-merge and needs-human, and MergeJudge lanes it block. It never approves or merges it, so nothing changes until a human merges it. No pattern, no PR.

5. Learning never lowers a lane. Lanes come from the rules, the AI review and the rubric. Learning only changes how many repeat comments you see, and config only changes when a human merges a proposal.

Learning starts when MergeJudge is installed. Older pull request history is not read. Details: learning loop.

The review rubric

Every review is the same few checks, in the same order. Some are deterministic rules, some are model answers. The rules set a floor; everything else can only push the lane up.

Check	What it looks at	Effect on the lane
Protected paths	Files matching lanes.protected (default: .github/**, deploy/**, **/auth/**, **/migrations/**, wrangler.toml, _headers, **/.env*)	at least review
Secrets	Added lines that look like an AWS access key, a private key block, a GitHub token or a hard-coded api_key = "…"	block
Safety guards	A protected-path change that removes --match-head-commit, persist-credentials: false, set -e or --frozen-lockfile / npm ci	block
File type	Source files (by extension), dependency manifests and lockfiles, files with no diff or over 5000 changed lines, and anything outside the auto paths	review
Auto paths	Docs, markdown, CSS and public/** (default), up to 200 changed lines	stays auto
Labels	do-not-merge and needs-review (names are configurable)	block / at least review
AI review	Real bugs, security and logic issues in the diff (plus surrounding code, depending on the review mode), guided by your policy files	can raise a file to review or block
Rubric	Yes/no criteria, each with an id and a severity. A high-severity failure only counts if the model quotes a line that really is in the diff	review, or block for criteria that say so
Opt-in: PR description	pr_template: the description must answer the sections you list	review (or block), only when switched on
Opt-in: trusted authors	lanes.trusted_authors: only listed GitHub logins get the light auto paths	takes auto away, only when switched on

How the final lane is chosen

Rules floorAI reviewRubric evidenceLabels

strictest lane wins → auto, review or block

1. Each file gets its own lane from the rules, then the AI and the rubric may raise it.
2. The PR lane is the strictest of all its files and labels. One block file makes the whole PR block.
3. Nothing lowers a lane (apart from the opt-in code auto exception below). If the model call fails or its answer can't be read after a retry, MergeJudge uses the rules alone and says so in the comment.
4. Your config is read from your main branch, never from the PR. Changing .mergejudge.yml in a PR has no effect on how that PR is judged.
5. Policy files are guidance. REVIEW_POLICY.md, APPROVAL_POLICY.md, AGENTS.md and CLAUDE.md from your main branch are given to the model as instructions. See what a good REVIEW_POLICY.md looks like.
6. Approvals follow the lane. Optionally MergeJudge approves auto-lane PRs as its own app and you can require human approvers per lane; if a later push raises the lane, its approval is withdrawn (approvals).

What a rubric is

A rubric is a list of yes/no questions a reviewer should ask. Each question is a criterion with an id, a severity (low, medium or high) and a scoring style: pass/fail, or a 1–5 score where 1–2 counts as a fail. Criteria are grouped into packs:

Pack	On by default	Criteria (severity)
security	yes	sec.no-secrets (high, raises to block), sec.no-exfiltration (high, block), sec.no-unsafe-exec (high), sec.input-handling (medium)
correctness	yes	cor.logic (high), cor.errors (medium), cor.tests (low, 1–5)
maintainability	no (opt-in)	mnt.readable (low, 1–5), mnt.scoped (low)

The model answers each criterion with pass, fail or n/a, with a short reason. For a fail it also gives the file, the line and a quote. MergeJudge checks that the quote really appears in that file's diff. A fail without a matching quote is shown as unverified and does not change the lane.

Only a failed high-severity criterion raises the lane, to its raise_to (review by default). Low and medium failures are reported in the comment but never change the lane. You can switch packs on and off and add your own criteria in .mergejudge.yml: see rubric packs.

The model's findings are line comments with a line number and a short message. They do not carry a severity of their own; severity belongs to rubric criteria.

What can never be auto

These are enforced by code, not by the model:

• Protected paths (the list above) are at least review, and _headers files (which set security headers such as CSP and caching) always need at least review, even if you remove them from your protected list.
• Possible secrets in added lines are block. Obvious fakes (documentation keys, EXAMPLE, PLACEHOLDER, DUMMY, FAKE, XXXX, repeated characters) are noted but not blocked. This is a coarse screen, not a secret scanner.
• Removing a safety guard from a protected path is block.
• Source files (.js, .ts, .py, .go, .sql, .sh and many more) are review, unless you switch on lanes.code_auto.
• Dependency manifests and lockfiles (package.json, package-lock.json, go.mod and similar) are review.
• Files with no diff (binary or very large) and changes over 5000 lines are review.
• Anything outside your auto paths, and auto-path files over max_auto_lines (200), are review.
• Learning proposal PRs (head mergejudge/proposal-*) and the mergejudge-learning branch are always block.
• Hold labels force block (do-not-merge) or at least review (needs-review).

Low-risk code auto (opt-in)

lanes.code_auto is off by default: code files always need review. If you turn it on, a code file can only become auto when all of this holds:

• the rules found nothing else wrong with it (not protected, not a dependency file) and it was added or modified (not deleted or renamed);
• it is not a CI/workflow, .env, _headers, shell, SQL or wasm file (those are never eligible);
• it matches lanes.code_auto.paths (if you set any), has at most max_lines changed lines (default 30), and the candidate files together have at most max_total_lines (default 100);
• the AI says auto for that file with a reason, and reports no findings on it;
• the rubric was scored, and no criterion in the whole PR failed or came back unverified;
• the author is trusted, if you use lanes.trusted_authors.

Any doubt means review. This is the one place an AI answer can bring a file down to auto.

PR description and trusted authors (opt-in)

Both are off by default and both are raise-only. Details: PR template and trusted authors.

• pr_template checks the PR description for the sections you list (default Testing, Risk, How to revert, with at least 15 characters of real text each; comments, placeholders and checkbox-only lines do not count). By default it only checks PRs made by coding agents (marked with the mergejudge:agent label or a MergeJudge-Actor: agent line); apply_to: all checks every PR. A missing section raises the PR to review (or block, if you set missing_lane: block).
• lanes.trusted_authors limits the light auto paths to GitHub logins you list. The author comes from GitHub, so a PR cannot fake it. With the default applies_to: code_auto it only affects lanes.code_auto candidates: an untrusted author's code file stays at review. With applies_to: all_auto, docs/style auto-path files go to review too. Bots are untrusted unless listed. Agent-marked PRs are judged by the author's login (agents: as_author, default) or never trusted (agents: untrusted). It can only take auto away.

Privacy and your code

Your code is always sent to your own AI provider under your own account. What MergeJudge's servers see depends on the mode:

• Deep-local (review.mode: deep with runner.code_source: local): the AI review runs on your machine with your own login and reads your local clone. The AI step and code index run on your machine. Our servers still read the pull request diff from GitHub to run the rule checks, and don't store it. They keep the review result (lane, reasons, summary, findings). It needs a runner that supports it; without one, the review falls back to a hosted deep path.
• Default modes (diff, context): the diff, your policy files and related code snippets pass through the hosted job queue. They are removed from the job when the review finishes, and a daily sweep removes anything missed. Finished job rows are deleted after 90 days, and a repo's stored code index after 30 days without activity.

More in Run reviews on your own machine.

What you get

• A floor the AI can't lower. The Worker re-applies the rules floor, labels and any opt-in PR checks to every runner result before posting, so an old or modified runner can't lower a lane.
• Config from your main branch. One .mergejudge.yml per repo (reference). A PR can't loosen its own rules.
• Evidence for rubric failures. A high-severity failure only counts with a quoted line that is in the diff.
• Editing the PR description re-checks the PR. The deterministic checks (labels, pr_template) re-run on the new text, and the stored model result for the same commit is reused (hosted AI and runner), so there is no new model call.
• Learning from outcomes, with a human in charge. Merges over a lane, reverts, follow-up fixes and label overrides are recorded. A scheduled run (weekly on the hosted service) may open one proposal PR that adds protected paths or rubric notes to .mergejudge.yml. It is always block for MergeJudge, and a human merges it. It can't remove protected paths or change packs, severities or raise_to (learning loop).
• Agents are told apart from humans. A coding agent that merges through a person's login can mark itself with the mergejudge:agent label or a MergeJudge-Actor: agent line in the PR body or merge message (gh pr merge 123 --merge --body "MergeJudge-Actor: agent"). Such actions are not counted as human decisions.
• An audit trail. The hosted panel keeps an audit log of changes: sign-ins, repo links, provider key changes (name only, never the value), runner enrollment and revocation, runner jobs and the results posted, and manual learning runs. It stores no code.
• One comment, capped inline comments, one check. A sticky summary, at most 20 inline findings, and a mergejudge/lane check. MergeJudge never merges (approvals).

How rules and rubrics change

Rules never change in the PR being judged. They change like any other code: through a PR a human approves.

1. Feedback. Tick a box on the comment (Correct, Too lenient, Too strict…) or comment /mergejudge learn …. Only people with write access count. On the hosted service, merges over a lane, reverts and follow-up fixes count too.
2. Lessons. Each signal becomes a line in lessons/lessons.jsonl on the mergejudge-learning branch.
3. Proposal PR. When enough evidence builds up, a scheduled run opens a mergejudge/proposal-learning PR with small config changes.
4. A human decides. You review it like any PR. MergeJudge never merges it and always lanes it as block.

Details: learning loop and security model.