MergeJudge learning loop

false alarmsuseful findsnotesprotected paths

Animated illustration of the learning loop: example feedback signals drift in and are absorbed into a stylised brain, where lessons glow as regions. Example signals, not real data

What gets collected

The sticky comment ends with feedback checkboxes:

• Correct / Too lenient / Too strict / Wrong reason / Missed something
• Per failed or low-scoring rubric criterion, under "Rubric feedback (tick any you disagree with)": one - [ ] <criterion id>: wrong box. Ticking it records a "disagree" lesson; unticked records nothing. Older comments with 👍/👎 pairs are still understood.

Writers (repo write access) can also comment:

/mergejudge learn <free-text note>

Bot comments are ignored. Checkbox ticks arrive as issue_comment edited events.

Where it is stored

Lessons are appended as JSONL on a dedicated branch:

• Branch: mergejudge-learning
• File: lessons/lessons.jsonl

Never written into the reviewed PR. Storing lessons needs Contents: write on the token (App: raise Contents to Read and write, or give the workflow contents: write).

Proposals

A scheduled (or manual) run with mode: propose (or mode: auto on schedule / workflow_dispatch) reads the lessons and, once learning.propose_after (default 10) is reached, opens or updates one PR (head mergejudge/proposal-learning). Two shapes:

• Edits (when evidence crosses a threshold): the PR changes .mergejudge.yml with small text patches, and the PR body lists the evidence (counts, PR numbers, logins). The report lives in the PR body only, so no stray file lands in the repo if the PR is merged.
• Record-only (below every threshold): the PR adds MERGEJUDGE_LEARNING_PROPOSAL.md with the stats, as before.

Either way the PR is labelled do-not-merge and needs-human (created if missing), is always block for MergeJudge itself (head mergejudge/proposal-*, rules only, no LLM), and is never approved, auto-merged or merged by MergeJudge.

What it can suggest

Evidence	Threshold (defaults)	Edit
"too strict" on a criterion: rubric <id>: wrong votes, humans merging a PR over a failed check, dismissed findings that name the criterion	at least min_prs 3 different PRs and min_actors 2 different people, and "right" votes on fewer than half of those PRs	adds rubrics.notes.<id>: a fixed sentence the model sees after the question
"too lenient" on a path: a human reverted an auto PR, or merged a fix soon after one, touching the same directory	at least min_path_events 2	adds <dir>/** to lanes.protected (writes the default list too, if the file had none)

Set under learning.proposal in .mergejudge.yml (edits: false turns edits off). The Action path has lessons only; the Worker also reads outcomes and dismissed findings from D1.

Outcomes alone can open a proposal (Worker). With fewer than propose_after lessons, the weekly job still loads the D1 evidence. If at least one real edit crosses the thresholds and passes the guards below, it opens or updates the same proposal PR with only the edits and the evidence in the body (no report file, labels do-not-merge and needs-human, never approved or merged). If nothing qualifies, it writes nothing to GitHub: no branch, no commit, no PR. The cron queues a linked, non-archived repo when it has lessons, or when learn_outcomes holds outcomes on at least 2 distinct PRs (the smallest default threshold, which is also the config floor). That is only a cheap pre-filter; the job applies the repo's own thresholds. If the learn_outcomes table is missing, only the lessons rule applies.

Run the weekly job now (Worker admin)

The cron's code path can be triggered on demand, in two ways that share one handler:

• POST /admin/v1/learning/run with Authorization: Bearer $ADMIN_RUN_TOKEN. The token is a Worker secret named ADMIN_RUN_TOKEN: generate it with openssl rand -hex 32, set it with npx wrangler secret put ADMIN_RUN_TOKEN. Unset or shorter than 32 characters disables the route (404). A missing or wrong token is 401; other methods are 405.
• POST /app/api/admin/learning/run from the panel session of a platform admin (PLATFORM_ADMINS; CSRF and Origin checks as for every panel POST). Non-admins get 404, like the other admin endpoints; signed out is 401; GET is 405.

Body (JSON, optional): {"dry_run": true, "repo": "owner/name"}. dry_run defaults to false. A dry run performs every read-only check and reports what would be queued, but sends no job and writes nothing except the audit row. repo limits the run to one repo. Invalid types are 400.

curl -sS -X POST https://app.example.com/admin/v1/learning/run \
  -H "Authorization: Bearer $ADMIN_RUN_TOKEN" -H "content-type: application/json" \
  -d '{"dry_run": true}'

The 200 response is {ok, dryRun, installations, considered, queued: [{repo, installationId}], skipped: [{repo, reason}]}; reasons are archived, not_linked, no_lessons_or_evidence or error: <message>. If listing installations fails the response is 502 {"error":"run_failed"}.

A real run takes the KV lock learning:run:lock (10 minutes); another real run inside that window is 409 recently_run. Dry runs ignore the lock and never set it. The cron sets the lock too but never skips because of it. Every manual run, dry or real, writes an audit_log row with action learning.run_manual (actor is the panel login, or token) and detail {dry_run, repo, queued, skipped, considered}.

What it can never do

guardConfigEdit (src/proposals.ts) re-parses the file before and after with the config parser and refuses the edit unless the only differences are more lanes.protected entries and new rubrics.notes. It cannot: remove a protected path, add one that is **, starts with * or is under .github/, change lanes.auto, lanes.code_auto, max lines, approval, labels, provider, packs, custom criteria, severity or raise_to, or touch a note that exists. Security criteria (sec.*, secret/token/credential ids) and anything that raises to block never get a note; they are only mentioned in the PR body. The write guard only lets a proposal branch carry .mergejudge.yml and the report file, never .github/**. Edits are text insertions, so comments and layout stay; a layout it cannot edit safely (flow lists, flow maps) is reported, not edited. Re-running with the same evidence adds nothing twice: an id that already has a note is skipped.

Rubric feedback boxes

Each failed criterion now has two boxes: <id>: wrong (false alarm) and <id>: right (real problem). If both are ticked, "wrong" wins. This is what makes per-criterion accuracy meaningful.

Workflow wiring

on:
  pull_request:
    types: [opened, synchronize, labeled, unlabeled, reopened]
  issue_comment:
    types: [created, edited]   # checkbox ticks = edited
  schedule:
    - cron: "0 9 * * 1"        # weekly propose
  workflow_dispatch:            # manual propose

Permissions: pull-requests: write, checks: write, and contents: write (for the learning branch + proposal PR). Prefer the MergeJudge GitHub App token.

Per-repo false-alarm learning (hosted Worker)

The sections above are the lesson loop: human feedback on the sticky comment, stored on a git branch, turned into a proposal PR for a human to review. This section is the outcome loop: it watches what happens to MergeJudge's inline findings and uses it automatically so that repeated false alarms go away. It needs D1, so it runs on the Worker only (the Action has no per-repo store; see "Open decisions" in the PR).

Signals

Read from the PR's MergeJudge review threads (GraphQL reviewThreads) at the start of every review (not for label-only re-runs) and when a PR is closed or merged. Only users with write access to the repo count; bots and drive-by accounts are ignored, so an outside contributor cannot teach the bot by resolving their own PR's threads.

What happened to the finding	Signal
A writer replies "false positive", "not a bug", "by design", "won't fix", "noise" …	dismissed
A writer gives the comment a thumbs-down	dismissed
A writer resolves the thread and the code at that spot is unchanged	dismissed
A writer resolved an outdated thread (the code at that spot changed)	fixed
MergeJudge resolved its own finding, the thread is outdated, the head moved, and the same reviewer (engine / mode / model) made both runs	fixed
MergeJudge resolved its own finding in any other case (code unchanged, or the reviewer changed, or no context recorded)	nothing
A writer gives the comment a thumbs-up	fixed (confirmed useful)

Each inline comment yields at most one signal (learn_events, unique per repo and comment id), so re-reading a PR never double counts. GitHub sends no webhook for reactions, which is why signals are read at review and close time instead of event by event.

The hidden marker

Every inline comment starts with <!-- mergejudge:inline key=... state=... -->. It now also carries sha (head when posted) and rv (reviewer id, mode:model). When MergeJudge resolves a finding, the visible text is replaced by "no longer flagged", so the marker of the resolved comment keeps the original text (orig, URL-encoded, 200 chars max), sha/rv from posting, and sha2/rv2 from the resolving run. The fingerprint is not stored: it is computed from orig, so it is always the same as for the live comment. The harvester reads orig, flattens and defangs it (no links, mentions, fences or HTML). Comments resolved before this change have no orig and are skipped, not stored as junk. An unchanged finding keeps its comment untouched, so sha/rv stay those of the first post.

Storage

D1 table learn_events (migration 0011_learning.sql), per repo: the finding text (200 chars), a fingerprint, the signal, the PR, the acting writer. A learned item is the group of events with the same fingerprint. The fingerprint is the sorted set of meaningful words of the message (code spans, numbers and stopwords removed), so "Missing await on tx" and "missing await on db" are one item; lookups use word-overlap (Jaccard >= 0.6). At most 2000 events are kept per repo. Data is removed with the repo/installation by the retention purge.

How it is used

score = dismissed - 2 * fixed.

• Prompt: one shared "Team preferences" block (src/learnedctx.ts), at most 1500 characters / 20 lines, used on the hosted path and sent to runners. It has up to three parts: known false alarms (score >= 2, max 8), the last 8 writer notes from lessons.jsonl on the learning branch (read with one GET per review, never creates the branch, failure = none), and a hint when people merged 3+ PRs left in review/block (from the outcome stats). Every line is flattened, defanged (no tags, links, mentions, fences, delimiter) and capped. The block says it is data, not instructions, and that it may not change a lane or relax any rule, security or secret check.
• Runners: the Worker puts the block's lines in the job payload (learnedLines, additive; runners before the release that reads it ignore it). The runner re-sanitises them. Downgrade/suppress is applied by the Worker when it posts the runner's result, so it works with old runners too.
• Downgrade (score >= 2): a repeat is not posted as an inline comment; it appears in the sticky comment under "Other LLM findings" marked "(dismissed before in this repo)".
• Suppress (score >= 3): a repeat is dropped; the sticky comment notes how many findings were hidden.
• Never: lanes, labels, rubric results and the check run are not changed by learning, and findings in a file the model laned block are never hidden. learning.enabled: false turns the whole thing off for a repo.

Outcomes (what humans actually did)

Migration 0012_learning_outcomes.sql adds two tables (0013 extends them, below). This is observation only: no lane, label or check run changes because of it. Only humans count; [bot] accounts, bot-typed users, dependabot, renovate and github-actions are ignored as human actors. Everything is best effort: if the tables do not exist yet (Worker deployed before the migration), the Worker logs a warning and carries on.

Agents are not humans. Coding agents often act through a human's GitHub login, so the login looks human. They never count as approval, but a revert or a clearly linked fix still tells us something went wrong, whoever made it. So every outcome stores who acted (actor_kind) and the rules below decide what each kind may add. The acting PR is the merged one (the PR itself, the revert PR or the fix PR). Its actor kind is:

1. agent if it is marked as an agent's. Either marker is enough: the PR label mergejudge:agent (in the webhook payload), or the text MergeJudge-Actor: agent (case-insensitive, anywhere in the text) in the PR body or the merge commit message;
2. else bot if the merger is not human ([bot] accounts, bot-typed users, dependabot, renovate, github-actions, ...) or is in learning.exclude_actors (case-insensitive, default []);
3. else human if the merge commit message was read and has no marker;
4. else unknown (no merge SHA in the payload, or the GET failed; a warning is logged).

The merge commit message is not in the closed payload, so the Worker fetches it (GET /git/commits/{sha}) at most once per merge, only when a row would be written and the actor is not already settled by 1 or 2. The label mergejudge:agent is never a hold label, adding or removing it is never an outcome, and label changes on a PR that carries the marker are not recorded. Commands for agents: gh pr merge N --merge --body "MergeJudge-Actor: agent", gh pr edit N --add-label mergejudge:agent.

Event	Acting actor	Recorded?
merged_over_lane (PR merged while its last lane was review or block)	human only	agent, bot, unknown: nothing
reverted (a revert PR names PR N)	any	yes, any lane of N
followup_fix, link explicit (fix PR says "Fixes #N" and similar)	any	yes, any lane of N
followup_fix, link hunks (fix intent and the fix's changed lines overlap N's merged lines)	any	yes, any lane of N
followup_fix, link title_paths (fix intent and a shared file only)	human only	agent, bot, unknown: nothing
label_stricter / label_looser	human only	unchanged

Softening evidence (a merge over a lane, and so rubric-note "too strict" proposals) is therefore human-only; "something went wrong" evidence may come from anyone, and says so in actor_kind.

• learn_reviews: one row per (repo, PR), replaced on every review. Final lane (after the server floor on the runner path), reviewer id, failed rubric criterion ids, normalised rule reasons (digits folded to N), non-auto paths (max 10). Filled in at merge time, for every lane, with merged_at, the changed paths (max 100) and merged_hunks: the line ranges (after the change) of each file, from the @@ -a,b +c,d @@ headers only (max 100 files x 20 ranges, 8000 characters; files without a patch are skipped; line numbers only, never code). Max 500 PRs per repo.
• learn_outcomes: one row per (repo, PR, kind), so a webhook delivered twice counts once. Max 500 per repo. lane is the bot's lane for that PR then; actor is the merger's login; actor_kind is human | agent | bot | unknown (NULL on rows from before migration 0013 means human); source is live (a later step may add backfill); evidence is a small JSON with slugs and numbers only: {"link":"revert|explicit|hunks|title_paths","ref_actor":"...","target_lane":"auto|review|block","days":2.5,"files":1} (files = shared files for hunks / title_paths, else 0). Kinds:

Kind	Signal	Rule
merged_over_lane	lane may be too strict	a human merged a PR whose final lane was review or block
reverted	lane too lenient	a merged revert PR names PR N by number (Reverts o/r#N body, revert-N-... branch, or title starting "Revert" with (#N)), N has a recorded review and was merged within 30 days before. A "Revert" title with no number is ignored
followup_fix	lane too lenient	a merged fix PR is linked to an earlier merged PR N (reviewed by MergeJudge, merged before it) by one of the links below. Never a PR's own fix, never a target merged after the fix
label_stricter	lane too low	a human added a hold label (do-not-merge, hold, blocked: always; needs-*, mergejudge:*: only on an auto lane)
label_looser	lane too high	a human removed a hold label while the lane was review or block

Fix links, strongest first (one merge records at most 5 targets, preferring the order below):

1. explicit: the fix PR's title or body says fix/fixes/fixed #N, hotfix for #N, regression from|in|of|introduced in #N, broken by|in #N or bug from|in #N (or follow-up to|for|of|on #N when the PR has fix intent, or a fix word such as fix/bug/broken/regression follows on the same line, within 120 characters), optionally as owner/repo#N of this repo (other repos are ignored). N must be a PR with a recorded review, merged within 30 days. No shared file is needed. see #N and closes #N do not count, and an issue number records nothing.
2. hunks: fix intent (title has fix / hotfix / bugfix / regression, or branch fix/...), N merged within 7 days, and a changed line range of the fix (before its change) is within 3 lines of N's merged range in the same file. Lockfiles, manifests and docs do not count.
3. title_paths: fix intent, N merged within 7 days and at least one shared non-trivial file. Human only.

Only a PR whose lane MergeJudge recorded can produce an outcome (as the earlier PR N, or as the PR merged over a lane). The signals come from webhooks the Worker already handles (pull_request closed / labeled / unlabeled, and the review run); no new event or permission. The 7-day window is a constant (FOLLOWUP_DAYS), not a config option. To read the fix PR's changed lines the Worker lists its files once, only when a rule or the bookkeeping needs them.

What proposals use (still narrower than what is recorded). Reverts and fixes of review/block PRs are recorded but not used yet; a later step turns them into prompt watch-outs. Today:

• "too lenient" protected-path proposals count reverted / followup_fix rows whose lane was auto (rows without a lane do not count, any actor kind), plus label_stricter rows as one event each for the paths stored on them, with the same min_path_events threshold and the same glob and guard rules;
• "too strict" rubric notes use merged_over_lane only, and ignore any row whose actor_kind is agent, bot or unknown;
• label_looser is still unused;
• the weekly cron queues a repo on outcome evidence only for those same rows (merged over lane, auto-lane reverts and fixes, hold labels).

Migration 0013 (0013_learning_outcomes_v3.sql) adds actor_kind, source, evidence to learn_outcomes and merged_hunks to learn_reviews. A Worker deployed before it is applied still records every outcome, with the 0012 columns only (a warning is logged), and loses just the new fields.

loadOutcomeStats(db, repoId) (packages/worker/src/outcomedb.ts) returns counts per kind, bot lane, failed criterion id, rule reason and path, plus the 10 most recent examples. It is empty when there is nothing yet or the table is missing.

Panel

Learning page (replaces Learned; /app/i/<id>/learning, the old /learned URL still works). Per repo it shows the learnings (false alarms with their status collecting / in summary / hidden from reviews and "confirmed useful" counts, notes and feedback ticks from the learning branch, outcome counts), a plain list of the last 50 signals, the next scheduled run and any open proposal PR, plus a "Run learning now" button. Only signals that really happened are shown; a part that cannot be loaded is named ("Data unavailable for notes") and never replaced by made-up rows.

• Who sees it: installation admins only (same session and scope checks as the rest of the panel; other installations and repos answer 404).
• No usernames: the API replaces every actor with "you", "a maintainer" or "an agent". Lesson summaries, head SHAs, URLs, PR titles and outcome evidence are never sent; PR links are built in the browser.
• GitHub reads are cached for 5 minutes in KV (lv:lessons:<repoId>, lv:config:<repoId>, lv:proposal:<repoId>, revalidated with If-None-Match), so a new note can take up to 5 minutes to show. The repo cards never call GitHub.
• Polling: the page asks /learning/:repo/signals?after=<cursor> every 60 s while the tab is visible (5 min after 10 empty polls); new signals are prepended and marked "New signal" for a minute.
• Delete (false alarms only) asks for confirmation and hides all events of that fingerprint (kept as tombstones so the same comments are not learned again); new dismissals start the item over. Notes and outcomes cannot be deleted yet.
• Brain view: above the lists, a canvas draws one region per lesson (the 12 newest) and lights it when a signal for it arrives. It animates recorded signals only: on opening a repo the last 10 real signals from the past 7 days float in once, each with its date ("Replay of recent signals", then a "Replay again" button); a signal that arrives while the page is open floats in once and the chip says "New signal" for a minute. A quiet repo only breathes and says "Listening". No made-up text, no demo data, no spinner. The "What it's thinking about" strip shows the next run and, only when something is really counted, how close a rubric note or a protected folder is to a proposal ("cor.tests: 2 of 3 PRs, 2 of 2 people"); it uses the same thresholds as the weekly run (proposalProgress) and never lists security checks.
• Calm view / reduced motion: with prefers-reduced-motion or the Calm view button (remembered in this browser) the brain is drawn once and nothing moves or floats; the signal list and an aria-live line ("New signal: false alarm on PR #53", at most one per 10 s) carry the same information. The animation also pauses while the tab is hidden or the canvas is off screen, and on a slow device it first halves the work and then switches to a static view ("Static view (slow device)"). The code is a separate asset, learn.js (≤ 15 KB), loaded only on this page.