Methodology: our small offline test
This page describes the one test behind the numbers on the MergeJudge home page ("Diff vs context vs deep"). It is our own smoke test of 10 pull requests. /results redirects here.
The 50-PR Martian Code-Review-Bench run is a separate measurement: MergeJudge Runner --deep scored 75.9% precision, 38.0% recall and 50.6% F1, and the fast default scored 75.0% precision, on the Core profile. Scripts and raw data are in the mergejudge-runner benchmark directory. The upstream submission is withmartian/code-review-benchmark#70. How to read the scores, and the 7-dimension comparison with other open-source tools: open-source code review tools.
What we ran
- Cases: 10 pull requests: 9 with one planted bug each, and 1 clean control (a copy edit that should stay
auto). - How bugs were seeded: we wrote them ourselves. Most are cross-file: a function signature, a unit, a config key or an event name changes in the diff, and the breakage is in a file the diff does not touch.
- Modes compared:
diff,contextanddeep(see review modes), one run per mode. - Model:
claude-sonnet-5-5. We repeated the run withglm-5.3. - When: October 2026. The exact MergeJudge build was not recorded with the results.
How it was scored
Scoring is strict. A bug counts as caught only when a finding names the actual breakage, meaning the affected downstream file or symbol. A generic hedge such as "check the callers" on the changed line does not count. The clean control must stay in the auto lane.
Results
| Mode | Planted bugs caught | Clean control | Avg tokens in / out | Avg latency |
|---|---|---|---|---|
diff | 2 / 9 | stayed auto | ~1.1k / 0.65k | 7.0 s |
context | 4 / 9 | stayed auto | ~3.2k / 0.72k | 8.5 s |
deep | 9 / 9 | stayed auto | ~11.6k / 1.6k | 13.1 s |
deep made 2.4 tool calls per review on average. Averages are over all 10 runs per mode. With glm-5.3, deep also caught 9 / 9 but was slower, and some diff and context runs failed to return valid JSON.
Deep here is the hosted tool-loop version. Deep on a runner or in the Action is a single pass without tools and was not part of this test.
Limitations
- Ten cases with bugs we planted ourselves. The same people wrote the bugs and the scoring.
- One run per mode and one model per run; no variance run.
- Real pull requests are messier. Your numbers will differ with your repo, model and config.
- This shows that deep mode finds cross-file breakage that diff-only review can miss. It is not an accuracy number.