Methodology: our small offline test
This page describes the one test behind the numbers on the home page ("Diff vs context vs deep"). It is our own smoke test, not a benchmark.
What we ran
- Cases: 10 pull requests: 9 with one planted bug each, and 1 clean control (a copy edit that should stay
auto). - How bugs were seeded: we wrote them ourselves. Most are cross-file: a function signature, a unit, a config key or an event name changes in the diff, and the breakage is in a file the diff does not touch.
- Modes compared:
diff,contextanddeep(see review modes), one run per mode. - Model:
claude-sonnet-5-5. We repeated the run withglm-5.3. - When: October 2026. The exact MergeJudge build was not recorded with the results.
How it was scored
Scoring is strict. A bug counts as caught only when a finding names the actual breakage, meaning the affected downstream file or symbol. A generic hedge such as "check the callers" on the changed line does not count. The clean control must stay in the auto lane.
Results
| Mode | Planted bugs caught | Clean control | Avg tokens in / out | Avg latency |
|---|---|---|---|---|
diff | 2 / 9 | stayed auto | ~1.1k / 0.65k | 7.0 s |
context | 4 / 9 | stayed auto | ~3.2k / 0.72k | 8.5 s |
deep | 9 / 9 | stayed auto | ~11.6k / 1.6k | 13.1 s |
deep made 2.4 tool calls per review on average. Averages are over all 10 runs per mode. With glm-5.3, deep also caught 9 / 9 but was slower, and some diff and context runs failed to return valid JSON.
Limitations
- Ten cases with bugs we planted ourselves. The same people wrote the bugs and the scoring.
- One run per mode and one model per run; no variance run.
- Real pull requests are messier. Your numbers will differ with your repo, model and config.
- This shows that deep mode finds cross-file breakage that diff-only review can miss. It is not an accuracy number.