Eru: An AI Code Review Quorum
We split AI code review into a deterministic gate, fresh single-purpose reviewers and a computed tally, because one long-running reviewer skims its checklist and never asks the big questions.
Founder / CEO
Bottom Line
- A single long-running AI reviewer optimizes for tokens, skims its checklist, loses context on what deleted code used to do, and rarely asks whether the change should exist.
- Eru splits review into three tiers: a deterministic gate, a quorum of fresh context-free reviewers (one per dimension), and a tally computed in tested PHP.
- A failed hard check blocks the review regardless of any agent opinion, and the gate also builds every reviewer prompt, so the controller cannot be talked out of the work.
- The verdict leads with premise, architecture and parity problems, not style; surface mechanics are left entirely to deterministic tools.
- Eru can run unattended under the PR supervisor, but only stages an unsubmitted draft; it does not post to GitHub on its own and does not replace human approvals; builder self-review is a separate, lighter kind.
Our first AI code review was one long prompt and a checklist that produced tidy line-level comments. It felt thorough. Then a pull request on the trade-execution path passed about a dozen of those reviews and a human still sent it back, for reasons none of the reviews had come near. That is the failure this post is about: AI code review that is busy but never asks the big questions.
The rebuild is a skill we call Eru, which reviews to our lead developer's standards, architecture first, as part of the system in how to build an AI software factory. The shape: a deterministic gate that cannot be skipped, a quorum of fresh single-purpose reviewers, and a tally computed in code, with the verdict written last by the one agent that has the full picture.
Why one reviewer is not enough
A single long-running reviewer has three compounding problems.
- It optimizes for tokens. Under pressure to finish, it quietly drops whole dimensions, and nobody notices the security pass that never happened.
- It loses context. Absence is invisible. A reviewer reading new code does not remember what the deleted code did, so a dropped redaction filter or lost empty state never shows up as a line to comment on.
- It reviews at the wrong altitude. Line-level reading finds line-level problems. The questions a lead developer asks first (should this exist, does it fit what we have, did it keep every old behavior) are the ones a skimmer skips.
So rigor must be enforced by code and independent agents, not by an instruction to try harder. Determinism cannot be skipped, and a fresh agent cannot lose context it never accumulated.
The altitude map
We wrote down what a lead developer actually reviews, from "should this exist" down to "is this name right", and gave every concern exactly one owner.
| Altitude | The question | Owner |
|---|---|---|
| Premise | Should this change exist, in this form? Root cause or cosmetic? | Quorum: premise |
| Architecture | Does it reuse what exists, in the right module and layer? | Quorum: architecture |
| Behavior | Does it work, and does it preserve what the replaced code did? | Quorum: correctness, parity |
| Implementation quality | Least code? Honest names? Tests that fail on regression? | Quorum: simplicity, tests |
| Production | Safe and operable: security, queries, memory, rollout? | Quorum: security, performance-operability |
| Surface mechanics | Formatting, style rules, static types | Gate only, zero agents |
Human-style judgment goes at the top; the bottom is fully automated, since a tool formats better than an agent.
Tier 1: the deterministic gate
The gate is an ordinary command-line program, checked by Psalm and a formatter and unit-tested like the rest of the codebase. It runs before any agent and does two jobs.
It produces facts
The gate parses the diff into a facts file every reviewer works from. It lists what the change deleted: files, templates, removed public methods and removed tests. It also pairs new classes with similar existing ones, so a reviewer can ask whether the author forked an abstraction we already had. That solves the absence problem: reviewers get the list of what vanished instead of having to remember.
It runs the hard checks
The gate runs the code formatter, static analysis and complexity checks inside the Docker container, plus our migration safety checker. If any hard check fails, the review is blocked whatever any agent later says. A skipped toolchain (the container was down) is explicitly not a pass: those checks did not run, so the review is incomplete.
The gate also builds the reviewer plan. Every reviewer prompt is assembled by a tested class, not improvised at review time, and so is the decision about which dimensions run. Parity, for example, is included only when something was removed: a deleted file, template or API, a removed function, or a kept file that dropped lines in place. The plan makes that call from the facts, not by judgment.
Tier 2: the quorum
The plan lists one job per dimension: premise, architecture, correctness, parity, simplicity, tests, security and performance-operability. Each job runs in a fresh agent with no memory of the conversation and no view of the other reviewers, given only the diff, the gate facts and one rubric.
That narrowness is the point: a reviewer with one rubric cannot drift into another dimension or absorb an earlier reviewer's framing. The rubrics are short and imperative. The parity rubric treats the gate's removal lists as a worklist: read each deleted file at its old revision and map every old behavior and test to the replacement, watching for things that vanish silently: redaction wrappers, empty and loading states, status styling, ordering and accessibility attributes.
Width: scan or quorum
Width 1 runs a single reviewer per dimension, a quick scan for a trivial diff. Width 3 runs three independent reviewers per dimension, and a dimension only blocks when at least two of the three independently call the same finding blocking. That agreement count separates findings several fresh agents reach on their own from the noise one agent produces. Width 3 is the default for the review that gates merge. The deterministic review routing sets a floor on width from the risk of the change, and an explicit request can raise it but never lower it.
Two tiers of model
The plan labels each job as frontier or balanced. Premise, architecture, parity, correctness and security get the frontier tier. Simplicity, tests and performance-operability get the balanced tier. Each host maps the label to a model it supports. On the same PR, a cheaper model took roughly 2.2x the turns, and moving two dimensions down a tier saved 9%, not the multiple the rate difference implied. Turns are what a review costs; measurements are in what an AI code review costs.
Tier 3: tally, then verdict
Reviewers' structured findings go to a tally step. Agreement and verdict votes are computed in tested PHP, not estimated by the orchestrating agent. Per dimension, it reports how many agents responded, how many voted to request changes, and whether a majority independently called the same finding blocking.
The tally had a subtle failure mode that we hit and fixed. Our first version matched findings by their prose, so three agents describing one blocker in three sentences produced three single-agent findings and the quorum silently never fired. The fix merges findings that share a concern tag and name the same file, line range stripped, since reviewers cite one defect as a range, a different range or a bare path. That errs toward merging, so every agent's wording is kept and the orchestrator can split a bad merge. The opposite error, a real blocker that never reaches a quorum, is the dangerous one.
The orchestrator, which holds the full conversation, the gate facts, the tally and every raw finding, then writes the verdict. It leads from the top of the altitude map: premise, then architecture, then behavior, then implementation, then production. A premise, architecture or parity violation is blocking even at lower agreement. Never lead with a naming nit while a duplicated abstraction or lost parity goes unnamed.
Why the controller must not decide
The orchestrator is the one agent that can be argued with: it has a long context, a friendly author and a plausible reason to cut a corner. So it does not choose dimensions, write reviewer prompts, count agreement or waive a failed hard check; those live in unit-tested code and agents that start from nothing. Its job is the part that needs full context: reading reports, splitting bad merges, resolving ambiguous candidates, honoring a won't-fix ledger so settled calls are not re-litigated, and writing the verdict.
Like the rest of our guardrails, a skippable step gets skipped under pressure, so make it part of the machinery.
Where the rules came from
We distilled the rules from our lead developer's review of a rejected branch and his fix commits, not a generic list. Fresh agents then re-reviewed that branch repeatedly until they caught what he caught, and any rubric that missed a known problem was rewritten.
The first real run on a live pull request found a merge-blocking bug, which is encouraging, not proof. Review quality is sampled and even a clean quorum is not absolute proof of completeness, so for a high-stakes change we repeat it until two consecutive passes surface nothing new.
Running the review without a person
Eru is also the default review worker of our pull request supervisor, so no one has to type the command. When the supervisor decides a pull request needs review, the executor claims it and hands the claim's fixed brief, verbatim, to a background worker, because added words such as "triage by risk" made workers skip evidence the review skill requires. The worker runs Eru at the router's recommended width in its own review worktree, torn down afterward.
The output is exactly one unsubmitted draft review at the current head. It carries inline feedback for the author or, when the change is clean, a body stating approval intent with the evidence. A person inspects and submits it. The review skill, the model and the cap on live reviews (two by default) are configurable, because each end-to-end review may bring up its own Docker stack.
A pending draft is visible only to the person it belongs to until they submit it, so Eru still does not post on its own. The supervisor never approves, requests changes, merges or resolves a thread for them. The loop around it is in GitHub as the bus for AI agents.
Findings can also carry a recorded repro, a command with its buggy and fixed outcomes. When the author pushes a fix, the repro is rerun first, then a fresh review runs at the new head. Nothing is replied to or resolved along the way, so a closed thread without a passing repro is not proof of a fix.
What Eru does not do
- It does not post. The verdict is delivered in the conversation, or staged as an unsubmitted draft under the supervisor. Eru never comments, submits a review or writes to a tracker unless the person asked in that session; permission is per request, not standing.
- It does not replace humans. Team approvals required by code ownership are still required, and an agent verdict does not satisfy them.
- It is not self-review. A builder's pre-PR check answers "am I finished?"; Eru answers "should this merge from a lead developer's perspective?" A clean self-review does not imply a clean Eru run, or the reverse.
- It is not free. A width-3 run is eight dimensions times three reviewers, each reading the diff on its own. Handing reviewers the whole diff in a file is worse than letting each pull its own slice, a story for the cost post.
We also tried the always-on version of review automation, and reverted it. That story is in the AI review bot we built and reverted. Eru is the opposite: started on purpose, deep when it runs, quiet otherwise.
Lessons
- Put rigor in code, not in instructions. Tested code, not an agent, should decide which checks and dimensions run and how agreement is counted.
- Use fresh, narrow reviewers. One rubric and no shared context stop dimensions being dropped and framing leaking.
- Hand reviewers the removals. A deterministic list of deletions turns parity from a memory test into a checklist.
- Count agreement on the right key. Merge findings by concern and file, not by prose, and keep every report so a human can undo a bad merge.
- Lead the verdict with design. Premise, architecture and parity problems outrank style, and style belongs to the gate.
- Keep humans accountable. Eru is an input to the people who own the outcome, not a substitute.
If you build something like this, start with the gate: it is the cheapest tier, needs no agent, and would have caught the deletions our first reviews missed. Add the quorum once you have a facts file worth reviewing against, and measure a real pull request before setting the width.
