What an AI Code Review Costs
A developer burned a token budget on one multi-agent review, so we measured where the tokens went. Here is what an AI code review costs, what cut it, and what did not.
Founder / CEO
Bottom Line
- An AI code review bills prefix times turns times reviewers, so carried context and turn count matter more than the price per token.
- Giving reviewers a purpose-built agent type with a small tool allowlist cut their context from about 81k to about 23k tokens, halved total tokens, and cut cost 45% while holding the blocking findings and the quorum that carried them.
- Asking reviewers for fewer, fuller steps gave 39% fewer turns and 31% less cost on a same-day two-dimension control; tool calls per turn stayed at 1.00, but the turns got denser.
- Three plausible optimizations failed on real runs: a cheaper tier saved 9%, a newer model cost 2.7x, and handing over the whole diff cost 12% more.
- The same configuration cost 3.5x three days later, so we quote ratios from same-day runs, not dollars.
One developer ran our multi-agent review once and used up their token budget. That is the honest answer to what an AI code review costs: more than you expect, and not for the reasons you would guess. We took the complaint as a measurement problem and went looking for where the tokens actually went.
This post covers what we found, the three fixes that worked, and three optimizations that looked obviously right and lost on real data. It is part of a series on how we build software with agents, which starts with the hub post on building an AI software factory. The review itself is described in how our gate plus quorum review works. Here we only care about the bill.
Where the bill comes from
Our review runs a deterministic gate first, then fans out fresh, context-free reviewers across eight dimensions. The default is three reviewers per dimension, and a dimension blocks only when at least two of the three independently agree on the same finding. That is 24 reviewers per run. Every one of them reads the diff on its own, because independent judgment is the point of a quorum.
A reviewer is an agent loop. On every turn it re-reads its entire system prefix: instruction files, tool schemas, whatever it was handed. So one reviewer costs roughly prefix times turns, and the whole review multiplies that by the number of reviewers. On a 28-file, 62KB pull request, the run measured 24 reviewers, 364 turns, and roughly 17M tokens after our fixes, against 34M before them.
That formula explains the rest of this post. A change helps only if it shrinks the prefix, removes turns, or removes reviewers. Anything else is rearranging the same bill.
Fix one: the context nobody used
Our reviewers inherited the default agent configuration of the host. That meant the repository instruction files, a catalog of every skill, and a schema for every connected MCP server. Together that was about 81k tokens of context. A reviewer reading a diff for a missing null check used none of it, and paid to re-read it on every turn.
We defined a purpose-built reviewer agent type with a small tool allowlist. Naming a few tools drops the schema for every connected MCP server, and the skill catalog goes with it. The prefix fell from about 81k to about 23k tokens. Total tokens halved and cost fell 45%. We checked that the blocking findings and the quorum that carried them were unchanged, so the cut did not come out of review quality.
Two details are worth copying. First, we kept the repository instruction files in the reviewer context. The recorded runs showed they were still injected, and reviewers need the project invariants that no rubric restates. We cut what was unused, not what was load-bearing. Second, the host snapshots its agent registry at session start, so a session that has just added the agent file cannot resolve it until it restarts. The skill now says so.
Fix two: reviewers that never batched
When we counted tool calls per turn across 1,315 turns, the number was exactly 1.00. Reviewers never issued two reads in one step. Each file read, each search, each scoped diff was its own turn, and each turn re-read the prefix.
The fix was in the prompt, not the code. We asked reviewers for fewer, fuller steps: read what you need together, then reason. On a same-day control over two dimensions, that gave 39% fewer turns and 31% less cost. Tool calls per turn stayed at 1.00; the turns themselves got denser. The saving is real either way, so the wording asks for the outcome rather than a batching mechanism the hosts do not deliver.
The lesson generalizes. If you have never histogrammed tool calls per turn for your agents, do it. A value of exactly 1.00 is a signal to measure, not proof of a mechanism. Change the prompt, re-measure, and keep the claim that still matches the data.
Fix three: the tally that never fired
This one is not about tokens, but it is about waste. We spend the money on 24 reviewers to get one thing: an agreement count that separates real findings from single-agent noise. Our tally was discarding it.
Agreement was keyed on the first 50 characters of the issue prose. Reviewers rarely open a sentence the same way, so three of them describing one defect produced three keys holding one reviewer each. The quorum never fired on a finding all three had raised. We were paying for a signal and then not counting it.
The key is now the concern tag plus the file, with the line range stripped. Reviewers cite one defect as a range, a different range, or no range at all, and keying on those splits one finding into three. The tally also counts distinct reviewers per key, so one reviewer restating a finding does not vote twice. The trade-off is deliberate: concern plus file can merge two different problems in one file, but a merged finding still carries every reviewer's full text for the controller to read. Under-merging silently destroys the count. Over-merging does not.
We verified the fix on a live 24-reviewer run, not only on recorded fixtures. Quorum now fires on the premise and tests dimensions, where before it fired on nothing.
Three optimizations we rejected
Each of these was plausible, and each had a defender. We ran them on the same pull request and wrote down the result.
A cheaper model tier
Correctness and security are our most expensive dimensions, so we moved them to the balanced tier. The rate card implied a 2.5x saving. The measured saving was 9%. On identical prompts, the cheaper model took roughly 2.2x the turns, and every extra turn re-read the whole prefix, so most of the per-token discount was spent back. It also traded one real finding for a different one rather than keeping the original. The tiering for the lower-judgment dimensions stays. The move for the expensive ones did not.
A newer frontier model
Going the other way, a newer and more thorough model tripled turns per reviewer on the same prompts and cost 2.7x. It did buy something: a blocking premise quorum that the cheaper run had missed. So this was not a bad trade in the abstract. It was an unpredictable one, and nothing on a price list foretold it. We keep it as a measured option for high-stakes changes, not a default.
Handing reviewers the whole diff
This was the most tempting. On one pull request, scoped diff calls made up 44% of every tool call. Each reviewer running its own scoped diff looks wasteful. So we wrote the full diff to a file and pointed reviewers at it. That removed 96% of those calls and still cost 12% more.
The reason is the formula again. A scoped diff pulls only the slice a reviewer needs. The whole diff enters the reviewer context and is then re-read on every remaining turn, and turns did not fall. A smaller call count is not a smaller bill. Anything that grows the carried context has to remove turns to break even, and handing over a document removes none.
| Change | Expected | Measured |
|---|---|---|
| Lean reviewer agent type | Less context | 45% less cost; blocking findings and quorum held |
| Batching prompt | Fewer turns | 39% fewer turns, 31% less cost |
| Balanced tier for correctness and security | 2.5x cheaper | 9% saved |
| Newer frontier model | Better findings | 2.7x cost, better findings |
| Whole diff in a file | Fewer calls | 96% fewer diff calls, 12% more cost |
Why we quote ratios, not dollars
Early on we wrote real figures into the skill. Then we re-ran one unchanged configuration three days later: same prompts, same pinned model identifiers, same pull request. It produced 2.2x the turns and 3.5x the cost. The per-reviewer prefix held at about 23k tokens across every run. Everything downstream of it moved.
We do not know all the reasons, and we have not tried to explain the variance away. Sampling is stochastic, and a reviewer that wanders takes more turns. The practical conclusion is that an absolute number in a document is stale the day after you write it. The skill now quotes ratios, tells the reader to date anything they do quote, and says to compare runs from the same day or not at all. It also says to re-measure before relying on any figure there.
What this does not fix
None of this makes reviews faster. Wall clock is set by waves, not total work. Hosts cap concurrent reviewers, so 24 reviewers on a 12-core machine run as three waves of 10, and the slowest reviewer in each wave sets the pace. Four dimensions carry the tail: simplicity, tests, performance, and correctness each take around 21-29 turns per reviewer, against 4 for parity and security. Raising the cap to 16 cuts the makespan by about a third. Past 16, a single reviewer's own turn count is the floor and more cores buy nothing.
Cost also still scales with diff size, with width, and with every extra turn. The same settings on a 3,000-line pull request cost several times what the 28-file one did. That is why the skill tells people to keep width 3 for the review that gates merge, and to use a single reviewer per dimension or a subset of dimensions while iterating. Repeating the quorum until it goes quiet is advice for a high-stakes change, not a habit.
When reviews run unattended, a supervisor bounds the work: at most 2 live reviews by default (configurable from 1 to 10), 2 attempts per head, and a watchdog notification for any review running over 2 hours. Cost and concurrency need an explicit ceiling, because the bill is prefix times turns times reviewers and nobody is watching it grow.
The cheapest agent work is the work you never start. We apply the same idea to the agents that watch pull requests, covered in our post on zero-token wakes.
How we verified the change
A cost change to a review pipeline is dangerous in a specific way: it can quietly make the review worse while the bill improves. So the checks were on findings as well as spend. The agent unit tests passed, and static analysis and the coding-standard checks were clean on the changed files. Mutation testing scoped to the changed lines killed 13 of 13 mutants. The tally fix was exercised on the live 24-reviewer run described above. The one failing set we saw, three gate tests for migration safety, was a missing PHP extension on hosts outside the container, and not a result of this change.
Lessons
- Cost is prefix times turns times reviewers. Reason about every proposed optimization against that formula before you build it.
- Audit what an agent carries. Default agent configurations load instruction files, skill catalogs, and every connected tool schema. Name the tools a job needs and drop the rest.
- Histogram your tool calls per turn. A value of exactly 1.00 is a signal to measure, not proof of a mechanism.
- Check that the expensive signal is counted. Quorum was paying for agreement and keying it on prose. Verify on a live run, not only on fixtures.
- Measure before you trade models. A rate card did not predict the cheaper tier or the newer model. Turns did.
- Quote ratios and date them. The same configuration cost 3.5x three days later. Compare only same-day runs.
If someone on your team has already blown a budget on an agent run, resist the urge to start with the model. Count turns first, then look at what each turn re-reads. In our case the model was never the problem, and the fixes that worked were the ones that left it alone.
