Risk-Proportional Review for AI Agents

Once coding agents started opening pull requests daily, "every pull request needs a human reviewer" became a queue: a typo fix in an agent instruction file waited behind a change to order handling. Risk based code review for AI agents is our answer: review proportional to risk, with rules, not the authoring agent, deciding the risk.

This post is one part of the system described in how to build an AI software factory. The limit, up front: we have not removed the human-approval requirement. The labeled pilot covers a narrow slice of low-risk changes and is advisory.

The principle: a floor that only rises

The mechanism is a single rule. Review requirements come from deterministic policy plus escalation signals. The authoring agent may report evidence and escalate risk, but must never downgrade a deterministic requirement or decide on its own that human review can be skipped.

Since an agent can only make review stricter, a confident, wrong, or manipulated one cannot talk its way out of scrutiny; the worst it does is waste a reviewer's time. The highest applicable risk wins.

Three tiers

• Low: agent and docs changes, tests, developer tooling, and mechanical maintenance with no runtime change. Target: self-review, required CI, and independent agent deep review; candidates for no mandatory human review once automation and repository rules support it.
• Medium: ordinary application behavior, UI, CRUD, non-critical APIs. Same stack, though signals or policy may still require a human while we build confidence.
• High: trading execution, broker behavior, auth and security, billing and money movement, account ownership, migrations, deployment and runtime infrastructure, and compliance-sensitive behavior. Same stack, plus human review always.

The deep review is the gate-plus-quorum process from our post on Eru; tiers decide how much runs and whether a person also signs off.

Path-based risk sets the floor

A function in our change summary tooling assigns every changed file to a category by path and sums additions, deletions, and file counts per category. Each category carries a risk level; overall risk is the highest present. It never reads the description, commit messages, or the agent's explanation. A path is a path.

We reused existing change summary and code ownership categories rather than invent a competing map, since two maps of "what is dangerous" drift apart in favor of the easier one to game. One rule matters for agents: the files defining our CI workflows are high risk, because an agent that can edit the pipeline enforcing its own review can weaken it.

Escalation signals only add requirements

Paths are a floor, not a ceiling. Some risk shows only in what a change does, so escalation signals raise requirements when present; absence never lowers anything: protected paths or specialist code owners; authorization, account scoping, credentials, secrets, external input, or security boundaries; migrations or deployment and runtime changes; order lifecycle, broker integration, billing, or compliance behavior; API or persisted-data contract changes; unusually broad call-graph impact or a large cross-domain diff; new abstractions where prior art or system fit needs judgment; replacement or removal where behavior parity is hard to prove; missing or weak regression coverage; deep-review findings exposing material architecture, correctness, security, or operability concerns; and an agent explicitly naming unresolved uncertainty or a need for human judgment.

The last matters most culturally: an agent saying "I am not sure about this" is doing the right thing, and the system treats it as a reason to bring in a person.

What the agent can say about itself

After the final push and a builder self-review, the authoring agent writes a short human-readable section into the pull request description plus a machine-readable marker, an HTML comment holding JSON, with five fields: the head commit it describes, the self-review result, a risk level, whether human review is required, and a list of reasons. The rules around it:

1. The head commit must equal the pull request's current head; the next push makes the evidence stale until the builder re-reviews and updates the marker.
2. The agent's risk level can only raise the effective risk from changed files (the higher wins), and any agent risk above low forces human review even if the agent says it is not required.
3. The agent sets the human-review flag whenever it finds an escalation or open judgment question, with concise reasons.
4. The marker does not encode CI or deep-review results; the harness or repository automation produces those, so an agent cannot self-certify them.
5. A missing, malformed, failed, or stale marker blocks a low-risk candidate.

The self-review behind the marker is one small pass asking "am I actually finished?", with one rule: never claim a check that was not performed. It does not replace independent review, and a clean one does not predict a clean deep review.

Routing review depth

Risk decides who must approve; a second function sets baseline review depth as a number of independent reviewers.

Overall risk above low gets width 3, a quorum. Low risk with only low-risk categories and at most about 120 changed lines and 10 files gets width 1, a scan, as does a smaller footprint (at most 80 lines, 6 files, no deletions). Everything else gets width 3.

An unsure agent can always choose 3, and code pull requests never skip deep review. Its cost is measured in what an AI code review costs.

Asking for more, never less

An author or reviewer can add a second marker, a review request, asking for a specific width, a subset of review dimensions, a free-text focus such as "check for N+1 queries in the new repository methods", or a particular reviewer skill for the first pass. It can only raise the ceiling. It cannot downgrade deterministic human review, skip builder self-review, or run the deep review narrower than baseline (a lower requested width is ignored, with a router note saying so). Like the first marker, it is ignored if tied to a stale head.

No field makes merging cheaper than the paths allow.

Where the route is visible

A workflow keeps one sticky comment on each pull request, the Change Summary, updated on every push. It shows the risk buckets, recommended route, human-review requirement and reasons, and which review kinds run at which depth. Code an author can run locally prints the same route as JSON, so agents can read their obligations early.

When a human is required, the summary names the teams from the code ownership file: a universal merge-approver team on every pull request, plus specialists for trading execution, platform and security, and published content. A picker proposes a primary reviewer without changing who must approve.

The guarded bot approval

One place produces something that looks like an approval, so we are careful. When the policy yields a candidate, a workflow can submit a pull request approval on behalf of the repository's automation account. A candidate means low deterministic risk, valid current-head agent evidence, and no human-review escalation, and it posts only when required CI is green on that exact head. The approval is managed, not fire-and-forget. It is dismissed when required CI fails or is pending on the head, or when the status stops being a candidate (a new push, a stale marker, higher risk, or an escalation). After a new head, older approvals are dismissed and one fresh one is posted if it is still a candidate. It is skipped on forks and for human-required or evidence-pending statuses.

What it is not matters more. It is an advisory signal for rollout metrics and future rules. It does not satisfy code owner team requirements or replace ruleset approvals: the automation token cannot impersonate a named team, and authors cannot approve their own pull requests. The repository enforces those rules, not our prompts, which is why we trust them.

What stays human

The high end is not negotiable. Trading execution, broker abstraction, auth and security, billing, migrations, and compliance-sensitive changes require a human however clean the automated loop ended, with no "auto-merge candidate" path. A required approval from each applicable team still gates the merge, and approvals go stale on new commits. Our Postgres migration safety checker fails deterministic hazards, but a person still owns shipping a migration.

Other triggers also send a pull request to a person: required CI still failing after five author pushes without a green snapshot, unresolved blocking findings on the current head, or a reviewer reopening a concern the author addressed. These escape hatches, covered in stopping AI agents from arguing forever, stop agent loops volleying forever.

Risk also decides what ships

The same classifier routes work into a release. A releaser, a role of our background agent, assembles the daily release from merged work. Sensitive, core or complex work waits for the weekly release; web and UI work can go daily. The split is mechanical and precedes any agent: a merged candidate touching any path the classifier puts in the trade execution, signals and webhooks, or trade planning categories ships only with the weekly release.

Nothing overrides that: no label, issue tracker nomination, manual add by a human, retry, or accepted risk. Every merge into the development branch is considered once; a label can only nominate more. A failed read of the file list fails the poll rather than counting as "no trade-execution files", because the failure we fear is a guess that the diff was harmless.

The releaser never merges, force-pushes or requests review; merging into production is always a human. Its one mark-ready is the release pull request, once the daily freeze window begins, main CI has finished with no failing check on that head, and no releaser run is live. It never acknowledges migration warnings, which stays with a different human reviewer. In default prepare mode every trigger is a recommendation and the chat asks first.

Release Notes Check

A CI job, Release Notes Check, fails a release pull request into production unless it adds or changes a dated customer note in the release notes folder or carries an explicit no-notes label. A README or summary alone does not count. Other pull requests skip the job and report success, so the check can be required without blocking everything else. An API read error fails closed.

Like our Postgres migration safety checker, it is a policy-as-code gate: a small, tested decision module in a thin read-only workflow. Skipping notes takes a visible label, not a silent default.

Rollout: evidence before rules

We have not changed the repository rules to let anything merge without a person. The sequence:

1. Label and report auto-merge candidates while humans keep reviewing.
2. Pilot unattended loops on narrow, low-risk slices only: agent and product docs, chores, developer tooling, and test-only changes.
3. Track every case where a human finds a blocking issue after clean self-review, CI, and deep review.
4. Expand eligibility only for categories with a demonstrated low miss rate, and change repository rules as a separate, deliberate step.

During the pilot we spot-check a sample of resolves: all high-risk pull requests and a fixed fraction of low-risk docs and chores. An unexpected resolve is an incident and a reason to tighten scope. "Green CI and the agent said it was fixed" is not our bar for trading-hot paths.

Lessons

1. Compute the floor from paths, not prose. A description can be wrong; a path cannot.
2. Keep every agent input one-directional. Evidence, risk, and review requests only raise requirements, so bad input costs time, not safety.
3. Tie evidence to the head commit. Staleness is the default after a push.
4. Keep outcomes out of the agent's mouth. The harness reports CI and deep-review results.
5. Label advisory signals as advisory. A bot approval that quietly satisfies a gate is a policy change nobody voted on.

Next in this series is compliance: encoding our independent-trader model as a constraint agents cannot reason around, the main reason several High paths exist. Until then, start with the floor: write down which paths are expensive to get wrong, turn that into executable rules, and only then argue about what else a person must see.