The AI Review Bot We Reverted

We built an automated AI code review GitHub Action in a single day, and we reverted it before the day was over. The idea was simple: someone comments a trigger phrase on a pull request, and a workflow runs our deepest review and posts the result. It took seven small pull requests to get that far, and one revert pull request to undo them.

This post is an honest failure story, so it has a limit we want to state up front. The revert does not explain itself, and we did not write a postmortem. We will not invent a root cause now. What we can give you is what we built, what the sequence of fixes tells you, and the principles we have applied since. For the wider picture of how we run agents on a production codebase, start with the hub post on building an AI software factory.

What we built that morning

We already had a deep review, which we call Eru: a deterministic gate plus a quorum of independent reviewers working through eight dimensions, from premise and architecture down to performance and operability. We cover it in the Eru quorum post. It ran in an agent session, on a developer's machine, when someone asked for it.

The Action tried to move that review onto GitHub. The pieces were small:

• A workflow that fired on a pull request comment containing a trigger phrase, filtered so it only ran on pull requests, not plain issues.
• A shared prompt of roughly 85 lines that told the model to read each rubric in altitude order, review the diff, and post a single GitHub review with inline comments.
• Optional focus text after the trigger phrase, such as a request to check a specific code path, which the prompt treated as an explicit item to report on.
• A deliberate omission: it skipped the PHP gate, because CI already covers formatting, static analysis, and migration safety.

The prompt was careful in places. It told the reviewer not to duplicate CI checks, not to fabricate findings, and never to approve, since only a human approves.

Seven pull requests in one day

The first pull request described a manual test plan: merge it, comment the trigger on another open pull request, and verify that inline review comments appear. A workflow triggered by comments runs from the default branch, so you cannot fully test it before it lands. Every fix therefore needed its own merge.

What followed was a run of small, tightly spaced changes:

1. The initial workflow and prompt, using a static API key stored as a repository secret.
2. A fix to the action's inputs and to the permissions the job requested.
3. A minimal stripped-down test workflow, added purely to debug why the action was failing.
4. Another small change to the same workflow.
5. A switch from the static key to workload identity federation, where the job exchanges a GitHub OIDC token for a short-lived access token. The stated motivation was that there would be no static key to rotate or fund separately. The same change pinned a smaller model for initial testing, with a note to move to a larger one for production.
6. Two more small changes after that switch.
7. The revert, which removed the workflow and its prompt file and undid all seven earlier changes.

You can read that sequence without any explanation from us. The failures were not in the review logic. The prompt that told the model how to review never got a real run that we judged worth keeping. The time went into three plumbing problems: what the action accepts as inputs, which permissions the job needs, and how the job authenticates to the model provider. Each fix exposed the next layer.

What we do not know

We should be precise about the evidence. The revert pull request records what it removed and which pull requests it reverted. It does not say why. Nobody wrote a postmortem. We did not collect a table of review comments the bot produced, measure their precision, or compare cost against the existing session-based review.

So we cannot tell you the bot failed because of noise, or cost, or authentication friction, or a change of mind about where review should live. Any of those stories would be tidy, and we would be making it up. What we can say is that a feature that takes seven merges to reach a first attempt, and then gets removed, was not earning its place in that form.

Cost is a real part of the question. Our measured numbers for a single review are in the cost post. A review that anyone can trigger with a comment is a review whose cost anyone can trigger.

Principles we applied afterward

These are the beliefs our current setup reflects. We present them as our values, not as a proven explanation of the revert.

A noisy bot costs attention

A review bot that posts on every pull request can waste more attention than it saves. Humans have to read the comments, decide which are right, and resolve threads. A reviewer who has learned that the bot is often wrong learns to skim, and then skims the good findings too. We wrote this down in our guardrails overview in one sentence: when the automation is wrong, turn it off and keep the standards.

Deep review stays in the author's session

Our review skill delivers its verdict in the conversation. That is the whole deliverable. Posting a comment or a formal review to GitHub requires the user to ask for it in that session, and permission is per request, not standing. A request-changes review blocks the pull request, so we treat it as the most expensive version of a posting mistake.

That rule actually predates the bot. An earlier change to the skill had already said that a review is an opinion, not a published artifact, and that publishing is visible to the whole team and awkward to retract. An Action that posts a review automatically runs against that rule by design. We did not weigh that explicitly at the time, and it is one reason the principle is now written down.

Deterministic gates first

The bot's own prompt skipped formatting, static analysis, and migration safety because CI already enforces them. That instinct was right and it points at the real priority. Checks that return pass or fail belong in CI, where they are cheap and repeatable. Model judgment is reserved for what a gate cannot decide. Our request-only CI post covers how we run those gates.

Durable instructions beat commentary

The same day as the revert, we merged review instructions for the code review feature GitHub already provides. This is the part we kept. There is one main instruction file that applies to every review, and four path-scoped files that load only when a pull request touches matching paths:

• PHP files, for code conventions with correct and incorrect examples.
• Database migrations, for safety judgment calls beyond what CI checks.
• Test files, for test quality rules and base classes.
• Twig templates, for component reuse and naming.

The main file mirrors our eight review dimensions in the same altitude order, in condensed form. It tells the reviewer what CI already owns so it does not repeat those findings, and it asks it to flag any attempt to weaken a gate. The maintenance doc says what does not belong: anything CI already checks, external links the reviewer cannot follow, and vague directives like "be more accurate."

Notice what changed. Instead of operating our own review infrastructure, with its inputs, permissions, and credentials, we put our standards where a hosted reviewer reads them. The standards are the durable asset. The runner is replaceable. Copilot review is a medium-depth path that does not block merges.

What we do instead today

Review is a pipeline with several kinds, not one button. The builder verifies with targeted checks and runs a quick self-review. CI runs the deterministic harness. The deep review runs on request in an independent session, with the verdict returned to whoever asked. The author's watch loop handles threads and formal re-requests. Humans own the required approvals, and bot or agent verdicts do not satisfy them. We describe the author and reviewer loops in the GitHub bus post.

That arrangement has costs. A developer has to remember to ask for the deep review, and no review fires by itself on every pull request. We accept that on purpose, because a review someone requested is a review someone intends to read.

Lessons

1. Count the plumbing before you build. If wiring an agent into CI takes several merges just to authenticate, the review prompt is not your real project.
2. Comment-triggered workflows are hard to test. They run from the default branch, so every fix is a merge. Plan for that or avoid the trigger.
3. Write the postmortem. We did not, so we cannot say why we reverted. Ten minutes of notes would have turned a vague memory into evidence.
4. Keep review output private by default. A verdict returned to the requester costs no one else's attention. Posting should be an explicit, per-request choice.
5. Invest in instructions, not infrastructure. Path-scoped, example-driven review instructions travel to whatever reviewer you use next.

If you are tempted to build the same bot, start by writing down the instructions you would give it and the gates that should already have caught the easy findings. In our case, that exercise produced something we kept, while the Action did not survive its first day.