Stopping AI Agents Arguing Forever
Two agents with opposing roles can ping-pong forever, and an agent that resolves its own threads grades its own homework. Here are the caps, escalation triggers, and defaults we use.
Founder / CEO
Bottom Line
- An AI agent review loop needs hard exits: without a cap, an author agent and a reviewer agent can volley indefinitely and burn tokens while nobody decides anything.
- The default mode drafts a packet (fixes, replies, a resolve recommendation) and waits for a human; unattended execution is opt-in, scoped to a named role, and never covers merge or force-push; the only thing any role marks ready is the release pull request under stated CI conditions.
- Authors never resolve review threads, so the agent that wrote the fix is never the one that declares it fixed.
- Escalation is a list of concrete triggers (5 stuck CI pushes, 2 disputed rounds, 3 re-request rounds, high-risk paths), not a judgment call about whether things feel ambiguous.
- This is a pilot: the policy is documented as advisory first, human approval is still required, and unattended loops start only on narrow low-risk slices.
An AI agent review loop has a failure mode single-agent workflows lack: two agents with opposing jobs can disagree forever. The author fixes what the reviewer flagged, the reviewer finds something new or re-opens the same concern, and the author pushes again. Every turn costs tokens and none is a decision. Quieter version: an agent resolving the thread on its own fix is grading its own homework.
These are the guardrails we wrote after external review of our workflow surfaced four operational risks, part of a series on how we build an AI software factory, with a caveat up front: it is a pilot. The policy calls itself advisory first, and repository rules still require human approvals.
Why agent loops do not converge
Human reviewers get tired, which is a feature: after three rounds a person picks up the phone. Agents do not tire. A reviewer will always find something to say about a fresh diff, and an author will address every comment. Our author and reviewer loops run on GitHub as the shared bus, woken by pull request events. That symmetry needs asymmetric exits: numbers and named conditions a human can read and an agent can check.
Four risks from external review
The outside review did not fault the agents' code; it found four weaknesses in the loop between them, each now covered by a rule. Self-verify and resolve: a reviewer resolving its own threads without an independent check creates a trust boundary, and bugs hide behind "verified on my prior comment." Vague escalation: an agent cannot reliably evaluate its own ambiguity. Infinite volley: with no cap, author and reviewer can ping-pong without end. Cost at scale: polling, duplicate watchers, and re-review on every push multiply across open pull requests.
Prepare mode is the default
Our review-watching skill has two modes, and prepare is the default for both author and reviewer. On every wake the agent builds a packet: proposed fixes, draft replies, verification notes, and a resolve recommendation. It does not commit, push, post a comment, submit a review, re-request a reviewer, or resolve a thread until a human approves the packet.
Auto mode runs unattended only when the user explicitly asks, and three properties keep it narrow:
- It is scoped to a role: the author gets its own pull request, and a separate releaser gets the daily release and branch-sync fix pull requests.
- Arming it counts as commit and push authorization for that scope only, overriding our "do not commit or push unless asked" rule there and nowhere else.
- For author and reviewer roles it never covers force-push, merge, marking a draft ready, or anything like a human approval; those need a separate ask every time. The releaser never merges (merging into the production branch is always a human), force-pushes, or requests review. It marks ready only the release pull request, once the daily freeze window begins, main CI has finished with no failing check on that head, and no releaser run is live.
Most sessions never arm auto mode; we would rather pay in latency than find an unattended loop rubber-stamping itself.
The author never resolves a thread
The simplest rule is the most useful: authors never resolve review threads, by any tool or in the GitHub UI. The author replies, notes the thread is left open for reviewer verification, and stops.
Resolution belongs to the reviewer, and only after required CI is green on the current head. The reviewer re-fetches the diff and the lines it named, confirms the fix matches the finding (not merely that a new commit exists), then replies with a verification marker tied to the head SHA and resolves; an author-resolved thread lacking the marker still needs verification. In prepare mode even the reviewer only recommends a resolve for a human to approve. The party that made the claim never accepts it.
Spot-checking the resolves
Humans spot-check resolves on a sample: every pull request with high deterministic risk, and a fixed fraction of low-risk docs and chore changes. An unexpected resolve is an incident, answered by tightening prepare mode or shrinking the pilot scope. "Green CI plus the agent said fixed" is explicitly not enough for trading-hot paths during the pilot.
Escalation is a list, not a feeling
The policy replaces "escalate when ambiguous" with checkable triggers. Any one routes the pull request to a human via a packet, a comment, or a human-review-required flag in the description's evidence block:
- Deterministic high risk. Our change summary computes review policy from changed files; an agent cannot downgrade it.
- Agent escalation marker. The author flags a human-review requirement, rates risk above low, or lists reasons.
- CI stuck. Required CI still failing after 5 author pushes on the same pull request with no green snapshot. The reviewer can also raise it after several red pushes with no fix attempt.
- Unresolved blocking findings. Deep review reports some still open on the current head.
- Reviewer disagreement. After the author re-requests review, the reviewer re-opens the same concern, requests changes, or declines to resolve, or deep review and an agent reviewer disagree on whether a finding blocks. A human decides.
- Repeat finding. The same thread is disputed across 2 full rounds of author fix, re-request, and re-verify. Escalation carries both sides' summaries.
- Sensitive paths. Trading execution, broker abstraction, auth and security, billing, migrations, and compliance-sensitive behavior always need a human.
Agents may add context but may not clear a trigger by lowering the risk rating or resolving threads outside policy. This follows risk-proportional review: risk can only go up from deterministic analysis, never down because an agent feels good about the diff.
Hard caps on the volley
Triggers catch disagreement; caps catch loops that never end though nobody disagrees.
| Loop | Cap | On cap |
|---|---|---|
| Solo session, one agent alternating author and review passes on its own pull request | 5 full author-then-review cycles | Stop, post a short escalation summary with last findings, SHAs and CI state, and hand over |
| Cross-session author and reviewer | 3 formal re-request rounds for the same reviewer without that reviewer resolving all their threads or handing off | Escalate and send no further re-request until a human resets the round |
| CI-only fix pushes | No cap on pushes | These do not count toward the re-request cap |
The CI exception is deliberate: fixing a red build is convergent work, and capping it would punish the agent for flaky infrastructure. A new round starts only after the author addresses feedback, CI is green, and a re-request goes out. The stuck-CI trigger still stops thrashing. Caps reset only when a human says the round restarted, for example after a scope change or a rejected approach. An agent cannot reset its own counter.
What the solo loop looks like
One session alternates an author pass that implements and fixes with an adversarial review pass that outputs findings only, returning to the author pass while blocking findings remain, up to the cap of 5. It exits when the review pass has zero blocking findings, deep review has run on the current head, self-review is current, and required CI is green. Then it stops and asks for external reviewers, never claiming merge eligibility.
Cost is a design constraint
The fourth risk is the least dramatic and the most predictable: loops that poll or wake on every push cost money in proportion to open pull requests. The wake side is in zero-token wakes. The policy adds loop-level rules:
- End the turn after arming subscriptions; do not busy-poll CI. CI is request-only on pull request branches here, so an unrequested run is a missing step.
- Batch re-requests: at most one per reviewer per round, after fixes, replies, and green CI.
- Reviewers wake primarily on a formal re-request, treat a new head as a backup signal, and defer re-verification until required CI is green.
- One armed watcher per role per pull request per session; re-arm after each wake and never stack poll loops.
Bounds for unattended work
A queue-wide supervisor now reads every open pull request, picks one next step each with pure, tested rules, and keeps a local ledger of what this checkout launched. The releaser works the same way. Both default to prepare, have an auto mode, and carry their own bounds:
- Review concurrency. At most 2 live reviews by default, configurable from 1 to 10. Each review can bring up its own Docker stack, so we raise it only with memory headroom.
- Attempts. Each head gets 2 review attempts, plus one per human grant; a retry is a human decision.
- Watchdog. A review over 2 hours trips it, as does a wake unmarked for 10 minutes, sending one native notification instead of waking an agent.
- Stopped means stopped. The executor never restarts a poller someone stopped on purpose.
- Auto means auto. In auto mode the executor runs the steps its table marks as auto without asking: claim a review, arm a watch, launch a re-verify. It hands the worker the brief verbatim, since added advice such as "triage by risk" made workers skip required evidence.
The supervisor stages exactly one unsubmitted draft review at the current head, and the human submits it. It never submits, approves, requests changes, merges, marks ready, or resolves threads for the human, and when the human does, it reads the new state and acts without asking them to confirm.
When an author pushes and replies, the reviewer watch reruns the finding's recorded reproduction on the new head and stages a fresh draft review. In both modes it posts no replies and resolves nothing, since GitHub would attribute them to the human. A closed thread with no recorded passing reproduction is not proof of a fix.
Stale pull requests stay silent
A pull request idle for 30 or more days and not ready is stale, and stale pull requests never ping anyone. A stale sweep is one read-only pass the human asks for, with no poller. The executor drafts one short comment per item with the human and posts only after the human approves that exact text, in every mode. The sole pre-approved text is a fixed merge-ready line per head, posted under a rollout flag after a live recheck.
The rollout is deliberately slow
The policy says plainly: do not remove the approval requirement yet; collect evidence first. The unattended pilot starts with agent docs, product docs, chore changes, developer tooling, and test-only changes at low deterministic risk. Core trading logic, broker integrations, auth, billing, and migrations are out of pilot: human review stays mandatory, and reviewers stay in prepare mode for resolves unless a human opts in per pull request.
We track the miss: a human finds a blocking issue after self-review, CI, and deep review were all clean. Eligibility expands only for categories with a low demonstrated miss rate. Repository rule changes happen separately, after the evidence, and high-risk categories are meant to stay human-reviewed even then. When a human catch reveals a mistake existing guidance could have prevented, the smallest durable fix goes into docs, a skill, a checklist, or a deterministic tool.
Lessons
- Give every loop a number. Agents will not reliably take an exit that is not a count or named condition.
- Separate the claim from the acceptance. The agent that fixes never resolves.
- Escalate on facts, not feelings. Stuck CI, repeat disputes, and sensitive paths are checkable.
- Do not count convergent work against the cap. CI fixes are not argument rounds.
- Treat cost as part of the protocol. Batch re-requests, one watcher per role, and cap supervisors too.
- Pilot narrow and measure misses. Expand only where humans keep finding nothing after the agents were clean.
We cannot yet claim these guardrails are sufficient, only that they turn the four risks into rules we can audit. Humans still own outcomes and still approve. The pilot will show how much approval work the loop can safely absorb on low-risk slices; for trading, auth, billing, and migrations the answer is already no.
