Request-Only CI for AI Agents

CI for AI agents has a cost model that human workflows never had to face. An agent edits quickly, commits often, and pushes in small batches. If every push to a pull request branch starts a full CI run, the bill and the queue scale with how fast the agent works, not with how much information you get back. Agents are slow where it counts (each step takes real time and tokens), and CI is expensive for the same reason: the flow assumed a check after every change.

We moved the gates to the end. Pushes to pull request branches no longer start CI. CI runs when someone requests it, and on every push to protected branches. A green result on the current head is still a hard requirement to merge. This post covers the flow, the commit status that makes it enforceable, the tooling bugs that surfaced when CI became optional, and the short failure output that made the loop usable. It is one part of the larger system we describe in how to build an AI software factory.

Why per-push CI stopped working

The old assumption was simple: a developer pushes when something is ready, so a run per push is a run per meaningful change. Agents break that. A session may commit after each finished unit, push to keep work safe, and fix a review comment in a handful of tiny commits. Each push is cheap for the agent and expensive for CI. Worse, most of those runs are noise. The agent knows the branch is mid-change, and the result tells it nothing it has not already learned from the one test it just ran.

The same logic applied to local gates. We had already scoped checks to changed files, which we described in the guardrails overview: changed files only, mapped tests, a coverage patch gate, and mutation testing on changed lines. That was the right unit of work. It was the cadence that was wrong. Running a file-scoped gate after every edit is still a gate after every edit.

The flow: loose, then strict

The rule we wrote into the always-loaded agent instructions fits in a few lines, with the detail in a pattern document. Iterate with no gates. Check once when done. Request CI once. Merge only on a green gate.

1. Iterate with no per-edit gates. Prove the change with the narrowest test, one test file or one filter, through the test runner. No full style pass, no suite.
2. Checkpoint at each commit or finished unit. Run the cheap static gates once, in a fast mode with no tests. That covers code style, complexity and nesting, static analysis, and formatting, with no tests, mutation testing, or coverage.
3. When the work is done, once: run the branch-wide baseline, with mapped tests, patch coverage, and mutation testing on changed lines.
4. Request CI with one command. The branch must already be pushed, because CI runs the remote head.
5. If CI is red, read the short failure summary, fix the root cause, run only the failing gate locally, push, and request again. Once per batch of fixes, never per edit.
6. Ready and merge need a green CI Gate on the pull request's current head.

The human still owns the decision to push and to spend a CI run. Requesting CI is a cost a person owns, so the instructions say not to request it per edit or "to see what happens". A human ask to open or update a pull request covers requesting CI for it. Without one, the agent stops after the local gate and reports.

The checkpoint gate

The middle step exists because "no gates per edit" and "everything at the end" leave a hole. Complexity and design findings are far cheaper to fix while a class is small than after a whole branch has been built around it. So the instruction is to run the static gates once per commit or finished unit, not per edit.

We attached one rule to it: when a gate fails, refactor the design. Do not extract trivial helpers to shave a number, and do not suppress the finding. An agent under a failing complexity gate will happily split a method into three that exist only to move the count. The changed-file scope still sees that, and we would rather the agent reconsider the shape of the code.

One mid-task gate, not a stack

We evaluated how agents behaved around these gates and found the same failure repeatedly: they stacked redundant diff gates. A file-scoped check, then a style diff, then a complexity diff, then a coverage diff, all covering the same files. So the instructions now coach one mid-task gate: pick the file-scoped check or a single narrower diff target, never several. Passing files are cached by content and tool configuration, so a second branch run rechecks only what changed.

Making CI a request

Requesting a run has three routes, all dispatching the same workflow. An agent or human runs one command that wraps a small request script. A human can add a label to the pull request; a small workflow reads the label event, dispatches the main workflow for the head branch, and removes the label so adding it again is a fresh request. Or someone starts the run from the GitHub Actions tab. The label workflow, a GitHub Actions workflow itself, never checks out or runs pull request code. It reads the event payload and calls the API, and it skips forks because dispatch needs a branch in our repository.

The script has a few guards that came from watching agents use it:

• It refuses when the local head differs from the remote head, because CI runs the remote head and a stale request would test the wrong commit.
• It skips the request when a run is already queued or in progress for that exact commit. A later change added a force option to replace a stuck run, with a note in the help text not to use it in retry or watch loops.
• A new head commit always dispatches, and the workflow concurrency group cancels the older run.

Protected branches (the ones we merge into and release from) still run CI on every push. The change only affects pull request branches.

The CI Gate commit status

Dropping per-push CI would have been unsafe if the merge requirement had gone with it. The catch is a GitHub behavior: the jobs of a manually dispatched workflow run do not appear in a pull request's checks list. A branch protection rule that names individual jobs would see nothing.

So the run posts a status itself. A start job sets CI Gate to pending on the head commit. A final job sets it to success or failure, linking to the run. It is failure unless every job it depends on passed, and a shell test keeps that dependency list complete when someone adds a job. The result is a single required check that exists only when a run has actually happened on that commit.

That last property is the point. A pull request with no CI run on its head has no CI Gate status, and it cannot merge. A push after green makes the result stale for the same reason, so the instruction is to request again before marking ready or merging. For job-level results, follow the status link or use a command that opens the run.

Bugs that optional CI exposed

Making CI optional changed an invariant that other tooling had silently relied on: there is always a run. We found the breakage by reading the code while making the change, not from an incident, and the four cases are instructive.

• The convergence tool treated "no checks" as pending. It reports whether a pull request is ready to land. With no run, it said CI was pending, and an agent would wait forever for something nobody had asked for. It now reports "CI not requested" as a distinct state, and the instructions say to request a run instead of waiting.
• The watcher read an empty check list as green. The opposite error, and the more dangerous one. An empty list is the absence of evidence.
• The change-summary workflow ignored dispatched runs. It looked for results from push-triggered runs, which no longer exist on these branches.
• The migration-warnings workflow looked only for push-triggered runs, for the same reason.

The shared lesson is that "no data yet" needs its own state. Any code that maps an empty result to pending or to green will do the wrong thing the first time CI is not requested. We cover the polling side of this in our write-up on zero-token wakes, where the watcher's reads are the cheapest part of the loop and have to be right.

Short failure output

The cadence change moved the expensive moment. When CI is red, an agent's first instinct is to pull the raw job log, which is thousands of lines. That spends context on noise and often buries the actual failure under setup output.

A short failure summary command replaces that. It lists the failed jobs and steps, each labelled with the command it ran, with a few lines of the real message. Identical failures across several jobs print once. Underneath, a shared log-condensing library does tool-aware compaction: a static analysis or code-style report becomes one line per finding, with the file and line and the rule, instead of banners and source excerpts. The same library feeds the local gates, so an agent sees the same short shape locally and in CI. Distinct exit codes separate "failures found" from "CI was not requested" and from a tool error, so a script does not confuse them.

We tested it against real failures, not hand-written samples. The repository holds 28 captured failing-step fixtures under tests/Shell/fixtures/ci-failures/, including non-ordinary endings such as a segfault, a silent exit, and a timeout. Those captures drive 29 of the 41 shell tests in ci-failures.bats; the rest use synthetic strings. Condensation is a line budget (STEP_LINES 25, TOTAL_LINES 400), not a measured megabyte ratio. The silent exit and the timeout case matter most, because a tool that dies without printing anything is exactly when an excerpt-based view can mislead. This is the same instinct we apply to instructions in regression tests for agent instructions: use real captured behavior as the test input.

Gaps we stated at the time

This was not a finished design when it shipped, and we said so in the change itself.

• An image build still ran on every pull request push. Request-only CI covered the main workflow, not every workflow.
• Nothing blocked marking a pull request ready before CI was green. The hard gate was at merge, so the instruction to wait for a green head before marking ready was a rule for agents rather than a mechanism.

Both are reminders that the merge gate is the only part of this that is enforced by the platform. Everything before it is cadence and instruction. We think that is the right split for the moment: be loose where the cost of a mistake is a retry, and strict where it is a bad merge.

Lessons

1. Match the check cadence to how the worker works. A flow built around a push per meaningful change breaks when the worker pushes per finished thought.
2. Move cost to the end, keep the requirement. Request-only CI saves runs. A required status on the current head keeps the merge guarantee.
3. Give "no run" its own state. Empty check lists read as pending or green caused real bugs in four places. Report "CI not requested" instead.
4. Checkpoint static gates once per unit, and refactor on failure. Early design feedback is cheap; trivial helpers and suppressions defeat the gate.
5. Make failure output short and test it on real failures. Captured logs from deliberate failures keep a condenser honest, including the ones that print nothing.
6. Say what is only instruction. Ready-before-green was a rule, not a block. Know which of your guarantees the platform enforces.

If you adopt this, start with the status, not the workflow change: decide what single check a pull request must carry to merge, and make sure it cannot exist without a real run. Then audit every tool that reads CI state for what it does when there is nothing to read. Agents will find those cases faster than you do.