How to Build an AI Software Factory

An AI software factory is not a bigger model or a cleverer prompt. It is a pipeline that takes a change from idea to shipped release, where agents do much of the work and every stage has a gate that does not depend on trusting the agent. Thesis: agents propose; deterministic gates and accountable humans dispose.

We build this on a production Symfony codebase that handles trade execution. This post is the hub of a series and a map: per stage, what goes wrong without a gate, the gate we built, and the deep dive. The earlier overview of our guardrails covers the first pieces.

What we mean by a factory

A factory is a sequence of stations, each checking something specific, so defects are caught early and cheaply. The workers are agents: fast, tireless, and confidently wrong in ways humans are not, since an agent produces plausible output even when unsure. So we do not ask agents to be careful; we build stations that catch carelessness mechanically and reserve human attention for judgment.

Three ideas that recur

• Scope verification to the change. Run the checks that cover the files you touched, not the whole repository.
• Make policy code, not prose. A parser or status check enforces a rule that matters; a paragraph is a suggestion an agent weighs against the shortcut.
• Make silence the default agent output. An agent with nothing new to say should say nothing and spend nothing.

What this is not

It is not lights-out software. Humans own outcomes; agents are leverage, not authority.

The whole pipeline at a glance

Stage 1: a sandbox for every agent

What goes wrong. Agents sharing a working tree fight over branch, containers, database, and queues, and one in a half-provisioned environment implements anyway, producing unverifiable changes. The gate. A deterministic create and verify pair; no implementing until verification passes, and failures return machine-readable remediation. Each parallel agent gets its own sibling worktree, container, and state. See Every AI Coding Agent Needs Its Own Sandbox.

Stage 2: context an agent can afford

What goes wrong. Teams answer agent mistakes by adding always-on rules until agents pay a large token cost per prompt and still miss the rule that applied. The gate. Always-on context is a scarce budget: the root file holds only universal invariants and a router, detail lives in on-demand patterns, checklists, and skills, a rule earns a place only by passing an always-on test (it must constrain the agent before the task type is known, or missing it could cause irreversible, security, financial, compliance, environment, or collaboration harm), and a size budget enforced in CI stops creep. The root file shrank about 76%, from 9,229 to 2,223 proxy tokens. See How We Cut Always-On Agent Instructions.

Stage 3: build and verify

What goes wrong. An agent runs everything on every edit or declares victory after a narrow command that missed the change, and hazards like a lock-heavy migration slip past reviewers. The gates. Request-only CI: no gates per edit, a fast file-scoped check at each finished unit, one branch check when done, then an explicit CI request. A CI Gate status on the current head is required to mark ready or merge, so a later push goes stale. See Request-Only CI for AI Agents. Policy as code for migrations: the checker parses SQL with a real PostgreSQL grammar, errors always block, and warnings block until a different human acknowledges them, so the author cannot self-approve. See our migration safety checker post. Regression tests for instructions: when an agent misses a rule, we encode the bad and good answer and score deterministically, so a documentation change cannot silently make agents worse. See Regression Tests for Agent Instructions. Browser verification: for any UI change the agent must open the real page, read the screenshot and check for page errors, because passing source-level tests once missed a broken flow. See Browser Verification for AI Agents.

Stage 4: review in layers

What goes wrong. One review pass has blind spots, and an agent reviewing its own work shares them. The gate. Our deepest review, Eru, starts with a deterministic gate so mechanical problems never reach a model, then uses a fresh reviewer per dimension, a quorum, and a synthesized verdict. See Eru, an AI Code Review Quorum. We measured what deep review costs (unused context, unbatched tool calls, a tally that never fired, rejected optimizations) in What an AI Code Review Costs. We also reverted a review Action that anyone could trigger with a comment phrase on a pull request: a review anyone can trigger is a review whose cost anyone can trigger. See The AI Review Bot We Built and Reverted.

Stage 5: handoff without chatter

What goes wrong. With several agents and humans on one pull request, authors push and walk away, reviewers resolve their own threads, agents volley indefinitely, and watchers burn tokens polling. Four conventions became mechanics.

• Completion loops. GitHub is the bus. Authors stay until required CI is green and re-request review; reviewers stay until findings are verified; authors never resolve threads. See GitHub as the Bus for AI Agents.
• Zero-token wakes. A wake gate and pre-wake filter decide whether an agent wakes. See Zero-Token Wakes for AI Agents.
• Caps and escalation. A pull request goes to a human when required CI fails after 5 author pushes or a finding is disputed across 2 full rounds. Watchers default to prepare mode, drafting a packet and waiting for a human. See Stopping AI Agents From Arguing Forever.
• Packets, not spawns. The dispatcher decides and records a packet in a ledger; in version 1 it spawns nothing. See Dispatcher Packets, Not Spawns.

Above these, a queue-level supervisor (a role of our watch skill) picks one next step for each open pull request: start or recheck a review, ask the author to fix CI or conflicts, or wait. Bounded workers stage one unsubmitted review draft for a human to submit; authors merge their own pull requests. Unattended action needs an explicit human request.

Stage 6: govern by risk

What goes wrong. Uniform review makes humans rubber-stamp low-risk work, and an agent rating its own risk sometimes decides wrong in the convenient direction. The gate. The risk floor is deterministic, from changed-file analysis and code-ownership categories. Agent evidence can only raise effective risk, never lower it, and a router picks which review kinds apply. For a narrow class of low-risk changes, a guarded advisory bot approval can mark an auto-merge candidate once CI is green on the current head. See Risk-Proportional Review for AI Agents.

The other constraint is product policy: agents must never introduce leader and follower fill mirroring, credential sharing, or remote operation of another person's account, and ambiguous cases escalate. See Compliance as an AI Agent Constraint.

Stage 7: shipping without surprises

What goes wrong. Trade-execution work rides a daily release because a label said so, customer release notes get forgotten, and a failed branch sync goes unnoticed. The gate. A releaser (another watch skill role) runs our daily and weekly cycle in prepare mode, so each trigger is a recommendation until a person arms auto, and the daily release skill's gates decide, never a label. Trade-execution work is excluded mechanically: a candidate touching a path the change-summary classifier puts in trade execution, signals and webhooks, or trade planning ships only weekly. It is the same classifier that sets risk in risk-proportional review, nothing overrides it, and a failed file-list read fails the poll rather than counting as "no sensitive files". A Release Notes Check job passes on production-branch release pull requests only when the diff adds a dated note or a label says there is none; API errors fail it. When the branch sync fails on a conflict, a worker opens a fix pull request, and the releaser follows downstream syncs, waking a human only when a hop is stuck. The releaser never merges, force-pushes, requests review, or acknowledges migration warnings; its one write is marking the release pull request ready once the daily freeze begins and CI has no failing check.

The pilot, stated honestly

The goal of stage 6 is "review proportional to risk" instead of "every pull request needs a human." We are not there: repository rules still require code-ownership approvals, which the bot approval does not satisfy. High-risk paths keep mandatory human review whatever any agent says: trading execution, broker behavior, auth and security, billing, migrations, and compliance-sensitive work. The auto-merge-candidate path is an advisory pilot gathering evidence on whether low-risk changes such as documentation, tests, and developer tooling could skip human review. Humans spot-check a sample of agent resolutions, and an unexpected resolve is an incident. A green draft is not ready; a human decides that.

What we deliberately do not automate

• Marking a pull request ready or merging it. A person decides (the releaser marks only its release pull request ready), on a green CI Gate.
• Merging into production. Always a human.
• Submitting reviews. Workers stage one unsubmitted draft; a person submits it.
• Force-pushing. Agents never do it.
• Downgrading risk. Agent evidence only raises it.
• Resolving threads without oversight. Authors never resolve, and reviewer resolves are spot-checked during the pilot.
• Acknowledging migration warnings. A different human does it, never the author.
• Judgment on compliance boundaries. Ambiguous cases escalate.

Where to start

You do not need all seven stages at once. On a new codebase, adopt them in this order, cheapest first.

1. Sandbox and verify. Give each agent its own environment and a verification command to pass first.
2. Shrink context. Move detail out of the always-on file; budget-test what remains.
3. File-scoped gates and CI on request. Cheap per-change checks, explicit final check.
4. One policy-as-code gate. Encode your scariest hazard, such as migrations, in a parser.
5. A deep review. Independent review behind a deterministic gate; measure its cost.
6. Then watchers. Completion loops, wake filters, and caps, once earlier stages are solid, because automation on a weak gate multiplies the weakness.
7. Supervisor and releaser last. They need review, risk routing, and caps beneath them.

What is still open

• Wake transport varies by host. Zero-token wakes behave differently across harnesses, so the shared layer states capabilities and fallbacks rather than assuming one vendor.
• Auto-merge is a pilot. The advisory approval does not replace required human approvals, and we have not changed repository rules to rely on it.
• Instruction evaluation measures a rubric on sampled answers. It shows whether agents find and apply a rule on the tasks we wrote, not whether production outcomes improved.

Lessons

1. Put a gate at every stage. A factory is stations with checks, not an agent with a long prompt.
2. Let agents raise risk, never lower it.
3. Turn rules into code.
4. Default to silence. Wakes, comments, and reviews should cost nothing with nothing new.
5. Keep a human accountable. Prepare mode, caps, and escalation put a person in the decision.
6. Be honest about pilots. Label advisory paths as such and collect evidence before changing who approves.

The real structure is the three recurring ideas, and they transfer. For a new failure, ask whether the check can be scoped to the change, the rule made code, and the default output made nothing; if so, you have found the next gate. Start with the sandbox, and work down the table.