Regression Tests for Agent Instructions

If you want to test AI agent instructions, start from a miss, not from a guess. When an agent skips a rule that your documentation already states, the useful question is whether you can reproduce the miss, write down the wrong answer and the right answer, and make a machine tell them apart. If you can, you have a regression test. If you cannot, you have an opinion about prose.

We run this loop in the same way we handle a code bug. A real review finding becomes a pair of fixtures and a rubric, the pair runs in CI forever, and only then do we ask whether the guidance itself needs to change. This post walks through the loop, the four gaps that seeded it, the false green that taught us the most, and what the whole exercise does not prove. It is part of our broader approach to building a software factory around coding agents.

A missed rule is a bug

Agent instructions fail in two different ways, and they need different fixes. Guidance can be wrong: it says something false, contradicts another document, or points at a command that no longer exists. Or guidance can be untested: it is correct, a narrow owner document already teaches the right behavior, and nobody has ever checked that an agent following the routing actually lands there.

Those look identical in a review comment. An engineer sees an agent do the wrong thing and says "the docs need to be clearer." Sometimes that is true. Often the docs were fine and the miss came from somewhere else: a route that nobody exercised, or a check that would have passed the wrong answer anyway.

Our gap loop forces the distinction before anyone edits a document. The order is find, encode, fix, prove. Collect misses from reviews, steers, and CI failures. Encode the observed bad answer as a negative fixture that must fail and a correct answer as a positive fixture that must pass. Only then fix the narrowest canonical owner, whether that is a router row, a pattern, a checklist, a skill, or a deterministic gate. Editing guidance first, before you can fail on the miss, means you cannot tell whether the edit did anything.

Anatomy of one gap

Each gap has an ID, the evidence that opened it, the bad behavior we encoded, the score of the bad answer against the score of the good one, the canonical owner of the correct behavior, and a status. They live in a backlog file next to the audits, and the rule at the bottom of it is deliberate: add a row only with a concrete review, steer, CI failure, or reproduced miss. A hypothetical failure mode does not get a row.

The four seed gaps came from review findings on real pull requests:

• Skipping browser verification. A reviewer found that source-only tests could not establish critical browser behavior, and the real flow then failed. The bad answer says passing Twig and JavaScript tests are enough and leaves the browser check to the reviewer. It scored 0%; the answer that loads the browser guidance, resolves a real route, inspects the screenshot and console, and says smoke does not replace granular tests scored 100%.
• Quality-scope overreach. The bad answer runs the full style and test suites plus a stack of separate branch-wide gates for a two-file mid-task change. It scored 21%; the one file-scoped check scored 100%.
• Approving migration SQL by reading it. A reviewer reproduced dangerous Postgres migration SQL passing the safety gate. The bad answer approves plausible SQL without loading migration guidance or running the branch safety gate, and let staging find any performance problem. It scored 0%; the answer that loads the migration pattern and checklist, runs the branch gate, rejects the type rewrite, requires a concurrent index, and bounds the backfill scored 100%.
• Treating a review worktree as author-side. A reviewer reproduced an agent treating a review worktree as its own and being told to push another engineer’s branch. The encoded bad answer uses the shared main checkout because it is clean. It scored 0%; creating an isolated sibling worktree non-interactively and working only there scored 100%.

The migration case pairs naturally with our migration safety checker. The checker is the deterministic gate; the golden answer checks that an agent knows to reach for it.

Encode a Bad and a Good Answer

The fixtures are plain text files, one positive and one negative per gap, in a golden directory. Each gap also has a fixed task prompt that describes the situation and asks the agent to decide and cite the routed guidance without executing anything. For the quality-scope gap, the prompt says only two files were edited, a teammate proposes the full suites plus coverage and mutation testing, and asks for the smallest correct command set.

The negative fixture is written to sound reasonable, because that is how real misses sound. "The broader suite gives more confidence, so the extra time and duplicate work are worthwhile" is not a strawman; it is what a cautious agent says. The positive fixture cites the pattern, gives one file-scoped check, and explains why the stack of separate gates is redundant for a mid-task round.

The rubric is the test

A rubric is a list of criteria. Each has a name, a regular expression, a weight, and a flag for whether the pattern should match. A criterion passes when the match result equals the flag. The score is the earned weight over the total weight.

Positive criteria check for evidence of the right behavior: the browser rubric looks for the skill reference, a route discovery step, and opening the page by host and path. Negative criteria (should-not-match) check for the wrong behavior: the quality-scope rubric penalizes a bare full-suite test run, a full style run, a full static analysis run, branch scope for a mid-task check, and the stacked branch-wide gates. That is how the quality-scope negative lands at 21%. It earns weight only on a non-empty answer and on two avoid-this criteria, and misses the file-scoped check entirely.

The false green

The quality-scope gap was not opened by an agent forgetting a rule. It was opened by the rubric. A review demonstrated that an answer combining a file-scoped check with a redundant mutation testing gate scored 100%. The rubric rewarded the right command and did not penalize the extra one, so a wrong answer got a perfect score.

We reproduced it first, then tightened the rubric. The current rubric has criteria that reject the stack of branch-wide gates and reject passing a file list to a gate that does not accept one. The adversarial answer stays in the suite as a negative fixture that must remain rejected, so the rubric cannot quietly lose its teeth again.

The lesson is uncomfortable but useful: a rubric that only looks for presence is a test that cannot fail. Every positive criterion needs a partner that fails the plausible wrong answer, and the only reliable way to find the missing partner is to write the wrong answer down and score it.

How the scorer works

The scorer is a command with distinct modes. The mode that matters for CI is the golden mode. For every task with fixtures it scores the positive file against the rubric and requires a perfect score at the default threshold of 1.0. Then it scores the negative file at a fixed 0.8 threshold and fails the run if the negative passes. Both directions are enforced in one run, which is the whole point: the positive proves the rubric accepts correct behavior, the negative proves it rejects the miss.

The other modes do not need CI. One mode writes the fixed task prompts and a manifest without spawning anything. Another scores an existing directory of saved agent outputs. A live run launches a coding-agent command-line tool, one process per sample, defaulting to ten agents with five in parallel. Outputs are saved, scored against the same rubric, and written to a scores file with an optional HTML report. If the agent is not installed, the command refuses to run rather than pretending.

There is also a feedback loop: run the agents, aggregate the most frequently failed criteria, ask a controller agent to propose documentation changes without making them, and wait for a human to answer yes or no before anything is applied. It stops after three iterations. It is a proposal tool with an approval gate, not an auto-editor, and given our own finding that these gaps were not documentation problems, we treat its proposals with suspicion.

Cheap in CI, periodic with agents

Scoring a text file against a handful of regular expressions takes milliseconds and no model calls. That is why the golden suite runs as a step of the main CI run, with a pass threshold of 100%, alongside the unit tests for the scorer itself and the always-on budget checks. The cost profile is the same as any other fast deterministic test, and it fits the rule behind our request-only CI: cheap checks can be broad, expensive checks are requested deliberately.

Live agents are the opposite. They cost money and minutes, and their answers vary between runs, so a single sample proves little. The local script runs a fixed pack of fifteen tasks, three fresh runs each, with a pass threshold of 0.8, and refuses to run when it detects CI. It is meant to run monthly, with an affected task rerun the same day when a miss hurts. The output feeds a scorecard that records pass rate, mean score, always-on budgets, and a failed-criteria histogram.

A closed deterministic regression does not leave the real-agent pack. That is the part people want to skip. A green golden file shows the rubric is sound; it does not show that agents stopped making the mistake. Keeping the gap in the monthly pack means a recurrence shows up as a count in the histogram. In the backlog, three of the four gaps are marked as still waiting for the monthly real-agent confirmation, for exactly this reason: closed in the deterministic suite is not the same as confirmed in a real-agent run.

We also ran a live quality-scope evaluation with ten agents per run. Coaching moved agents toward a single mid-task gate. That is a result from one task, and we would not generalize it beyond that.

Decide whether guidance changes

With the fixtures in place, the decision about guidance becomes empirical. If the positive passes, the negative fails, and the canonical owner already teaches the correct behavior, the guidance is not wrong. It was untested, and the fix is the test. That was the outcome for all four seeds: the backlog records that no canonical guidance rewrite was needed, because the gaps were in coverage and rubric teeth.

If instead the positive fixture cannot be written without inventing behavior the docs do not describe, the guidance is wrong or missing, and you fix the narrowest owner. This ordering also protects against the failure we described when we cut always-on agent instructions: adding rules to the root instruction file every time an agent slips is how instruction files bloat. A failing case that routes to an existing pattern argues for a route test, not a longer root file.

What this does not prove

A passing suite proves that a rubric accepts our hand-written good answers and rejects our hand-written bad ones. That is a narrow claim. It does not prove an agent will behave correctly on a task we did not encode, or in production with real tools and real pressure. The tasks ask agents to describe what they would do and not to execute commands, so the suite measures knowing and routing, not doing.

Regex rubrics can also be gamed by accident. An answer can include the right keywords and still be wrong in spirit, and a correct answer phrased unexpectedly can score badly. The false green was one example; there will be others. Live runs are small samples, and the failed-criteria histogram tells you where to look, not what is true. Humans still own outcomes and still review the work.

Lessons

1. Reproduce before you edit. A miss you cannot reproduce as a failing negative fixture is not yet a gap.
2. Write the wrong answer down. A rubric with only positive criteria cannot fail. The negative fixture is what gives it teeth.
3. Separate wrong from untested. Most of our misses were coverage gaps in correct guidance, so the fix was a test, not a rewrite.
4. Run cheap checks always and expensive ones periodically. Deterministic scoring belongs in every pull request; live agents belong on a schedule.
5. Keep closed gaps in the real-agent pack. Deterministic green is not confirmation, and recurrence only shows up if you keep looking.
6. Add rows only with evidence. A backlog of imagined failures is just another document nobody trusts.

The backlog is small on purpose, four rows, each tied to something that actually happened. The next entry will come from a review comment, and it will start the same way: reproduce it, score the bad answer, score the good one, and keep both.