The eval that would have caught this: turn one bad output into a regression test

A customer-facing answer was wrong last week. The team fixed the prompt, checked a few examples by hand, and shipped. Two weeks later, a different customer received the same class of error. The first fix addressed the sentence, not the failure.

A regression test starts from the opposite direction: one bad output becomes a case that runs before every change. It does not prove the system is good. It proves the team will notice when the same failure returns.

A bad output is not a bug report yet

“The model gave a bad answer” cannot be run. A regression case needs enough structure that another reviewer can reproduce the failure:

• the exact input the user provided;
• the context the model received, including source versions;
• the expected behavior, stated as an observable property;
• the actual output, kept verbatim;
• the failure label and severity;
• the reviewer who classified it and the date.

Without the context version, the case may pass later for the wrong reason: the source changed, not the model. Without an expected behavior, the team will argue about tone instead of the failed property. Without a severity, every failure becomes equally urgent and the gate becomes ignored.

The case does not need the full production record. It needs enough to replay the decision without exposing personal data. Redact the customer name, keep the policy version, and store the input that actually drove the failure.

Turn the output into a case, not an anecdote

A useful template has six fields:

Field	What to record	Why it matters
Case ID	A stable identifier	Links the failure to the test and the fix
Slice	Refund, eligibility, owner attribution, etc.	Prevents the case from becoming a one-off
Input	The user message and relevant context	Makes the failure replayable
Expected property	What must be true, not the exact wording	Allows legitimate variation
Actual output	The verbatim failure	Shows what “bad” means
Severity	Critical, major, minor	Decides whether the gate blocks a release

The expected property is the hard part. “Should be better” is not testable. “Must cite the current refund policy version and must not state approval” is. A property can be checked by a rule, a rubric, or a human reviewer; the method matters less than the agreement about what counts as failure.

Example: the refund-policy answer that passed review

A support assistant answered: “Your annual plan is refundable within 30 days.” The cited page was a real policy—two versions old. The current policy used a 14-day window and required a prorated calculation after the first month.

The team added a case with three properties:

1. the answer cites the current policy version;
2. the answer does not claim the refund is approved;
3. the answer routes an edge case to a human.

The case runs as a small check, not a full answer-comparison. A rule verifies the presence of the current version ID and the absence of approval language; a reviewer reads the edge-case routing. The failure is now visible before a prompt change ships.

The important detail is that the test does not require the exact sentence. A different explanation passes if it preserves the three properties. This keeps the test useful when the model’s style changes.

Counterexample: the meeting notes that invented an owner

A second team tested meeting notes by comparing them with a reference summary. The model’s notes were fluent and close in wording, so the test passed. In production, the notes assigned an action to the wrong person.

The reference summary also contained the wrong owner, because it was written from memory. The test had reproduced the error instead of catching it.

The team replaced the reference with a transcript-grounded check:

• every action item must quote the line where the owner accepted it;
• an action with no explicit owner must be marked “unassigned”;
• a name may not be inferred from who spoke first.

This case shows why “compare to a golden answer” is not enough. The reference can be wrong, and a wording similarity score can reward the same mistake. Prefer properties that can be checked against the source.

The regression gate

A regression set becomes useful when it runs automatically. The team added a gate to three change types:

Change	Run	Block when
Prompt or system message	Full regression set	Any critical case fails
Model or version	Full set plus latency/cost check	Critical failure or cost ceiling exceeded
Retrieval source or chunking	Retrieval cases plus answer cases	A cited source is missing or stale
Output parser or schema	Format cases plus a sample of answer cases	Schema errors or silent field loss

The gate does not require a perfect average score. It requires zero critical failures and a reviewed explanation for major failures. Averages hide the one case that matters: the refund approval, the wrong owner, the leaked document.

The team also records a baseline. A new model can be better on average and worse on a critical slice. The release decision compares per-slice results, not a single number.

What to run on every change

Start with the failure that actually happened, then add two neighbors:

• a near miss that is structurally similar but not identical;
• a control that should keep passing.

For the refund case, the near miss might be a monthly plan with a different window. The control might be a simple billing-address change that should not mention refunds at all. This prevents the team from overfitting to one sentence.

Run the set on the frozen baseline and the candidate. Record:

• critical pass rate;
• pass rate by slice;
• false passes found by human review;
• reviewer disagreement;
• latency and cost, because a safer model that is ten times slower may not be deployable.

The test is not a one-time acceptance document. It is a living regression suite. When a case stops being relevant, retire it with a reason; when a new failure appears, add it before the next release.

When not to add a test

Not every bad output deserves a regression case. Skip it when:

• the task has no stable acceptance property (open-ended brainstorming, one-off creative work);
• the “failure” is a style preference that reviewers cannot agree on;
• the input cannot be stored or replayed without exposing data that should not be retained;
• the case would test a model behavior that the product does not rely on.

In those situations, use a review rubric or a qualitative sample instead. A regression suite full of ambiguous cases will be disabled by the team, which is worse than no suite.

Boundary

This workflow fits classification, extraction, policy answers, support drafts, and other tasks with observable acceptance properties. It does not replace a full evaluation program, human review, or security testing. It is not a guarantee that the system is safe; it is a guard against repeating a known failure. If the team cannot state the expected property or keep the context version, fix that before adding more cases.