Build an AI evaluation set that stays useful

An evaluation set is not a gallery of prompts that once made a model look impressive. It is a maintained instrument for deciding whether a specific workflow is ready to change. To serve that purpose, its cases must resemble the work people actually bring, its expected results must be reviewable, and its score must expose important failures instead of hiding them inside an average.

A ten-case pilot can reveal whether an idea deserves more work. A durable evaluation set answers a later question: did this prompt, model, retrieval source, or tool change make the operating workflow better without breaking a critical slice?

Name the release decision first

Write the decision the set will support. Examples include “replace the current classifier,” “allow drafts to reach the review queue,” or “update the retrieval index.” Without a decision, teams collect cases indefinitely and cannot explain what a score permits.

Define the unit of evaluation at the same time. One unit might be a customer request plus its assigned route, or a question plus an answer and cited passages. Do not mix whole conversations, single messages, and extracted fields under one denominator. They fail in different ways.

Set non-negotiable gates before choosing metrics. A higher average score cannot compensate for exposing a restricted document, inventing a refund promise, or taking an irreversible action. These failures deserve their own count and a release stop rule.

Sample slices from the work, not from memory

Choose a recent traffic window that represents the use case. Remove duplicates and incidents that belong to a different workflow, then label meaningful slices. Useful slices often include language, input length, task subtype, source availability, ambiguity, urgency, and risk level. Select only dimensions that could change a release decision.

Give each slice a target share or a minimum case count. A simple random sample can omit rare cases that carry most of the harm. Deliberately include those cases, but keep their results separate from estimates of ordinary traffic. That preserves both operational realism and safety visibility.

Keep a sealed holdout for the final comparison. If every failed case becomes a prompt example and remains in the test set, the system can improve on familiar questions without becoming more dependable on new ones.

Write references that allow more than one good answer

For extraction, routing, and calculation tasks, an expected value may be exact. For drafting and question answering, a single “gold answer” can punish valid alternatives. Record required claims, permitted evidence, forbidden claims, and acceptable uncertainty instead.

Create a short reviewer guide with examples of pass, partial, and fail. Then have two reviewers independently score a small shared sample. Discuss disagreements until the guide—not private intuition—contains the distinction. Repeat calibration when the task or reviewer pool changes.

Automated graders can help with scale, but first compare them with the calibrated human decisions. Track where the grader is too lenient, too strict, or sensitive to style. A fluent explanation from a grader does not prove that its judgment matches the workflow’s acceptance rule.

Example: route online-shop returns

An operations team wants to route return requests into standard return, damaged item, possible fraud, or human review. Historical data is dominated by ordinary returns, while the costly mistakes live in sparse combinations such as a delivery dispute plus a high-value item.

The team samples recent requests and defines slices for route, evidence completeness, order age, language, and suspected abuse. The reference is not the previous automation label; two experienced agents re-check the policy and record the intended route plus the decisive evidence. A wrong standard-return route on a possible-fraud case is a release-blocking failure, even if overall accuracy rises.

After launch, overturned routes enter a candidate pool. They are not silently appended: an owner reviews the case, assigns a slice, removes unnecessary personal data, and records which evaluation version first contains it.

Example: answer questions from internal policy

A policy assistant must answer from approved documents and show support. Its evaluation unit contains a question, the allowed document version, the required conclusion, supporting passage locations, and any qualification the answer must preserve.

The set includes straightforward lookups, questions whose answer changed between versions, conflicts between documents, missing-answer cases, and requests from someone without permission to see the source. The team reports answer support, citation correctness, qualification preservation, and appropriate refusal separately. One blended score would conceal a system that sounds accurate while citing the wrong version.

When a policy changes, affected cases receive a new reference and version note. Old cases remain available for reproducing earlier releases, but they no longer count toward the current decision.

Start with a usable evaluation blueprint

The first version does not need to be large. It needs to cover the decision path and make catastrophic failure visible. Give every case the following fields:

Field	Purpose	Detail teams often miss
case_id	Connect results, discussion, and remediation	Do not use the full question as an identifier
task_slice	Mark task family and risk segment	One case may carry several labels
input_snapshot	Preserve what the workflow actually received	Include system instructions, context, and tool results
expected_behavior	State what must and must not happen	Allow multiple valid phrasings in open-ended work
blocking_rule	Say whether the case can stop release	Link to a numbered acceptance rule
reviewer_note	Explain the edge judgment	Record how disagreement was resolved
source_version	Freeze the valid reference material	Never overwrite a case silently when sources change

On the first run, review every case manually and confirm that the harness sends the complete input to the intended workflow version. Only after those results align should an automated grader take over repetitive decisions. This catches the subtle situation where the scoring code works perfectly while measuring something other than the system being released.

Read an evaluation report as a release record

The opening should answer four questions: which version was tested, which baseline it was compared with, which gates passed, and which release decision followed. Slice results, failed cases, and reviewer disagreements belong after that summary.

When an aggregate rises, identify the slices responsible. When it falls, check whether the dataset gained harder, more representative cases. Raw scores from different dataset versions are not directly comparable. Every blocking failure should name the case identifier, violated rule, likely reach, owner, and retest condition instead of merely pasting a model response.

Common questions

How often should cases change? Expand by event, not by quota. A new business branch, input format, source revision, or production failure mechanism deserves a versioned update.

Can another model grade the answers? It can assist after a calibration sample shows alignment with human judgment on critical rules. Record both the grader model and its instruction version with the run.

Should every negative production report become a test? No. Confirm that it belongs to the task, remove irrelevant sensitive material, and establish a reference decision first. It may enter the regression set, a watch set, or remain evidence for one incident.

Operate the set as a versioned asset

Every evaluation run should record the dataset version, workflow version, configuration, grader version, and date. Add cases when production reveals a genuinely new task slice or failure mechanism. Retire or rewrite cases when the task disappears, the source becomes invalid, or the wording leaks into training and prompt examples.

Review slice coverage on a schedule. More cases are not automatically better; a smaller set with known provenance and current references is more useful than a large folder no one can explain. Publish the result by slice, include the blocking failures, and state the decision taken. That creates a trail a later reviewer can reproduce.

Boundary

An evaluation set estimates behavior inside its sampled task definition. It does not certify a model for unrelated users, languages, tools, or risk levels. It also cannot reveal failures that the sampling and acceptance rules never name.

Do not treat benchmark scores or a vendor’s aggregate result as a substitute for workflow cases. Do not reuse production inputs without permission and minimisation. For high-impact decisions, domain, safety, privacy, and legal owners may require independent testing beyond this blueprint. If reviewers cannot agree on what counts as acceptable, pause the model comparison and repair the task contract first.