Run a ten-case AI pilot before you scale

A single successful example proves that a workflow can work once. It does not show how often it works, which inputs break it, or how expensive review becomes. Ten deliberately chosen cases create a better decision than a hundred convenient examples.

Build a set with four kinds of pressure

Use four ordinary cases, two difficult cases, two ambiguous cases, and two cases the workflow should refuse or hand off.

• Ordinary: common inputs that justify building the workflow.
• Difficult: long, noisy, incomplete, or unusual but still valid inputs.
• Ambiguous: inputs where two reasonable reviewers may disagree.
• Stop cases: missing permission, prohibited content, or an action outside the brief.

Record the expected behavior before running the workflow. Otherwise a fluent answer can quietly change the team’s definition of success.

Measure review work, not only output quality

For every case, record pass or fail, the reason, review minutes, retry count, and whether the fallback worked. A result that needs fifteen minutes of correction may be technically correct and operationally useless.

Do not average away a critical failure. Keep a separate list of conditions that block release even when the total score looks good.

Example: invoice field extraction

The ordinary set contains clear invoices from frequent suppliers. Difficult cases include a rotated scan and a page with several tax totals. Ambiguous cases include a credit note and a bilingual invoice. Stop cases contain a file without the required company identifier and a request from an unauthorized folder.

The pilot passes only if required fields are traceable to the source, ambiguous totals are flagged, and unauthorized files are rejected. “Nine out of ten fields correct” is not enough when the missing field controls payment.

Example: editorial outline assistant

Ordinary cases contain a clear audience and source packet. Difficult cases include conflicting sources and a narrow word budget. Ambiguous cases use a broad goal such as “make this more strategic.” Stop cases request claims without evidence or reuse restricted material.

Here, review time and unsupported claims matter more than exact wording. A draft can be imperfect and still useful if it exposes uncertainty and keeps source links intact.

Keep the ten cases from becoming a demo reel

Draw candidate cases from the input scope in the task brief, then assign the four categories before viewing model output. Do not label a case “difficult” or “ambiguous” after the result arrives. Record why it was selected, its original source, expected behavior, and unacceptable failure. If a case changes during the pilot, retain the earlier version so the team cannot quietly lower the difficulty.

Pre-register a short release rule. Name the failures that block continuation, the minimum behavior expected on ordinary cases, the maximum acceptable review effort, and the handoff state expected for stop cases. A rule written before execution prevents one fluent answer from moving the standard.

Field	Question to answer	Often-missed signal
Expected behavior	Should this answer, refuse, or hand off?	Describing an ideal answer but not a correct stop
Actual result	Which field, claim, or action changed?	Replacing observation with “looks good”
Failure type	Did input, context, capability, tool, or handoff fail?	Calling every problem a bad prompt
Review cost	How long did a person spend and what changed?	Counting inference while ignoring repair work
Recovery	Could the task still be completed after failure?	Sending to a person without useful state

One run cannot expose stochastic variation, while repeating every case may be wasteful. Prioritize repeat runs for consequential decisions, previously unstable cases, and recently repaired failures. Record configuration, time, and dependency versions. The purpose is an evidence-backed investment decision, not an attractive percentage.

Decide what happens after the pilot

• Proceed: ordinary cases pass, difficult cases are recoverable, and stop cases stop.
• Narrow: value exists, but one input class creates disproportionate failure or review work.
• Redesign: failures point to missing context, unclear acceptance, or a broken handoff.
• Stop: the output cannot be verified cheaply or the fallback is not safe.

Common questions about a ten-case pilot

Why ten cases instead of a larger sample?

Ten is not a statistical claim. It is a compact working format that forces ordinary, difficult, ambiguous, and stop behavior into the first review. Use it to screen a proposal and discover scope. Performance claims, close model comparisons, and consequential actions require a larger evaluation that represents real work.

Can the workflow continue when one case fails?

The failure type matters more than the total. A harmless formatting miss may enter a revision queue. Restricted-data exposure, a prohibited action, or an unrecoverable handoff should block release. Define those rules before running the set instead of averaging critical failures away.

When should the pilot become a formal evaluation set?

Once the task has earned further investment and the team needs version comparisons, regression checks, or production thresholds, turn cases into stable task slices. Add real frequency, scoring guidance, reviewer calibration, and version history. Preserve early failures because they often become the most valuable regression tests.

Boundary

Ten cases are a scoping instrument, not a production guarantee. Use them to decide whether the workflow deserves a larger, representative evaluation. Replace convenient examples with real distributions before making performance claims.