Open · structured · practical
Search 中文 Find your starting point

Evaluation

What evidence would make this workflow safe to expand?

Turn opinions about AI quality into representative cases, acceptance checks, failure categories, and a release decision another reviewer can reproduce.

AI VISTA / DECISION01 / 03
WORKING DECISIONTest ordinary and uncomfortable cases, then fix the failure source rather than the most visible symptom.
01INPUT
02CHECK
03BOUNDARY
RETURN TO THE REAL TASK✓

DECISION BRANCH

Do not begin with the tool name.

  1. ASK FIRST↓

    Is the missing piece a fact, a standard, or model capability?

  2. IF↓

    The result is verifiable and the failure cost is bounded, expand carefully.

  3. OTHERWISE✓

    Narrow the action, keep the human handoff, and preserve recovery.

WORKED CASE

Meeting notes read smoothly but assign actions to the wrong owner.

The team calls the notes ‘mostly usable’ without separating omissions, wrong ownership, invented deadlines, and style defects. Every iteration is judged from memory.

WORKING ARTIFACTAn evaluation-set blueprint with task slices, failure labels, severity, and reviewer notes.
01WEAK SHORTCUT

Choose ten clean recordings, ask one reviewer for a single overall score, and scale when the average rises.

02BETTER JUDGMENT

Sample routine meetings, interruptions, unclear ownership, and sensitive decisions. Count factual errors, omissions, attribution failures, and formatting issues separately.

03ACCEPTANCE

No consequential decision invents an owner or date, every action traces to the transcript, and two reviewers reach the pre-agreed pass or fail judgment.

ENTRY & EXIT SIGNALS

Know when to enter—and when the decision is good enough to leave.

A topic is not an endless knowledge directory. Entry signals tell you whether the problem belongs at this layer. Exit signals decide whether to continue instead of substituting time spent reading for work completed.

AI VISTA / DECISION STATUSEvaluation
01ENTER HERE
  • The team judges quality by overall impression
  • Different cases are used before and after a change
  • A score does not reveal which risks remain
02LEAVE WHEN
  • Real task slices and boundary cases are represented
  • Failure types, severity, and review rules are repeatable
  • The same set compares baseline and candidate

If failures concentrate in missing knowledge, version conflict, or unsupported claims, take the evaluation set into evidence work.

3 PRACTICES

Build the judgment, make the artifact, then review it with real cases.

01 Run a ten-case AI pilot before you scaleBuild a deliberately small test set with ordinary, difficult, ambiguous, and unsafe cases so a promising demo becomes evidence. Evaluation 02 Build an AI evaluation set that stays usefulSample real task slices, write reviewable references, calibrate graders, and version the set so model changes produce trustworthy comparisons. Evaluation 03 Diagnose an AI answer before changing the modelTrace a bad answer to missing input, conflicting instructions, weak evidence, capability limits, or a broken handoff. Evaluation

BOUNDARY

What this topic page will not do

An evaluation set supports a release decision; it is not a statistical guarantee. Consequential use still needs domain-owner review.

01Do not let one average score erase a critical failure02Do not test only demo cases while skipping refusal and boundary cases03Do not claim improvement without a repeatable review rule
When failure types, severity, and responses are reproducible, move into evidence, authority, or operations work.