Make AI answers reliable enough to use
Preserve the right context, define content and format gates, test untrusted and difficult inputs, then diagnose failures in evidence, instructions, parsing, or handoff.
BEFORE LESSON ONE
Bring real material so the course produces a real result.
- 01 · WHO IT IS FOR
- For teams whose prompt works in a demo but changes quality across real inputs.
- 02 · WHAT TO BRING
- Bring one prompt, three good outputs, and three disappointing outputs.
- 03 · HOW TO FINISH
- Each failed case is assigned to a fixable category instead of a vague quality complaint.
WHEN THIS COURSE FITS
Start here when any of these situations is true.
You do not need a level label. Enter when the work shows one of these signals, bring real material, and use course acceptance to decide when you are done.
- 01✓
The demo works but real inputs produce variable answers
A live version exists, yet quality changes with documents, users, or formats.
- 02✓
Every failure leads to another prompt edit
The team does not separate missing context, conflicting acceptance, model capability, and broken handoff.
- 03✓
A model switch is proposed without a fixed baseline
You need the same task slices and failure labels to judge whether a change is actually better.
COURSE MILESTONES
Every stage leaves a change someone else can review.
These milestones are not another reading list. They show whether the lessons form one piece of work: real material at the start, connected artifacts in the middle, and evidence another person can reproduce at the finish.
- 01 BEFORE↓
Freeze the current version before changing the prompt.
Save the live prompt, three satisfying outputs, three disappointing outputs, and the complete input that produced each one. This becomes the baseline for judging improvement.
Starting evidence: every good and bad case can be reproduced with its original context. - 02 MIDPOINT↓
Turn quality complaints into diagnosable failures.
Map context, write the acceptance contract, test untrusted content with paired normal and injected cases, and build an evaluation set across real task slices.
Midpoint evidence: every acceptance rule has at least one case and one failure label. - 03 FINISH✓
Rerun the same set and look for newly introduced regressions.
After fixing the leading failure source, compare against the frozen baseline. Define when changing inputs, dependencies, or human overrides should trigger another evaluation.
Completion evidence: gains trace to a change, failures have causes, and drift signals have owners.
- 01✓LESSON 1
Map the context before rewriting the prompt
Separate instructions, source material, examples, history, tools, and output rules to see what the model actually receives.
LESSON OUTPUT · context mapUnreadOpen lesson → - 02✓LESSON 2
How many tokens should one LLM call reserve for input and output?
Count the next complete request, reserve room for tool results and all generated tokens, then check model input, output, and context limits before sending it.
LESSON OUTPUT · per-call context budgetUnreadOpen lesson → - 03✓LESSON 3
When should you compact a long AI conversation?
Set a compaction checkpoint from measured context use, preserve decisions and unresolved actions, then test whether the next turn still follows the facts.
LESSON OUTPUT · compaction checkpointUnreadOpen lesson → - 04✓LESSON 4
Write the acceptance contract before the prompt
Define required fields, evidence, tone, uncertainty, and rejection conditions before optimizing prompt wording.
LESSON OUTPUT · acceptance contractUnreadOpen lesson → - 05✓LESSON 5
Choose JSON Schema, Markdown, or plain text for AI output
Match an AI response format to its downstream consumer, then route refusals, truncation, parser errors, schema errors, and factual errors separately.
LESSON OUTPUT · output protocol checkUnreadOpen lesson → - 06✓LESSON 6
Test prompt injection in documents and tool outputs
Build a regression set for untrusted pages, files, retrieval snippets, and tool results; check intended behavior and permission gates before release.
LESSON OUTPUT · injection test setUnreadOpen lesson → - 07✓LESSON 7
Run a ten-case AI pilot before you scale
Build a deliberately small test set with ordinary, difficult, ambiguous, and unsafe cases so a promising demo becomes evidence.
LESSON OUTPUT · pilot sheetUnreadOpen lesson → - 08✓LESSON 8
Build an AI evaluation set that stays useful
Sample real task slices, write reviewable references, calibrate graders, and version the set so model changes produce trustworthy comparisons.
LESSON OUTPUT · evaluation-set blueprintUnreadOpen lesson → - 09✓LESSON 9
How do you calibrate an AI grader against human review?
Compare rule checks, an LLM judge, and independent human decisions on the same outputs; find false passes by task slice before trusting automated scores.
LESSON OUTPUT · grader calibration packUnreadOpen lesson → - 10✓LESSON 10
How to set AI classification thresholds with a human review queue
Route calibrated binary probabilities into automatic approval, automatic rejection, and human review using error costs and reviewer capacity, then validate the rule on held-out cases.
LESSON OUTPUT · classification routing sheetUnreadOpen lesson → - 11✓LESSON 11
How to test LLM temperature and sampling settings
Compare sampling controls on repeated, frozen cases; measure task errors and useful variation without assuming temperature zero guarantees identical answers.
LESSON OUTPUT · sampling settings trialUnreadOpen lesson → - 12✓LESSON 12
Diagnose an AI answer before changing the model
Trace a bad answer to missing input, conflicting instructions, weak evidence, capability limits, or a broken handoff.
LESSON OUTPUT · diagnostic treeUnreadOpen lesson → - 13✓LESSON 13
The eval that would have caught this: turn one bad output into a regression test
A practical regression workflow that turns one production failure into a labeled case, a pass condition, and a gate before the next prompt or model change.
LESSON OUTPUT · evaluation-set blueprintUnreadOpen lesson → - 14✓LESSON 14
Monitor an AI workflow before drift becomes an incident
Watch task mix, inputs, dependencies, outcomes, and human overrides against a named baseline so every alert leads to an inspectable response.
LESSON OUTPUT · drift watchboardUnreadOpen lesson →
COURSE ACCEPTANCE
Each failed case is assigned to a fixable category instead of a vague quality complaint.
Reading is not the finish line. Assemble the lesson outputs and ask another person to review them against the standard above.