Automate with permission, recovery, and cost controls
Build a bounded tool-using workflow with control ownership, data and action authority, stop rules, human review, recovery, service fallback, cost, and drift checks. Test delegation, MCP, memory, models, and deployment only when your task needs them.
BEFORE LESSON ONE
Bring real material so the course produces a real result.
- 01 · WHO IT IS FOR
- For teams moving an agent or tool-using workflow from prototype toward production.
- 02 · WHAT TO BRING
- Bring a diagram of the current workflow, its tools, owners, and irreversible actions.
- 03 · HOW TO FINISH
- Every automated action has an owner, approval rule, observable result, and recovery path. A reviewer can validate the core controls and the branch evidence relevant to the task.
WHEN THIS COURSE FITS
Start here when any of these situations is true.
You do not need a level label. Enter when the work shows one of these signals, bring real material, and use course acceptance to decide when you are done.
- 01✓
An agent is about to change external state
The workflow writes to systems, notifies people, submits payment, or triggers irreversible action.
- 02✓
Human review exists but happens too late
The approver cannot see evidence or changes and upstream errors can no longer be contained.
- 03✓
The system can succeed but cannot fail safely
Duplicate execution, partial success, timeouts, and drift do not yet have recovery owners.
COURSE MILESTONES
Every stage leaves a change someone else can review.
These milestones are not another reading list. They show whether the lessons form one piece of work: real material at the start, connected artifacts in the middle, and evidence another person can reproduce at the finish.
- 01 BEFORE↓
Inventory actions and decide who chooses the next step.
Use real cases to compare fixed branches with a bounded loop. Then record the tool, owner, impact, and reversibility for reads, drafts, writes, notifications, and payments.
Starting evidence: the control-loop choice has case evidence and each irreversible action has an owner and approval route. - 02 MIDPOINT↓
Design authority, tool contracts, human review, and recovery together.
Set data boundaries and the permission ladder. For each tool, define arguments, preconditions, effects, and result states before placing human review where it can still alter the outcome.
Midpoint evidence: every tool action has validation, approval, repeat behavior, and a reviewable result. - 03 FINISH✓
Rehearse failure in a controlled setting, then calculate operating cost.
Simulate a timeout, duplicate request, and rejected approval. Once recovery works, include retries and review in cost. Run model, local serving, or cache trials only when the task requires those branches.
Completion evidence: the drill recovers, relevant branches have measurements, and cost and drift signals have owners.
HOW TO USE THIS COURSE
Complete the control path, then choose the branches your task needs.
The syllabus lists every lesson. Its progress bar counts read marks; a cloud API workflow does not need local VRAM, quantization, or concurrency trials. Finish by passing the control and recovery checks for your real task.
- 01 · 12 lessons
Core control path
For every tool-using workflow: control ownership, data boundaries, authority, stop rules, human review, recovery, and operating cost.
Open this path → - 02 · 11 lessons
If needed: delegation and durable state
Use the relevant lessons when the task needs a provider-specific tool loop, multiple agents, memory, long runs, trace debugging, MCP, reusable skills, or executable plugins.
Open this path → - 03 · 10 lessons
If needed: model and deployment
Run the scorecard when changing models; as needed, test memory, quantization, adapters, engine compatibility, load, and GPU replicas. Test caching only for reusable prefixes.
Open this path → - 04 · 4 lessons
If needed: implementation cases
Read one case when your workflow resembles it; each case shows the review gate, the rejected version, and the recovery path.
Open this path →
- 01✓LESSON 1 · CORE PATH
Workflow or AI agent? Choose the control loop before building
Decide when a fixed AI workflow is enough and when an agent needs bounded control, tool feedback, stop rules, and a human handoff.
LESSON OUTPUT · control-loop decisionUnreadOpen lesson → - 02✓LESSON 2 · CORE PATH
Draw the data boundary before AI sees the input
Route every field through allow, transform, isolate, or exclude so a useful workflow does not quietly become an uncontrolled data transfer.
LESSON OUTPUT · data boundary mapUnreadOpen lesson → - 03✓LESSON 3 · CORE PATH
Give an agent the smallest useful permission
Place each action on a five-level permission ladder, from read-only suggestions to explicitly approved irreversible work.
LESSON OUTPUT · permission ladderUnreadOpen lesson → - 04✓LESSON 4 · CORE PATH
How do you test an AI agent sandbox's file, network, and credential boundaries?
Run observable allow-and-deny probes for an agent execution environment's mounts, egress, commands, resource limits, and credentials before it handles untrusted work.
LESSON OUTPUT · sandbox boundary trialUnreadOpen lesson → - 05✓LESSON 5 · CORE PATH
Design an AI agent tool contract before exposing the function
Specify an agent tool's input schema, preconditions, authorization, side effects, idempotency, timeout results, and trace before making it callable.
LESSON OUTPUT · tool-calling contractUnreadOpen lesson → - 06✓LESSON 6 · CORE PATH
When should an AI agent continue, retry, or stop?
Use observable progress, separate turn and cost budgets, error classes, and duplicate-action checks to govern each step of an agent run.
LESSON OUTPUT · runtime loop guardUnreadOpen lesson → - 07✓LESSON 7 · CORE PATH
Put human review where it can change the outcome
Choose review gates from impact, reversibility, and uncertainty, then give reviewers the evidence and actions needed to make a real decision.
LESSON OUTPUT · review gate mapUnreadOpen lesson → - 08✓LESSON 8 · CORE PATH
Design recovery before the agent fails
Write the stop signal, saved state, owner, rollback action, and safe retry rule before automation reaches production.
LESSON OUTPUT · recovery runbookUnreadOpen lesson → - 09✓LESSON 9 · CORE PATH
How should an AI model service fail over? Plan retries, queues, and human handoff
Route a failed model request by error type, user deadline, data boundary, and an evaluated backup model instead of treating every outage as a retry.
LESSON OUTPUT · model service fallback ladderUnreadOpen lesson → - 10✓LESSON 10 · CORE PATH
Map the full cost of an AI workflow
Count input, output, retries, tools, waiting, review, and failure recovery instead of comparing a single token price.
LESSON OUTPUT · cost mapUnreadOpen lesson → - 11✓LESSON 11 · CORE PATH
Your AI agent's real cost is cost per accepted task
A cost model that counts tokens, retries, tools, human review, and failure recovery per accepted outcome—not per API call.
LESSON OUTPUT · cost mapUnreadOpen lesson → - 12✓LESSON 12 · CORE PATH
Monitor an AI workflow before drift becomes an incident
Watch task mix, inputs, dependencies, outcomes, and human overrides against a named baseline so every alert leads to an inspectable response.
LESSON OUTPUT · drift watchboardUnreadOpen lesson → - 13✓LESSON 13 · IF NEEDED
How to implement an AI agent tool-calling loop
Build a provider-aware round trip that assembles complete calls, validates before execution, pairs every result, and resumes from durable state.
LESSON OUTPUT · agent loop execution contractUnreadOpen lesson → - 14✓LESSON 14 · IF NEEDED
Subagent or multi-agent? Choose the delegation topology
Compare one agent, a manager calling specialists, a conversation handoff, and parallel delegation by control ownership, shared state, cost, and failure recovery.
LESSON OUTPUT · agent delegation topologyUnreadOpen lesson → - 15✓LESSON 15 · IF NEEDED
When should an AI agent save, correct, or delete long-term memory?
Design a cross-session memory rule for what to save, how to handle a changed preference, and how to verify deletion across retrieval and retained copies.
LESSON OUTPUT · memory lifecycle contractUnreadOpen lesson → - 16✓LESSON 16 · IF NEEDED
How can a long-running AI agent task resume without repeating work?
Design a durable checkpoint, approval and operation ledger, and single-owner resume gate for agent jobs that outlive one process or wait for a person.
LESSON OUTPUT · long-task resume-state contractUnreadOpen lesson → - 17✓LESSON 17 · IF NEEDED
How to debug one AI agent run from its observable trace
Reconstruct a single agent run with spans, tool call IDs, approval changes, and external state to find the first evidenced divergence without exposing sensitive trace data.
LESSON OUTPUT · single-run trace reviewUnreadOpen lesson → - 18✓LESSON 18 · IF NEEDED
What should you check before connecting a third-party MCP server?
Vet a third-party MCP server's discovered tools, OAuth or local credentials, data destinations, version changes, failure behavior, and revocation before an agent can use it.
LESSON OUTPUT · MCP server intake recordUnreadOpen lesson → - 19✓LESSON 19 · IF NEEDED
When should a repeated AI task become an agent skill?
Decide whether a recurring task has a stable trigger, inputs, steps, acceptance test, and owner before packaging it as a reusable agent skill.
LESSON OUTPUT · agent skill decision cardUnreadOpen lesson → - 20✓LESSON 20 · IF NEEDED
How do you load and unload an agent plugin safely?
Test registrations, event errors, resource cleanup, two hot reloads, disablement, partial initialization, and external effects before shipping an agent extension.
LESSON OUTPUT · plugin release acceptance recordUnreadOpen lesson → - 21✓LESSON 21 · IF NEEDED
Your MCP server works locally. What breaks when a team depends on it?
A production-readiness review for MCP servers covering per-user credentials, tool allowlists, tenant boundaries, rate limits, audit logs, and failure isolation.
LESSON OUTPUT · MCP server intake recordUnreadOpen lesson → - 22✓LESSON 22 · IF NEEDED
Claude Code without the chaos: a review-first workflow for coding agents
A team workflow for coding agents that gates the task, the diff, and the runtime before a human merges—without turning review into a rubber stamp.
LESSON OUTPUT · review gate mapUnreadOpen lesson → - 23✓LESSON 23 · IF NEEDED
Log the decision, not the prompt: a trace schema for AI agents
A practical trace schema for agent runs that records identity, tool effects, approvals, state changes, and costs—without storing secrets or full prompts.
LESSON OUTPUT · single-run trace reviewUnreadOpen lesson → - 24✓LESSON 24 · IF NEEDED
Choose a model with a job-specific scorecard
Weight quality, latency, tool use, privacy, recovery, and operating constraints using cases from the workflow you will ship.
LESSON OUTPUT · selection scorecardUnreadOpen lesson → - 25✓LESSON 25 · IF NEEDED
How much VRAM does a local LLM need? Budget weights and KV cache
Estimate raw model weights and KV cache separately, test context length and concurrency, then measure the missing peak costs on your serving stack.
LESSON OUTPUT · VRAM budget and peak testUnreadOpen lesson → - 26✓LESSON 26 · IF NEEDED
Is 4-bit LLM quantization worth it? Test fit, quality, and speed
Compare an unquantized BF16 reference with a named 4-bit package on fixed task cases and serving load before trading quality for memory.
LESSON OUTPUT · BF16 versus 4-bit trialUnreadOpen lesson → - 27✓LESSON 27 · IF NEEDED
LoRA or QLoRA fine-tuning? Run a small, reviewable pilot
Admit a stable behavior problem, split training data without leakage, probe real training memory, and compare adapters with a frozen baseline on unseen cases.
LESSON OUTPUT · LoRA/QLoRA pilot recordUnreadOpen lesson → - 28✓LESSON 28 · IF NEEDED
Ollama, llama.cpp, vLLM, or SGLang? Choose a local inference engine
Shortlist a local LLM server by model package, hardware, API behavior, and operating needs, then run the same acceptance and service trial before committing.
LESSON OUTPUT · inference engine selection trialUnreadOpen lesson → - 29✓LESSON 29 · IF NEEDED
How many concurrent requests can a local LLM serve?
Find the highest tested arrival rate that meets first-token, output-speed, queue, error, and memory targets on your actual inference server.
LESSON OUTPUT · inference serving load trialUnreadOpen lesson → - 30✓LESSON 30 · IF NEEDED
AI inference latency debugging: trace the wait from request to screen
Separate client, gateway, model, tool, and rendering delays in one request trace before changing a model or adding hardware.
LESSON OUTPUT · inference latency traceUnreadOpen lesson → - 31✓LESSON 31 · IF NEEDED
How many GPUs does a local LLM service need?
Turn a tested per-replica arrival rate into a candidate replica count, then verify GPU placement, failure capacity, warm-up time, and rollout peaks.
LESSON OUTPUT · GPU replica capacity worksheetUnreadOpen lesson → - 32✓LESSON 32 · IF NEEDED
Local LLM or cloud API? Choose by data, peak load, and operations
Compare owned hardware, a team-managed rented GPU, and a managed model API using data paths, measured peak capacity, operating ownership, and cost per accepted task.
LESSON OUTPUT · inference deployment matrixUnreadOpen lesson → - 33✓LESSON 33 · IF NEEDED
Why prompt caching misses: run a three-request test
Measure cache writes, reads, latency, and accepted-task cost with a stable prefix and a changed-prefix control, using provider-specific rules.
LESSON OUTPUT · cache trialUnreadOpen lesson → - 34✓LESSON 34 · IF NEEDED
AI support triage: the 10-case pilot that earned a review gate
A ten-case support triage pilot that separates queue accuracy from authority and turns a wrong refund promise into a review gate.
LESSON OUTPUT · pilot sheetUnreadOpen lesson → - 35✓LESSON 35 · IF NEEDED
AI content review: the citation gate that stopped a confident wrong answer
A content team's citation gate for AI drafts: every material claim needs a current source, version, owner, and a pass, revise, or block decision.
LESSON OUTPUT · citation auditUnreadOpen lesson → - 36✓LESSON 36 · IF NEEDED
AI research synthesis: when the source changes after the brief
A source-change drill for AI research briefs: re-run retrieval, mark dependent claims stale, and hand off a no-answer note when evidence is missing.
LESSON OUTPUT · retrieval trialUnreadOpen lesson → - 37✓LESSON 37 · IF NEEDED
The approval agent that could move money—until we removed the payment tool
An invoice-approval agent that prepares packets but cannot pay, with approval bound to invoice scope and drills for duplicate and unknown payment outcomes.
LESSON OUTPUT · permission ladderUnreadOpen lesson →
COURSE ACCEPTANCE
Every automated action has an owner, approval rule, observable result, and recovery path. A reviewer can validate the core controls and the branch evidence relevant to the task.
Assemble the core artifacts and the branches that apply, then ask a colleague to review them. Deployment trials that do not apply are not a course gate.