Design recovery before the agent fails

An automation plan usually describes the successful route in detail and compresses failure into “retry.” Recovery needs its own design: what stops, what state survives, who owns the incident, and which actions must not repeat.

The difficult condition is often not explicit failure but an unknown outcome. After a timeout, a message may have been sent, a payment may have reached the provider, or half the records may be changed. The runbook must help an operator establish external facts before choosing resume, compensation, or continued stop.

Write the runbook in six fields

• Stop signal: the observable event that pauses new work.
• Last safe state: the checkpoint from which recovery begins.
• Impact view: affected records, users, tools, and external actions.
• Owner: the person or role authorized to decide what happens next.
• Rollback or compensation: how completed actions are undone or balanced.
• Retry rule: which steps are idempotent, which need review, and which must never repeat automatically.

Store this information beside the workflow, not in a document nobody sees during an incident.

Example: bulk email preparation

An agent prepares and sends renewal reminders. The stop signal is a recipient mismatch or bounce rate above the agreed limit. Draft generation may be repeated; sending may not.

The last safe state stores the approved recipient list, message hash, and send ledger. Recovery suppresses already-sent recipients, routes disputed records to the account owner, and requires a new approval if message content changes.

Example: inventory synchronization

An agent reads stock from several systems and updates a storefront. One source becomes delayed and reports zero for every item. The stop signal is an abrupt coverage drop, not an API error.

The workflow freezes writes, keeps the last known values, and records which items were updated before the stop. Recovery compares the delayed source with a second signal before resuming. Blind retries would repeatedly publish the wrong zero.

Run a tabletop failure

Choose one credible failure and walk through the runbook without touching production. Ask what the operator can see after five minutes, whether credentials are available, and how they distinguish completed from pending work.

If the team must reconstruct state from logs during the exercise, add a checkpoint to the product.

Record the exercise outcome, owner, and next retest condition together.

Assign recovery semantics to every step

Break the workflow into observable steps and select one recovery behavior for each:

• Safe replay: the same idempotency identifier cannot create a second effect.
• Query before retry: inspect remote state after a timeout and submit again only when non-execution is confirmed.
• Compensating action: the original effect cannot be erased, but another controlled action can reduce harm.
• Human adjudication: the system cannot establish the correct state and must provide evidence to an owner.
• Permanent stop: a safety threshold makes this batch ineligible for automatic resume.

Do not treat an API’s “retry supported” label as proof of safe replay. Verify how duplicates are detected, how long identifiers live, whether changed parameters remain the same operation, and what concurrent requests do.

Copy this recovery-state record

Record field	State to retain
run_id / step_id	Stable identifiers for batch, object, and step
Input fingerprint	Critical parameters and versions actually used
External operation id	Key that can query authoritative remote state
Status	Pending, running, succeeded, failed, or unknown
Observed effects	Message sent, record changed, charge submitted, and so on
Next permitted action	Query, undo, compensate, hand off, or terminate
Takeover packet	Evidence, owner, timestamp, and decision

Success and failure are not enough states. Unknown must be explicit: it blocks automatic replay and requires reconciliation. Batch workflows need an object-level ledger so one failed aggregate does not replay objects already completed.

Define the resume gate

Before resuming, determine whether the trigger’s cause is known, affected objects are inventoried, incorrect effects are compensated, queued requests remain valid, credentials and dependencies are trustworthy, and focused tests pass. An authorized owner then chooses checkpoint continuation, partial regeneration, or cancellation.

Reopen with a constrained slice and watch the original stop signal plus human correction before expanding. Do not release an entire backlog because an endpoint looks healthy again. Old requests may have expired sources, approvals, or timing windows.

Common questions

How does compensation differ from rollback? Rollback restores the earlier state. Compensation accepts that the effect occurred and uses a new action—refund, withdrawal notice, or corrective record—to reduce impact. It also needs authority and audit.

How many automatic retries should a failure receive? Start with replay safety and failure class, not a universal count. A confirmed transient read may receive bounded retries; an external write with unknown outcome should receive zero blind retries.

When should the runbook be exercised again? Repeat after changes to actions, tools, identity, queue mechanics, or ownership. During stable operation, use a risk-based cadence and record findings plus the next trigger.

Boundary

A runbook is not a substitute for preventing dangerous actions. Remove unnecessary authority first. Recovery is the second line: it limits damage when assumptions, tools, or inputs still fail.