How should an AI model service fail over? Plan retries, queues, and human handoff

When a model API becomes unavailable, an application needs more than “try again” or a silent switch to another model. A rate limit may clear soon or reflect an exhausted budget. An asynchronous drafting job may wait; a customer seeking an account explanation needs a truthful status and an owner. The aim of graceful degradation is a trustworthy result or an honest incomplete state for each request, not a dashboard that calls every attempted route a success.

This page decides how an application routes a failed inference request among bounded retries, admission control, a time-limited queue, an approved backup model, and human handling. The serving load trial measures capacity under arrivals. The agent recovery runbook addresses an external action that might already have taken effect. Switching a model cannot tell you whether an email was sent or a payment was submitted.

Classify the failure before choosing a route

Keep the HTTP status, provider error code, Retry-After, request and model identifiers, timing, and whether streaming output had already begun. A status code alone is not a retry policy. In the vendor documentation checked in October 2026, OpenAI describes a temporary 429 rate limit and a 503 model overload, but spend and usage limits may also return 429 and require a different action. Anthropic documents 529 for temporary overload and notes that some 429 responses can mean a monthly spend cap. These are provider-specific examples, not universal meanings for every model API.

Observation	Candidate response	Mistake to prevent
Confirmed temporary throttling or overload with time left	Respect a valid Retry-After; otherwise use a bounded delay; consider an approved alternate route	Retrying before the stated delay or multiplying attempts across layers
Credit, billing, authentication, permission, or invalid-input failure	Stop this route and ask its operator to resolve the cause; tell the user what remains available	Treating an intervention error as a transient one
Connection timeout or interrupted stream	Record what the user has already seen and apply the full-response acceptance rule	Joining two partial model outputs into one “completed” answer
HTTP success with an invalid schema or unacceptable content	Reject the result under the task’s quality rule	Counting transport success as task completion

A stream can fail after an HTTP 200 response. If the user has already seen a partial answer, mark it interrupted. Decide in advance whether the product may clear it and start a new answer; do not append another model’s continuation as though it were one validated result. If inference triggers a tool with side effects, use the recovery runbook to check the external state before considering any replay.

Budget retries from the user’s deadline

Give each task slice an end-to-end deadline, leaving time to validate the output, update the user, and prepare a handoff. A timeout for one HTTP attempt is not the whole task deadline. If Retry-After exceeds the remaining useful time, defer or take another authorized route instead of retrying early to keep a spinner moving. Without a valid server hint, increasing delays with some randomness can be reasonable, but cap both attempts and elapsed time. Official OpenAI and Anthropic SDKs can retry some transient failures automatically; check the installed SDK version so its attempts and your application attempts share one budget.

For persistent failures, an operator can set an observation window and threshold that opens a circuit breaker: new requests quickly enter the fallback decision instead of hammering the failing dependency. After a cooldown, admit a small number of probes and expand only after recovery evidence. Choose thresholds from the product’s traffic and deadlines, not from a copied universal number. Bound intake as well. When every permitted path is full, a visible rejection or delay is better than an unbounded invisible queue.

Approve a backup on quality and data separately

Approve a backup route before an incident and for a particular task slice. Run the same held-out cases through primary and backup paths. Check required fields, factual support, no-answer behavior, structured-output validity, critical slices, severe errors, latency, and cost per accepted task. A prompt, tool call, or JSON contract that works on the primary may behave differently on the backup. Record revisions and blocking failures in the model selection scorecard, and rerun the cases after a model or prompt change. Passing easy examples does not qualify a backup for all traffic.

Then check each field against the data boundary map: destination, provider, region where relevant, logs, retention, and connected tools. Even a different endpoint from the same provider may have different controls. If the required account fields may not leave the current approved environment, an outage does not authorize sending them to a new API. A missing quality or data approval keeps that route closed. The product may still give a fixed, nonpersonal status and transfer the case to a qualified person.

Make a task-specific fallback ladder

These are conditional branches, not a five-step sequence every request must traverse. For each route, name its signal, maximum duration, eligible work, user-visible state, and exit gate.

Route	When it can be used	What must remain visible
Bounded retry on the primary	A recoverable error, time left, and attempts remaining	Processing state, original request ID, attempt count, server hint
Expiring asynchronous queue	The user accepted later delivery and the task stays useful until expiry	Job ID, status lookup, cancellation, expiry and owner
Evaluated backup model	This task slice passed quality and data gates; backup capacity and budget remain	Actual model revision, validation result, known limitations
Human handling or explicit unavailability	Risk blocks substitution, or all useful automated paths failed	Verified facts, missing result, owner, next contact time

One task may queue briefly and then move to a person. Another may skip the queue and go straight to an approved backup. The route follows the task and current evidence, not a ranking by model price. Include retries, waiting, review, and abandoned work in the workflow cost map.

Give a queue capacity, expiry, and an exit

Queues suit drafts, batch transformations, and reports that can finish later. Azure’s queue-based load-leveling guidance warns that a sustained producer rate above the consumer rate makes backlog and latency grow. It also notes that a low-latency synchronous caller may be a poor fit. Define maximum pending work, maximum age, job expiry, worker concurrency, and downstream rate. Track the oldest job and the share past its user deadline, not just the queue count.

Enqueue a stable job ID, expiry, minimal necessary payload or controlled reference, and a status the user can query and cancel. A browser refresh should not create a second copy of the same draft. Sensitive raw input should not sit indefinitely in the queue. Before draining a recovered backlog, recheck source versions and user intent. Expired, canceled, or approval-dependent jobs must not restart merely because the model endpoint responds again. Publishing, sending, and charging remain separate permission and recovery decisions.

Example 1: public event copy in three languages

This is a synthetic rehearsal. An events team turns already public venue, date, and registration rules into draft copy in three languages; an editor still publishes it. The team prepares held-out cases with changed dates, capacity limits, negated rules, and an absent venue. A backup route must preserve those facts and treat an invented venue or deadline as a blocking failure. Only after that matched test may the backup handle this public-draft slice.

Suppose the primary API returns a temporary 429. The router reads the specific error and any valid Retry-After, then compares the wait with the remaining editorial deadline. If waiting still leaves review time, it can retry within the total budget. Otherwise an explicitly asynchronous job can enter an expiring queue or use the preapproved backup. An expired job becomes an incomplete case reported to the editor rather than outdated copy generated later. These are rehearsal branches; no vendor availability or model pass rate is being claimed here.

Example 2: private account charge explanation

In a second synthetic task, a support worker explains a charge using account identifiers and internal adjustment records. The primary API reports temporary overload, but a backup provider is not authorized to receive those fields. Even a fast, high-scoring backup fails the data gate. The interface says the charge needs a human check and offers a support handoff; its packet contains a controlled case ID, the failure class, and a pointer to the verified source, not the full account record copied into a general queue.

The rehearsal also cuts a stream after half a sentence is displayed. The partial explanation is not a final answer, and a second model must not guess the ending. The support worker uses the authoritative billing system to respond. If another provider is approved later, the team still checks its exact endpoint, retention controls, and task cases before making the route available. An emergency workaround is not automatically a production policy.

Rehearse failure and return to the primary gradually

Use the workbench below for one task’s ladder. Inject temporary 429, spend-cap 429, 503 or 529, a single-attempt timeout, an error after HTTP 200 streaming begins, invalid backup output, prohibited data transfer, an expired queue item, and recovery while a backlog remains. Count original arrivals as well as accepted completions, timeouts, cancellations, human transfers, and explicit rejections; a success-only denominator hides the people who received no result. Watch user wait distribution, oldest queued job, serious backup-quality failures, and cost per accepted result.

Once the primary returns, send controlled probes before routing normal traffic back. Verify error behavior, latency, and output acceptance, then increase traffic gradually. Dumping the whole backlog may recreate throttling and crowd out new work. A critical quality failure, changed data route, new backup revision, or persistent backlog should pause the affected branch for another rehearsal. Ongoing model-quality changes can be tracked with the drift watchboard; this ladder records which delivery route is permitted during a service incident.

Questions teams ask

The backup is faster and cheaper. Can we turn it on during an outage? First show that this task slice passes quality, format, and data-flow gates. One impressive ordinary answer is not evidence for critical failures.

Should every 429 wait in a queue? No. Inspect the provider’s error body for temporary throttling versus spend or billing intervention. Even a recoverable request should expire if waiting makes its answer useless.

Can work continue after the user closes the page? Only if the product truly supports persistent asynchronous delivery, lookup, cancellation, and expiry. Otherwise mark this attempt incomplete rather than promising a result nobody can track.

Boundary

This ladder does not prove a provider’s availability guarantee and does not replace load testing, model evaluation, privacy review, or incident response. It governs inference delivery for a defined task. External effects such as tool writes, payments, and email still need their own authorization and recovery rules. Error mappings, SDK retries, and provider data controls change; verify the actual API, model, account, and version before deployment and rehearse again after material changes.