AI inference latency debugging: trace the wait from request to screen

When someone says “the AI answer is slow,” the bottleneck may be the model, a client queue, connection setup, proxy buffering, a tool call, or page rendering. One end-to-end duration cannot tell you whether the first character is late, generation pauses mid-stream, a tool is waiting, or the answer already reached the browser but never appeared. Trace one request before choosing a fix.

This guide focuses on diagnosing latency for one request: build a timeline of observable stages and distinguish evidence from inference. It does not estimate how much concurrent traffic a local service can carry. For arrival rates, bursts, and sustained queue boundaries, use the local inference concurrency trial. For the first process divergence in an agent run, use the agent trace review.

Define when the user sees work begin and end

A request has at least three different “starts”: the service emits its first token, the client receives its first content, and the interface paints text on screen. The first byte in a streamed response might only be an HTTP header or event wrapper. Total completion time also depends on output length; a longer answer with normal generation speed is not necessarily slower inference.

Record a consistent request ID and timing boundaries for each stage:

Stage	Useful events	What it can answer
Client	Local queue, connection ready, request sent, first byte, first visible content, last byte, render complete	When did the request leave, and when could a person actually see and finish reading it?
Gateway and proxy	Request received, upstream forwarded, first chunk returned, buffering or retry	Was it limited, queued, retried, or held for batching?
Model service	Received, scheduled, queue when exposed, first token, stream events, complete	Does visible evidence support a claim about internal waiting or generation?
External tools	Start, end, status, tool name, and call ID for every call	Was time spent in search, a database, or another downstream service?
Result handling	Parsing, validation, post-processing, UI render complete	How long after model completion until the application showed the result?

Use a monotonic clock for durations on one machine so a system clock adjustment cannot produce a negative interval. When joining traces across hosts, retain clock sources and synchronization status. A parent span already includes its child spans; do not add both when summing user wait. Parallel branches can overlap in time as well.

Capture request shape and retries

Latency depends on input, output, and the execution path. Record the model and revision, input and output token counts, streaming mode, context size, tool-call count, cache use, cold-start state, region, HTTP status, and retry count. “About a thousand characters” is not a substitute for token count: retrieved context and images change the workload. Keep sensitive request content in an approved observability system and join it with sanitized IDs or controlled evidence pointers.

Time to first token (TTFT) at the client includes all waits before that event. If a model service exposes its request-queue duration, use it to explain part of that wait; the queue is already inside client TTFT and must not be added again. A long TTFT alone does not prove slow prefill. Network setup, cold loading, rate limiting, or a gateway retry can look similar.

Self-managed inference servers may expose queue, prefill, decode, and token-duration histograms. A hosted API may not reveal its internal spans. If server metrics are unavailable, label the stage “unobserved” instead of inferring what happened inside a provider from client duration. Evidence such as a 429 response, a retry at a particular proxy, or a server request ID is more useful than an invented cause.

Route from the earliest abnormal boundary

Compare request sent, service received, first byte, first token, screen display, and completion; then inspect errors and request shape. These patterns suggest the next measurement, not a final diagnosis:

Observed pattern	Check first	Still rule out
Long gap between user action and request send	Client work queue, concurrency limit, upload preparation	Blocked browser main thread, client retry
Request sent, but gateway forwards it late	Gateway queue, connection pool, rate limit, retry	DNS, TLS, or cross-region network
Service receives request late, then token generation proceeds normally	Exposed model queue, cold start, input length, prefill metrics	Whether server queue is already included in TTFT
First token is timely, then output repeatedly pauses	Stream event timestamps, output length, scheduler, tool handoff	Proxy flush behavior for particular chunks
Model emitted content but browser displays it late	Proxy buffering, application event parsing, main-thread rendering	Whether “first content” actually means first visible character
Model response is timely but the task completes late	Tool calls, retries, dependencies, post-processing	Whether tool wait overlaps model time

Find the earliest boundary that violates the target, then add a measurement that separates causes at that boundary. Segment distributions by model revision, input size, streaming, region, tool path, and cache state; one P95 that mixes unlike requests can hide a slow region or long-input slice. A single trace explains a failure path but cannot establish service capacity.

Example: a short question waits, then answers quickly

An internal product assistant takes a long time to begin showing a short answer, then finishes within seconds. The trace shows that the client sent promptly but the model service did not begin processing until later; this request also coincided with an instance cold start and visible loading activity. The reviewer can establish that “receive-to-first-token was extended while a load occurred.” One observation does not prove that cold start explains every slow request.

Next, run the same sanitized input several times on the first request to an instance and after warm-up, while recording service state and the same timestamps. If only the first request is delayed, set a separate cold-start expectation or change warm-up. If both are slow, inspect queue and input length. This diagnosis does not require adding a GPU first, and its result is not a capacity estimate.

Example: inference streams continuously, but the user sees one block

A support-summary service records continuous streaming events locally, yet the browser always receives visible content at the end. Client capture shows that events reached the application server promptly, while a reverse proxy buffered them until the response closed. Model generation is not the main source of the user’s wait; prompt edits or model quantization will not remove the proxy policy.

The team changes streaming behavior on a test endpoint, verifies that the first event reaches the browser early, and checks incremental rendering and cancellation. It then repeats the test on a slow network and with a long answer. If there are no event timestamps on both sides of the proxy, collect that evidence before concluding that buffering is the cause.

Choose a measurement that can disprove the guess

For the first delayed boundary, design a small comparison: what happens to the same request on a cold versus warm service, short versus long input, with versus without tools, and direct versus proxied? Change one condition at a time and keep model revision, content length, and sampling configuration fixed. Compare streaming and non-streaming separately because “response began” means different things to a user.

Every fix should have a rollback and retest trigger. More warm instances consume resident resources. Disabling proxy buffering changes connection occupancy. Splitting long requests may lose cross-section context. Compare end-to-end user wait, errors, and task quality on the same route before and after the change; do not declare victory from one internal timer.

Common questions

Is user-visible first-token delay the model’s TTFT?

No. Model-service TTFT, first byte at the client, first text received, and first screen paint are different events. Instrument each layer and state which one the product uses as its service target.

Is the experience good if the first token is fast?

Not necessarily. The answer may be long, streaming may stall, a tool may delay completion, or the interface may render late. Monitor first content, stream gaps, end-to-end completion, and task acceptance separately.

Boundary

Some hosted model services do not expose internal queue or prefill/decode timings. In that case you can identify unobserved stages, but you cannot attribute a black-box total to a hidden component. One trace helps select the next measurement; it does not replace concurrency testing, a tail-latency SLO, long-term trend analysis, or a safety review. Verify timer definitions and clock relationships before comparing across machines.