How many concurrent requests can a local LLM serve?

A model that fits on one GPU can still make a team wait too long. The VRAM budget checks a necessary memory condition; the 4-bit trial checks whether a smaller package preserves task quality. This page asks a different question: at what sustained request arrival rate does this exact service stop meeting its response targets? The result belongs in the local-versus-cloud decision, not in a universal claim that an 8B model “supports N users.”

Keep task acceptance separate from service performance. A fast but wrong answer does not pass the product gate. The load trial below assumes the chosen model and package have already passed a fixed task set; it then tests whether the server can deliver that accepted behavior at the intended traffic shape.

Four quantities that must stay separate

Quantity	What to record	What it cannot prove
Offered arrival rate	Requests scheduled to arrive per second, plus the rate actually sent	That the server completed them or that the load generator kept up
In-flight concurrency	Requests currently admitted and unfinished, including running and waiting where visible	A fixed requests-per-second capacity; each request holds resources for a different time
Completed throughput	Successful responses finished per measurement second	That their latency met a user target or that the queue remained stable
SLO-qualified goodput	Completed requests meeting all predeclared service limits, divided by measurement seconds	That the answer itself passed task review

An admission limit can move waiting from the server to the test client. In vLLM, --request-rate controls the intended arrival process, while --max-concurrency limits outstanding benchmark requests. If that limit binds, report client queue time and arrival-to-completion time alongside server metrics. A high completed throughput with a backlog that grows throughout the run is not a sustainable operating point.

Set the service target before raising the load

Name the use case and traffic slice first. Record the model revision, weight package, tokenizer and template, server and driver versions, GPU count, full GPU versus CPU residency, context limit, KV and prefix-cache settings, batch policy, replicas, and gateway admission limit. Freeze the prompt and output rules. A change in any of these can move the capacity boundary.

Use a representative mix of input and generated-token lengths, including the long tail; record shared prefixes separately. Specify streaming because a first-token target makes sense only when a client can receive an early output. Set the measurement window, warm-up, failure policy, and predeclared limits, such as P95 time to first token (TTFT), P95 time per output token (TPOT), P95 end-to-end response, maximum rejected-request share, no OOM, and a queue that does not keep rising. The numbers are application choices, not vendor defaults.

For a request with at least two output tokens, vLLM defines TPOT = (end-to-end latency − TTFT) ÷ (output tokens − 1). Inter-token latency (ITL) measures gaps between streamed output events. An event can contain several tokens, so ITL and TPOT are not always interchangeable. Record which measure the target uses. Treat one-token outputs as not having a useful request-level TPOT rather than silently mixing zeroes into a percentile.

Run a rate ladder, then replay a burst

1. Prove the generator and endpoint work. Send a small, low-rate stream. Compare intended and actual arrivals, response status, token counts, and timestamps. If the client cannot generate the planned traffic, repair the test before assessing the server.
2. Separate loading from steady operation. Record a cold load and warm-up, then measure warm traffic in multiple windows. If the real product frequently unloads models, add cold arrivals back into the production mix instead of hiding them.
3. Increase one dimension at a time. Hold the model, traffic mix, context, cache state, and client cap fixed while stepping up finite arrival rate. At every step, save sent, completed, failed, cancelled, and rejected counts. Wait for outstanding work to drain before the next step; otherwise one stage contaminates the next.
4. Recheck the last passing range. Repeat around the first failed step and replay a representative burst or real arrival trace. A single mean rate misses the short periods when many long requests arrive together.

For a vLLM service, vllm bench serve can drive a finite --request-rate and report TTFT, TPOT, ITL, and end-to-end distributions. Use its random input and output lengths to check instrumentation, then replace them with approved representative requests for the capacity decision. Its default infinite request rate pushes all requests immediately and answers a saturation question instead. Repeated identical prompts or a fixed random seed can warm prefix cache and inflate throughput; record cache state, vary the seed or reset the cache when reuse is not part of the real workload.

On vLLM, inspect /metrics at the same time: running and waiting requests, server request queue time, vllm:kv_cache_usage_perc, TTFT, TPOT, and end-to-end histograms. The KV value is a fraction of allocated KV blocks, from 0 to 1; it is not a physical GPU-memory percentage. Also capture device memory peak, preemptions or offload when applicable, OOM, and HTTP errors. TTFT measured from client send includes service-side delay, but excludes any time the benchmark client held the request before sending it. Do not add server queue time to TTFT as if it were a separate user wait; use queue metrics to explain where the delay came from.

Ollama exposes different evidence. Its generate response includes load, prompt-evaluation, and generation durations and token counts. A streaming client must timestamp intended arrival, actual send, first output, and completion to calculate user-visible delay and client waiting. ollama ps or /api/ps shows loaded models, their configured context, and size_vram; that value is not a per-request KV-block occupancy metric. Record OLLAMA_NUM_PARALLEL, OLLAMA_CONTEXT_LENGTH, and OLLAMA_MAX_QUEUE explicitly: parallel requests expand context memory demand, and a full queue can return HTTP 503. Keep 503 separate from OOM and from an answer-quality failure.

Example 1: short FAQ answers hit a queue boundary

A team serves public product questions with a frozen model and 500–1,000 input tokens per request, capped at 120 output tokens. Before testing, it writes a hypothetical trial target: P95 TTFT at most 1 second, P95 TPOT at most 80 milliseconds, P95 client waiting at most 200 milliseconds, P95 server queue at most 300 milliseconds, no OOM or 503, and no rising backlog. These are the team’s proposed limits, not measured performance.

The team schedules 0.5, 1, and 2 requests per second in separate steady stages, leaving result cells blank until a real run. Suppose the 2-request stage completes more answers per second but the waiting count climbs through successive windows and P95 TTFT exceeds the agreed limit. It fails even if KV blocks are not full and no OOM occurs. The recorded result is “the last tested stage that passed,” followed by another run between that stage and 2 requests per second. It is not an estimate of the maximum number of simultaneous users.

Example 2: the same mean hides a long-document burst

A document assistant has an average of one request per second across a work period. Some requests bring about 8,000 input tokens and ask for up to 600 output tokens; arrivals bunch around a review deadline. The team keeps the mean rate but compares an even stream with the observed burst trace, using the same server configuration and task mix. Before either run, it sets its TTFT, full-response, error, and queue limits. It does not fill in results from a hardware guess.

For an Ollama candidate, the record includes actual context length and OLLAMA_NUM_PARALLEL. Raising parallelism is a separate configuration trial: it may let more requests run immediately while increasing memory allocated for their contexts. A burst may instead wait until OLLAMA_MAX_QUEUE is exhausted and return 503. Whichever happens must be observed. A passing even stream does not certify the burst. If the burst fails, test an admission rule, a separate lane for long documents, or more serving capacity, then repeat the exact trace and quality checks.

Fill the serving load trial

The workbench below can be edited, copied, and exported. Repeat its stage row for every rate and burst stage; keep raw counts and metric units beside the verdict. These fields make an unrun experiment visibly incomplete:

Group	Minimum evidence
Fixed setup	Model and package revision, server version, GPU placement, context, KV/cache policy, client and server admission limits
Traffic	Task slice, input/output token distributions, prefix reuse, intended and actual arrival rate, burst shape, warm-up and measurement duration
Each stage	Sent/completed/rejected/failed/OOM counts; P95 TTFT, TPOT, and end-to-end; client versus server queue; running/waiting trend; KV fraction and device peak
Decision	Predeclared limits, SLO-qualified completions per second, highest tested sustained passing arrival rate, failing dimension, owner and retest trigger

Count a rejected, timed-out, or cancelled request in the arrival cohort when calculating the share that met the user target. If a benchmark reports goodput only over completed requests and omits client waiting, show those missing counts separately. Do not declare a stage sustainable until the client actually delivered the intended rate, enough requests completed to evaluate the tail, and the waiting queue stopped accumulating.

Common questions

Can I compute maximum concurrency from free VRAM? No. Free memory does not specify prefill compute, decode speed, request lengths, scheduler behavior, or the acceptable wait. It helps choose a safe starting configuration for a measured load trial.

Should I use requests per second or tokens per second? Keep both. Arrivals and completed requests describe demand and service; input and output token distributions explain why two equally sized request batches consume different work. Output tokens per second alone can reward longer answers that your task did not ask for.

What if the average latency looks fine? Check the agreed percentile, failed requests, client waiting, and backlog over time. Averages can improve while the slowest useful slice becomes unacceptable.

Boundary

The last passing stage is valid only for the recorded hardware, model, software, context, workload mix, cache state, and service target. It is not a guarantee under a different traffic burst, multimodal input, CPU offload, extra replicas, or a model update. A very low-traffic service may need longer runs to estimate tail latency. A load generator, network, or client connection limit can be the bottleneck; verify actual arrivals and repeat from a suitable client before purchasing hardware or publishing a capacity claim. For a rate limit or outage after launch, use a separate service fallback ladder to plan retries, queues, and handoff.