How many GPUs does a local LLM service need?

“Our model handles four concurrent requests” is not a GPU purchase plan. Concurrent work, average latency, and GPU utilization cannot be converted into a reliable number of replicas without the request-length distribution, target response times, queue behavior, and arrival pattern. Start with a measured service boundary, then test whether several replicas still meet it when routing, loading, failures, and placement enter the picture.

The single-instance serving load trial finds the last tested sustained arrival rate that meets the task’s service objectives. This page begins with that result and asks how many ready replicas and GPU slots the whole service needs. The local-versus-cloud decision chooses the deployment path, and the workflow cost map handles the wider operating bill; neither substitutes for a placement and failure test.

Define a comparable unit of capacity

Let c_test be one replica’s highest tested sustained request arrival rate that passes the declared quality, P95 first-token and full-response limits, rejection rule, and stable-queue rule. It is not an advertised requests-per-second figure or the peak rate of an unbounded benchmark. Record the exact model package, engine version, GPU count per replica, context, batch and cache settings, input/output length mix, streaming behavior, gateway limit, and warm state. If those change, the number no longer describes the candidate replica.

Let λ_design be the arrival rate the service must accept for the same workload mix and measurement window. Obtain it from a real or forecast traffic trace, including relevant bursts and long requests. A day-long average is often smaller than the minute that matters. If the mix changes during a burst, either measure that mix separately or replay the trace; one scalar rate cannot make long and short requests equivalent.

Choose a fractional reserve h between 0 and 1 for normal operation and state why it exists: measurement uncertainty, drift in request lengths, routing imbalance, or a planned buffer. The simple starting point is:

effective tested capacity per ready replica = c_test × (1 − h)
N_base = ceil(λ_design / [c_test × (1 − h)])
running GPU slots for this candidate = N_base × GPUs_per_replica

The denominator must be positive. Do not apply this formula if the traffic mix, latency target, model, or replica hardware differs from the trial. N_base is a candidate, not a guarantee: it assumes traffic can be distributed usefully among ready replicas and that the measured per-replica limit survives the gateway and cluster environment. Write the reserve explicitly so the team does not quietly add it twice or hide it inside a rounded number.

Add failures and placement before naming a GPU count

Choose the failure the service must tolerate: one Pod, one GPU, one node, or a whole zone. For each scenario, count the ready replicas remaining, not only the desired replica setting. If losing the largest failure domain leaves R_ready, its candidate capacity is R_ready × c_test × (1 − h); that must still cover λ_design under the same task and response gates. For a one-replica failure with identical independent replicas, a simple arithmetic candidate is N_base + 1. A node that hosts two replicas removes two, and a zone may remove more. Redundancy is a placement property, not a percentage in a spreadsheet.

Convert replicas to GPU slots using the real per-replica placement. A tensor-parallel replica using two GPUs is one serving replica consuming two GPU slots; a failure of either GPU may take that entire replica out of service. Kubernetes schedules GPU resources exposed by a device plugin as extended resources, with the requested GPU count constraining placement. Check that each node has the required GPUs together, suitable device memory and interconnect, and capacity for the intended topology. Topology spread constraints can help distribute Pods across nodes or zones, but verify the actual schedule and the number remaining after a simulated loss.

Also separate ready running GPUs from cluster peak or reserved GPUs. A rolling update may temporarily start an extra replica; a failed node may require a free slot elsewhere; a standby Pod may be scheduled but not yet ready. State max surge or equivalent rollout policy, replacement placement, and whether cloud capacity is actually obtainable. Counting only steady-state Pods can leave a deployment unable to update or heal.

Make scaling respond before the queue fails

Autoscaling is not instant capacity. Measure image pull, model download, weight load, warm-up, health checks, and routing delay from scale request to a replica that can pass a real inference probe. Kubernetes startup probes can allow a slow-starting container to initialize without a premature liveness failure; readiness controls whether a Pod receives service traffic. A healthy HTTP process is not necessarily a loaded model, so test the readiness signal against the actual route.

Kubernetes HPA can use resource, custom, or external metrics. The vLLM Production Stack documents a KEDA/Prometheus example using vLLM metrics such as waiting requests. This is an implementation option, not a universal threshold. vLLM exposes running/waiting requests, first-token latency, and end-to-end latency metrics, which help explain a scaling trigger and its result. A waiting queue is often a late signal for a burst shorter than model warm-up; prewarmed minimum replicas, a scheduled ramp, admission control, or a backlog with an honest user delay may be needed. Test the trigger, stabilization and cooldown behavior under your traffic trace.

Example 1: one-GPU review service

This is a fictional teaching calculation, not measured hardware performance. A review assistant’s one-GPU replica passed a matched sustained stage at c_test = 0.8 requests/s. Its design burst is λ_design = 2.0 requests/s, and the team chooses h = 0.20. The effective planning rate is 0.8 × 0.8 = 0.64 requests/s, so N_base = ceil(2.0 / 0.64) = 4 ready replicas, using four GPUs.

The requirement also includes loss of any one replica. Four replicas would leave three, with a planning total of 3 × 0.64 = 1.92 requests/s, short of the target. Five ready replicas leave four after one failure and offer a candidate 2.56 requests/s; that means five one-GPU replicas in distinct failure placements if the failure assumption is one node. If a rolling update can create one extra readying Pod, the scheduler may need a sixth GPU slot at update peak. None of these numbers proves P95 latency: the team must replay the burst with five replicas, remove one, and confirm the remaining four pass the same service rules.

Example 2: two-GPU replicas and a zone failure

This is another fictional teaching calculation. A model uses two GPUs per replica. A matched load trial gives c_test = 0.5 requests/s; the service needs λ_design = 1.2 requests/s with h = 0.25, so one ready replica contributes a planning 0.375 requests/s. The base count is ceil(1.2 / 0.375) = 4 replicas, or eight running GPU slots. If only one replica can fail, five ready replicas would leave four, using ten GPUs normally.

The actual requirement is stronger: survive the loss of any one of three zones. Five replicas spread 2, 2, 1 leave only three when a two-replica zone fails; 3 × 0.375 = 1.125, short of the design rate. Six replicas spread 2, 2, 2 leave four, a candidate 1.5 requests/s after a zone loss. That means 12 ready GPU slots across suitable nodes and zones. If a rollout needs one extra two-GPU replica while six remain ready, peak placement may require 14 GPU slots. The team must still test routing, queue, and P95 response after removing a zone, including whether replacement capacity can be scheduled.

Suppose a new replica needs seven minutes to download, load, and warm the model, while the burst reaches its peak in two minutes. Reactive scale-up cannot rescue that burst. Keep the required replicas ready, prewarm on a known schedule, or change the user-facing admission policy; do not count an unready replica in failure capacity.

Fill the capacity worksheet and replay failure

The workbench below records the trial source, arithmetic, placement, warm-up, and observed cluster result. Use the actual rate units and a time window; copy the scenario row for every failure domain. A blank measured result remains untested even when the arithmetic passes.

Worksheet row	Minimum entry
Comparable trial	Task and model versions, per-replica GPUs, input/output mix, response gates, c_test, queue behavior, source trace
Demand and reserve	λ_design, window and burst shape, h with reason, effective capacity and N_base
Failure placement	Ready replicas by node/zone, largest loss, remaining ready count, GPU slots, rollout surge and replacement slot
Scaling	Signal and threshold, minimum warm replicas, pull/load/warm time, readiness probe, maximum usable backlog
Proof	Normal peak replay, loss replay, warm replacement, accepted results, P95 delays, rejects, waiting trend, decision and owner

Common questions

Can average latency or GPU utilization give the replica count? No. Both can help diagnose the service, but neither specifies arrival shape, output length, tail latency, queue stability, or accepted results. Measure c_test under the real target first.

Can I divide concurrent users by requests per GPU? No. Users vary in activity, and in-flight requests vary in prefill and decode work. Use a request arrival trace with lengths and response targets.

Does HPA or KEDA replace warm standby? Only if new replicas become ready before the service target is missed. Measure the full scale-out delay against the burst, including available GPU placement.

Boundary

This worksheet gives a starting replica and GPU-slot candidate for a specified model, workload mix, engine, hardware, and objective. It does not certify cluster networking, load balancing, storage, node availability, zone independence, or the scheduler’s ability to place replacements. Re-run the single-replica and cluster failure trials when model weights, context, quantization, traffic shape, routing, GPU type, autoscaling policy, or rollout settings change. Commit capacity only after normal, burst, and required failure replays meet the task and service gates.