How much VRAM does a local LLM need? Budget weights and KV cache

A local model can fail to start even when the weight file looks smaller than the GPU’s advertised memory. The file is only one part of the running process. Generated tokens keep key and value tensors; concurrent requests keep their own cache; the framework also needs buffers and memory for work that this page cannot infer from a model name.

Use the calculator below as a bounded arithmetic check, not a fit certificate. It answers a useful first question: do the raw weights plus a conventional KV cache already exceed the physical VRAM entered for one GPU? Only that condition rules out full residency under the stated single-GPU assumptions. Crossing a separate reserve line means the plan missed your preferred margin; neither result below physical capacity proves the model will run.

Read the model configuration before calculating

The input called “14B” is a rounded family name, not enough for a memory budget. Record the model version, total parameter count, number of transformer layers, number of KV heads, and dimension of each head. The last three describe the shape of a conventional attention cache. For a model with grouped-query attention, use KV heads, not query heads.

The official Qwen2.5-14B-Instruct card reports 14.7 billion parameters, 48 layers, 40 attention heads and 8 KV heads. Its configuration reports a hidden size of 5,120, so 5,120 ÷ 40 gives a head dimension of 128. These values make a reproducible example. Another model, revision, adapter, or serving format needs its own checked configuration.

Keep weight and cache arithmetic separate

The calculator uses these raw payload equations, with decimal billions for the parameter count:

Term	Calculation	What it covers
Weight bytes	parameters in billions × 1,000,000,000 × weight bits ÷ 8	A theoretical value for all named parameters at one bit width
KV cache bytes	2 × layers × KV heads × head dimension × bytes per KV element × cached tokens per request × resident requests	Conventional full-attention K and V tensor elements

“Cached tokens” means the peak input plus generated token slots held for a request, not just the prompt length. Concurrency means requests resident at the same time. If requests have different lengths, add their cached token counts rather than multiplying one average by the number of requests. The form deliberately assumes each request reaches the same entered peak; that makes the load assumption visible.

Results show GiB and GB because they are different units: 1 GiB is 1,073,741,824 bytes; 1 GB is 1,000,000,000 bytes. Enter the GPU’s actually reported usable physical capacity in GiB, or convert a decimal GB specification before entering it. The calculator shows physical capacity minus known terms separately from your planning line. The prefilled 4 GiB reserve is just a worked assumption that you may change; it is not an official or universal overhead coefficient.

Example: BF16 versus 4-bit for four Qwen requests

For the official Qwen2.5-14B-Instruct values above, BF16 weights have a raw payload of 27.38 GiB. Each 8,192-token request with 16-bit KV values adds 1.50 GiB. Four resident requests bring KV to 6.00 GiB and the known total to 33.38 GiB, already more than an entered 24 GiB of physically available VRAM. Full single-GPU residency with those exact assumptions is ruled out before runtime overhead is added.

Choosing “4-bit” changes only raw weight arithmetic: 14.7 billion parameters at four bits produce 6.85 GiB of raw weights, or 12.85 GiB with the unchanged four-request KV cache. That falls below 24 GiB physical capacity and a separate 20 GiB planning line after an illustrative 4 GiB reserve. A real 4-bit package can hold scales, zero points, grouping metadata, and tensors at other precisions. Neither positive margin proves full residency, speed, or task quality.

Example: 16 concurrent requests change the answer

Keep 4-bit raw weights and 8,192 cached tokens per request, but make 16 requests resident. One request’s KV cache is still 1.50 GiB; concurrent KV rises to 24.00 GiB. Added to 6.85 GiB of raw weights, the known total is 30.85 GiB, above even the entered 24 GiB physical capacity. Reducing weight precision did not cancel the cache load created by concurrency.

Qwen2.5-14B-Instruct scenario	Raw weights	KV per request	Concurrent KV	Known total	24 GiB physical minus known
BF16, 4 requests	27.38 GiB	1.50 GiB	6.00 GiB	33.38 GiB	−9.38 GiB
4-bit raw, 4 requests	6.85 GiB	1.50 GiB	6.00 GiB	12.85 GiB	+11.15 GiB
4-bit raw, 16 requests	6.85 GiB	1.50 GiB	24.00 GiB	30.85 GiB	−6.85 GiB

The table remains readable without JavaScript. The calculator’s three presets reproduce these cases; you can then replace the model dimensions and load with your own values.

Measure the costs the formula leaves out

Run the same model revision, quantization format, server version, target context and concurrent load you plan to use. Record a cold load, warm requests, peak GPU allocation or device memory, out-of-memory events, first-token latency and sustained throughput. PyTorch exposes a peak allocated-memory counter, but that counter alone does not describe every CUDA, framework, or device allocation; also inspect the GPU’s reported usage. Keep the batch and cache settings beside each result so another person can repeat it.

Write down whether the model and active cache were fully resident on this GPU. If the server silently offloads to CPU or distributes tensors across devices, a “fit” observation answers a different question. Treat a reported free margin as a trigger for the peak test, and keep the measured margin for traffic spikes and software updates.

Common questions

Can the weight file size replace the parameter calculation? It is useful evidence for a specific artifact, but compression, serialization, shards, and mixed precision affect disk size. Measure loaded device memory for the exact artifact.

Does an 8-bit KV setting always halve memory? The raw KV element term halves when one byte replaces two. Cache blocks, scales, supported kernels, quality, and the rest of the process do not automatically halve.

Can I add several GPUs together? This calculator models one GPU and does not assign layers, weights, or KV across devices. A distributed plan needs per-device placement and measured communication overhead.

Boundary

Use this calculation for dense decoder models with a conventional per-layer K/V cache and a known single-GPU placement. It does not model mixture-of-experts active and total parameter behavior, multi-head latent attention, sliding-window or hybrid cache layouts, speculative decoding, multimodal tensors, adapters, CPU offload, tensor or pipeline parallelism, or training. A positive remaining budget is never proof that the model will run. The workbench below records the missing configuration and the real peak test that can resolve that uncertainty.