Local model deployment

Can this model serve our workload on local hardware, and when is that deployment worth it?

Budget memory, test quantization and adapter training, choose an engine, measure one-instance capacity, plan GPU replicas and failure reserve, then compare with a cloud API.

AI VISTA / DECISION01 / 03

WORKING DECISIONA model file fitting in memory is only a starting estimate; require task quality, peak memory, latency, and recovery to pass the same trial.

01INPUT

02CHECK

03BOUNDARY

RETURN TO THE REAL TASK✓

DECISION BRANCH

Answer the decision this topic actually owns.

ASK FIRST↓
How much VRAM will target precision, context, and concurrency need at peak?
IF↓
Quality, peak memory, and latency pass the same load test, then compare local and cloud total cost.
OTHERWISE✓
Reduce concurrency or context, change precision, or continue with the cloud API.

WORKED CASE

Can one 24 GiB GPU handle the internal assistant at peak?

A team considers a 14B model for policy questions. Eight sessions may overlap during the day, source passages vary in length, and a working cloud API provides a quality and latency baseline.

WORKING ARTIFACTA VRAM budget, quantization quality trial, load record, and local-versus-cloud deployment matrix.

01WEAK SHORTCUT

Treat a model file that fits in VRAM as ready to serve, assume 4-bit performs like BF16, and infer peak capacity from one successful request.

02BETTER JUDGMENT

Budget weights, KV cache, and serving margin separately. Compare BF16 with a named 4-bit package on the same policy cases, then raise context length and concurrency in steps while recording peak VRAM, queuing, and time to first token.

03ACCEPTANCE

At target load, severe errors, latency, and failure rate meet agreed thresholds. Measured peak use leaves headroom, cloud fallback works, and a second reviewer can reproduce the comparison.

ENTRY & EXIT SIGNALS

Know when to enter—and when the decision is good enough to leave.

A topic is not an endless knowledge directory. Entry signals tell you whether the problem belongs at this layer. Exit signals decide whether to continue instead of substituting time spent reading for work completed.

AI VISTA / DECISION STATUSLocal model deployment

01ENTER HERE

Local serving is proposed for data or cost constraints
Precision, context length, or concurrency may exceed VRAM
Peak load and response-time requirements are known

02LEAVE WHEN

Weights, KV cache, and measured headroom are recorded separately
The quantized candidate clears the same holdout quality gate
Peak load and fallback are compared with a cloud baseline

7 PRACTICES

Build the judgment, make the artifact, then review it with real cases.

01 How much VRAM does a local LLM need? Budget weights and KV cacheEstimate raw model weights and KV cache separately, test context length and concurrency, then measure the missing peak costs on your serving stack. Local model deployment 02 Is 4-bit LLM quantization worth it? Test fit, quality, and speedCompare an unquantized BF16 reference with a named 4-bit package on fixed task cases and serving load before trading quality for memory. Local model deployment 03 LoRA or QLoRA fine-tuning? Run a small, reviewable pilotAdmit a stable behavior problem, split training data without leakage, probe real training memory, and compare adapters with a frozen baseline on unseen cases. Local model deployment 04 Ollama, llama.cpp, vLLM, or SGLang? Choose a local inference engineShortlist a local LLM server by model package, hardware, API behavior, and operating needs, then run the same acceptance and service trial before committing. Local model deployment 05 How many concurrent requests can a local LLM serve?Find the highest tested arrival rate that meets first-token, output-speed, queue, error, and memory targets on your actual inference server. Local model deployment 06 How many GPUs does a local LLM service need?Turn a tested per-replica arrival rate into a candidate replica count, then verify GPU placement, failure capacity, warm-up time, and rollout peaks. Local model deployment 07 Local LLM or cloud API? Choose by data, peak load, and operationsCompare owned hardware, a team-managed rented GPU, and a managed model API using data paths, measured peak capacity, operating ownership, and cost per accepted task. Local model deployment

BOUNDARY

What this topic page will not do

The budget estimates theoretical payload. Quantization, drivers, serving frameworks, and the actual request mix still require measurement on the target hardware.

01Do not treat model file size as peak serving VRAM02Do not infer task quality from a 4-bit file's size03Do not infer multi-request capacity from one successful request

Once quality, peak VRAM, latency, and fallback pass target-load tests, compare the benefits and cost of the entire workflow.

Courses and topics open directly from the main navigation.

Courses