Ollama, llama.cpp, vLLM, or SGLang? Choose a local inference engine

Choosing a local model answers what weights should produce the response. Choosing an inference engine answers how those weights are loaded, called, scheduled, and observed. A team can like a model in a notebook and still fail when its chosen server cannot load the available package, return the fields its application expects, or stay responsive to real traffic.

This is an engine selection exercise, not a claim that one project is always fastest. First decide whether the task belongs on local hardware or a managed API. Then use the VRAM budget to narrow packages. This page turns four plausible local servers into a small, reproducible acceptance trial. The separate serving load trial establishes the capacity of the final configuration.

Start with the package and the caller

Write down the exact model revision, weight format, quantization, tokenizer, chat template, context requirement, and license. The phrase “same model” is insufficient if one candidate loads a different quantization or template. Record what the application actually calls: chat, completions, embeddings, streaming, tool calls, structured output, or an engine-specific API. A route being described as OpenAI-compatible does not promise that every parameter, response field, error, streaming event, or model feature behaves identically. Run a contract test against the features you use.

The candidates occupy useful starting positions, with overlap:

Engine	Good first reason to trial it	First mismatch to investigate
Ollama	A person or small team wants a short route from model acquisition to a local chat/API prototype. Its native API and model packaging reduce initial setup work.	The chosen model package, context setting, and actual application calls; test any OpenAI-compatible route you plan to depend on.
llama.cpp server	You have a suitable GGUF package and need to control CPU/GPU placement or run on constrained hardware. Its server exposes compatible chat and embedding routes and detailed runtime options.	Whether the exact GGUF, offload split, chat template, and context fit; measure latency after CPU fallback rather than treating “loads” as “usable.”
vLLM	You are building a shared accelerator-backed service and need serving controls and production metrics alongside a supported model.	The specific architecture, weight format, hardware backend, chat template, and required API behavior under your pinned version.
SGLang	You are evaluating a shared model service with its OpenAI-compatible endpoint and runtime features as a candidate.	Support for the chosen model and deployment stack, plus application contract and operations evidence under the exact release.

These are shortlist cues, not capability boundaries. Each project changes quickly, and model support varies by version and backend. A missing row in an old comparison chart is not proof a current release cannot do the job; a feature on a documentation page is not proof it works with your selected model. Keep the version and source link beside every claim you use to accept or reject a candidate.

Run a three-gate selection trial

Gate 1 — can it load and stay resident? For each candidate, record the server build, hardware and driver, model file or repository revision, context length, and memory behavior. Start with one representative request, then the longest permitted input. Confirm that it runs in the intended GPU/CPU placement without an out-of-memory event. If one route requires a different weight package, record the change and rerun quality checks. The 4-bit comparison explains why package size alone cannot settle that trade.

Gate 2 — does it satisfy the caller’s contract? Replay the same small set of accepted task cases through the real client. Check role and template handling, streaming order, stop reason, token usage, timeouts, error status, and every required structured or tool field. Mark an unsupported feature as a blocking mismatch, a client adapter task, or an unused feature; do not count a successful curl response as application compatibility. Keep request and response fixtures synthetic or sanitized.

Gate 3 — can the team operate the intended service? Apply the same prompt-length distribution, generation cap, arrival pattern, and quality gate to each surviving setup. Measure first-token delay, complete latency, completed good requests, queue growth, rejects, and peak memory. Identify who owns model updates, access controls, logs, health checks, and recovery. For a single-user offline tool, this gate may be a short cold-start and repeat-use check. For a shared service, use the full load experiment rather than a one-request speed demo.

Score only candidates that passed mandatory gates. An attractive throughput result cannot rescue an incompatible package or a missing application field. If none pass, revise the model, interface, hardware, or deployment requirement explicitly instead of declaring the least-bad candidate ready.

Example 1: an offline research laptop

An analyst needs to classify public research notes during travel without a network connection. The laptop has limited accelerator memory; its approved model is available as a particular GGUF file. The client needs ordinary streamed chat, not server-side tool execution. The team tests llama.cpp server first because GGUF loading and an adjustable GPU offload path match the package and hardware. It also tests an Ollama package of the same base model for setup and maintenance effort, while recording any quantization or template difference.

Suppose both launch but the longest note pushes one route into a very slow CPU-heavy path. The decision depends on the analyst’s prewritten wait limit and correct classification on the same notes, not on which command was shorter. If the Ollama package differs, a quality difference cannot be attributed to the engine alone. The record distinguishes engine effect from package effect and keeps the faster setup only if its actual result passes.

Example 2: a shared GPU review queue

A support team drafts answers from a permitted knowledge set. Its application sends streaming chat requests, requires a specific JSON result shape, and must show a reviewer a draft within a set time. It has a dedicated GPU service and bursty arrivals before shift changes. vLLM and SGLang enter the shortlist because both offer API-serving paths suited to this deployment shape. The team pins releases, a supported model package, and the client request contract, then runs ten task cases including malformed input and an unsupported claim.

If one server completes more requests during an unbounded saturation test but fails the declared first-token or queue target during the shift burst, it does not win. If a required JSON field or error behavior differs, the team either adds a tested adapter or removes that candidate. The resulting decision cites accepted answers, caller-contract results, a repeatable arrival trace, and an owner for service recovery. It does not generalize the result to a different model or GPU.

Fill the inference engine trial

Use the workbench below for one task and one hardware envelope. Duplicate its candidate row and attach controlled evidence pointers for logs and fixtures. Record not tested rather than filling an empty cell with a feature claim.

Record	Evidence required
Frozen task	Input and output types, two or more representative cases, blocking errors, response target, privacy and license gates
Candidate identity	Engine and release, model revision, weight package and quantization, tokenizer/template, hardware, driver, context, placement
Caller contract	Exact routes and fields used, streaming and error expectations, fixture result, adapter work if needed
Operating trial	Cold/warm load, peak memory, quality result, first-token and full-response time, burst queue, rejects, monitoring and recovery owner
Verdict	Pass, adapter-and-retest, or reject; evidence pointer, reviewer, and retest trigger

Common questions

Is an OpenAI-compatible endpoint enough to switch engines? No. Run your actual client requests against each pinned server and compare the fields and failure cases it needs. Compatibility is a contract for the subset you use.

Should a laptop user benchmark vLLM and SGLang too? Only when a supported package and hardware path, plus a real service need, make the setup relevant. A shortlist is meant to save trial time, not require four installations for every task.

Can I choose from a public tokens-per-second ranking? Use it to find candidates, not to sign off. Package, precision, hardware, context, prefix reuse, request mix, and acceptance criteria can all change the result.

Boundary

This trial chooses an inference engine for a named model, task, client, hardware envelope, and release. It does not establish model quality in general, software security, license permission, or production capacity. Treat exposed endpoints, model downloads, telemetry, and saved prompts as part of the deployment’s data boundary. Rerun the relevant gates after changing weights, quantization, templates, server version, hardware, API features, or traffic shape.