Local LLM or cloud API? Choose by data, peak load, and operations

“Run it locally or call a cloud API?” sounds like a two-way choice, but there are at least three operating routes. You can serve a model on hardware your organization owns, manage the inference server yourself on a rented GPU, or send requests to a provider-managed model API. The middle route is still cloud infrastructure: moving the server configuration into your team’s hands does not keep data inside your own facility. For each route, the useful question is whether an acceptable task can be completed under the permitted data path and the real peak load, with an owner for failures.

Use the data-boundary map to classify fields before they move. The VRAM budget checks a narrow hardware constraint, and the model scorecard tests candidate model behavior. This page decides where inference runs and who operates it. The workflow cost map remains the detailed ledger for tools, review, and recovery; its accepted-task denominator is used here so the three deployment routes share a fair unit.

Compare three routes, not three marketing labels

Route	Data and control to verify	Capacity and operating owner	Main cost evidence
Owned hardware	Facility, network egress, model downloads, access, logs, cache, backups, and deletion	Your team sizes GPUs, queues requests, patches the stack, watches incidents, and replaces failed parts	Hardware amortization, power and cooling, storage, network, support, and idle capacity
Team-managed rented GPU	Cloud region and contract, storage and telemetry path, administrative access, backups, and network egress	Your team still configures serving, security, monitoring, scaling, and recovery; the infrastructure provider supplies the rented machine	GPU instance time while running, attached storage, transfer, supporting services, and operator time
Provider-managed API	Exact organization, project, model, endpoint, retention control, region, tools, and subprocessors	Provider runs inference; your team owns integration, keys, limits, retries, fallbacks, evaluation, and incidents at the application boundary	Current input, cached-input, cache-write, output, tool, storage, and reserved-capacity terms where applicable

An API vendor’s statement that submitted data is not used to train its models does not mean zero storage. For example, OpenAI says API data is not used for training by default unless the customer opts in, while default abuse-monitoring logs may contain prompts and responses and are retained for up to 30 days, subject to stated exceptions. Its Modified Abuse Monitoring and Zero Data Retention controls require eligibility and approval; endpoint and feature rules can still permit application state. Data residency also has specific coverage and exclusions. Check the actual provider, contract, project, endpoint, feature, and date before passing a sensitive-data gate. A self-hosted server likewise needs authentication, network isolation, logging rules, backup controls, and deletion tests. vLLM’s security guide warns that its --api-key flag protects only certain route prefixes, so an exposed server needs an allowlisted proxy and network controls.

Put hard gates before the cost calculation

Start with the task and its acceptable result. Record data classes, allowed locations and retention, model or weight license, must-pass cases, P95 first-token and completed-response targets, peak arrival pattern, and who must respond to an outage. Mark a route excluded when it cannot pass a hard gate; do not give it extra points for a low quoted price. Keep unknowns visible instead of writing “compliant,” “fast,” or “private” without evidence.

For owned or rented GPUs, weights fitting in memory do not prove the server can handle concurrent traffic. Measure complete GPU residency, KV usage, queue wait, out-of-memory events, time to first token, end-to-end latency, and completed-request rate under the expected prompt and output lengths. vLLM offers serving, benchmark, and production metrics for such a test. Set the benchmark’s request rate and burst shape to the observed traffic: an unlimited send rate and client-side queue are not the same as the service’s arrival pattern. Record prefix-cache behavior so repeated prompts do not create a misleading throughput result. For a managed API, inspect the actual project’s request and token limits and run the same arrival pattern through the integration. OpenAI documents separate RPM and TPM limits, along with rate-limit responses and retry guidance; those are provider-specific examples, not a guarantee that every API offers unlimited burst capacity.

Example 1: confidential contract field extraction

A legal operations team needs to extract a payment amount, currency, effective date, payee, and the clause that changes when payment is due. Some documents contain confidential counterparty details. First write a field-level route: what may remain in the organization, what may be redacted, and whether a named provider, region, endpoint, log policy, backup path, and human reviewer are approved. If policy forbids sending the required raw clause to a provider-managed API and no approved transformation preserves the task, that API route fails the data gate. A rented GPU still processes data in an infrastructure provider’s environment and needs its own contract and telemetry review. Owned hardware is a candidate only after access and backup paths are checked; physical ownership alone does not make a deployment private.

The team then freezes documents with a changed date in a late appendix, a negated payment condition, and a no-answer case. Each route’s model must return the right fields and supporting passage without a blocking error. The quantization trial can test a smaller self-hosted package, but no memory saving can excuse a wrong amount or payee. Only routes that pass the same quality and data gates move to load and cost comparison. This is a decision procedure, not a claim that any candidate has already passed.

Example 2: a public FAQ with campaign peaks

A retailer’s public product FAQ is quiet outside promotions and receives bursts during a campaign. The content is public, but customer questions and logs still need a retention decision. Use actual minute-by-minute arrival traces and distributions of input and output tokens, rather than multiplying the daily average by an arbitrary peak factor. Freeze the answer set, escalation rule, and a small no-answer slice so a cheap or fast route cannot win by answering incorrectly.

For an owned or rented GPU, replay ordinary and campaign traffic while recording queue growth, KV pressure, P95 latency, rejected requests, cold starts, and the operator action when capacity is exhausted. For an API, test the current project’s RPM, TPM, spend controls, 429 or 503 behavior, retry budget, and observed latency. A provider-managed API may avoid paying for an idle dedicated GPU between campaigns, but its variable price and rate limits must be checked. A self-managed GPU may be economical at sustained use, but only if the team can meet the campaign SLO and support the queue or scaling plan. Neither route wins from the scenario alone; fill the matrix with measured outcomes.

Use the same cost unit for all routes

Compare one period with the same task mix, acceptance criteria, traffic trace, and human review policy. For owned hardware, include a stated amortization period, power, cooling, storage, network, spares, security, and on-call time. For a rented GPU, count the hours it remains allocated, attached services and transfer, plus the team’s serving and recovery work; do not also count a purchase price for the rented card. For a managed API, apply the dated model and feature price schedule to actual input, cached input, cache writes where applicable, output, retries, and tools. Subscription-seat prices are not API rates. Prices and entitlements change, so retain the source, region, currency, and effective date instead of hardcoding a break-even number in this guide.

For each route, calculate period cost ÷ tasks that passed acceptance. Keep rejected tasks, retries, human corrections, and incidents in the numerator; do not divide by requests sent. If a route produces no acceptable tasks or fails a hard data or service gate, mark it unfit rather than forcing a finite unit cost. Record non-monetary limits separately: a privacy prohibition, a critical extraction error, or an outage owner cannot be erased by arithmetic. The workbench below gives each route the same evidence rows and ends with an owner and retest trigger.

Common questions

Is a cloud GPU “local”? It is self-managed inference on rented cloud infrastructure. Its operator may control the model and serving code, but data, storage, telemetry, and administrative access still cross an infrastructure boundary.

Is an API automatically less private? The answer depends on the exact provider controls and approved data path. Some API configurations offer strong contractual and technical controls; a self-hosted server can still leak through exposed endpoints, logs, backups, or outbound tools. Verify the actual configuration.

Is self-hosting always cheaper once traffic grows? No. The result changes with utilization, peak spare capacity, hardware life, energy, staff coverage, quality, retries, and provider prices. Measure the same accepted workload before claiming a break-even point.

Boundary

This matrix selects an inference operating route for a specified task and workload. It does not certify legal compliance, guarantee a vendor’s future retention policy, or replace model evaluation and security review. A rented GPU, a managed inference endpoint, and a general model API can have different contracts and billing units even when all are called “cloud.” Recheck provider terms, model license, hardware support, prices, and measured traffic after material changes; keep any route replaceable when its evidence changes.