Why prompt caching misses: run a three-request test

Prompt caching reuses eligible input work from an earlier request. It does not reuse an answer: the model still processes new input and generates new output. A lower bill or a faster response is plausible only when a later request can read a sufficiently long, matching prefix under that provider’s rules. The first question is therefore what did the API actually count as a cache write or read?

This lesson produces a three-request trial record. Keep the trial separate from a production savings claim. One apparent hit cannot establish a typical hit rate, and one quick response cannot establish a latency improvement.

Prepare one eligible, safe prefix

Choose one provider, API, model version, and region for the whole trial. Use a public or invented reference document that would genuinely be reused in your workflow. Call the unchanged instructions and document P. Put the task-specific question after P as a short suffix. Do not put a clock value, request ID, user name, or changing search result in front of P. Hold tool definitions, output schema, and relevant generation settings fixed.

Count P with the selected provider’s tokenizer or token-counting API. Cache eligibility is measured in tokens, and the minimum differs by model and sometimes by request configuration. Do not add filler solely to cross a threshold; a longer prompt can cost more than the cache saves. Check that P is worth keeping and fits the model’s context limit. Keep sensitive material out of this demonstration and review the provider’s retention terms before testing real documents.

Provider route	Enablement and eligibility to check	Read the result from
OpenAI	Current guidance gives GPT-5.6 and later a 1,024 visible-input-token minimum and offers implicit or explicit breakpoints; earlier models differ. Put an explicit breakpoint after P when that model supports it, or verify that the implicit boundary covers P.	usage.input_tokens_details.cached_tokens and, where reported, cache_write_tokens
Anthropic	Use top-level automatic or block-level cache_control; the minimum is model-specific. For a changing suffix, mark the last stable block explicitly. The default cache lifetime is five minutes; a one-hour option has a different price.	usage.cache_creation_input_tokens, cache_read_input_tokens, and input_tokens
Google Gemini	Gemini 2.5 and newer support implicit caching, but a hit is not guaranteed. The Interactions API uses implicit caching; generateContent also offers explicit cache objects in beta, with storage-duration charges. Check the selected model’s minimum.	Interactions: usage.total_cached_tokens; generateContent: usage_metadata
DeepSeek	Disk context caching is on by default, but reads must fully match a previously persisted prefix unit. A common prefix may become a unit only after another request.	usage.prompt_cache_hit_tokens and prompt_cache_miss_tokens

These fields have different meanings and billing treatment. Do not combine their raw values into one cross-provider “hit rate.” Consult the price page for the exact model, date, and cache mode before converting tokens to money.

Run three requests in order

1. Request 1, write opportunity: send P followed by question A. Save the exact serialized request, response usage, start time, time to first token if available, total latency, output, and acceptance result. A response that contains no reported write may mean P is too short, the boundary is ineligible, or this route does not expose writes.
2. Request 2, same prefix: after the first response has begun, send the identical P and a different short question B within the documented cache lifetime. Keep model, tools, schema, and other prefix-affecting settings unchanged. Record the same fields. A read is evidence of reuse; a miss is a diagnosis to pursue, not proof that caching is unsupported.
3. Request 3, changed-prefix control: change a harmless phrase near the start of P to create P′, then ask question B again. Preserve the document’s meaning and acceptance rule. Compare the reported read with Request 2. A drop supports the prefix-match explanation; it need not fall to zero if an earlier system or tool prefix still matches.

Run the requests sequentially. Anthropic states that a new cache entry becomes available only after the first response begins, so firing Requests 1 and 2 together is a poor test. Other providers also describe routing, retention, or best-effort behavior that can prevent a hit even when the visible text is unchanged. Save the raw API response rather than inferring a hit from speed.

Example: two questions over a public support guide

A support team uses the same public product guide for cancellation and rescheduling questions. P contains the approved guide text and answer format. Request 1 asks, “Can a customer cancel after booking?” Request 2 asks, “Can a customer move the date?” Only the question changes. On a route with an eligible breakpoint after the guide, the trial tests whether the second call reads the guide prefix while processing its new question normally.

The reviewer checks each answer against the guide, including exceptions and the cited section. If the second answer is wrong, a reported cache read is not a workflow success. Record the cost of both calls and the review needed to produce acceptable answers. The guide’s length, questions, and outputs are trial inputs; any token counts must come from the actual API response.

Example: a timestamp defeats the intended prefix

An agent prepends “Report generated at 09:14” to its instructions, then appends the same reference guide. A later request says “09:15.” The rendered prefix now differs before the reusable guide begins. Request 2 may show little or no guide reuse despite the guide itself being unchanged. Move the timestamp and current user question after the stable guide; where supported, put the breakpoint on the last stable block, not on the changing block.

Repeat the three requests after this one change. A provider might still report some reuse from earlier tool or system content, and DeepSeek may persist a newly detected common prefix for a later call. Interpret the reported token counts at the documented boundary instead of demanding that every control read exactly zero tokens.

Decide with accepted-task cost

For each request, retain provider, API, model, time, prefix token count, cache mode or breakpoint, retention setting, input and output token usage, cache read and write fields, price version, latency, and acceptance status. Include explicit cache storage charges when the route has them. Compute the monetary total using that route’s actual prices, then divide a comparable batch’s model, storage, tool, and review cost by the number of tasks that pass the same acceptance check. If none pass, report that the batch has no deliverable cost per accepted task.

Compare a few repeated task batches with and without the intervention, using the same task mix and quality gate. Report hit tokens, total cost, completion time, and rejected answers together. A cache improvement that worsens answer quality or requires extra review may not improve the full workflow.

Boundary

This trial diagnoses a reusable input prefix; it does not benchmark all traffic, prove a provider’s future price or retention behavior, or guarantee a latency gain. It is a poor candidate when prompts rarely share eligible content, the shared content changes often, or storage and privacy requirements rule out the cache route. Recheck provider documentation and the response fields whenever the model, API, prompt structure, or pricing changes.