Prompt caching is an optimization where an LLM API provider stores the computed key-value (KV) cache for a prompt prefix so that subsequent requests sharing that prefix can skip recomputation. The model reuses the cached attention state instead of re-encoding identical tokens, reducing both latency and per-token cost for the cached portion. This mechanism is transparent to the model weights — it only affects inference-time compute.
How prompt caching works
When a transformer processes input tokens, it computes attention scores between every token and all preceding tokens. The intermediate key and value projections for each layer — collectively the KV cache — grow linearly with sequence length. Without caching, every request recomputes these projections from scratch, even when the first N tokens are identical across requests.
With prompt caching enabled, the provider checks whether the incoming prompt’s prefix matches a previously cached entry. If a match exists (typically requiring an exact token-for-token match including whitespace and special tokens), the server loads the precomputed KV cache for that prefix and begins generation from the first uncached token. The cache key usually includes the model identifier, the exact token sequence, and sometimes system-level parameters like temperature or top-p if they affect the attention computation.
# Conceptual flow on the provider side
def generate_with_cache(request):
cache_key = make_cache_key(
model=request.model,
prompt_tokens=request.prompt_tokens,
# Some providers include sampling params in the key
sampling_params=request.sampling_params
)
cached_kv = kv_store.get(cache_key)
if cached_kv:
# Resume from cached position
start_pos = cached_kv.length
logits = model.forward(
input_ids=request.prompt_tokens[start_pos:],
past_key_values=cached_kv
)
else:
# Full forward pass, then store for next time
logits, new_kv = model.forward(
input_ids=request.prompt_tokens,
return_kv_cache=True
)
kv_store.set(cache_key, new_kv, ttl=provider_ttl)
return sample(logits)The cache lives in GPU memory (HBM) or offloaded to CPU RAM / NVMe depending on the provider’s architecture. Because KV cache size scales with layers × heads × sequence length × hidden dimension, a 32k context on a 70B model can consume 1–2 GB per sequence. Providers evict entries using LRU or TTL policies — typically 5 to 60 minutes of inactivity — to bound memory pressure.
Why it matters for cost and latency
The compute savings are proportional to the cached prefix length. On a 10k-token prompt where 8k tokens are cached, you avoid roughly 80% of the prefill FLOPs. Prefill is memory-bandwidth bound on modern GPUs, so the wall-clock speedup is often 3–10× for the cached portion compared to cold prefill. For workloads with long shared contexts — system prompts, few-shot examples, retrieved documents, or conversation history — this translates directly to lower per-request latency and lower provider compute cost, which some APIs pass through as discounted cached-token pricing.
Anthropic, Google, and OpenAI all expose prompt caching with slightly different semantics:
- Anthropic: Explicit
cache_controlbreakpoints in the message array. You mark which blocks are cacheable. Cached tokens billed at ~10% of standard input price. TTL ~5 minutes. - Google (Gemini): Automatic prefix matching on the full prompt. No explicit markup. Cached tokens billed at reduced rate. TTL ~60 minutes.
- OpenAI: Automatic prefix matching on the
messagesarray (including system prompt). Cached tokens billed at 50% of input price. TTL ~5–10 minutes.
// Anthropic explicit cache control example
{
"model": "claude-3-5-sonnet-20241022",
"messages": [
{"role": "user", "content": [
{"type": "text", "text": "You are a senior engineer...", "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "Here is the 50k token codebase..."},
{"type": "text", "text": "Now answer: ..."}
]}
]
}The pricing model matters. If your workload sends the same 20k-token system prompt + RAG context 100 times/hour, caching turns a $2–4/hour prefill bill into $0.20–$0.40. But if every request has a unique prefix, caching does nothing — you pay full price and the provider wastes memory tracking useless entries.
Concrete example: RAG with a fixed corpus
Consider a support bot that injects the same 50-page product manual (≈30k tokens) into every request alongside a short user question. Without caching, each request prefill processes 30k + question tokens. With caching, the first request pays full prefill; subsequent requests within the TTL window pay only for the question tokens.
# First request - cold
request_1 = {
"model": "claude-3-5-sonnet-20241022",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": MANUAL_TEXT, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "How do I reset the device?"}
]
}]
}
# Response headers show: cache_creation_input_tokens: 30000, cache_read_input_tokens: 0
# Second request within 5 min - cached
request_2 = {
"model": "claude-3-5-sonnet-20241022",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": MANUAL_TEXT, "cache_control": {"type": "ephemeral"}},
{"type": "text", "text": "What's the warranty period?"}
]
}]
}
# Response headers show: cache_creation_input_tokens: 0, cache_read_input_tokens: 30000The manual text must be byte-for-byte identical — same tokenizer output, same whitespace, same special tokens. A single character change (extra newline, different unicode dash) breaks the prefix match and forces full recomputation. This is the most common operational pitfall.
Common misconceptions
“Caching works across different models”
False. KV cache is model-specific — different architectures, layer counts, head dimensions, or even quantization schemes produce incompatible cache tensors. A cache entry for gpt-4o cannot be reused for gpt-4o-mini or a fine-tuned variant.
“Caching works across different sampling parameters”
Depends on the provider. Anthropic includes sampling params in the cache key by default. OpenAI and Google currently do not — the same prefix with temperature=0 and temperature=1 hits the same cache. But this is an implementation detail that could change; don’t build logic assuming either behavior.
“Cached tokens are free”
They’re discounted, not free. Anthropic charges ~10%, OpenAI 50%, Google ~25% of standard input token price. The discount reflects the remaining compute (attention over cached KV + output generation) and the GPU memory opportunity cost of holding the cache.
“Longer TTL is always better”
Longer TTL increases cache hit rate but consumes more GPU memory, which constrains max concurrent requests. Providers tune TTL to balance hit rate against memory pressure. You cannot configure TTL directly (except Anthropic’s ephemeral vs persistent tiers on some models).
“Prompt caching and KV cache offloading are the same thing”
Prompt caching is a product feature with pricing and API semantics. KV cache offloading (moving cache to CPU/NVMe between requests) is an infrastructure technique providers use to support longer TTLs or larger caches without OOM. Users see the former; the latter is invisible.
“You need to change your prompt structure to benefit”
For automatic prefix matching (OpenAI, Google), you only need consistent ordering. Put the large static context first — system prompt, then retrieved docs, then few-shot examples, then the variable user query. For explicit marking (Anthropic), you add cache_control blocks but the ordering rule still applies: cached prefix must be contiguous from the start.
When not to rely on it
Prompt caching helps when you have a stable, long prefix repeated across many requests. It doesn’t help for:
- Unique prompts every request (creative writing, diverse user queries with no shared context)
- Short prompts where prefill is already sub-millisecond
- Workloads where the shared context changes frequently (daily rotating docs, per-tenant customization)
- Streaming-first UX where first-token latency matters more than total prefill time — caching doesn’t accelerate the first request
If your p99 prefill latency is already acceptable and your input token spend is a small fraction of total cost, the engineering effort to structure prompts for caching may not pay off. Measure first.
Debugging cache hits
All three major providers return cache metadata in response headers or usage objects:
# OpenAI response usage
{
"usage": {
"prompt_tokens": 5000,
"completion_tokens": 200,
"prompt_tokens_details": {
"cached_tokens": 4500 # These were served from cache
}
}
}
# Anthropic response headers
# anthropic-cache-creation-input-tokens: 30000
# anthropic-cache-read-input-tokens: 0Log these fields. A dropping cached_tokens / prompt_tokens ratio signals cache misses — often due to prompt template drift, tokenizer version changes, or TTL expiration under load.
Summary
Prompt caching reuses KV cache for identical prompt prefixes, cutting prefill compute and latency roughly in proportion to the cached length. It requires exact token matches, has provider-specific pricing and TTL, and only benefits workloads with long, stable shared context. Structure your prompts with static content first, monitor cache hit rates via response metadata, and treat the discount as a bonus — not a guarantee — in your cost model.