Prompt caching vs KV cache: what's the difference?

Prompt caching and KV cache solve different problems at different layers of the inference stack. Prompt caching is a provider-level feature that avoids recomputing the same prompt prefix across requests. KV cache is a model-level optimization that avoids recomputing attention keys and values for tokens already generated during a single request. Confusing them leads to wrong capacity planning and missed optimization opportunities.

What prompt caching actually does

Prompt caching stores the computed hidden states (or the pre-fill computation result) for a prompt prefix that appears repeatedly across independent requests. When you send a request whose first N tokens match a cached prefix, the provider skips the pre-fill for those tokens and jumps straight to generation. The cache key is typically the exact token sequence — system prompt, few-shot examples, RAG context, or any fixed prefix you control.

Providers implement this differently. Anthropic caches the first 1024 tokens (configurable) with a 5-minute TTL. Google caches prefixes of 1024+ tokens with longer TTLs. OpenAI’s implementation caches any prefix of 1024+ tokens automatically. All three return cache hit/miss metadata in the response headers so you can measure effectiveness.

The critical constraint: prompt caching only helps when multiple requests share the same prefix. It does nothing for the first request, and it does nothing for the continuation of a single conversation. If your traffic pattern is one long conversation per user, prompt caching buys you almost nothing.

{
  "model": "claude-3-5-sonnet-20241022",
  "system": "You are a helpful assistant...",
  "messages": [
    {"role": "user", "content": "Analyze this document: [10KB document]"}
  ],
  "metadata": {"cache_control": {"type": "ephemeral"}}
}

What KV cache actually does

KV cache lives inside the model’s forward pass. During pre-fill, the model computes attention keys and values for every token in the prompt and stores them in GPU memory. During decoding, each new token attends to all previous tokens — but the keys and values for previous tokens never change. So the model reuses the cached K and V tensors instead of recomputing them.

This is why generation is fast (one token per forward pass) while pre-fill is slow (quadratic in sequence length). Without KV cache, every decoding step would re-run attention over the entire history, making generation O(n²) instead of O(n).

KV cache memory scales with: batch size × sequence length × layers × heads × head dimension × 2 (K and V) × precision bytes. For Llama-3-70B at FP16 with 4K context: roughly 1.7 GB per sequence. This is why long-context inference eats VRAM and why techniques like KV cache quantization, sliding window attention, and paged attention (vLLM) exist.

# Simplified KV cache update in a decoder layer
def forward(self, x, past_kv=None):
    # x: [batch, seq_len, hidden]
    q = self.q_proj(x)
    k = self.k_proj(x)
    v = self.v_proj(x)
    
    if past_kv is not None:
        past_k, past_v = past_kv
        k = torch.cat([past_k, k], dim=1)
        v = torch.cat([past_v, v], dim=1)
    
    # attention with full k, v
    attn_out = self.attention(q, k, v)
    
    return attn_out, (k, v)  # return updated cache

Head-to-head comparison

Capabilities and behavior differences

Prompt caching is a traffic optimization. It reduces load on the provider’s pre-fill kernels when many requests share context — think: same system prompt across thousands of chat turns, same RAG chunks across document QA, same few-shot examples across classification tasks. The cache key is discrete: either the prefix matches exactly (token-for-token) or it doesn’t. Whitespace, casing, or a single token difference invalidates the match.

KV cache is a compute optimization. It is mandatory for any practical autoregressive decoding. You cannot disable it without rewriting the attention kernel. The cache grows token-by-token during generation. For multi-turn conversations, the entire history (system + all turns) occupies KV cache until the request ends or the context window fills.

Some providers let you reserve prompt cache space (Anthropic’s cache_control) or pin KV cache for a session (via session IDs or sticky routing). These are orthogonal: prompt cache reservation guarantees your prefix stays hot; KV cache pinning keeps a conversation’s tensors resident across requests to avoid re-prefill. The latter is rare and usually requires dedicated instances.

Cost and pricing models

Prompt caching directly reduces your bill. Anthropic charges ~10% of the normal input token price for cache hits. Google charges ~25%. OpenAI charges 50% for cached tokens. The savings apply only to the cached prefix tokens — the uncached suffix and all output tokens bill at full rate.

# Example: 2000-token system prompt, 500-token user query, 200-token response
# Without caching: 2500 input + 200 output tokens
# With caching (second request): 500 input + 200 output + 2000 cached @ 10%

KV cache has no per-token price. Its cost is capacity: each concurrent sequence reserves VRAM proportional to context length. On self-hosted GPUs, this limits batch size. On serverless APIs, it’s baked into the per-token price (which is why long-context models cost more). KV cache quantization (FP8, INT4) reduces memory 2-4x with minimal quality loss — a lever you control when self-hosting, but not via most APIs.

Latency and throughput implications

Prompt caching cuts pre-fill latency for cache hits. A 2000-token prefix that takes 200ms to pre-fill might take 5ms on a hit. The first request (cold) sees no benefit. Throughput improves because the pre-fill kernels are freed for other requests.

KV cache determines decode latency and maximum throughput. Decode is memory-bandwidth bound: each token reads the entire KV cache for all layers. Longer context = more bandwidth per token = slower decode. Throughput (tokens/sec) drops as context grows because the same GPU memory bandwidth serves fewer concurrent sequences.

# Rough decode latency scaling with context (A100, Llama-3-70B, FP16)
# 1K context:  ~45 tok/s
# 4K context:  ~38 tok/s  
# 8K context:  ~30 tok/s
# 32K context: ~18 tok/s

Prompt caching and KV cache interact: a prompt cache hit means the provider also has the KV cache for that prefix resident (or can reconstruct it instantly). So a cache hit skips both the pre-fill compute and the initial KV cache population. This is why cache hits feel disproportionately fast.

Ergonomics and integration

Prompt caching requires prompt discipline. You must structure requests so the cacheable prefix is stable and at the beginning. Dynamic content (user queries, retrieved chunks that vary per request) must come after the cached prefix. This often means reordering your prompt template:

# Good: stable prefix first
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},           # cached
    {"role": "user", "content": FEW_SHOT_EXAMPLES},         # cached
    {"role": "user", "content": RETRIEVED_CONTEXT},         # cached if stable
    {"role": "user", "content": USER_QUERY},                # NOT cached
]

# Bad: variable content breaks the prefix
messages = [
    {"role": "user", "content": USER_QUERY},                # breaks cache
    {"role": "system", "content": SYSTEM_PROMPT},           # never cached
]

KV cache ergonomics matter when you self-host or use frameworks like vLLM, TGI, or SGLang. You tune: block size (paged attention), quantization (FP8/INT4), CPU offloading, chunked pre-fill. Most APIs expose none of this — you get the provider’s defaults. n4n.ai forwards provider cache-control hints and honors client routing directives, so you can express preferences without locking into one provider’s KV cache configuration.

Limits and gotchas

Prompt caching limits:

• Minimum prefix length (usually 1024 tokens) before caching activates
• TTL expiration (5 min to 1 hour) — bursty traffic loses cache benefit
• Exact token match required — template interpolation breaks it
• Provider-specific: cache doesn’t transfer across models or providers
• No visibility into eviction policy — “cold” requests are unpredictable

KV cache limits:

• VRAM is the hard ceiling — OOM kills the request or the server
• Context window = max KV cache length (sliding window exceptions aside)
• Quantization adds decode latency (dequantize overhead)
• CPU offloading kills throughput (PCIe bandwidth << HBM)
• Multi-GPU: KV cache sharding adds communication overhead

A subtle gotcha: prompt caching increases peak KV cache pressure on the provider. Cached prefixes mean more concurrent requests can start decoding simultaneously, each holding their full KV cache. Providers manage this with admission control, but it can manifest as higher tail latency or 429s during cache-heavy bursts.

Which to choose

Use prompt caching when:

• You control the prompt template and can stabilize a ≥1024 token prefix
• Traffic volume is high enough that cache hit rate exceeds ~30%
• Cost reduction on repeated prefixes matters (high-volume classification, RAG with shared corpus, multi-tenant system prompts)
• You’re using a provider API and cannot touch the model runtime

Use KV cache optimization (self-hosted or dedicated) when:

• You need long context (16K+) and hit VRAM limits
• Decode latency is the bottleneck (real-time streaming, low-latency SLAs)
• You want to quantize KV cache to FP8/INT4 for 2-4x capacity
• You need custom eviction (sliding window, attention sinking) for ultra-long context
• You run batch inference and need maximum throughput per GPU

Use both when:

• High-volume API workload with stable prefixes and you self-host for control
• You pin KV cache for active sessions and prompt-cache the shared system prompt
• Example: a coding assistant with a fixed system prompt (prompt cached) and long conversations (KV cache pinned per session)

Ignore prompt caching when:

• Every request has a unique prefix (personalized prompts, unique document per request)
• Traffic is too low for cache warmth (<10 req/min per prefix)
• You’re already self-hosting and can optimize KV cache directly

Ignore KV cache tuning when:

• You only use serverless APIs with no runtime control
• Context stays under 4K and batch size fits comfortably in VRAM
• Latency requirements are loose (>500ms TTFT acceptable)

The shortest path: start with prompt caching on your provider — it’s a configuration change, not an architecture change. Measure hit rate and cost savings. If you hit VRAM walls or decode latency floors, move to self-hosted with KV cache quantization and paged attention. The two optimizations compose cleanly because they operate at different layers.