How prompt caching speeds up multi-turn conversations

Prompt caching multi-turn conversations is the single highest-leverage optimization for latency-sensitive LLM applications. When you reuse the same system prompt, retrieved context, or conversation history across turns, you avoid recomputing attention over identical tokens — cutting both time-to-first-token and provider costs by 50-90% in typical workloads. The mechanism is straightforward: providers hash the prefix of your request, check for a cache hit, and serve the precomputed KV cache instead of reprocessing. But the devil lives in the details of cache keys, invalidation, and provider-specific behaviors.

How the cache key actually works

Every provider implements prompt caching slightly differently, but the core concept is identical. The cache key is derived from the exact token sequence at the start of your request — typically the system message, any developer instructions, and the first N user/assistant turns. A single character change (extra whitespace, different newline style, reordered JSON keys in a tool definition) produces a different hash and a cache miss.

{
  "model": "gpt-4o",
  "messages": [
    {"role": "system", "content": "You are a precise SQL generator."},
    {"role": "user", "content": "Schema: users(id, name, email)\nQuery: find users named Alice"},
    {"role": "assistant", "content": "SELECT * FROM users WHERE name = 'Alice';"},
    {"role": "user", "content": "Now find users named Bob"}
  ]
}

In this example, the first three messages form the cacheable prefix. The fourth message is the new turn. On the second request, the provider computes attention over the prefix once, stores the resulting key-value tensors, and only processes the new tokens. Subsequent turns in the same session reuse that cached prefix.

OpenAI’s implementation caches prefixes of 1024 tokens or more. Anthropic caches any prefix (minimum 1024 tokens for Claude 3.5 Sonnet, 2048 for Opus). Google’s Gemini caches context windows of 2048+ tokens. The token threshold matters: short system prompts won’t trigger caching unless you pad them or combine multiple requests.

Where the savings materialize

The latency win comes from skipping the prefill phase for cached tokens. Prefill scales quadratically with sequence length in standard attention (linear with flash attention, but still nonzero). For a 4k-token system prompt plus 2k tokens of conversation history, you’re avoiding ~36M attention operations per turn. At typical GPU throughput, that’s 200-800ms saved per request.

Cost savings follow the same logic. Providers charge a fraction of the normal input token price for cache hits:

• OpenAI: 50% discount on cached input tokens
• Anthropic: 90% discount on cached input tokens (but you pay a 25% premium to write to cache)
• Google: 75% discount on cached input tokens

A concrete scenario: a coding assistant with a 3k-token system prompt (instructions, few-shot examples, tool definitions) and 10-turn conversations averaging 500 tokens/turn. Without caching: 3k + 10×500 = 8k input tokens per conversation. With caching: 3k (first turn) + 10×500 = 8k, but 3k are cached after turn 1. At $2.50/M input tokens (GPT-4o), that’s $0.02 vs $0.0125 per conversation — 37% cheaper. At scale, this compounds.

Cache key design patterns that work

The most common mistake is treating the cache key as opaque. You control it through message ordering and content stability. Three patterns cover most production cases:

Pattern 1: Stable system prefix, dynamic suffix Put everything that never changes in the system message and the first few user/assistant turns. Tool definitions, few-shot examples, retrieved document chunks — these belong in the prefix. The active conversation grows at the end.

def build_messages(system_prompt: str, few_shots: list, history: list, current_query: str):
    messages = [{"role": "system", "content": system_prompt}]
    messages.extend(few_shots)  # stable, cacheable
    messages.extend(history[-6:])  # recent context, semi-stable
    messages.append({"role": "user", "content": current_query})  # dynamic
    return messages

Pattern 2: Versioned prompts with explicit cache busting When you deploy a new system prompt version, you need a clean cache miss. Append a version token to the system message:

SYSTEM_PROMPT_V3 = """You are a precise SQL generator.
[INSTRUCTIONS...]
CACHE_VERSION: 2024-01-15-v3"""

Changing v3 to v4 guarantees a miss. Don’t rely on implicit invalidation — providers don’t document TTL, and cache eviction is opaque.

Pattern 3: Deterministic serialization for structured inputs If you inject JSON schemas, retrieved chunks, or tool definitions, serialize them deterministically. json.dumps(obj, sort_keys=True, separators=(',', ':')) in Python. Any non-deterministic ordering (dict iteration, set ordering) breaks the cache key.

The multi-turn trap: growing context evicts your prefix

Here’s the catch that bites teams: most providers implement prefix caching with a fixed window. As your conversation grows, the oldest turns fall off the end of the cached prefix — but the beginning (your system prompt) stays cached. That’s the intended behavior. However, some providers (and some proxy layers) implement sliding-window caching where the entire conversation must fit in the cacheable region.

Anthropic’s documentation is explicit: “The cache is associated with the first N tokens of the conversation.” OpenAI is less specific but behaves similarly. Google’s context caching is a separate product — you explicitly create a cached context object with a TTL, then reference it. Different model, different semantics.

If you’re building a long-running agent (50+ turns), you’ll eventually exceed the cacheable prefix length. At that point, you have two options:

1. Summarize early turns and inject the summary as a new system message (cacheable, but loses detail)
2. Accept cache misses on older turns and optimize for the recent window

The right choice depends on your latency budget. For interactive chat, summarize aggressively. For batch processing, eat the miss.

Provider-specific gotchas

OpenAI: Cache hits only apply to the input_tokens portion of billing. cached_tokens appears in the usage object. The cache is per-model, per-organization, and survives across API keys. No explicit TTL documented — assume hours to days. Cache writes happen automatically on cacheable requests; you cannot force a write.

Anthropic: You must opt in with cache_control: {"type": "ephemeral"} on specific messages. This gives you control but adds complexity. The 25% write premium means very short conversations can cost more with caching enabled. Do the math: if your conversation is 2 turns, you pay 1.25× for the prefix write, then 0.1× for the read — net 1.35× vs 1.0× without caching. Break-even is around 3-4 turns.

Google: Context caching is a separate API (cachedContents.create). You create a cache with a TTL (default 1 hour, max 24 hours), get a cache name, then reference it in GenerateContentRequest. More verbose, but explicit control over lifetime. Useful for RAG pipelines where the retrieved corpus is stable for hours.

n4n.ai: The gateway normalizes these differences. You send standard OpenAI-format requests with cache_control hints; the gateway translates to each provider’s caching API and surfaces unified cached_tokens in the response. Fallback across providers preserves cache hits where the target provider supports equivalent prefix caching.

Measuring cache effectiveness in production

Don’t guess — instrument. Log these fields on every request:

{
  "request_id": "req_abc123",
  "model": "gpt-4o",
  "prompt_tokens": 4200,
  "cached_tokens": 3100,
  "cache_hit": true,
  "latency_ms": 420,
  "turn_number": 3
}

Track cache hit rate by turn number. You should see ~100% on turn 2+, dropping only when context window pressure evicts the prefix or when you deploy prompt changes. If hit rate is low on turn 2, your prefix is unstable — audit serialization, message ordering, and any dynamic content leaking into the “stable” region.

Correlate cached_tokens with latency. The relationship should be roughly linear: each 1k cached tokens saves ~50-150ms depending on model and provider load. If you see high cache hits but no latency improvement, you’re hitting a different bottleneck (queueing, rate limits, network).

When not to use prompt caching

Three scenarios where caching hurts more than helps:

1. Highly dynamic prefixes: If your system prompt includes the current timestamp, user-specific data, or per-request retrieved chunks that change every turn, you’ll never get a hit. The cache write overhead (Anthropic’s 25% premium, or just the provider’s internal bookkeeping) becomes pure cost.

2. Single-turn workloads: Classification, extraction, one-shot generation. No reuse = no benefit. The cache write (explicit or implicit) adds latency on the first request with no payoff.

3. Adversarial or privacy-sensitive contexts: Cached KV tensors persist in provider infrastructure. If your prefix contains PII, secrets, or user-specific proprietary data, you’re trusting the provider’s cache isolation. Some compliance regimes forbid this. Anthropic’s ephemeral cache is designed for this (TTL ~5-10 minutes), but verify your requirements.

The decisive takeaway

Prompt caching multi-turn conversations is not optional for production LLM applications — it’s table stakes. The implementation is trivial (stable prefix, deterministic serialization, versioned prompts), the latency gains are immediate (200-800ms/turn), and the cost reduction is real (30-90% on input tokens). The only engineering work is instrumenting hit rates and designing your message structure so the cache key stays stable.

Start with Pattern 1: stable system prompt + few-shots + recent history. Measure. If hit rate exceeds 80% on turn 2+, you’re done. If not, audit the prefix for instability. The providers have done the hard systems work; your job is feeding them a cacheable request.