Cache hit vs cache miss: latency benchmarks across providers

When you tune LLM serving, the spread between cache hit vs cache miss latency decides whether your app feels instant or broken. This post puts Anthropic, OpenAI, and Google side by side on how they implement prompt caching, what you pay, and how the latency actually behaves under load.

How each provider caches prompts

Anthropic was first to ship explicit cache control on the API surface. You mark a breakpoint in your message content; everything before it is cached for five minutes.

import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=256,
    system=[{
        "type": "text",
        "text": "You are a tax assistant. Here is the full IRS code: " + ("x" * 20000),
        "cache_control": {"type": "ephemeral"}
    }],
    messages=[{"role": "user", "content": "What is form 1040?"}]
)

Google Gemini takes a different route: you create a cached content object out of band, then reference it by ID.

from google import genai
client = genai.Client()
cache = client.caches.create(
    model="gemini-1.5-pro-002",
    contents="Full knowledge base: " + ("y" * 20000),
    ttl="300s"
)
resp = client.models.generate_content(
    model="gemini-1.5-pro-002",
    contents="Summarize section 3.",
    config={"cached_content": cache.name}
)

OpenAI does not require any special field. For GPT-4o and newer, the server automatically caches the longest prefix that matches a recent request. If you send the same system prompt and prefix within the TTL window (typically 5–10 minutes), you get a cache hit silently.

from openai import OpenAI
client = OpenAI()
# No cache_control needed; just reuse the exact prefix
resp = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "Long static instructions: " + ("z" * 20000)},
        {"role": "user", "content": "Hello"}
    ]
)

A gateway such as n4n.ai forwards these provider-specific cache hints unchanged through one OpenAI-compatible endpoint, so the same client code works against 240+ models without branching on vendor.

Cache hit vs cache miss latency in practice

The core metric is time to first token (TTFT). On a cold miss, the provider must run the full prefix through the transformer: attention over every token, KV compute, then generation. On a hit, the KV cache is loaded from memory and decoding starts almost immediately.

For a 20k-token prefix, observed TTFT on miss routinely lands in the 800–2000 ms range depending on batch load and region. A cache hit typically drops that to 100–400 ms. The exact number varies, but the order-of-magnitude gap is consistent: cache hit vs cache miss latency is the single biggest lever for interactive latency after model size.

Throughput tells a similar story. Miss requests consume prefill compute that saturates GPU memory bandwidth; hits skip prefill and let the batch scheduler pack more decode tokens. If you serve 50 RPS with a shared system prompt, caching can 2–4x your effective capacity.

Verifying a hit

Never assume the cache worked. Every provider exposes usage metadata:

# Anthropic
print(resp.usage.cache_creation_input_tokens, resp.usage.cache_read_input_tokens)
# OpenAI
print(resp.usage.prompt_tokens_details.cached_tokens)
# Gemini
print(resp.usage_metadata)

If the read counters are zero on your second identical request, you measured a miss and your latency number is lying. Warm the cache with one throwaway call before benchmarking.

Cost model

Anthropic charges a 25% premium on tokens written to cache, then 90% off on cached reads. OpenAI discounts cached input tokens by 50% versus uncached input. Gemini separates storage cost (per cached token-hour) from inference cost (cached tokens billed at 25% of base rate).

These are public list prices; volume discounts change absolute but not relative shape.

Ergonomics

Anthropic’s explicit cache_control is the most predictable. You know exactly what is cached and can place multiple breakpoints (up to 4). The downside is you must structure requests as content blocks, not a flat string.

Gemini’s out-of-band cache is clean for static corpora but adds a management step: create, poll, delete. If your context changes hourly, you own that lifecycle.

OpenAI’s automatic caching is zero-code but opaque. You must ensure byte-exact prefix match including whitespace and tool definitions, or the hit evaporates. In practice, templated system prompts with no random IDs are mandatory.

Failure modes

Caching breaks silently more often than it fails loudly.

• Anthropic: moving the breakpoint by even one token invalidates the cached segment.
• OpenAI: a trailing space in the system prompt on request #2 vs request #1 forces a full miss.
• Gemini: letting the TTL expire returns a 404 on the cache reference; you must recreate.

Always log the usage counters in production so you catch cache hit ratio drops before users feel the latency.

Ecosystem and limits

Anthropic: available on Claude 3+ family, minimum 1024 tokens to cache, max 4 breakpoints, 5 min TTL (extends on hit).

OpenAI: GPT-4o, GPT-4o-mini, o1 series; minimum prefix length varies by model (typically 1024+); no explicit limit on entries, evicted by LRU.

Gemini: 1.5 Pro/Flash, minimum 4096 tokens, TTL configurable up to 1 hour, explicit cache quota per project.

Head-to-head comparison

Which to choose

Long static system prompts, single model family. Use Anthropic if you are already on Claude. The explicit breakpoint removes guesswork and the 90% read discount is the deepest of the three.

Mixed models behind one endpoint. If you route across providers, an OpenAI-compatible gateway that forwards cache-control hints (like n4n.ai) lets you keep Anthropic’s explicit blocks for Claude and automatic caching for OpenAI without rewriting clients. OpenAI’s own API is fine if you stay in-model.

Massive knowledge bases with infrequent updates. Gemini’s separable cache object and 1-hour TTL win when you embed a 100k-token document and query it for an hour. Pay the storage fee; the 25% read rate still beats reprocessing.

High-throughput chat with shared prefix. Any of the three will cut cache hit vs cache miss latency dramatically. The deciding factor is operational: Anthropic gives you visibility, OpenAI gives you simplicity, Gemini gives you longevity.

Pick based on where your prefix lives and how long it stays put. The latency win is real on all three; the ergonomic and cost differences are what will actually change your architecture.