Claude prompt caching: latency savings measured

Claude prompt caching latency savings come from skipping prefill on repeated context prefixes, but the practical impact varies wildly with request shape. This analysis breaks down when the cache actually hits, what it costs, and how to measure it in production. We will show request structures, usage fields, and the tradeoffs you must weigh before adopting it.

How the cache actually works

Anthropic treats the prompt as a sequence of tokens processed left to right. When you mark a prefix with cache_control: {type: "ephemeral"}, the provider stores the computed key/value states for that prefix on its accelerators. Subsequent requests with the exact same prefix can load those states instead of recomputing the attention passes.

The mechanism is a KV cache keyed by the token sequence. It is not semantic; it is byte-exact. The constraints are non-negotiable:

• Minimum cached prefix length is 1024 tokens for Claude 3.x models. Shorter prefixes are ignored.
• Idle entries expire after 5 minutes. A single request after expiry forces a full recompute and write.
• The prefix must match byte-for-byte, including whitespace, unicode normalization, and JSON key order.
• Write cost is 1.25× the base token price; read cost is 0.1×. These are published Anthropic rates.

That pricing means a cache miss costs more than a never-cached request. Latency on a miss includes the prefill plus the cache write. On a hit, you skip prefill and pay the discount.

Where the latency win comes from

Time-to-first-token (TTFT) on long prompts is dominated by prefill. In a transformer, every generated token requires attention over the full prefix. A 12k-token system prompt with a 20-token user question forces the model to process 12,020 tokens before emitting anything. With a cache hit, only the 20 new tokens are processed forward; the 12k prefix is a memory lookup.

The difference is typically seconds versus hundreds of milliseconds for large prefixes. That is the core of Claude prompt caching latency savings: you trade a one-time write penalty for repeated prefill avoidance. The savings scale with prefix length and repeat count.

Here is a minimal Anthropic request that caches a system prompt:

{
  "model": "claude-3-5-sonnet-20241022",
  "system": [
    {
      "type": "text",
      "text": "You are a strict SQL reviewer. Rules: ... [long static content] ...",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    { "role": "user", "content": "Review: SELECT * FROM users;" }
  ]
}

If the next request keeps the same system block and changes only the user message, the prefix hits. The model never re-reads the SQL rules.

Prefill math

Assume prefill throughput of ~2,000 tokens/sec on the serving hardware (a rough industry norm, not a measured claim). A 12k-token miss takes ~6 seconds before first token. A hit with 20 new tokens takes ~10 ms of compute plus lookup. The gap is what makes agentic systems feel responsive.

Measuring hits without guessing

You cannot tune what you cannot see. Anthropic returns cache_creation_input_tokens and cache_read_input_tokens in the usage object. A hit shows zeros in creation and non-zero in read.

import anthropic

client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    system=[{"type": "text", "text": LONG_PROMPT, "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": query}],
)
usage = resp.usage
print(usage.cache_creation_input_tokens, usage.cache_read_input_tokens)

When cache_read_input_tokens equals your prefix size, you got the win. When cache_creation_input_tokens is non-zero and read is zero, you paid the write premium and got nothing reusable yet. Tracking these per request is the only honest way to quantify Claude prompt caching latency savings for your traffic.

Pipe these fields into your metrics system. A dashboard showing read/write ratio per route exposes dead caches quickly.

Failure modes that silently kill hits

The 5-minute TTL is brutal for low-traffic endpoints. A cron job that calls Claude every 10 minutes will never hit. A user session that pauses for six minutes resets the cache.

Prefix drift is worse. Injecting a timestamp into the system block, or reordering JSON keys, invalidates the match. We have seen teams put a request_id in the cached prefix and wonder why savings vanished. Even trailing whitespace differences between serialized configs bust the key.

Short prompts below 1024 tokens are never cached. If your static context is 800 tokens, the feature does nothing but add overhead.

Model changes also bust the cache. Routing from claude-3-5-sonnet to claude-3-7-sonnet mid-session forces a miss. So does changing the anthropic-version header in some SDK paths.

Cost versus latency tradeoff

The write multiplier means caching is a bet on reuse. If you write 10k tokens at 1.25× and then read them once at 0.1×, you spent more than not caching. You need at least two reads in the TTL window to break even on token cost, ignoring latency gains.

Break-even reads ≈ (write_multiplier) / (write_multiplier - read_multiplier) ≈ 1.25 / (1.25 - 0.1) ≈ 1.09, so two reads clears it. But that is token cost only.

Latency is its own currency. In an agentic loop where a 10k-token instruction set is reused across 20 tool calls in a minute, the TTFT collapse changes UX from sluggish to snappy. There, Claude prompt caching latency savings justify the write cost even at modest hit counts because human wait time drops.

Gateway and routing implications

If you sit behind an inference gateway, cache behavior depends on request fidelity. n4n.ai forwards provider cache-control hints unchanged and meters per-token usage so the read discount appears in your bills. But automatic fallback to a different provider on degradation will route around Anthropic entirely, and the new provider has no copy of the prefix. Your client routing directives must pin the model when cache locality matters.

That is a real trade: resilience versus cache warmth. Decide per workload. For batch jobs, pin and cache. For latency-critical user flows with fallback, accept misses.

Structuring prompts for hits

Put the largest stable block first. Place volatile content—user input, timestamps, ephemeral state—after the breakpoint. If you need multiple cached sections, Anthropic allows breakpoints on multiple content blocks, but each is billed independently.

{
  "model": "claude-3-5-sonnet-20241022",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "STATIC CONTEXT A", "cache_control": { "type": "ephemeral" } },
        { "type": "text", "text": "STATIC CONTEXT B", "cache_control": { "type": "ephemeral" } },
        { "type": "text", "text": "dynamic user input here" }
      ]
    }
  ]
}

This nests caches, but watch the 1024-token minimum per block. If block A is 500 tokens, it will not cache and may push block B’s effective start later.

Agentic example

A coding agent loads a 15k-token repo summary as system context, then issues 30 tool calls per session. Without caching, every call pays 15k prefill. With caching, the first call writes, the next 29 read. The cumulative TTFT drop is massive. That is the poster child for Claude prompt caching latency savings.

A decision framework

Use Claude prompt caching when all hold:

1. Static prefix ≥ 1024 tokens.
2. Expected ≥ 2 requests reusing it within 5 minutes.
3. Prefix is byte-stable (no injected nonce, no reordered keys).

Skip it when prompts are single-shot, highly dynamic, or short. The write surcharge is pure tax.

For batch evaluation of a fixed rubric across 1k examples, the cache is mandatory. For a chat app with unique per-user system prompts, it is useless.

Takeaway

Claude prompt caching latency savings are real but earned. Mark a stable, sufficiently long prefix, watch cache_read_input_tokens to confirm hits, and never pay the write premium without a reuse plan. In workloads with repeated long context, it is the highest-leverage optimization available; elsewhere it is dead weight. Measure, then commit.