Prompt caching latency at scale: thousands of requests

Prompt caching latency at scale is rarely the clean win that vendor benchmarks suggest. Once you push past a few thousand requests per minute, cache hit rate, prefix alignment, and provider eviction policies dictate whether you see a 10ms tail improvement or a 10x regression in time-to-first-token (TTFT). This analysis breaks down where the wins come from, where they fall apart, and how to engineer prompts that survive contact with production traffic.

What prompt caching actually saves

A transformer prefill step computes the KV cache for every token in the input prompt. That cost scales with prompt length and batch size. Prompt caching lets the provider reuse a previously computed KV cache for a matching prefix, skipping the prefill compute for those tokens.

The latency win is concentrated in TTFT, not token generation. Generation still runs the full decode loop regardless of cache. So if your workload is inference-heavy with short prompts and long outputs, caching moves the needle less than a long-system-prompt, short-output pattern.

Anthropic’s documentation cites up to 85% TTFT reduction on long cached prefixes. In mixed production traffic with 1–4k token prefixes, observed reductions typically land in the 40–60% range because partial hits and cache warmup eat into the theoretical max.

Latency under load: from single digits to thousands of RPS

At low request rates, a cache hit is a cache hit. The provider has abundant GPU memory to pin your prefix. At thousands of requests per second, the dynamics change:

• Cache footprint competition. Every distinct prefix consumes KV memory. With 100 byte-distinct system prompts across 1k RPS, you are asking the provider to hold 100 large KV blocks. They evict.
• Prefix fragmentation. A single trailing whitespace difference forces a miss. At scale, minor template drift (timestamps, UUIDs in the prefix) silently drops hit rate from 95% to near zero.
• Batching interference. Providers batch prefills. A cache miss on a hot prefix stalls the batch behind it, inflating TTFT for unrelated requests.

Cache hit rate is a function of prefix discipline

The cache key is the exact token sequence from the start of the conversation. If you put a nonce at the top of your system prompt, you have zero hits. The fix is rigid prefix ordering: static instructions first, mutable data last.

{
  "model": "anthropic/claude-3.5-sonnet",
  "messages": [
    {
      "role": "system",
      "content": "You are a SQL analyst. Always output valid Postgres. Use UTC. Never drop tables.",
      "cache_control": {"type": "ephemeral"}
    },
    {
      "role": "user",
      "content": "Generate a report for tenant 8821 from 2024-01-01 to 2024-03-31"
    }
  ]
}

The cache_control breakpoint tells the provider to cache everything up to that message boundary. At scale, you want exactly one breakpoint on the static prefix, not per-message breakpoints that multiply cache entries.

Eviction and contention at scale

Providers use LRU or frequency-based eviction on KV blocks. If your prefix is requested every 30 seconds but the provider serves 10k other tenants, your block may be gone. The only reliable mitigation is high request density on a single prefix or a dedicated capacity arrangement.

A gateway such as n4n.ai forwards provider cache-control hints on its OpenAI-compatible endpoint, so the same breakpoint logic works across 240+ models without per-provider SDK branching. That removes one excuse for prefix inconsistency, but does not solve eviction.

Measuring prompt caching latency at scale

You cannot tune what you do not measure. A minimal harness sends paired requests: one with a cold prefix (forced miss) and one with a warm prefix (forced hit), under load.

import asyncio, time, random
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://api.example.com/v1", api_key="key")

async def ttft(messages):
    t0 = time.monotonic()
    stream = await client.chat.completions.create(
        model="anthropic/claude-3.5-sonnet",
        messages=messages,
        stream=True
    )
    async for chunk in stream:
        if chunk.choices[0].delta.content:
            return time.monotonic() - t0
    return time.monotonic() - t0

async def main():
    static = {"role": "system", "content": "STATIC PREFIX " * 200,
              "cache_control": {"type": "ephemeral"}}
    for i in range(5000):
        msgs = [static, {"role": "user", "content": f"query {random.randint(0,9999)}"}]
        # alternate cold by mutating prefix occasionally
        if i % 100 == 0:
            msgs[0] = {"role": "system", "content": "STATIC PREFIX " * 200 + str(i)}
        await ttft(msgs)

asyncio.run(main())

Run this against your production-like prefix length and concurrency. Plot TTFT percentiles for hit vs miss. If p99 miss TTFT is 3x p99 hit TTFT at 2k RPS, caching is earning its keep. If the gap collapses under load, your prefix is getting evicted.

Tradeoffs: when caching hurts

Caching is not free:

• Memory tax. Cached prefixes occupy GPU memory that could batch more requests. Over-caching short prefixes wastes capacity.
• Stale context. Ephemeral caches expire (often 5–10 min). If your static prefix embeds a version string that changes hourly, you pay miss cost on every rotation.
• Breakpoint overhead. Some providers bill a small write fee on cache creation. At 10k RPS with a 1k-token prefix, that is a non-trivial per-token surcharge if hit rate is low.

The decisive factor is hit rate. Below ~70% hit rate on a given prefix, the operational complexity rarely pays off unless TTFT is a hard SLA.

Designing for cache-friendly prefixes

Engineer the prompt, not the cache config.

1. Freeze the first N tokens. Put boilerplate, role, and constraints first. Put user variables, retrieved docs, and timestamps last.
2. Canonicalize. Trim whitespace, sort JSON keys, avoid randomized separators.
3. Single breakpoint. Use one cache_control at the end of the static block. Multiple breakpoints split the cache and reduce eviction resilience.
4. Monitor hit rate. Log the provider’s cache read/write metadata if exposed. Without it, you are guessing.

# Example: extract cache metrics from response headers (provider-dependent)
curl -s -D - -o /dev/null https://api.example.com/v1/chat/completions \
  -H "content-type: application/json" \
  -d '{"model":"anthropic/claude-3.5-sonnet","messages":[{"role":"system","content":"static","cache_control":{"type":"ephemeral"}},{"role":"user","content":"hi"}]}'
# Look for x-cache-read: hit / x-cache-write: miss

Takeaway

Prompt caching latency at scale is a function of prefix discipline and request density, not a toggle you flip. At thousands of requests per minute, treat your static prefix as a fixed memory address: keep it immutable, monitor hit rate relentlessly, and accept that below 70% hits the complexity costs more than the saved milliseconds. Build the measurement harness before you build the cache strategy, and let the p99 TTFT delta make the call.