Prompt caching TTL: latency impact as caches expire

The prompt caching TTL latency you observe in a production LLM pipeline is not a static number. After a cached prefix expires, the next request absorbs the full prefill cost, turning a sub-second time-to-first-token into a multi-second stall that surfaces only in p99 charts. If you treat caching as a free win and ignore the expiry window, you will misread your own telemetry and ship regressions that look like “slow models.”

How prefix caching changes the latency curve

Most inference providers reuse the KV cache for identical prompt prefixes. On a transformer, the prefill step computes attention keys and values for every token in the context. If your system sends an 8,000-token system prompt plus variable user input, the provider can compute those keys/values once and reuse them for subsequent calls that share the same prefix. The client signals this intent via cache-control markers.

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {
      "role": "system",
      "content": "Long static instructions, policy text, and tool schemas...",
      "cache_control": {"type": "ephemeral"}
    },
    {"role": "user", "content": "{{variable input}}"}
  ]
}

On a hit, TTFT drops because the GPU skips prefill for the cached span. On a miss, the provider must process the entire prefix from scratch. The gap between those two states is the prompt caching TTL latency penalty, and it scales with prefix length rather than with how little your user input changed.

What expires, and when

Providers attach a time-to-live to cached prefixes. Anthropic documents a 5-minute TTL for ephemeral caches; OpenAI applies automatic prompt caching with a similar window for most models, though exact durations vary by model family. The cache is also scoped per model, per region, and per provider account. If no request repeats the prefix within the TTL, the entry is evicted.

That eviction is silent. Your code receives no error; it just gets billed for cache_write_input_tokens on one call and cache_read_input_tokens on the next hit. The latency difference is invisible to the API contract but brutal to user experience. A cache that worked perfectly at 10:00 can be cold at 10:06 if traffic dipped.

Quantifying the miss penalty

Prefill latency scales roughly linearly with prompt length because the hardware must materialize the full KV tensor. A 10k-token prefix that hits cache might return first token in 150–300 ms on current GPU classes. A cold miss on the same prefix can take 2–4 seconds depending on batching, model size, and provider load. Those ranges are qualitative observations from common deployments, not a published benchmark, but the order-of-magnitude gap is consistent.

To track this, wrap the call and parse the usage object:

import time, openai
client = openai.OpenAI()

start = time.monotonic()
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=msgs,
)
latency_ms = (time.monotonic() - start) * 1000
usage = resp.usage
cache_read = getattr(usage, "cache_read_input_tokens", 0)
cache_write = getattr(usage, "cache_write_input_tokens", 0)

if cache_read == 0 and cache_write > 0:
    # this request paid the full prefill cost
    record_cache_miss(latency_ms)

Logging cache_read_input_tokens over time reveals your real prompt caching TTL latency exposure. If you only chart mean latency, the hits mask the misses.

The periodic spike pattern

Consider a support bot that gets 50 req/min during business hours but drops to 2 req/min at night. The long system prompt caches reliably while traffic is high. At 2 a.m., the interval between identical prefixes exceeds the TTL. The first morning request after a quiet period pays the cold cost.

Users perceive a “slow server” exactly when they are least tolerant. Worse, if you route through a gateway that load-balances across regions, a cache hit in us-east-1 does nothing for a request sent to eu-west-1. The prompt caching TTL latency benefit is local to the cache instance.

Gateway routing and cache locality

Sticky routing is the simplest fix. If your gateway supports client routing directives, pin a session or tenant to a single region and provider for the cache lifetime. n4n.ai forwards provider cache-control hints and honors routing directives, so a header can keep prefixes warm where they were written:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-n4n-route: provider=anthropic;region=us-east-1" \
  -d '{"model":"claude-3-5-sonnet","messages":[...]}'

Without that pin, automatic fallback—useful when a provider is rate-limited—can silently move traffic and erase your cache. The fallback protected your throughput but quietly imposed the full prompt caching TTL latency tax on the next request.

Should you send keepalive requests?

A tempting mitigation is to periodically replay the static prefix to reset the TTL. This works, but it costs input tokens and compute. At typical input rates, a 10k-token keepalive every 4 minutes is not free at scale, and providers reserve the right to evict caches under memory pressure regardless of TTL. A keepalive reduces but does not eliminate miss risk.

Tradeoff: if your p99 latency SLA is strict and traffic is bursty, a warm-up before predicted spikes (e.g., a cron that replays the prefix at 7:55 a.m.) is cheaper than angry users. If traffic is steady, natural repeats handle it. Do not blindly keepalive every tenant; measure miss rate first.

Cache versioning pitfalls

Any change to the cached prefix invalidates it. Embedding a timestamp, rotating a schema, or adding whitespace breaks the match. We have seen teams append a request ID to the system prompt “for tracing” and wonder why cache hits never occur. The prompt caching TTL latency problem is secondary to a cache that never forms.

Version your static prefix explicitly. If you must change it, accept a one-time cold period and consider dual-writing old and new prefixes during transition.

Streaming and perceived latency

With streaming responses, total generation time is dominated by decode steps, but perceived responsiveness is set by TTFT. A 3-second cold prefill before the first token arrives feels broken even if the subsequent tokens stream at 80 tps. The prompt caching TTL latency cliff therefore hurts UX disproportionately compared to its share of wall-clock time.

A decision framework

• Measure hit rate, not just latency. Tag histograms by cache_read_input_tokens > 0.
• Pin routes for cache-sensitive flows. Use routing directives for long-prefix tenants.
• Alert on miss rate. A rising miss ratio predicts latency complaints before p99 crosses threshold.
• Warm deliberately. Cron-driven keepalives for known low-traffic windows beat reactive retries.
• Account for cache writes in cost. cache_write_input_tokens is billed; factor it into unit economics.

from collections import defaultdict
buckets = defaultdict(list)
buckets["hit" if cache_read > 0 else "miss"].append(latency_ms)

Takeaway

Prompt caching TTL latency is a hidden variable that turns quiet periods into latency cliffs. Measure cache hit rate per route, pin traffic to preserve locality, and warm caches deliberately before demand spikes. Ignore the TTL and your p99 will quietly betray you.