GPT-4o prompt caching cache hit latency is the difference between waiting for the model to recompute attention over your entire system prompt and reusing a precomputed KV cache from a previous request. In practice, a cache hit strips most of the prefill cost for repeated prefixes, but the real-world speedup depends on prefix length, request shape, and provider scheduling. This analysis breaks down where the latency win is real, where it is negligible, and how to measure it without guessing.
The mechanism: what a cache hit actually skips
Decoder-only transformers spend the first phase of inference—prefill—computing key/value tensors for every token in the input context. For a 4,000-token system prompt, that is 4,000 memory-bound matmuls before the first output token is generated. GPT-4o prompt caching stores those KV tensors server-side, keyed on the exact prefix hash. On a cache hit, the provider loads the cached KV and jumps straight to processing the uncached suffix (your per-request user message).
The saved time is roughly the prefill compute for the cached span. Prefill is memory-bandwidth bound, not compute bound, so the wall-clock saving scales near-linearly with cached token count up to the point where network and scheduling dominate. Fixed overhead—TLS handshake, load balancer, queueing, kernel launch—does not disappear on a cache hit.
# Order-of-magnitude mental model of prefill cost
def prefill_latency(total_tokens, cached_tokens=0):
fresh = total_tokens - cached_tokens
# ~0.1-0.2 ms per token prefill on H100-class hardware
return fresh * 0.15 + 25 # 25ms fixed overhead
print(prefill_latency(4000, 0)) # cold: ~625ms
print(prefill_latency(4000, 3500)) # warm: ~100msThe decode phase (generating completion tokens) is identical whether or not the prefix was cached. Therefore cache hits only affect time-to-first-token (TTFT), not tokens-per-second during generation.
Measuring GPT-4o prompt caching cache hit latency in practice
You cannot trust vague marketing claims about “up to 80% faster” without measuring your own traffic. The only authoritative signal is the cached_tokens field in the usage details, combined with client-side time_to_first_token measurements.
Setting up a reproducible test
Send two back-to-back requests with identical long prefixes. The first warms the cache; the second should hit it. Use the OpenAI Python SDK:
from openai import OpenAI
import time
client = OpenAI() # or point base_url at a gateway
system_prompt = "You are a meticulous contract reviewer. " * 700 # ~3.2k tokens
def call():
start = time.perf_counter()
resp = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Summarize the indemnification clause."}
]
)
ttft = time.perf_counter() - start
cached = resp.usage.prompt_tokens_details.cached_tokens
return ttft, cached
# warm-up (cold)
call()
# measured hit
ttft, cached = call()
print(f"TTFT: {ttft*1000:.0f}ms, cached_tokens: {cached}")If cached_tokens equals the prefix length (minus any non-cacheable trailing tokens), you got a hit. Run the second call many times and record percentiles; provider load will add variance.
Reading cache hit signals from the API
OpenAI exposes prompt_tokens_details in the usage object. A gateway such as n4n.ai forwards these fields unchanged and meters per-token usage, so you can plot cache hit ratio without instrumenting every client.
{
"usage": {
"prompt_tokens": 3400,
"completion_tokens": 12,
"prompt_tokens_details": {
"cached_tokens": 3200
}
}
}When cached_tokens is zero on the second call, either the prefix changed, fell below the 1024-token minimum, or the cache entry expired. The minimum cacheable prefix for GPT-4o is 1024 tokens; shorter static content will never hit.
When the speedup matters (and when it doesn’t)
Long static prefixes
If your application sends a 3k-token system prompt—legal boilerplate, codebase context, RAG retrieval results—on every request, GPT-4o prompt caching cache hit latency is a massive win. Public traces and our own logs show time-to-first-token dropping from over one second to sub-300ms once the prefix is stable and warm. The longer the prefix, the larger the fraction of total prefill eliminated.
Example: a support agent with a 5k-token knowledge base prefix and a 50-token user question. Full prefill processes 5,050 tokens; cache hit processes 50. The decode phase is identical, so the latency gap is almost the entire prefill. For interactive chat, that difference is the difference between “snappy” and “sluggish”.
Short or dynamic prompts
For a 200-token system prompt and a 300-token user turn, caching yields nothing—the prefix is below the minimum threshold. Even if you pad to 1k tokens, the saved prefill is a small part of total request time. Adding static filler to chase cache hits wastes tokens and can increase cost without meaningful latency gain.
# Anti-pattern: padding to force cache
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-d '{
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "Padding text repeated to exceed 1024 tokens..."}
]
}'The latency benefit is dwarfed by network RTT and queueing. Do not distort your prompt design for a cache that will not pay off.
Tradeoffs and operational caveats
Cache TTL and eviction
OpenAI holds cached prefixes for a short window (reported as 5–10 minutes of inactivity). If your traffic is bursty with gaps longer than the TTL, you pay full prefill on the first request after idle. Design for warm-up requests or accept the cold-start tax. For low-QPS services, the cache may rarely be warm.
Cost vs latency interaction
Cached input tokens are billed at a discount (50% on GPT-4o). Latency and cost move together: a cache hit is both cheaper and faster. But the discount only applies if the prefix is reused. Unique per-request prefixes get no discount and no speedup. The break-even point is when the same prefix is sent at least twice within the TTL.
Hash sensitivity
The cache key is the exact token sequence of the prefix. A single different token—a timestamp, a UUID, or reordered messages—invalidates the cache. Keep dynamic content at the end of the context, after the static block.
# Good: static first, dynamic last
messages = [
{"role": "system", "content": STATIC_POLICY},
{"role": "user", "content": user_query}, # varies, but after system
]
# Bad: timestamp inside the prefix
messages = [
{"role": "system", "content": f"Current time: {datetime.now()}. {STATIC_POLICY}"},
]Streaming and perceived latency
Most production apps stream completions. A cache hit does not make tokens stream faster, but it makes the stream start much sooner. For UX, TTFT is often more important than throughput. Cutting TTFT from 900ms to 250ms changes the feel of the product even if total generation time is unchanged.
How to structure prompts to reliably hit cache
- Put all immutable instructions, few-shot examples, and retrieved context at the top of the context window.
- Use a single system message for the static block; multiple messages are fine but must be byte-identical across calls.
- Append user-specific content after the static prefix.
- Monitor
cached_tokensin production; alert if it drops below expected. - For multi-turn conversations, treat the growing history as non-cacheable unless you truncate or pin the early turns.
For RAG pipelines, embed the retrieved documents in the system prompt rather than interleaving them with varying user text. The prefix remains constant; only the final user question changes.
Takeaway
GPT-4o prompt caching cache hit latency is a real, measurable win for workloads with long, repeated prefixes—expect a 2–4x reduction in time-to-first-token for prefixes beyond a few thousand tokens. For short or highly dynamic prompts, it is irrelevant overhead. Engineer your request shape to keep static content contiguous at the front, watch cached_tokens in usage, and let the cache discount pay for the latency gain. If you route through a gateway, ensure it preserves usage metadata so you can verify hits without custom instrumentation. The decisive move: measure your own TTFT with and without warm cache before assuming the optimization applies to your traffic.