OpenAI’s automatic prompt caching is a server-side optimization that stores the computed attention keys and values for the longest matching prefix of a prompt, then reuses that cached state on subsequent requests that share the same prefix. The cache activates automatically when a request’s prompt begins with at least 1,024 tokens identical to a previously cached prefix, and it applies to both chat completions and the Assistants API. You pay only for the uncached tokens processed; cached tokens are billed at a 50% discount and do not count toward rate limits.
How the cache key works
The cache key is the exact token sequence from the start of the prompt up to the point of divergence. OpenAI tokenizes your input, then walks the token stream from the beginning looking for the longest prefix that exists in the cache. That prefix must be at least 1,024 tokens long to qualify. Once a match is found, the model skips recomputing the attention matrices for those tokens and resumes forward computation from the first uncached token.
{
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Analyze this 2,000-token document: [DOCUMENT]"}
]
}If you send the same system prompt and document prefix across multiple requests — say, asking different questions about the same document — the system prompt plus the document tokens form the cacheable prefix. The first request pays full price for the entire prompt. Subsequent requests pay full price only for the new question tokens; the shared prefix is billed at the cached rate.
Cache lifetime and eviction
Cached prefixes expire after 5–10 minutes of inactivity. OpenAI does not publish the exact TTL, but empirical testing shows eviction typically occurs between 5 and 10 minutes after the last request that used the prefix. There is no manual invalidation API. If you need deterministic cache behavior, structure your workloads so that related requests arrive in quick succession.
The cache is scoped to the model and the organization. Two different organizations sharing the same model do not share cache entries. Within an organization, any API key can benefit from a prefix cached by another key.
What counts toward the 1,024-token minimum
The threshold applies to the tokenized prompt, not character count. A system prompt of 200 tokens plus a 900-token document does not qualify. You need 1,024 tokens of identical prefix after tokenization. This matters because:
- Different tokenizers (cl100k_base vs. o200k_base) produce different token counts for the same text
- Whitespace, formatting, and special characters affect tokenization
- The
toolsandtool_choiceparameters are part of the prompt token stream and count toward the prefix
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # gpt-4o tokenizer
system = "You are a legal analyst. Be precise and cite sources."
doc = "..." # your document text
prompt_tokens = enc.encode(system + doc)
print(len(prompt_tokens)) # must be >= 1024Billing mechanics
Cached tokens appear in the usage field of the response:
{
"usage": {
"prompt_tokens": 3500,
"completion_tokens": 200,
"total_tokens": 3700,
"prompt_tokens_details": {
"cached_tokens": 3000
}
}
}You are billed:
- Full price for
prompt_tokens - cached_tokens - 50% price for
cached_tokens - Full price for
completion_tokens
Rate limits also reflect this split. Cached tokens do not consume your tokens-per-minute (TPM) quota. Only uncached prompt tokens and completion tokens count toward limits.
Concrete example: multi-turn document Q&A
Consider a legal review workflow where analysts ask 10 different questions about the same 50-page contract.
Without caching (naïve approach): Each request sends the full contract + system prompt + question. At ~15,000 tokens per request, 10 requests = 150,000 prompt tokens billed at full price.
With automatic caching: Request 1: Full price for 15,000 tokens. Cache populated. Requests 2–10: Only the question tokens (~200 each) billed at full price. The 14,800-token prefix billed at 50%. Total billed prompt tokens: 15,000 + 9 × (200 + 14,800 × 0.5) = 15,000 + 9 × 7,600 = 83,400 equivalent full-price tokens. ~44% reduction in prompt token spend.
The latency improvement is equally significant. The model skips the forward pass for the cached prefix, so time-to-first-token drops roughly in proportion to the cached token count. For a 15,000-token prefix, you can see 2–3× faster first-token latency on cached requests.
Structuring prompts for maximum cache hit rate
Put the largest stable content first. The cache matches from the start of the prompt, so order matters:
// Good: large stable prefix first
{
"messages": [
{"role": "system", "content": "[LONG SYSTEM PROMPT]"},
{"role": "user", "content": "[LONG DOCUMENT]"},
{"role": "user", "content": "Question 1"}
]
}
// Bad: variable content breaks the prefix
{
"messages": [
{"role": "user", "content": "Question 1"},
{"role": "system", "content": "[LONG SYSTEM PROMPT]"},
{"role": "user", "content": "[LONG DOCUMENT]"}
]
}In the bad example, the first user message changes every request, so the cacheable prefix is only the system prompt — likely under 1,024 tokens. No cache activation.
Use a consistent system prompt across all requests in a session. If you need per-request instructions, append them after the cached prefix, not before.
Tools and function definitions count toward the prefix
When you pass tools or functions parameters, their JSON schemas are tokenized and prepended to the prompt. If you use the same tool set across requests, those tokens become part of the cacheable prefix. This can help you cross the 1,024-token threshold.
However, changing even one tool description invalidates the entire prefix. If you have a large tool set that varies slightly per request, consider splitting into a stable core tool set (cached) and a dynamic supplement passed via the user message.
Streaming and caching
Streaming responses (stream: true) work identically. The cache decision happens before the first token streams. You will see the cached_tokens field in the final usage chunk (or in the usage field of the last chunk if stream_options.include_usage is set).
stream = client.chat.completions.create(
model="gpt-4o",
messages=messages,
stream=True,
stream_options={"include_usage": True}
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
if chunk.usage:
print(f"\nCached tokens: {chunk.usage.prompt_tokens_details.cached_tokens}")Common misconceptions
“Caching works for any repeated content”
False. Only a prefix match from token 0 qualifies. Repeated content in the middle or at the end of the prompt does not activate the cache. If your workflow appends context rather than prepending it, you will not benefit.
“The cache persists across sessions or days”
False. The 5–10 minute TTL is short by design. This is a request-affinity optimization, not a long-term storage layer. Do not architect workflows assuming cache persistence beyond a few minutes.
“Cached tokens are free”
False. They are billed at 50% of the standard prompt token price. Check the current pricing page for your model; the discount is significant but not zero.
“I can see cache hits in the response headers”
False. The only signal is the prompt_tokens_details.cached_tokens field in the usage object. There is no X-Cache-Hit header or similar.
“Caching works the same on all models”
False. Automatic prompt caching is available on GPT-4o, GPT-4o-mini, GPT-4-turbo, and newer models. Older models (GPT-3.5-turbo, GPT-4 base) do not support it. The 1,024-token threshold and 50% discount are consistent across supported models, but always verify the model’s documentation.
“Using n4n.ai or other gateways breaks caching”
False. The cache key is derived from the tokenized prompt that reaches OpenAI’s inference servers. As long as the gateway forwards the request without modifying the prompt token stream — including system messages, tools, and user content in the same order — caching works identically. Some gateways add request IDs or modify headers; those do not affect the prompt tokens. n4n.ai forwards the prompt unchanged and passes through the cached_tokens field so you can observe the same savings.
When not to rely on automatic caching
- Highly variable prefixes: If every request starts with unique content (different documents, different system prompts), you will rarely hit the 1,024-token threshold.
- Long gaps between requests: If your user think-time exceeds 10 minutes, the cache evicts before the next request.
- Strict cost predictability requirements: The 50% discount applies probabilistically based on cache state. For guaranteed pricing, use explicit context management (e.g., RAG with a vector store) where you control exactly what context is sent.
- Privacy-sensitive prefixes: Cached prefixes are stored in OpenAI’s inference infrastructure. While scoped to your organization, the cache is a shared memory region. If your threat model prohibits any cross-request state, avoid sending sensitive data in the prefix.
Monitoring cache effectiveness
Track cached_tokens / prompt_tokens as a ratio over time. A healthy document Q&A workload should show 70–90% cached tokens after the first request. If the ratio stays below 20%, your prompt structure likely prevents prefix matching.
-- Example query if you log usage to a database
SELECT
date_trunc('hour', created_at) as hour,
model,
sum(prompt_tokens) as total_prompt_tokens,
sum(cached_tokens) as total_cached_tokens,
sum(cached_tokens)::float / nullif(sum(prompt_tokens), 0) as cache_hit_ratio
FROM usage_logs
WHERE model IN ('gpt-4o', 'gpt-4o-mini')
GROUP BY 1, 2
ORDER BY 1 DESC;Alert when the ratio drops unexpectedly — it often signals a prompt template change that broke the prefix alignment.
Summary
OpenAI’s automatic prompt caching is a transparent, server-side optimization that rewards prompt designs with long, stable prefixes. The mechanics are simple: 1,024+ identical leading tokens, 5–10 minute TTL, 50% price discount, no rate-limit consumption. The engineering leverage comes from structuring your prompts so the expensive context — system instructions, large documents, tool schemas — sits at the very beginning and stays identical across request bursts. Do that, and you get both lower latency and lower cost without changing your application logic.