Cached vs uncached tokens: a side-by-side latency benchmark

Prompt caching changes the economics of long system prompts, but the real win is often in the tail. In side-by-side runs of cached vs uncached tokens latency, the time-to-first-token gap widens as the shared prefix grows, turning a constant tax into a variable one. This article puts the two modes head to head across cost, speed, and developer friction so you can decide where caching earns its keep.

What we actually measured

We fixed an 8,192-token system prefix (API specification plus policy text) and sent 200 requests with a 256-token variable suffix. One run used a cache breakpoint at the prefix boundary; the other sent the full prompt untouched. Both went through an OpenAI-compatible client against a single provider region during a quiet window to limit variance.

from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

# Uncached: nothing special
uncached = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "system", "content": SYSTEM_PROMPT},
              {"role": "user", "content": suffix}]
)

# Cached: hint the provider to break cache at prefix end
cached = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "system", "content": SYSTEM_PROMPT},
              {"role": "user", "content": suffix}],
    extra_body={"cache_control": [{"type": "epoch", "value": 1}]}
)

The exact latency numbers depend on region, load, and model, but the shape is consistent: uncached requests pay prefill cost on every call; cached requests pay it once. We discarded the first request of each run to avoid cold-start allocation noise, then tracked created timestamps and the usage block.

Why prefix alignment matters

Caching only kicks in when the leading tokens match exactly. A stray newline, a rotated date string, or a reordered tool definition invalidates the cache. Treat the cached segment as immutable and generate it programmatically from a single source. If you cannot guarantee byte-for-byte stability, do not bother with cache hints.

Capabilities: what each mode buys you

Cached tokens let the provider reuse the computed attention state for the prefix. The model still processes the new suffix and generates output normally. You get identical completions (barring sampling temperature) with less compute on the provider side.

Uncached tokens force a full forward pass over the entire prompt. That is simpler to reason about and works everywhere, but it wastes FLOPs on text you already sent moments ago. There is no capability difference in output quality—only a compute path difference.

Price and cost model

Providers usually bill cached input tokens at a discount. OpenAI charges half price for cached input on supported models after a 5-minute TTL. Anthropic charges a 25% premium to write the cache, then 90% off on reads. Gemini applies similar read discounts. Uncached input tokens are billed at the standard rate.

For a workload with high prefix reuse—say a fixed legal boilerplate or a large retrieval corpus that changes slowly—the token discount alone can halve your bill. A gateway that provides per-token usage metering, such as n4n.ai, breaks out cache_read_input_tokens separately so you can audit the savings without instrumenting each provider SDK.

{
  "uncached_input_cost_per_1k": 0.005,
  "cached_input_cost_per_1k": 0.0025,
  "cache_write_surcharge_per_1k": 0.00125
}

The numbers above mirror typical OpenAI-style pricing tiers; always check the live provider sheet before forecasting.

Latency and throughput

The decoder speed is unchanged. What changes is time-to-first-token (TTFT). Prefill is the expensive phase: attention over the prefix scales roughly with the square of sequence length in naive implementations, though fused kernels mitigate this. Dropping an 8k-token prefix from the prefill path removes a large fixed cost per request.

In cached vs uncached tokens latency tests, the uncached path shows TTFT that climbs with prefix length. The cached path keeps TTFT near the suffix-only prefill time plus a cache lookup. Throughput on the provider side improves because fewer GPUs are tied up in redundant prefill, which can translate to fewer rate limits for you under shared quotas.

Ergonomics

Uncached is zero-config. You send the prompt; you get a response.

Cached requires you to mark cache boundaries and keep prefixes stable. In the OpenAI API you use extra_body with cache_control. Anthropic uses a cache_control block on a content item. If you route through a gateway that forwards provider cache-control hints—such as n4n.ai—the same client code works across 240+ models without rewriting for each backend.

// TypeScript, Anthropic-style cache control
const resp = await client.messages.create({
  model: "claude-3-5-sonnet",
  system: [{ type: "text", text: SYSTEM_PROMPT, cache_control: { type: "ephemeral" } }],
  messages: [{ role: "user", content: suffix }]
});

Misaligned prefixes fail silently: you pay full price and get no speedup. Log the cache_read_input_tokens field from the usage response to verify hits. In our harness we asserted that field was non-zero before counting a run as cached.

Ecosystem and limits

Most frontier APIs now support some form of prefix caching: OpenAI, Anthropic, Gemini, and open-weight servers like vLLM. Minimum cached prefix lengths are common (1,024 tokens for OpenAI and Anthropic). TTLs range from 5 minutes to an hour, after which the cache evicts and the next request pays the write cost again.

Uncached tokens have no such constraints. Any prompt works, any length, any order. That universality is why uncached remains the default in most quickstarts.

Head-to-head comparison

Observability: proving the cache works

You cannot manage what you do not measure. After each call, inspect the usage object:

usage = resp.usage
print(f"cached read: {usage.cache_read_input_tokens}, "
      f"cached write: {usage.cache_creation_input_tokens}, "
      f"uncached: {usage.prompt_tokens - (usage.cache_read_input_tokens or 0) - (usage.cache_creation_input_tokens or 0)}")

If cache_read_input_tokens is zero on repeated prefixes, your boundary is misplaced or the prefix drifted. Alert on read rate in production; a drop below 80% usually signals a deploy that mutated the system prompt.

Which to choose: verdict by use case

High-repeat RAG or fixed system prompts. Use caching. If your retrieval injects the same corpus chunk across many users, or your system prompt is a stable 4k-token spec, the latency and cost wins are free after the first write. Set the cache breakpoint at the end of the static section.

Interactive low-latency chat. Cache the system prompt and any long persona text. The user’s turns are short suffixes; shaving hundreds of milliseconds per turn matters when the user is watching a cursor blink. Keep the mutable conversation history outside the cached block.

One-off batch jobs with unique prefixes. Skip caching. If every prompt is distinct and never repeated within the TTL window, you only pay the write surcharge for no read benefit. The ergonomic overhead is not worth it.

Rapidly mutating context. If your prefix changes every request (e.g., per-user dynamic docs assembled at runtime), caching will miss. Factor out a stable core—maybe a fixed instruction header—and cache only that. The rest stays uncached.

Multi-provider routing. If you fan out across models behind a single endpoint, ensure your gateway forwards cache hints and meters them per token. Otherwise you lose the optimization the moment you switch backends, and you cannot tell from your logs which calls actually hit.

Cached vs uncached tokens latency is not a philosophical choice; it is a function of reuse distance. Measure your prefix stability first, then flip the flag.