Prompt caching explained: cutting token costs for agents

Prompt caching is a provider mechanism that stores the computed key-value states for a prefix of your prompt and reuses them across subsequent requests that share that exact prefix. Prompt caching token costs drop because you pay a reduced rate for those reused tokens instead of the full compute price. For agentic systems that repeatedly send large system prompts, tool definitions, and conversation history, this is the difference between a viable and a bankrupt architecture.

What prompt caching is (and isn’t)

Prompt caching operates at the transformer attention layer, not at the HTTP response layer. It caches the intermediate KV states for a contiguous prefix of tokens so the model does not recompute attention for that segment on the next request.

This is distinct from a semantic cache or a response cache. A response cache stores completed generations and serves them back for similar inputs; prompt caching stores internal state and still runs the decode step for new suffix tokens. If you are building an agent, you almost certainly want both, but they solve different problems.

How prompt caching works

Prefix matching is exact

Providers key the cache on the exact token sequence of the prefix. Any change in whitespace, a reordered tool definition, or a different system prompt breaks the match. The prefix must be identical byte-for-byte after tokenization.

{
  "model": "claude-3-5-sonnet",
  "system": [
    {
      "type": "text",
      "text": "You are a helpful agent with access to tools.",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [{"role": "user", "content": "Ping"}]
}

The cache_control marker tells the provider where the stable prefix ends. Everything before that marker is eligible for caching.

Block alignment and fragmentation

Most inference stacks only cache at fixed token boundaries (commonly 1,024 or 2,048 tokens). If your marked prefix ends mid-block, the provider may round up or ignore the tail. Keep stable blocks sized as multiples of the provider’s minimum cache block to avoid fragmentation waste.

TTL and invalidation

Cached prefixes expire after a provider-defined idle window. Typical TTLs range from five to sixty minutes; if no request hits the prefix within that window, the entry evicts. Subsequent requests pay full price to rebuild it.

Pricing model

You usually see two meter lines: cache_creation_tokens (written once) and cache_read_tokens (charged on hits). The read rate is a fraction of the base token price—often an order of magnitude lower. Prompt caching token costs therefore split into a small write amortization plus cheap reads.

Why agents feel this immediately

An agent loop sends the same scaffolding every turn:

• System instructions (often 500–5,000 tokens)
• Tool schemas (JSON or function defs, easily 2k–10k tokens)
• Retrieved context or policy documents
• Prior conversation transcript (grows each turn)

Without caching, every turn re-encodes the entire scaffold. With a 4k-token stable prefix and 50 agent steps, you process 200k scaffold tokens unnecessarily. Prompt caching token costs in that scenario collapse to one write plus 49 reads.

Concrete example: a 10-turn research agent

Assume a fixed system + tool block of 8,000 tokens. Each user turn adds 200 new tokens; the model responds with 300 tokens.

Without caching:

• Per turn input: 8,000 (scaffold) + growing history + 200.
• Total input tokens across 10 turns ≈ 80,000 scaffold tokens processed at full price, plus history.

With caching:

• Turn 1: 8,000 cache write + 200 = full price on 8,200.
• Turns 2–10: 8,000 cache read (at reduced rate) + delta + history (history not cached if changing). The scaffold portion is cheap.

# Pseudocode for a loop that keeps stable prefix first
system_block = {"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}
for step in range(10):
    resp = client.messages.create(
        model="claude-3-5-sonnet",
        system=[system_block],
        messages=history,
    )
    history.append({"role": "assistant", "content": resp.content})

The cache_control stays on the system block; the history array changes each turn but sits after the cached prefix, so it does not invalidate the cache.

Reading the meter

A correct cached session shows the write on turn one, reads thereafter:

{
  "usage": {
    "input_tokens": 8200,
    "cache_creation_tokens": 8000,
    "cache_read_tokens": 0,
    "output_tokens": 300
  }
}

Turn two should report cache_read_tokens: 8000 and cache_creation_tokens: 0. If not, your prefix drifted.

Common misconceptions

“The provider caches everything automatically”

Some providers auto-cache prefixes above a token threshold; others require explicit cache_control markers. Even with auto-caching, you must place stable content at the front. Putting volatile user text before your tool definitions destroys the match.

“Cache hits are free”

They are cheaper, not free. You still pay for the storage write once and a reduced read rate. If your prefix changes every request, you pay write cost repeatedly with zero read benefit.

“Cached state shares across models”

A cache entry is bound to the model checkpoint. Switching from claude-3-5-sonnet to claude-3-opus invalidates the prefix. The KV dimensions differ; there is no cross-model reuse.

“Prompt caching stores my data for later training”

Cache entries are ephemeral KV tensors in inference memory, not persisted to disk for training. They expire per TTL. If your compliance regime requires data deletion, cache TTL is short enough to be irrelevant.

“Longer prefix is always better”

A prefix only helps if it is actually reused. Padding the cache with content that varies per request (e.g., a timestamp) forces a miss. Trim the cached region to the truly invariant core.

Operational patterns that actually work

Put the immutable first

Order your request: system prompt, static tool definitions, then slowly changing retrieved docs, then fast-changing conversation. Tag the boundary after the last stable block.

Watch the usage meter

Parse usage.cache_read_tokens and usage.cache_creation_tokens from responses. If cache_read_tokens is zero after the first call, your prefix is not matching—inspect for hidden whitespace or ordering drift.

Route deliberately

If you front your models with a gateway, ensure it forwards cache-control hints and respects model pinning. n4n.ai honors client routing directives and forwards provider cache-control hints, so a cached prefix built on one provider stays cached when you pin that provider for the session. Losing the pin silently upgrades your prompt caching token costs back to full price.

Batch writes when possible

If you bootstrap multiple agents with the same scaffold, issue the first request to warm the cache, then fan out. The write cost is paid once; the fan-out reads are cheap.

When not to bother

For one-shot completions or batch jobs where each prompt is unique, caching yields nothing. The write overhead is pure loss. Reserve the pattern for repeated prefixes: agents, multi-turn chat, recurring eval harnesses, and templated RAG queries.

Debugging cache misses: a checklist

1. Confirm the cached block is the first token sequence in the request.
2. Diff the raw request bodies between turns; JSON key order and whitespace count.
3. Verify the cache_control marker is present and not overwritten by a middleware serializer.
4. Check TTL: a gap longer than the provider window between turns resets the entry.
5. Ensure the model string is identical across calls.

Bottom line

Prompt caching token costs are a structural lever for any system that issues repeated requests with shared context. Get the prefix stable, tag it explicitly, and read your usage meter. The savings scale with agent steps, not with model cleverness.