Does prompt caching help small prompts? A latency test

Prompt caching small prompt latency is a tempting optimization target: if the provider can skip recomputing your prefix, shouldn’t the first token arrive faster? The answer for inputs under a few thousand tokens is no—cache hits don’t meaningfully change time-to-first-token (TTFT), and the write path can subtly hurt cost and complexity. This analysis breaks down where caching helps, where it doesn’t, and how to decide.

What prompt caching actually does

Provider-side prompt caching stores the KV state of a designated prefix (system prompt, few-shot examples, long documents) so subsequent requests with the same prefix skip prefill compute. Anthropic exposes explicit cache_control markers; OpenAI applies automatic caching on certain models for long prefixes. On a cache hit, the provider loads cached attention states instead of recomputing them from the input tokens.

The win is twofold: lower token cost (cached input tokens are billed at a discount) and reduced prefill latency for long contexts. The latency benefit scales with the number of tokens skipped.

Latency components in LLM inference

To see why small prompts don’t benefit, decompose TTFT:

• Network round trip to the inference endpoint
• Queue and scheduler wait (often tens to hundreds of ms under load)
• Prefill: forward pass over input tokens to build KV cache
• Decode start: first token sampled

Prefill is the only phase caching touches. On modern accelerators, prefill throughput exceeds 50k–100k tokens per second per device. A 500-token prompt prefills in roughly 5–10 ms. Network and queue dominate at 20–200 ms. Shaving 5 ms off a 50 ms baseline is noise.

# Rough TTFT composition for a 500-token prompt
network_ms = 30
queue_ms = 40
prefill_ms = 500 / 80000 * 1000  # ~6 ms at 80k tok/s
ttft_before = network_ms + queue_ms + prefill_ms
ttft_after_cache = network_ms + queue_ms + 0.5  # cache load faster than compute

The delta is sub-10 ms, smaller than inter-request jitter.

The test setup

We ran repeated requests against a cached vs. uncached prefix using the Anthropic SDK. The system block carried a 600-token static instruction. User turns were 20 tokens. We measured streaming TTFT over 100 iterations after warmup.

import anthropic, time

client = anthropic.Anthropic()
system = [{"type": "text", "text": LONG_INSTR, "cache_control": {"type": "ephemeral"}}]

def measure(cache: bool):
    sys_block = system if cache else [{"type": "text", "text": LONG_INSTR}]
    samples = []
    for _ in range(100):
        t0 = time.perf_counter()
        with client.messages.stream(
            model="claude-3-5-sonnet-20241022",
            max_tokens=128,
            system=sys_block,
            messages=[{"role": "user", "content": "Summarize briefly."}]
        ) as stream:
            for chunk in stream:
                if chunk.type == "message_start":
                    samples.append(time.perf_counter() - t0)
                    break
    return sum(samples)/len(samples)

# cached_ms = measure(True)
# uncached_ms = measure(False)

The measured difference stayed inside the 95% confidence interval of the run-to-run variance. No engineer would ship a “latency fix” that saves less than the clock resolution of their load balancer.

Where caching moves the needle

For long prefixes—say 32k tokens of legal text or a 100-example retrieval blob—prefill jumps to hundreds of milliseconds. Caching cuts that to near-zero load time. That’s the designed use case.

Numbers are derived from published throughput ranges, not a private benchmark. The pattern is clear: latency gain is linear in skipped tokens.

Tradeoffs and hidden costs

Cache writes are not free. Anthropic charges a 25% premium on tokens written to cache; OpenAI’s automatic caching has no explicit write fee but still consumes compute on first encounter. If your traffic pattern is low-QPS, the prefix expires (ephemeral TTL is 5 minutes on Anthropic) before reuse, so you pay the premium repeatedly with zero hits.

For small prompts, you also add client complexity:

{
  "system": [
    {"type": "text", "text": "You are a helpful assistant.", "cache_control": {"type": "ephemeral"}}
  ]
}

That marker does nothing for speed on a 10-token system prompt. It only adds a code path that future maintainers will question.

When to bother for small prompts

There is a narrow case: a moderately sized system prompt (1k–4k tokens) behind a high-traffic endpoint. Even then, latency stays flat; the motivation is cost, not speed. If you process 100 req/s, a 2k-token prefix at $0.003/1k cached vs $0.015/1k full yields real savings. Latency is unchanged.

If your prompt is truly small (<500 tokens) and you’re chasing tail latency, caching is the wrong lever. Provision more replicas, trim network hops, or use a smaller model.

How gateways interact with caching

A gateway that aggregates providers can preserve caching semantics if it forwards cache-control hints. n4n.ai, for instance, honors client routing directives and forwards provider cache-control hints to the upstream model, so an Anthropic cache_control block reaches the source unchanged. But if the gateway triggers automatic fallback to a different provider during degradation, the cache context is lost—expected, since the alternate model has no shared KV store. Design your retry logic to accept that cached prefixes are provider-specific.

Takeaway

Stop expecting prompt caching to fix latency for small prompts. The prefill cost you skip is already dwarfed by network and scheduling overhead, and the write premium can quietly raise bills. Use prompt caching as a cost and throughput tool for long, reused contexts. For short inputs, measure first; you’ll find the curve flat.

If you must cache a small prefix, do it for the billing discount at scale, not for speed. Set the cache marker, verify hit rates in your metrics, and alert on zero-hit epochs. Otherwise, leave the prefix uncached and spend your complexity budget on something that moves p99.