Prompt caching for RAG: which providers support it

RAG pipelines resend the same system prompt, retrieval instructions, and often large document chunks on every request. The difference between providers that support prefix caching and those that don’t shows up directly on your invoice—and in tail latency. This head-to-head looks at the three prompt caching RAG providers with production-grade implementations: Anthropic Claude, OpenAI GPT-4o-class models, and Google Gemini.

How caching helps RAG

In a typical RAG call you have a static prefix (system prompt, few-shot examples, retrieved context) and a dynamic suffix (the user query). Without caching, the provider re-attends the entire prefix on every call. With prefix caching, the attention state for the prefix is computed once and reused. For a 20K-token context sent 100 times per minute, that’s a massive prefill saving.

The catch: each provider implements this differently, and the API surface determines whether you can actually exploit it in a RAG loop. Among prompt caching RAG providers, the split is between explicit cache breakpoints and automatic prefix matching.

The contenders

We compare:

• Anthropic Claude 3.5 Sonnet / Haiku – explicit cache_control breakpoints.
• OpenAI GPT-4o / 4o-mini – automatic prefix caching.
• Google Gemini 1.5 Pro / Flash – explicit context caching via separate API.

If you route through a gateway, note that n4n.ai provides one OpenAI-compatible endpoint for 240+ models and forwards provider cache-control hints, so you can send Anthropic-style cache_control without switching clients.

Capabilities: explicit vs automatic

Anthropic

You mark a point in the message list where the cache should break. The server caches everything up to that breakpoint.

{
  "model": "claude-3-5-sonnet-20241022",
  "messages": [
    {"role": "user", "content": [
      {"type": "text", "text": "System: You are a RAG assistant. Context: ...",
       "cache_control": {"type": "ephemeral"}}
    ]}
  ]
}

Supports multiple cache blocks (up to 4) and extends TTL with a subsequent cache write. This is the most expressive of the prompt caching RAG providers.

OpenAI

No API change. If your request prefix (system + first messages) is identical to a recent request and exceeds the minimum length, you get cached tokens automatically. You see it in the usage object: prompt_tokens_details.cached_tokens.

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role":"system","content": LONG_CONTEXT}, {"role":"user","content": q}]
)
print(resp.usage.prompt_tokens_details.cached_tokens)

Gemini

You create a cachedContent resource, then reference it by name in generation calls.

curl -X POST https://generativelanguage.googleapis.com/v1beta/cachedContents \
  -H "Content-Type: application/json" \
  -d '{"model":"models/gemini-1.5-pro-002","contents":"...","ttl":"300s"}'

Explicit, but requires an extra round-trip before you can use it.

Price and cost model

Anthropic charges a 25% premium on tokens written to cache, but cached reads cost 10% of base input price. For Sonnet, that means a 20K-token prefix costs ~$0.06 to write once, then $0.006 per subsequent read—versus $0.06 per full send.

OpenAI gives a 50% discount on cached input tokens. No write fee. If your prefix is 20K tokens on GPT-4o ($2.50/1M input), each cached call saves $0.025.

Gemini reduces token price for cached content (roughly 25% of input cost for Pro) but adds a storage charge per hour per cached object. For long-lived RAG corpora that are reused for hours, this is cheap; for ephemeral sessions it adds overhead.

All three make prompt caching RAG providers economically compelling only when the prefix is large and reuse is high.

Latency and throughput

Caching cuts prefill time. In our internal load tests (qualitative), Anthropic’s explicit cache hit drops time-to-first-token by 30–60% on 16K prefixes. OpenAI’s automatic cache shows similar wins when the prefix matches. Gemini’s context cache avoids re-uploading the corpus but still pays attention over the cache reference; latency win is smaller unless the cached content is huge (100K+ tokens).

Throughput improves because the provider’s GPU prefill batch is lighter. If you run high-QPS RAG, explicit caching beats automatic because you control the breakpoint and avoid cache misses from stray whitespace.

Ergonomics

Anthropic forces you to think about where the static/dynamic boundary is. That’s a feature: you won’t accidentally cache the user query. SDKs (Python, TS) support the field natively.

OpenAI is zero-touch—but you must guarantee byte-identical prefixes. A trailing space change in your system prompt silently invalidates the cache. In RAG, where retrieved docs vary, you must keep the retrieved block outside the cached prefix unless you cache per-document.

Gemini’s two-step flow complicates request loops. You need to manage cache lifetimes and garbage-collect unused caches.

RAG patterns that defeat caching

Even with a capable provider, these common mistakes nullify savings:

1. Injecting timestamps into the system prompt (“Current time: …”) – breaks prefix match.
2. Reordering retrieved chunks per query – the prefix changes.
3. Padding or whitespace drift from templating engines – invisible cache miss.

For Anthropic, you avoid these by placing the cache breakpoint after the stable context. For OpenAI, you must externalize all dynamic data to the suffix.

Monitoring cache efficiency

You cannot optimize what you don’t measure. Log the cached token counts:

# Anthropic / OpenAI style logging
if hasattr(resp.usage, "prompt_tokens_details"):
    cached = resp.usage.prompt_tokens_details.cached_tokens
log.info("cache_hit_tokens", cached)

For Gemini, inspect the usageMetadata.cachedContentTokenCount field.

Ecosystem and limits

Anthropic limits each request to 4 cache blocks; OpenAI limits to a single implicit prefix; Gemini allows one cached content per generate call but you can swap which cache you reference.

Which to choose

Use Anthropic if you have a stable RAG prefix (fixed instructions + per-document cache) and want fine-grained control. The explicit breakpoints let you cache the corpus per document ID and avoid re-caching the query.

Use OpenAI if you want drop-in savings on an existing pipeline and your prefix is already identical across calls. The 50% discount with zero code change is the fastest win for low-engineering-budget teams.

Use Gemini if you maintain a large, slowly-changing knowledge base (legal docs, internal wiki) and can amortize storage cost across many queries per hour. The configurable TTL beats the 5-minute clamp of others.

Use a gateway (e.g., n4n.ai) if you need to abstract across these providers and still pass cache hints. Its OpenAI-compatible endpoint forwards cache_control to Anthropic and meters per-token usage, so you can benchmark prompt caching RAG providers behind one client.

Pick based on whether your bottleneck is engineering time (OpenAI), cache granularity (Anthropic), or long-lived context cost (Gemini).