Cutting RAG costs with prompt caching and batching

Retrieval-augmented generation pipelines waste money by resending the same retrieved documents on every request. To reduce RAG costs prompt caching is the highest-leverage change you can make, followed closely by batching independent queries that share a context prefix. This guide gives you ordered steps to implement both against an OpenAI-compatible endpoint, with runnable code and a way to verify the savings.

Step 1: Separate stable context from volatile queries

A RAG prompt has two parts: the retrieved corpus slice (stable for a given document set) and the user question (volatile). Providers cache by prefix, so you must place the stable block first and explicitly mark it.

For Anthropic models, use cache_control on the final stable block. For OpenAI models, the prefix is cached automatically once it exceeds a minimum length, but you still benefit from keeping it identical across calls.

# Build messages with a cached system/context block
messages = [
    {
        "role": "system",
        "content": "You answer from the following docs only.\n" + retrieved_docs,
        "extra_body": {"cache_control": {"type": "ephemeral"}}  # Anthropic-specific
    },
    {
        "role": "user",
        "content": user_question
    }
]

If you call multiple users against the same retrieved_docs, the second call hits cache instead of re-billing the full context.

Step 2: Send cache-control hints through your client

Most Python clients default to the OpenAI shape. When you route through an OpenAI-compatible gateway that forwards provider hints, pass extra_body so the directive reaches the upstream model.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.openai.com/v1",  # swap for your gateway
    api_key="sk-..."
)

resp = client.chat.completions.create(
    model="claude-3-5-sonnet",  # or any supported model
    messages=messages,
    extra_body={"cache_control": {"type": "ephemeral"}}  # forwarded if provider supports
)

The key rule: never mutate the stable string (whitespace, ordering, or interpolation) between calls. Any change invalidates the prefix cache. Use a content hash of retrieved_docs as a key in your app to reuse the exact same string.

To reduce RAG costs prompt caching must be deterministic. Log the hash alongside the request ID.

Step 3: Batch queries that share the same document set

When you have N independent questions over the same retrieved slice, submit them as a batch instead of N serial calls. OpenAI’s batch API accepts a JSONL file of request objects; the cached prefix applies per line, but you amortize fixed overhead and often get lower per-token rates.

{"custom_id":"q1","method":"POST","url":"/v1/chat/completions","body":{"model":"claude-3-5-sonnet","messages":[{"role":"system","content":"DOCS: ...","extra_body":{"cache_control":{"type":"ephemeral"}}},{"role":"user","content":"What is the refund policy?"}]}}
{"custom_id":"q2","method":"POST","url":"/v1/chat/completions","body":{"model":"claude-3-5-sonnet","messages":[{"role":"system","content":"DOCS: ...","extra_body":{"cache_control":{"type":"ephemeral"}}},{"role":"user","content":"How do I cancel?"}]}}

Upload and run:

curl -s https://api.openai.com/v1/files \
  -H "Authorization: Bearer $KEY" \
  -F purpose=batch \
  -F file=@requests.jsonl

curl -s https://api.openai.com/v1/batches \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id":"file-abc","endpoint":"/v1/chat/completions","completion_window":"24h"}'

The batch completes asynchronously. Each line still benefits from the prefix cache if the system block is byte-identical.

Step 4: Measure cache hits and token reuse

Cost reduction is invisible unless you read the usage telemetry. OpenAI returns prompt_tokens_details.cached_tokens on supported models; Anthropic returns usage.cache_read_input_tokens. Parse and emit it.

usage = resp.usage
cached = getattr(usage.prompt_tokens_details, "cached_tokens", 0)
total_prompt = usage.prompt_tokens
print(f"cached={cached} total={total_prompt} save={cached/total_prompt:.1%}")

If cached_tokens is zero on the second call, your stable block changed. Diff the serialized system content between calls to find the mutation.

A gateway such as n4n.ai forwards your cache-control hints to the provider and automatically fails over when a provider is rate-limited, so you keep cache locality without writing custom retry loops. That matters because a fallback to a different model family silently drops cache hits if the prompt shape isn’t portable.

Step 5: Route and fall back without breaking cache

In production, providers degrade. If you switch models mid-traffic, the new model may not honor the same cache key. Design your routing to prefer the same model class for a given document hash.

# Pseudocode for sticky routing
route_map = {}  # doc_hash -> model
def get_model(doc_hash):
    if doc_hash in route_map:
        return route_map[doc_hash]
    model = "claude-3-5-sonnet"  # default
    route_map[doc_hash] = model
    return model

When you reduce RAG costs prompt caching must be paired with routing discipline. A smart gateway can honor client routing directives (x-n4n-model) and keep the same upstream for the cache lifetime.

Verify success

You have working cost reduction when all of the following hold:

1. Cache hit on repeat context – Second request with same retrieved_docs shows cached_tokens > 0.
2. Batch throughput – A 100-query batch returns completions with no per-call latency penalty and identical quality to serial calls.
3. Token accounting – prompt_tokens - cached_tokens equals roughly the size of the volatile user question, not the full doc set.
4. Fallback safety – Forced provider outage triggers failover but logs a cache miss warning rather than silent full billing.

Run a before/after test: send 50 identical-context queries serially without cache, record spend via usage.total_tokens * price. Then enable caching and batching, send the same 50. The delta is your real reduction.

Caveats that bite in production

• TTL limits: Ephemeral caches expire in 5–15 minutes depending on provider. If your RAG corpus is static for hours, re-mark the block each call; the cache refreshes cheaply.
• Minimum cache length: OpenAI requires ~1024 tokens before prefix caching kicks in. Don’t bother caching tiny contexts.
• JSON escaping: If you embed retrieved text in JSON, normalize newlines and unicode first. A \n vs \\n mismatch breaks the prefix.
• Batch completion window: 24h batches are cheap but not real-time. Use them for offline augmentation (summaries, eval sets), not live chat.

The second technique to reduce RAG costs prompt caching further is batching, but only when the stable prefix is truly shared. If every user gets a different top-k retrieval, caching the system prompt alone saves less—consider caching the embedding index description or global instructions instead.

Ship the five steps above, watch cached_tokens, and you’ll cut RAG spend without touching model quality.