How caching cuts latency in production RAG pipelines

RAG caching latency reduction is the highest-leverage optimization you can apply to a production retrieval-augmented generation system after retrieval quality is acceptable. Most pipelines recompute embeddings and re-call the LLM for near-identical questions, paying full round-trip latency and token cost on every hit. This guide gives concrete steps to layer caches across the RAG stack and measure the win.

Step 1: Map the request path and isolate repeatable work

Trace one query through your pipeline and list every network call that produces the same bytes for similar inputs. In a typical RAG flow you have three candidates:

1. Embedding the retrieved chunks (often static unless documents change).
2. The vector search itself (same query vector → same top-k docs).
3. The final LLM completion (same system prompt + same context → same answer).

Cache only what is deterministic or near-deterministic. Do not cache raw user queries without normalization—whitespace, casing, and date stamps break naive keys.

Step 2: Cache embedding vectors for stable chunks

Document chunks do not change between requests. Compute a content hash and store the embedding in Redis. On cache miss, call the embedding model; on hit, skip the round trip entirely.

import hashlib
import redis
import numpy as np

r = redis.Redis(host="localhost", port=6379, db=0)

def get_embedding(text: str, embed_fn):
    key = "emb:" + hashlib.sha256(text.strip().lower().encode()).hexdigest()
    cached = r.get(key)
    if cached:
        return np.frombuffer(cached, dtype=np.float32)
    vec = embed_fn(text)
    r.set(key, vec.astype(np.float32).tobytes(), ex=86400)
    return vec

This cuts a 20–50 ms embedding call to sub-millisecond Redis lookup for any repeated chunk. In practice, a few thousand CMS articles generate the majority of retrieval hits.

Step 3: Build a semantic query cache for retrieval

Exact-match query caches miss on paraphrases. Store the query embedding in a small vector index and reuse the top-k document IDs when cosine similarity exceeds a threshold (0.98 works for factual questions).

from sklearn.metrics.pairwise import cosine_similarity

QUERY_CACHE = []  # list of (embedding, doc_ids)

def cached_retrieve(query_vec, retrieve_fn, threshold=0.98):
    for q_vec, doc_ids in QUERY_CACHE:
        sim = cosine_similarity([query_vec], [q_vec])[0][0]
        if sim >= threshold:
            return doc_ids  # cache hit
    doc_ids = retrieve_fn(query_vec)
    QUERY_CACHE.append((query_vec, doc_ids))
    return doc_ids

Keep the cache bounded (LRU by recency) to avoid memory blowup. This step is the core of RAG caching latency reduction because vector search latency often dominates p95.

Step 4: Use LLM-level cache-control hints

Modern providers support prompt caching via explicit cache markers. Anthropic’s cache_control ephemeral blocks let you pin the system prompt and retrieved context. If you front your models with an OpenAI-compatible endpoint such as n4n.ai, it forwards provider cache-control hints to the upstream and applies per-token metering, so the same request shape works across 240+ models without code changes.

import anthropic

client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=512,
    system=[
        {"type": "text", "text": "You answer from retrieved docs.",
         "cache_control": {"type": "ephemeral"}}
    ],
    messages=[{"role": "user", "content": context + "\n\n" + question}]
)

For an OpenAI-compatible gateway, send the same structure as JSON; the gateway passes the hint through. Set TTLs that match your document refresh cycle.

Step 5: Instrument and verify the latency drop

Add timestamps around each stage before and after caching. Log them as structured fields.

import time, logging

t0 = time.time()
vec = get_embedding(chunk, embed_fn)
t1 = time.time()
docs = cached_retrieve(vec, retrieve_fn)
t2 = time.time()
answer = generate(vec, docs)
t3 = time.time()
logging.info("emb_ms=%.1f ret_ms=%.1f gen_ms=%.1f", 
             (t1-t0)*1e3, (t2-t1)*1e3, (t3-t2)*1e3)

Verify success: run a representative query twice in succession. The second run should show emb_ms near zero and ret_ms near zero if both caches hit. Compare p95 over a 24-hour sample from your load test: a healthy RAG caching latency reduction shows retrieval stage dropping by 1–2 orders of magnitude and end-to-end p95 falling proportionally. No fabricated numbers—measure on your own traffic.

Step 6: Set invalidation and versioning

Caches without invalidation serve stale answers. Use these rules:

• Embedding cache: key includes a corpus version tag. Bump the tag on any document update.
• Query cache: TTL of 1 hour for volatile domains (news, stock), 24 hours for static docs.
• LLM cache_control: ephemeral TTLs (5–10 min) for context, longer for system prompt.

CORPUS_VERSION = "v12"
key = f"emb:{CORPUS_VERSION}:" + hashlib.sha256(...).hexdigest()

Route invalidation events from your CMS to a Redis DEL by prefix. This keeps RAG caching latency reduction safe without manual flushes.

Step 7: Load-test with realistic paraphrase rates

Synthetic exact-repeat load tests overstate cache hit rate. Generate paraphrases with a small model or use historical logs. Aim for 30–60% semantic cache hits; above that, you are likely overfitting to a narrow query set.

Run k6 or locust against the staged pipeline:

locust -f rag_load.py --headless -u 50 -r 5 -t 10m

Watch the logged stage latencies. If ret_ms stays flat under load, your semantic cache is sized correctly. If it climbs, shrink the similarity threshold or increase the LRU cap.

Step 8: Monitor cache ratio in production

Export cache hit/miss counters to Prometheus. Alert when embedding hit ratio drops below 70% after a deploy—usually a sign of a missing version bump.

from prometheus_client import Counter
EMB_HITS = Counter("emb_cache_hits", "embedding cache hits")
EMB_MISSES = Counter("emb_cache_misses", "embedding cache misses")

def get_embedding(text, embed_fn):
    key = ...
    if r.exists(key):
        EMB_HITS.inc()
        return ...
    EMB_MISSES.inc()
    ...

RAG caching latency reduction is not a one-time toggle. It is a layered system: embed once, retrieve once, and let the LLM provider cache the prompt. Each layer compounds. Ship the instrumentation first, then the caches, then the invalidation—never the other way around.