Measuring end-to-end RAG latency on a 1M document corpus

Measuring end-to-end RAG latency large corpus deployments is fundamentally different from profiling a toy demo with a few thousand embeddings. At 1M documents, the retrieval layer stops being a constant-time lookup and becomes a memory-bound, parallelism-sensitive subsystem where tail latency dominates user experience. If you ship without measuring the real pipeline on representative hardware, you will misallocate engineering effort.

Why a 1M document corpus breaks naive assumptions

When you scale from 10k to 1M vectors, the index crosses hardware thresholds that expose hidden costs. An HNSW graph with default M=16 stores roughly 1–2 bytes per vector per edge, plus the raw vectors. For 1M 1536-dim float32 embeddings, the vectors alone consume 6 GB; the graph adds several more GB of RAM. That forces either expensive high-memory instances or disk-backed ANN with seek penalties.

IVF indexes trade recall for memory but require careful nprobe tuning. A flat brute-force scan over 1M vectors takes tens of milliseconds per query on a single core, which violates interactive budgets. You cannot extrapolate latency from a 10k benchmark; the slope changes.

Sharding and replication

At 1M docs, many teams shard by tenant or hash. Cross-shard merge adds network round-trips. If your client serializes those calls, p95 blows up. Replication helps read throughput but doubles memory footprint. Decide based on read/write ratio: a mostly-read corpus can tolerate a single writer with read replicas.

Decomposing the pipeline stages

End-to-end RAG latency is the sum of independent contributions. Isolate them before optimizing:

1. Query embedding (model inference)
2. Vector search (ANN lookup)
3. Metadata filtering / re-ranking
4. Context assembly (chunk concatenation, token counting)
5. Generation (LLM completion)
6. Post-processing (citation extraction, streaming flush)

Wrap each in a timing context. A minimal Python decorator:

import time, functools

def timed(stage):
    def deco(fn):
        @functools.wraps(fn)
        async def wrapper(*a, **k):
            t0 = time.perf_counter()
            res = await fn(*a, **k)
            dt = time.perf_counter() - t0
            metrics.record(stage, dt)
            return res
        return wrapper
    return deco

Without per-stage instrumentation you will misattribute a generation slowdown to retrieval and waste weeks. When you set out to measure end-to-end RAG latency large corpus systems, start with this decomposition.

Instrumenting retrieval at scale

Generate a query set that mirrors production: sample from real user logs, not synthetic random vectors. Run at least 500 queries per configuration to get stable p95. Discard the first 50 runs to warm page caches.

Measure both hit rate and latency. A retrieval call that returns in 20ms but drops relevant docs forces the generator to hallucinate or abstain, which is a latency debt paid later in retries.

Example of a stratified benchmark loop:

import asyncio, random

async def bench_queries(queries, retriever, n=500):
    samples = random.sample(queries, n)
    latencies = []
    for q in samples:
        t0 = time.perf_counter()
        await retriever.search(q, top_k=10)
        latencies.append(time.perf_counter() - t0)
    latencies.sort()
    p95 = latencies[int(0.95*len(latencies))]
    return p95

On a 1M corpus, expect p95 vector search to range from 5ms (in-RAM HNSW) to 200ms (disk-backed IVF with filter). Those numbers are environment-specific; do not trust vendor defaults. Cache query embeddings separately—identical phrasings should skip the embed model entirely.

Generation latency and external dependencies

The LLM call is the most variable leg. A 70B model self-hosted on 2 GPUs behaves nothing like a managed API. External providers inject rate-limit and queue depth tail latency that dwarfs retrieval.

Using an inference gateway such as n4n.ai that provides automatic fallback when a provider is rate-limited or degraded lets you keep the generation leg from polluting your retrieval measurements. It forwards provider cache-control hints, so repeated system prompts don’t get re-billed or re-computed, stabilizing that stage’s timing.

Still, generation scales with output tokens and context size. Feeding 20 retrieved chunks instead of 5 multiplies prompt tokens by 4x and extends time-to-first-token. Measure both TTFT and total completion separately.

# approximate token count for context
def ctx_tokens(chunks, tokenizer):
    return sum(len(tokenizer.encode(c)) for c in chunks)

Streaming does not reduce total latency but improves perceived responsiveness; include it in user-facing specs.

The hidden cost of context assembly

Engineers often ignore the CPU work between retrieval and generation. Re-ranking with a cross-encoder over 50 candidates adds 30–100ms. Truncating to fit a 32k context window requires tokenization passes. If you serialize JSON metadata per chunk, parsing adds up.

At 1M scale, you may retrieve from sharded indexes; merging results across shards is network I/O. A misconfigured async client can serialize these calls. Profile with py-spy. I have seen assembly eat 40% of total latency in a pipeline that theoretically should be retrieval+generation only.

Parallelize independent steps with asyncio.gather:

async def assemble(query, retrievers):
    results = await asyncio.gather(*[r.search(query, top_k=20) for r in retrievers])
    merged = heapq.merge(*results, key=lambda x: x.score, reverse=True)
    return list(merged)[:10]

Benchmark methodology that holds up

Define a fixed hardware spec. Record:

• Index type and parameters
• Vector dimension and dtype
• Top-k and re-rank depth
• Generator model and max tokens
• Concurrency level (1, 8, 32)

Run each configuration three times; report median of p95 values. Never average p95 across runs; take the median of the p95 observations.

Include a cold run where the index is freshly loaded and an active run after 10 minutes of traffic. The gap reveals cache dependence. For end-to-end RAG latency large corpus measurement, separate single-stream interactive latency from batch throughput. They optimize differently: batch favors bigger fetches, interactive favors aggressive pruning.

Use statistical rigor: with 500 samples, bootstrapped confidence intervals on p95 are wide. Increase to 2000 queries if you need to detect a 10% regression.

Tradeoffs: recall versus latency

Increasing top-k from 5 to 20 improves answer correctness on complex queries but linearly increases context tokens. On a 1M corpus, a higher nprobe in IVF reduces false negatives but adds milliseconds per query.

A practical curve: at k=5, p95 retrieval 8ms, generation 600ms; at k=20, retrieval 14ms, generation 1100ms. The retrieval delta is minor; generation dominates. Therefore, optimize recall by better embedding or re-ranking, not by blindly fetching more.

If you need high recall, use a two-stage retrieve: cheap ANN for 50 candidates, fast re-rank to 5. This keeps generation context small while preserving accuracy. Quantizing vectors to int8 cuts memory and speeds distance calc with negligible recall loss at this scale.

Decisive takeaway

Measure every stage of your RAG pipeline in isolation before trusting any end-to-end number. On a 1M document corpus, retrieval is not the bottleneck unless you neglect indexing; generation and context assembly usually control the curve. Use stratified queries from production logs, report p95 under concurrency, and stabilize the LLM leg with fallback or caching. Treat end-to-end RAG latency large corpus targets as a systems problem, not a model problem, and you will ship a product that stays under 2 seconds when it matters.