A reliable RAG pipeline latency benchmark has to dissect the pipeline instead of quoting a single end-to-end millisecond figure. Retrieval-augmented generation splits cleanly into embedding, vector search, and generative inference, and the model you pick for each stage drives cost and latency in different ways. Treating “the LLM” as the only variable hides the fact that embedding model size and context padding often matter as much as decoder throughput.
Why latency in RAG is not a single number
A RAG call looks simple from outside: send a question, get an answer. Internally it runs three network-bound and compute-bound steps in sequence. First, you embed the query. Second, you search a vector index. Third, you assemble a prompt with retrieved chunks and run completion. Each step carries its own tail risk.
The generation step usually dominates wall-clock time for interactive apps, but only if retrieval stays under a few tens of milliseconds. If your embedding model is a 1B-parameter transformer running on a shared CPU instance, the first step can rival generation. Network round trips between your service, the embedding endpoint, and the vector store add fixed overhead that no model swap removes.
Measure each stage separately. A minimal Python harness:
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.example.com/v1")
def stage_latency(query, embed_model, gen_model, vector_db):
t0 = time.perf_counter()
emb = client.embeddings.create(input=query, model=embed_model).data[0].embedding
emb_ms = (time.perf_counter() - t0) * 1000
t1 = time.perf_counter()
docs = vector_db.search(emb, top_k=5)
ret_ms = (time.perf_counter() - t1) * 1000
t2 = time.perf_counter()
client.chat.completions.create(
model=gen_model,
messages=[{"role": "user", "content": f"{docs}\n\n{query}"}]
)
gen_ms = (time.perf_counter() - t2) * 1000
return emb_ms, ret_ms, gen_msRun that loop 200 times and you get distributions, not anecdotes. A RAG pipeline latency benchmark that reports only a mean is misleading by design.
Model classes and their latency profiles
Embedding models
Embedding models cluster into two useful buckets: sub-100M parameter sentence transformers and 300M–1B general-purpose embedding models. The smaller ones finish a single query in low double-digit milliseconds on GPU; the larger ones add 20–50ms for marginally better recall. For most RAG pipelines, the small bucket is the right default because embedding runs on the critical path before retrieval can start.
Batching changes the equation for offline indexing, not for live queries. A single user question cannot be batched with others without adding queue latency, so optimize the unbatched path.
Generation models
Generation is where model class explodes latency variance. An 8B-parameter model on a single modern GPU serves time-to-first-token (TTFT) in the low hundreds of milliseconds and sustains tens of tokens per second. A 70B-class model on equivalent hardware pushes TTFT past half a second and halves token rate. Frontier mixtures add scheduling and routing overhead on shared endpoints.
Switching models behind an OpenAI-compatible interface is trivial:
# small, fast RAG answerer
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages
)
# larger, slower, more nuanced
resp = client.chat.completions.create(
model="gpt-4o",
messages=messages
)The RAG pipeline latency benchmark should record both p50 and p95 for each model. Tail latency, not averages, breaks user trust.
Reranking: the optional tax
Many production RAG systems add a cross-encoder reranker after vector search. This is a second model call, usually a few hundred milliseconds for a 100M–400M parameter model. It improves precision but extends the critical path before generation starts.
If you benchmark with reranking disabled, you will ship a number that lies about production. Include it as a toggled stage:
if use_rerank:
t_re = time.perf_counter()
docs = reranker.rank(query, docs, top_k=3)
rerank_ms = (time.perf_counter() - t_re) * 1000A reranker that costs 300ms p95 may be worth it if it cuts generation tokens by removing irrelevant chunks. Measure the net effect, not the stage in isolation.
Benchmark methodology without fooling yourself
Cold starts distort everything. Warm your endpoint with 20 calls before timing. Use concurrent clients to simulate real traffic; a single-threaded loop hides queueing delay at the provider.
A routing config keeps experiments repeatable:
{
"embed_model": "text-embedding-3-small",
"gen_model": "llama-3-8b-instruct",
"retrieval": {"top_k": 5, "index": "prod-docs"},
"cache_control": {"reuse_kv": true}
}Track these metrics per stage:
- Embed p50 / p95 (ms)
- Retrieve p50 / p95 (ms)
- Rerank p50 / p95 (ms, if used)
- Generate TTFT p50 / p95 (ms)
- Generate tokens/s (observed)
- Total end-to-end p95 (ms)
Do not mix batch embedding latency with single-query latency. Batching embeds is cheaper per item but irrelevant to synchronous user queries.
The hidden cost of context padding
RAG injects retrieved text into the prompt. A chunk-heavy prompt of 4K tokens versus a 1K token prompt changes decoder math quadratically in some attention implementations. If your retrieval returns five 500-token passages, you force the model to attend over 2.5K extra tokens before it emits the first byte.
This is where provider cache-control hints matter. Mark static retrieved corpus prefixes as cacheable and the gateway can reuse KV states across similar queries. An inference gateway like n4n.ai that honors client routing directives and forwards provider cache-control hints lets you keep the same benchmark harness while trimming redundant compute. That directly lowers TTFT in repeated or overlapping RAG queries.
Test context scaling explicitly: fix the model, vary retrieved tokens from 500 to 4000, and plot TTFT. The slope tells you whether your deployment penalizes long context harder than the model card admits.
Gateway effects on measured latency
Provider degradation is real. A regional outage or rate limit turns your p95 into a timeout. A RAG pipeline latency benchmark run on a single provider on a good day will not match production. Gateways with automatic fallback to a secondary provider when the primary is rate-limited or degraded absorb those spikes.
The tradeoff: fallback may route to a slower model class, so your benchmark must record which model actually served the request. Log the resolved model in each response:
resp = client.chat.completions.create(model="auto", messages=messages)
print(resp.model) # actual model usedIf you do not capture that, you are benchmarking a phantom. Per-token metering also helps: if fallback silently shifts you to a pricier model, your cost-per-query creeps even when latency looks stable.
Tradeoffs: when to use big models
Large models answer ambiguous queries with less retrieval precision. If your vector store is weak, a 70B model might compensate. But for a mature RAG system with clean chunks and good reranking, an 8–14B model delivers most of the quality at a third of the latency.
Accuracy testing should run offline. Latency testing runs online. Do not conflate them. A RAG pipeline latency benchmark that only uses synthetic questions understates production variance because real users ask follow-ups that break cache reuse.
Decisive takeaway
Start every RAG build with a staged latency test, not a vendor speed claim. Default to a small embedding model and an 8–14B generation model; measure p95 end-to-end with realistic context sizes and reranking enabled if you ship it. Promote to larger models only after offline eval proves the smaller one misses answers you care about.
Use a gateway that exposes resolved model and supports cache hints so your benchmark reflects production failover. Capture these in your harness:
- Per-stage p50 and p95, not averages
- Resolved model name on every call
- Context length as a variable, not a constant
- Reranker toggled on and off
A RAG pipeline latency benchmark is only useful when it isolates stages, captures tails, and survives provider hiccups—anything less is a demo.