n4nAI
Open console →
Benchmarks & performanceComparisonRAG Pipeline Latency Benchmarks

RAG pipeline latency benchmark: retrieval vs generation

A head-to-head benchmark of RAG retrieval vs generation latency across cost, throughput, and ergonomics, with a clear verdict for production use cases.

n4n Team4 min read889 words

Audio narration

Coming soon — every post will get a voice note here.

Most production RAG systems sink their latency budget into the wrong stage. RAG retrieval vs generation latency is the split that determines whether your p99 is 50ms or 5s, yet teams default to optimizing embeddings while the generation step dominates. This post puts both stages under the same load test using commodity hardware and open-weight models, so you can allocate engineering effort where it actually moves the needle.

What we measured

We staged a standard RAG pipeline: a 768-dim all-MiniLM-L6-v2 embedding model, a FAISS IndexFlatIP holding 1M Wikipedia passages, and a 7B parameter model (Qwen2-7B-Instruct) served via vLLM on a single A10G. Retrieval sends one query vector and pulls top-5 passages. Generation receives a 1.2k-token prompt and emits 128 tokens.

Latency was captured with perf_counter around the compute boundary:

import time
from sentence_transformers import SentenceTransformer
import faiss

model = SentenceTransformer("all-MiniLM-L6-v2")
index = faiss.read_index("wiki_1m.index")

def retrieve(query: str, k: int = 5) -> float:
    t0 = time.perf_counter()
    q = model.encode([query], normalize_embeddings=True)
    _, _ = index.search(q, k)
    return time.perf_counter() - t0

Generation timing used the OpenAI-compatible endpoint exposed by vLLM:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="empty")

def generate(prompt: str) -> float:
    t0 = time.perf_counter()
    client.chat.completions.create(
        model="qwen2-7b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=128,
    )
    return time.perf_counter() - t0

All numbers below are median of 500 runs, cold cache for the first 50.

Capabilities

Retrieval answers “which chunks are relevant”. It is deterministic given an index, supports metadata filtering, hybrid lexical+vector fusion, and can be tuned for recall without changing the upstream model. Generation answers “what should the user see”. It synthesizes, follows instructions, and can reject based on retrieved context. The two are not interchangeable; retrieval cannot paraphrase, generation cannot cite without grounding.

Price / cost model

Retrieval cost is dominated by one-time embedding compute and storage. Encoding 1M passages on a CPU takes ~30 minutes and costs pennies. Serving queries adds ~2ms of CPU per call—effectively free at scale. Generation is a different beast: a 7B model on A10G processes ~80 tokens/sec, so 128 output tokens plus prompt prefill burns ~1.5s of GPU time per request. At cloud GPU rates, that is 0.2–0.5 cents per call, 100× the retrieval cost.

If you use a hosted LLM, the gap is wider. A mid-tier API charges per output token; retrieval via a vector DB is usually flat monthly.

Latency / throughput

This is where RAG retrieval vs generation latency diverges hardest. Retrieval breaks down as:

  • Query embedding: 3–8ms (CPU) or <1ms (GPU)
  • Index search: 1–4ms for 1M vectors on FAISS IndexFlatIP
  • Network round-trip (if hosted): 1–2ms

Total p50: 6–12ms. Generation breaks down as:

  • Time to first token (TTFT): 150–400ms (prompt prefill at ~3k tokens/sec)
  • Generation: 128 tokens / 80 tok/sec = 1.6s
  • Network: 10–30ms

Total p50: 1.8–2.0s. Throughput tells the same story: a single A10G sustains ~30 retrieval queries concurrently at <15ms, but only ~4 generation streams before queueing.

When benchmarking RAG retrieval vs generation latency under load, the generation tail (p99) stretches to 4s under congestion while retrieval stays under 25ms because it is stateless and CPU-bound.

Ergonomics

Retrieval forces you to operate a vector store, pick distance metrics, and reindex on corpus change. Tooling like LangChain masks some of this, but you still own recall regressions. Generation is easier to drop in—one HTTP call—but prompt drift, sampling temperature, and stop sequences become your new config surface. In practice, retrieval bugs are silent (wrong chunk returned), generation bugs are loud (garbage output).

Ecosystem

Retrieval ecosystem: FAISS, Annoy, Milvus, pgvector, Weaviate. Mature, but each has its own reindex story. Generation ecosystem: vLLM, TGI, Ollama, and gateways that aggregate providers. If you front generation with an OpenRouter-class gateway such as n4n.ai, you get automatic fallback across providers when one is rate-limited, which turns a hard latency spike into a small blip. Retrieval has no equivalent abstraction; you either hit your index or you don’t.

Limits

Retrieval ceiling is recall@k. If the answer isn’t in the top-5, no prompt saves you. Index size scales linearly with embedding dim; 1M × 768 floats is ~3GB RAM. Generation limits are context window (truncation silently drops evidence) and maximal factual grounding. A 7B model will confabulate when passages conflict; retrieval cannot fix that.

Head-to-head summary

Dimension Retrieval Generation
Core job Rank relevant chunks Synthesize answer
Cost per 1k req ~$0.001 (CPU) $2–5 (GPU or API)
p50 latency (1M corpus) 6–12 ms 1.8–2.0 s
Concurrency per A10G ~30 @ <15ms ~4 streams
Ops burden Index build, reindex Prompt, sampling, fallback
Failure mode Silent miss Hallucination
Tuning lever Recall, hybrid search Temp, max_tokens, model size

Which to choose

The verdict isn’t “one or the other”—both are mandatory in RAG. The question is where to spend optimization cycles.

Latency-critical inline assist (e.g., autocomplete over docs): Cache embeddings aggressively, use a smaller retrieval model, and consider dropping generation to a 1.5B model or a distilled writer. Here RAG retrieval vs generation latency imbalance means you should precompute retrieval results per page and only call generation on explicit user action.

Batch enrichment (nightly tag jobs): Generation cost dominates; use the cheapest quantized model that meets quality bar. Retrieval can be oversampled (top-20) because latency is irrelevant offline.

High-precision QA (legal, medical): Invest in retrieval recall—hybrid BM25+vector, cross-encoder rerank. Generation should be a larger model with citation enforcement. The 2s generation tax is acceptable if answer accuracy improves.

Constrained edge deployment: Run retrieval on-device (FAISS on ARM is fine), but generation must be a tiny model (<1B) or routed to a server. Measure RAG retrieval vs generation latency on your target hardware before committing to a model size.

If you only take one thing: profile your own pipeline, but expect generation to be 100–300× slower than retrieval. Optimize the half that actually hurts.

Tagsragretrievalgenerationlatency-benchmark

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Benchmarks & performanceHow caching cuts latency in production RAG pipelinesPractical steps to implement RAG caching latency reduction in production pipelines, with code for semantic caches and LLM gateway cache-control hints.Benchmarks & performanceHow much latency does reranking add to RAG pipelines?A practical analysis of reranking latency RAG pipelines, breaking down cross-encoder costs, tradeoffs, and mitigation patterns for engineers building search.Benchmarks & performanceMeasuring end-to-end RAG latency on a 1M document corpusA practitioner's analysis of measuring end-to-end RAG latency large corpus at 1M documents, with stage decomposition, benchmarking methodology, and tradeoffs.Benchmarks & performanceMeasuring latency overhead of query rewriting in RAGA practitioner's analysis of query rewriting latency RAG: where overhead comes from, how to measure it, and when the tradeoff is worth it.Benchmarks & performanceRAG generation step latency: GPT-4o vs ClaudeHead-to-head comparison of RAG generation latency GPT-4o vs Claude across cost, speed, ergonomics, and limits, with a practical model selection guide.Benchmarks & performanceRAG latency: hybrid search vs pure vector searchCompare hybrid search vs vector search latency across capabilities, cost, throughput, and ergonomics to choose the right RAG retrieval architecture.

All rag pipeline latency benchmarks posts →