How much latency does reranking add to RAG pipelines?

The reranking latency RAG systems incur is often the silent killer of otherwise snappy semantic search. Most teams optimize vector recall and generation speed, then watch p95 query time double when they bolt a cross-encoder onto the retrieval step. This analysis breaks down where that latency comes from, what it costs in practice, and how to keep it under control.

Where the milliseconds go

Reranking is not a single operation. It is a second scoring pass over candidates that the vector store already returned. Understanding the breakdown matters because optimization targets differ.

Baseline vector search

A typical ANN search over 1M–100M embeddings returns 20–100 neighbors in 5–30ms on a warmed index (FAISS, Pinecone, Weaviate). That cost is independent of reranking. If your pipeline already does hybrid search (BM25 + dense), add another 5–15ms for the lexical merge. None of this is avoidable once you have a corpus.

Cross-encoder inference

The dominant cost is the reranker model itself. Open-source cross-encoders (e.g., sentence-transformers/ms-marco-MiniLM-L-6-v2) score query–document pairs jointly. Unlike bi-encoders, there is no precomputed document embedding; you run the transformer forward pass for each pair at inference time.

On a single CPU core, a 22M-parameter MiniLM processes pairs with overhead that typically lands in the tens of milliseconds per 50-pair batch. Community-reported numbers range from 30–120ms for 50 docs on CPU to 10–40ms on a modest GPU (T4). Larger models (cross-encoder/ms-marco-MiniLM-L-12-v2 or BGE reranker base) can triple that due to deeper stacks and longer attention.

If you call a hosted rerank API, add network round-trip (1–10ms intra-region) plus provider queue time. Cold starts on serverless rerankers can add 200–500ms on the first call of a session. The model compute remains the bottleneck after warmup.

Orchestration tax

Naive implementations serialize steps:

docs = vector_store.search(query, k=50)  # 20ms
ranked = reranker.rank(query, docs)      # 80ms
context = ranked[:5]

Total: ~100ms. But the reranker cannot start until the search returns. That forced serialization is half the perceived latency problem. In a synchronous web handler, this blocks the event loop or thread.

Measuring reranking latency RAG pipelines

Instrument both stages separately. A minimal async wrapper exposes the split:

import time, asyncio

async def retrieve_and_rerank(query, vector_store, reranker, k=50, top_n=5):
    t0 = time.perf_counter()
    docs = await vector_store.asearch(query, k=k)
    t1 = time.perf_counter()
    ranked = await reranker.arank(query, docs)
    t2 = time.perf_counter()
    print(f"vector: {t1-t0:.1f}ms, rerank: {t2-t1:.1f}ms")
    return ranked[:top_n]

Run this against production-like candidate counts. The reranking latency RAG teams see in logs is usually 2–5x the vector search time for cross-encoders, but only if candidate count stays under 100. Push k=500 to improve recall, and cross-encoder cost scales linearly: 500 docs may add 300–800ms on CPU. That is where pipelines fall over.

Hosted vs self-hosted

Self-hosting gives you batching control and no per-call network tax, but you pay for GPU idle. Hosted APIs remove ops burden but introduce cold-start variance. If you use a gateway that forwards cache-control hints (e.g., n4n.ai), repeated queries can skip provider compute, yet the local scoring loop still runs if you self-host. The latency floor is the model, not the hop.

Tradeoffs: quality versus tail latency

Reranking exists because bi-encoder recall is imperfect. A cross-encoder recovers meaningful top-1 accuracy on MS MARCO-style queries. The question is whether that lift is worth the added p95.

When to skip reranking

• Sub-100ms p95 requirement and k<=20: a good dense + keyword hybrid often suffices.
• Static or low-variance corpora where you can pre-rank offline and store the top results.
• High-throughput bots where cost per query dominates; a bi-encoder with tuned threshold beats a cross-encoder on price.

When it pays off

• Legal, medical, or enterprise search where a wrong top result is expensive.
• Long-context generation where feeding 5 bad chunks wastes tokens and degrades answers.
• Multilingual queries where lexical match fails and dense retrieval recalls weakly.

Mitigation patterns that actually work

Two-stage candidate reduction

Use a cheap bi-encoder rerank (or second vector pass) to cut 200 candidates to 20, then run a heavy cross-encoder on those. This bounds cross-encoder cost.

coarse = await vector_store.asearch(query, k=200)      # 25ms
sifted = await light_reranker.rank(query, coarse, k=20) # 15ms on CPU
final = await heavy_reranker.rank(query, sifted, k=5)   # 20ms

Total reranking portion: ~35ms instead of 200ms+.

Batch and pad

Cross-encoders love batched tensors. If you serve multiple queries concurrently, batch across requests on the GPU. A T4 handles 64 query–doc pairs per batch with minimal per-query overhead. Pad sequences to the max length in the batch to avoid shape changes.

Cache reranker scores

Query strings repeat. Cache (query_hash, doc_id) scores in Redis with a TTL. For a support bot, 30% of queries are duplicates; this cuts reranking latency RAG traffic by a third without model changes.

cache_key = f"rerank:{hash(query)}:{doc.id}"
if (cached := redis.get(cache_key)):
    score = float(cached)
else:
    score = await reranker.score(query, doc)
    redis.setex(cache_key, 3600, score)

Overlap with generation

Start streaming the LLM answer using the top-1 vector result while the reranker scores the full set. If the reranker promotes a different doc, inject it into the context mid-stream or restart the completion. This hides reranking behind perceived responsiveness, at the cost of engineering complexity.

A decisive takeaway

Reranking adds 20–200ms to a RAG query in typical configurations (k=20–100, small cross-encoder). That is acceptable for chat-style assistants but toxic for autocomplete. Measure your own candidate counts and model; if p95 must stay under 100ms, use a lightweight reranker on ≤20 docs or drop reranking for a tuned hybrid retriever. The reranking latency RAG pipelines tolerate is a function of candidate volume, not the network. Engineer the candidate set first, then choose the reranker.