A reranker is a cross-encoder model that takes a query and a candidate document together as input and outputs a relevance score, enabling precise ordering of retrieval results. Unlike bi-encoders used for initial vector search, which embed queries and documents independently, rerankers perform full self-attention across both sequences simultaneously. This architectural difference lets them capture fine-grained semantic interactions that embedding similarity misses.
How rerankers work
The standard two-stage retrieval pipeline looks like this:
-
First-stage retrieval (bi-encoder): Embed the query and all corpus documents into the same vector space using a model like BGE, E5, or OpenAI’s text-embedding-3-large. Retrieve top-k candidates via approximate nearest neighbor search (HNSW, IVF, etc.). This stage is fast — milliseconds for millions of documents — but coarse.
-
Second-stage reranking (cross-encoder): Feed the query and each candidate document together into a cross-encoder (e.g., BGE-reranker-v2, Cohere Rerank, Jina Reranker). The model outputs a raw logit or probability per pair. Sort by score and take the top-n for your LLM context window.
# Pseudocode for a typical reranker call
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
query = "How do I configure connection pooling in PgBouncer?"
candidates = [
"PgBouncer config: pool_mode = transaction, max_client_conn = 100",
"PostgreSQL connection limits: max_connections = 200",
"Redis connection pooling with redis-py: ConnectionPool(max_connections=50)",
]
pairs = [(query, doc) for doc in candidates]
scores = reranker.predict(pairs) # Returns logits
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked:
print(f"{score:.3f} {doc[:60]}...")Output:
8.234 PgBouncer config: pool_mode = transaction, max_client_conn = 100
2.101 PostgreSQL connection limits: max_connections = 200
-1.456 Redis connection pooling with redis-py: ConnectionPool(max_connections=50)The cross-encoder attends to every query token against every document token. A bi-encoder compresses each into a single 1024-dim vector first — losing token-level alignment. That compression is why vector search retrieves “PostgreSQL connection limits” for a PgBouncer question: both share “connection” and “pool” tokens in embedding space. The reranker sees the full sequences and correctly downweights the irrelevant match.
Why rerankers matter for RAG
Retrieval quality bounds generation quality. If the top-5 chunks fed to your LLM contain two irrelevant documents, the model hallucinates or hedges. Rerankers directly improve precision@k — the metric that correlates with downstream answer accuracy.
Empirical pattern across benchmarks (BEIR, MTEB retrieval tasks): a strong bi-encoder + reranker consistently beats a larger bi-encoder alone. The compute tradeoff is favorable: encoding 100 candidates with a cross-encoder costs ~50-200ms on a modern GPU, while scaling the bi-encoder index to return 1000 candidates instead of 100 costs more latency and memory with diminishing returns.
Typical configuration:
- Bi-encoder retrieves top-50 to top-100
- Reranker reorders and selects top-5 to top-10
- LLM receives only the reranked top-n
This keeps context windows lean and token costs down. For a 128k context model, you could stuff 50 chunks — but you pay for every input token, and irrelevant context degrades instruction following.
Concrete example: hybrid search with reranking
Production RAG systems rarely rely on pure vector search. A common pattern combines sparse (BM25) and dense retrieval, then reranks the union.
# Hybrid retrieval + reranking pipeline
import bm25s
from sentence_transformers import SentenceTransformer, CrossEncoder
# 1. Sparse retrieval (BM25)
bm25 = bm25s.BM25(corpus=documents)
bm25_scores, bm25_idx = bm25.retrieve(query, k=50)
# 2. Dense retrieval (bi-encoder)
bi_encoder = SentenceTransformer("intfloat/e5-large-v2")
query_emb = bi_encoder.encode([query])
dense_scores, dense_idx = index.search(query_emb, k=50) # FAISS/HNSW
# 3. Merge candidates (reciprocal rank fusion or simple union)
candidate_idx = list(set(bm25_idx[0]) | set(dense_idx[0]))
candidates = [documents[i] for i in candidate_idx]
# 4. Rerank
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
pairs = [(query, doc) for doc in candidates]
scores = reranker.predict(pairs)
top_docs = [doc for _, doc in sorted(zip(scores, candidates), reverse=True)[:8]]
# 5. Build prompt
context = "\n\n---\n\n".join(top_docs)
prompt = f"Answer using only this context:\n{context}\n\nQuestion: {query}"Why this works: BM25 catches exact keyword matches (error codes, function names, version numbers). Dense retrieval catches semantic paraphrases. The reranker resolves conflicts — a document that scores high on both signals rises; a document that only matches keywords but is semantically off-topic drops.
Common misconceptions
“Rerankers are just slower bi-encoders”
False. They are fundamentally different architectures. A bi-encoder produces fixed embeddings independent of the query. A cross-encoder cannot precompute document representations — it must see the query. This means:
- No vector index for cross-encoders; you rerank a small candidate set
- Latency scales linearly with candidate count × sequence length
- Throughput is lower, but you only run it on 50-100 pairs per query
“You need a GPU for reranking”
Not strictly. Quantized ONNX models (int8, int4) run at acceptable latency on CPU for batch sizes of 1. BGE-reranker-v2-m3 quantized to int8 processes ~30 query-document pairs per second on a modern x86 CPU core. For higher throughput, batch inference on GPU or use a dedicated inference service. n4n.ai routes reranker calls to optimized inference endpoints alongside LLM traffic, but the principle holds: CPU inference is viable for lower-volume workloads.
“Rerankers replace the need for good chunking”
They don’t. A reranker cannot recover information that isn’t in the retrieved chunks. If your chunking strategy splits a logical unit (e.g., a function signature separated from its docstring), neither bi-encoder nor cross-encoder will reassemble it. Fix chunking first: use semantic chunking, preserve code structure, keep headers with their sections. Reranking amplifies good retrieval; it doesn’t substitute for it.
“Higher reranker score = ground truth relevance”
Rerankers output relative scores, not calibrated probabilities. A score of 8.2 vs 2.1 means “much more relevant,” not “82% chance of containing the answer.” Thresholding on absolute scores is brittle across domains. Instead, use rank position (top-3, top-5) or score gaps (drop-off after 3rd result) as signals. Some teams calibrate per-domain using a small labeled set, but don’t assume transferability.
“One reranker fits all domains”
General-purpose rerankers (BGE-reranker, Cohere Rerank, Jina Reranker) are strong out of the box. But domain-adapted rerankers — fine-tuned on legal, medical, or code retrieval tasks — can add 5-15% nDCG@10 over the base model. If you have labeled relevance data (even weak signals like click logs or user feedback), fine-tuning a cross-encoder is one of the highest-ROI investments in a RAG pipeline. The model is small (typically 300M-1B params), trains fast, and serves cheap.
When to skip the reranker
- Ultra-low latency requirements (<50ms p99 end-to-end): The extra 50-200ms may violate SLAs. Optimize bi-encoder retrieval instead (better index, hybrid search, query expansion).
- Tiny corpora (<10k documents): Exhaustive bi-encoder search with a strong model often suffices. The marginal gain from reranking diminishes when recall@100 is already near 1.0.
- Structured lookup: If queries map to exact IDs (error codes, SKUs, function names), deterministic lookup beats learned retrieval.
- Streaming/real-time UX: When you must show something in <100ms, return bi-encoder top-3 immediately, then async-rerank and swap in better results. This “speculative retrieval” pattern works well in chat interfaces.
Operational considerations
Batch inference: Always batch candidate pairs. Sending 50 separate API calls adds HTTP overhead. Send one request with 50 pairs.
Sequence length: Most rerankers max at 512 tokens (query + document). Truncate documents from the left (keep the end, which often contains conclusions/answers) or use sliding windows with max-pooling over chunk scores.
Caching: Reranker inputs are (query, doc) pairs. Cache scores for repeated queries. In multi-tenant systems, cache at the tenant level if queries overlap.
Monitoring: Track reranker score distributions per query type. A sudden shift toward low scores signals corpus drift or query distribution change. Log the top-1 score and the score gap between positions 1 and 3 — these correlate with answer confidence.
Summary
A reranker is a cross-encoder that scores query-document pairs with full self-attention, correcting the coarse approximations of vector search. It sits between retrieval and generation, takes 50-100 candidates, and returns the 5-10 most relevant. The latency cost is real but bounded; the precision gain is often the difference between a useful answer and a hallucination. Start with a strong off-the-shelf model (BGE-reranker-v2-m3, Cohere Rerank 3.5, Jina Reranker v2), integrate it into your hybrid retrieval pipeline, and measure nDCG@10 or answer quality before and after. If you have domain data, fine-tune. If you don’t, the general models are already strong enough to justify the extra hop.