How reranking fixes bad retrieval in RAG

Reranking fixes RAG retrieval by adding a second, more precise scoring pass over the documents your vector search returns. Vector similarity is fast but imprecise; a cross-encoder reranker evaluates query-document pairs directly, catching relevance signals that embeddings miss. This guide walks through adding reranking to a production RAG pipeline, from model selection to evaluation.

Step 1: Understand why vector search alone fails

Dense embeddings compress documents into fixed-size vectors. That compression loses nuance — negation, specific entities, temporal constraints, and multi-hop reasoning all degrade. A query like “Apple’s revenue in Q3 2023 excluding services” retrieves documents about Apple, revenue, and Q3, but the embedding cannot reliably enforce the exclusion or the quarter constraint.

Typical failure modes:

• False positives: Documents share topical vocabulary but answer a different question
• Missing constraints: Temporal, geographic, or logical filters that embeddings don’t encode
• Chunking artifacts: Relevant content split across chunks, none of which scores highly alone

Reranking fixes RAG retrieval by scoring each (query, candidate) pair with a model that attends over both sequences jointly. Cross-encoders see the full interaction, not two independent embeddings.

Step 2: Choose a reranker model

Start with a general-purpose cross-encoder. The most common open models:

For production, consider:

• Latency budget: Reranking 50 candidates adds 50 × latency. Target <200ms total.
• Context length: Some rerankers truncate at 512 tokens; others support 8k+. Match your chunk size.
• Domain adaptation: Legal, medical, or code-heavy corpora benefit from fine-tuned rerankers (e.g., BAAI/bge-reranker-v2-gemma for legal).

Start with bge-reranker-v2-m3 — strong zero-shot performance, 8k context, multilingual. Swap later if latency or domain accuracy demands it.

Step 3: Implement the reranking pipeline

The pattern: retrieve k candidates with vector search, rerank top n (where n ≤ k), pass top m to the generator. Typical values: k=50, n=20, m=5-8.

# rag_pipeline.py
from sentence_transformers import CrossEncoder
from typing import List, Dict, Any
import numpy as np

class Reranker:
    def __init__(self, model_name: str = "BAAI/bge-reranker-v2-m3", device: str = "cuda"):
        self.model = CrossEncoder(model_name, device=device, max_length=8192)
    
    def rerank(self, query: str, candidates: List[Dict[str, Any]], top_n: int = 20) -> List[Dict[str, Any]]:
        """
        candidates: list of dicts with at least 'text' and 'metadata' keys
        returns: top_n candidates sorted by reranker score, with 'rerank_score' added
        """
        if not candidates:
            return []
        
        pairs = [(query, c["text"]) for c in candidates]
        scores = self.model.predict(pairs, batch_size=32, show_progress_bar=False)
        
        for candidate, score in zip(candidates, scores):
            candidate["rerank_score"] = float(score)
        
        reranked = sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)
        return reranked[:top_n]


class RAGPipeline:
    def __init__(
        self,
        vector_store,
        embedder,
        reranker: Reranker,
        generator,
        retrieve_k: int = 50,
        rerank_n: int = 20,
        final_m: int = 6,
    ):
        self.vector_store = vector_store
        self.embedder = embedder
        self.reranker = reranker
        self.generator = generator
        self.retrieve_k = retrieve_k
        self.rerank_n = rerank_n
        self.final_m = final_m
    
    def answer(self, query: str) -> Dict[str, Any]:
        # 1. Vector retrieval
        query_embedding = self.embedder.encode(query)
        candidates = self.vector_store.search(query_embedding, k=self.retrieve_k)
        
        # 2. Rerank
        reranked = self.reranker.rerank(query, candidates, top_n=self.rerank_n)
        
        # 3. Select final context
        context_docs = reranked[:self.final_m]
        context = "\n\n".join([f"[Doc {i+1}] {d['text']}" for i, d in enumerate(context_docs)])
        
        # 4. Generate
        answer = self.generator.generate(query, context)
        
        return {
            "answer": answer,
            "context": context_docs,
            "rerank_scores": [d["rerank_score"] for d in context_docs],
        }

Key implementation notes:

• Batch inference: Always batch predict() calls. Single-pair inference wastes GPU.
• Max length: Set max_length to match your chunk size. Truncation loses signal.
• Score calibration: Cross-encoder scores are uncalibrated logits. Don’t treat them as probabilities. Use only for ranking.
• Metadata preservation: Keep original vector scores and metadata alongside rerank scores for debugging.

Step 4: Tune top-k and thresholds

The three knobs — retrieve_k, rerank_n, final_m — trade latency for recall and precision.

# tuning.py
import time
from dataclasses import dataclass

@dataclass
class TuningResult:
    retrieve_k: int
    rerank_n: int
    final_m: int
    latency_ms: float
    recall_at_k: float
    ndcg_at_10: float

def tune_knobs(pipeline: RAGPipeline, eval_queries: List[Dict], ground_truth: Dict) -> List[TuningResult]:
    """
    eval_queries: [{"query": "...", "relevant_doc_ids": [...]}]
    ground_truth: {doc_id: doc_text}
    """
    configs = [
        (20, 10, 4),
        (50, 20, 6),
        (50, 30, 8),
        (100, 30, 10),
        (100, 50, 12),
    ]
    
    results = []
    for retrieve_k, rerank_n, final_m in configs:
        pipeline.retrieve_k = retrieve_k
        pipeline.rerank_n = rerank_n
        pipeline.final_m = final_m
        
        latencies = []
        recalls = []
        ndcgs = []
        
        for eq in eval_queries:
            start = time.perf_counter()
            result = pipeline.answer(eq["query"])
            latencies.append((time.perf_counter() - start) * 1000)
            
            retrieved_ids = [d["metadata"]["doc_id"] for d in result["context"]]
            relevant = set(eq["relevant_doc_ids"])
            
            # Recall@final_m
            recall = len(set(retrieved_ids) & relevant) / len(relevant) if relevant else 0
            recalls.append(recall)
            
            # NDCG@10 (simplified)
            dcg = sum(1 / np.log2(i + 2) for i, doc_id in enumerate(retrieved_ids[:10]) if doc_id in relevant)
            idcg = sum(1 / np.log2(i + 2) for i in range(min(len(relevant), 10)))
            ndcgs.append(dcg / idcg if idcg > 0 else 0)
        
        results.append(TuningResult(
            retrieve_k=retrieve_k,
            rerank_n=rerank_n,
            final_m=final_m,
            latency_ms=np.mean(latencies),
            recall_at_k=np.mean(recalls),
            ndcg_at_10=np.mean(ndcgs),
        ))
    
    return results

Run this on a representative query set (50-100 queries minimum). Pick the config where:

• Latency meets your SLA (e.g., p95 < 500ms)
• Recall@final_m > 0.85
• NDCG@10 plateaus — increasing rerank_n further yields diminishing returns

Typical sweet spot: retrieve_k=50, rerank_n=20, final_m=6 on a T4 or A10G.

Step 5: Evaluate with real queries

Offline metrics (recall, NDCG) correlate with quality but don’t guarantee good answers. Run an A/B evaluation with human or LLM judges.

# evaluation.py
import json
from concurrent.futures import ThreadPoolExecutor
from typing import Callable

def llm_judge(query: str, answer_a: str, answer_b: str, context: str) -> Dict:
    """
    Returns {"winner": "A" | "B" | "tie", "reasoning": "..."}
    """
    prompt = f"""Compare two answers to the same question. Judge which is more accurate, complete, and grounded in the context.

Question: {query}

Context:
{context}

Answer A:
{answer_a}

Answer B:
{answer_b}

Respond with JSON: {{"winner": "A" | "B" | "tie", "reasoning": "..."}}"""
    
    # Use your preferred LLM client
    response = generator.generate(prompt, "")  # adapt to your client
    return json.loads(response)

def ab_test(
    pipeline_with_rerank: RAGPipeline,
    pipeline_without_rerank: RAGPipeline,
    eval_queries: List[str],
    judge: Callable = llm_judge,
) -> Dict:
    wins = {"with_rerank": 0, "without_rerank": 0, "tie": 0}
    details = []
    
    def evaluate_one(query):
        # Get context from both for fair judging
        result_with = pipeline_with_rerank.answer(query)
        result_without = pipeline_without_rerank.answer(query)
        
        # Combine contexts for judge
        combined_context = "\n\n".join([
            f"[With Rerank] {d['text']}" for d in result_with["context"]
        ] + [
            f"[Without Rerank] {d['text']}" for d in result_without["context"]
        ])
        
        judgment = judge(query, result_with["answer"], result_without["answer"], combined_context)
        return judgment
    
    with ThreadPoolExecutor(max_workers=4) as executor:
        judgments = list(executor.map(evaluate_one, eval_queries))
    
    for j in judgments:
        wins[j["winner"].lower().replace(" ", "_")] += 1
        details.append(j)
    
    return {"wins": wins, "details": details}

Run on 30-50 real user queries (anonymized). Expect 15-30% win rate for reranking on complex queries; simple lookups often tie. If reranking loses, check:

• Reranker model mismatch (domain, language)
• Chunk size too large for reranker context window
• Generator ignoring context (prompt issue, not retrieval)

Step 6: Monitor and iterate in production

Log everything needed to debug retrieval failures:

# logging_middleware.py
import logging
import uuid
from contextvars import ContextVar
from dataclasses import dataclass, asdict
from typing import Optional

request_id_var: ContextVar[str] = ContextVar("request_id", default="")

@dataclass
class RetrievalLog:
    request_id: str
    query: str
    retrieve_k: int
    rerank_n: int
    final_m: int
    vector_scores: list
    rerank_scores: list
    selected_doc_ids: list
    latency_ms: float
    generator_model: str

def log_retrieval(log: RetrievalLog):
    logging.info("retrieval", extra=asdict(log))

# In your pipeline.answer():
request_id = request_id_var.get() or str(uuid.uuid4())
start = time.perf_counter()

# ... retrieval & reranking ...

latency = (time.perf_counter() - start) * 1000
log_retrieval(RetrievalLog(
    request_id=request_id,
    query=query,
    retrieve_k=self.retrieve_k,
    rerank_n=self.rerank_n,
    final_m=self.final_m,
    vector_scores=[c.get("vector_score", 0) for c in candidates[:self.rerank_n]],
    rerank_scores=[d["rerank_score"] for d in context_docs],
    selected_doc_ids=[d["metadata"]["doc_id"] for d in context_docs],
    latency_ms=latency,
    generator_model=self.generator.model_name,
))

Build a dashboard tracking:

• Rerank score distribution: Should separate relevant from irrelevant. If scores cluster tightly, the reranker isn’t discriminating.
• Vector vs. rerank agreement: High agreement means vector search already works; low agreement means reranking adds value (or the reranker is noisy).
• Latency percentiles: p50, p95, p99 for each stage.
• Fallback rate: If you implement cascade (rerank only when vector confidence low), track how often it triggers.

Common failure patterns and fixes

Step 7: Advanced — hybrid retrieval with reranking

Reranking fixes RAG retrieval most effectively when the candidate pool is diverse. Pure vector search returns near-duplicates. Hybrid retrieval (BM25 + vector) expands coverage before reranking.

# hybrid_retriever.py
from rank_bm25 import BM25Okapi
from typing import List, Dict
import numpy as np

class HybridRetriever:
    def __init__(self, vector_store, embedder, documents: List[Dict], bm25_weight: float = 0.3):
        self.vector_store = vector_store
        self.embedder = embedder
        self.documents = documents
        self.bm25_weight = bm25_weight
        
        # Build BM25 index
        tokenized = [doc["text"].split() for doc in documents]
        self.bm25 = BM25Okapi(tokenized)
        self.doc_id_to_idx = {doc["metadata"]["doc_id"]: i for i, doc in enumerate(documents)}
    
    def search(self, query: str, k: int = 50) -> List[Dict]:
        # Vector search
        query_emb = self.embedder.encode(query)
        vector_results = self.vector_store.search(query_emb, k=k * 2)  # oversample
        
        # BM25 scores for same candidates
        query_tokens = query.split()
        bm25_scores = self.bm25.get_scores(query_tokens)
        
        # Fuse scores
        fused = []
        for vr in vector_results:
            doc_id = vr["metadata"]["doc_id"]
            idx = self.doc_id_to_idx.get(doc_id)
            bm25_score = bm25_scores[idx] if idx is not None else 0
            
            # Normalize both to [0, 1] per query
            vector_score = vr.get("score", 0)
            fused_score = (1 - self.bm25_weight) * vector_score + self.bm25_weight * bm25_score
            
            fused.append({**vr, "fused_score": fused_score, "bm25_score": bm25_score})
        
        fused.sort(key=lambda x: x["fused_score"], reverse=True)
        return fused[:k]

Replace vector_store.search in RAGPipeline.answer() with hybrid_retriever.search. The reranker then sees a more diverse candidate set — keyword matches that vector search missed, and semantic matches that BM25 missed.

Verification checklist

Before declaring success, confirm:

• Latency: p95 end-to-end < your SLA (typically 500-800ms for RAG)
• Recall@final_m: > 0.85 on held-out eval set
• NDCG@10: Improves > 5% over vector-only baseline
• LLM judge win rate: > 55% on complex queries (multi-hop, constrained, negated)
• Score separation: Rerank scores for relevant vs. irrelevant docs show clear gap (visualize with histogram)
• No regression on simple queries: Win rate ≥ 50% on factoid lookups
• Monitoring dashboards live: Score distributions, latency, agreement metrics alerting on drift

What to try next

1. Fine-tune the reranker: Generate (query, positive, negative) triples from your logs. Train with sentence_transformers.CrossEncoder — 1-2k pairs often yields 3-5 NDCG points.
2. Cascade reranking: Only rerank when vector search confidence is low (max score < threshold). Saves 40-60% reranker calls.
3. Listwise reranking: Replace pointwise cross-encoder with listwise (e.g., RankGPT, RankZephyr) for better ordering at top-k.
4. Query rewriting: Expand ambiguous queries before retrieval — reduces burden on reranker.

Reranking is the highest-ROI addition to a RAG pipeline. The code above is production-ready — deploy it, measure it, iterate.