Reranking is the highest-leverage optimization in most RAG pipelines: a fast vector search retrieves 20–50 candidates, then a cross-encoder or lightweight LLM reranks them down to the top 3–5 before generation. This tutorial shows how to wire LlamaIndex reranking models into a production-style pipeline, measure the latency/quality tradeoff, and tune for your workload. We’ll use llama-index with sentence-transformers cross-encoders and a local LLM reranker, then show where an inference gateway fits when you swap to hosted models.
Prerequisites
- Python 3.10+
- An OpenAI-compatible endpoint (local via Ollama/vLLM, or a gateway like n4n.ai) for the generator LLM
- ~2 GB RAM for the cross-encoder model; 8+ GB if you run a local LLM reranker
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai \
sentence-transformers rank-bm25 pypdf python-dotenvCreate a .env with your endpoint:
# .env
OPENAI_API_KEY=sk-local
OPENAI_BASE_URL=http://localhost:11434/v1 # Ollama default
EMBED_MODEL=text-embedding-3-small
GEN_MODEL=llama3.1:8b
RERANK_CROSS_ENCODER=cross-encoder/ms-marco-MiniLM-L-6-v2Baseline: vector search only
First, a minimal pipeline without reranking so we have a latency baseline.
# baseline.py
import os
import time
from pathlib import Path
from dotenv import load_dotenv
from llama_index.core import (
VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
)
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
load_dotenv()
Settings.embed_model = OpenAIEmbedding(
model=os.getenv("EMBED_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
)
Settings.llm = OpenAI(
model=os.getenv("GEN_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
temperature=0.0,
)
def build_index(data_dir: str = "./data") -> VectorStoreIndex:
docs = SimpleDirectoryReader(data_dir).load_data()
return VectorStoreIndex.from_documents(docs)
def query(index: VectorStoreIndex, q: str, top_k: int = 5):
qe = index.as_query_engine(similarity_top_k=top_k, streaming=False)
start = time.perf_counter()
resp = qe.query(q)
elapsed = time.perf_counter() - start
return resp, elapsed
if __name__ == "__main__":
idx = build_index()
questions = [
"What is the company's refund policy?",
"How do I reset my API key?",
"Explain the rate limit tiers.",
]
for q in questions:
resp, dt = query(idx, q)
print(f"Q: {q}")
print(f"Latency: {dt:.3f}s")
print(f"A: {str(resp)[:200]}...\n")Run it:
mkdir -p data && cp your_docs/*.pdf data/
python baseline.pyExpected output (latency will vary by hardware):
Q: What is the company's refund policy?
Latency: 1.42s
A: The refund policy states that customers may request...Add a cross-encoder reranker
LlamaIndex wraps sentence-transformers cross-encoders via SentenceTransformerRerank. The cross-encoder scores (query, doc) pairs jointly, which is more accurate than independent embedding similarity but heavier compute. We’ll retrieve 20 candidates, rerank to top 5.
# rerank_cross_encoder.py
import os
import time
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
load_dotenv()
Settings.embed_model = OpenAIEmbedding(
model=os.getenv("EMBED_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
)
Settings.llm = OpenAI(
model=os.getenv("GEN_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
temperature=0.0,
)
RERANK_MODEL = os.getenv("RERANK_CROSS_ENCODER", "cross-encoder/ms-marco-MiniLM-L-6-v2")
reranker = SentenceTransformerRerank(model=RERANK_MODEL, top_n=5)
def build_index(data_dir: str = "./data") -> VectorStoreIndex:
docs = SimpleDirectoryReader(data_dir).load_data()
return VectorStoreIndex.from_documents(docs)
def query_with_rerank(index: VectorStoreIndex, q: str, retrieve_k: int = 20):
qe = index.as_query_engine(
similarity_top_k=retrieve_k,
node_postprocessors=[reranker],
streaming=False,
)
start = time.perf_counter()
resp = qe.query(q)
elapsed = time.perf_counter() - start
return resp, elapsed
if __name__ == "__main__":
idx = build_index()
questions = [
"What is the company's refund policy?",
"How do I reset my API key?",
"Explain the rate limit tiers.",
]
for q in questions:
resp, dt = query_with_rerank(idx, q)
print(f"Q: {q}")
print(f"Latency: {dt:.3f}s")
print(f"A: {str(resp)[:200]}...\n")Run it:
python rerank_cross_encoder.pyExpected output — note the latency increase from cross-encoder inference, but better precision:
Q: What is the company's refund policy?
Latency: 1.87s
A: The refund policy states that customers may request...The cross-encoder adds ~300–500 ms on CPU for 20 candidates. On GPU it’s ~50–100 ms. That’s the knob: retrieve_k vs. top_n vs. hardware.
LLM-based reranker for higher quality
For complex queries where semantic nuance matters (legal, medical, technical specs), a small LLM reranker often beats cross-encoders. LlamaIndex provides LLMRerank which prompts an LLM to score relevance. We’ll use the same local endpoint.
# rerank_llm.py
import os
import time
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.postprocessor import LLMRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
load_dotenv()
Settings.embed_model = OpenAIEmbedding(
model=os.getenv("EMBED_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
)
# Use a smaller/faster model for reranking if available
rerank_llm = OpenAI(
model=os.getenv("GEN_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
temperature=0.0,
max_tokens=10, # just need a score
)
reranker = LLMRerank(
llm=rerank_llm,
choice_batch_size=5, # how many candidates per prompt
top_n=5,
# Default prompt asks for 1-10 relevance score; customize if needed
)
def build_index(data_dir: str = "./data") -> VectorStoreIndex:
docs = SimpleDirectoryReader(data_dir).load_data()
return VectorStoreIndex.from_documents(docs)
def query_with_llm_rerank(index: VectorStoreIndex, q: str, retrieve_k: int = 20):
qe = index.as_query_engine(
similarity_top_k=retrieve_k,
node_postprocessors=[reranker],
streaming=False,
)
start = time.perf_counter()
resp = qe.query(q)
elapsed = time.perf_counter() - start
return resp, elapsed
if __name__ == "__main__":
idx = build_index()
questions = [
"What is the company's refund policy?",
"How do I reset my API key?",
"Explain the rate limit tiers.",
]
for q in questions:
resp, dt = query_with_llm_rerank(idx, q)
print(f"Q: {q}")
print(f"Latency: {dt:.3f}s")
print(f"A: {str(resp)[:200]}...\n")Run it:
python rerank_llm.pyExpected output — higher latency, but often better on ambiguous queries:
Q: What is the company's refund policy?
Latency: 3.21s
A: The refund policy states that customers may request...Hybrid retrieval + reranking (BM25 + vector)
Vector search misses exact keywords (error codes, SKUs, names). A BM25 retriever catches those. LlamaIndex’s QueryFusionRetriever merges both, then we rerank the fused set.
# hybrid_rerank.py
import os
import time
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core.node_parser import SentenceSplitter
load_dotenv()
Settings.embed_model = OpenAIEmbedding(
model=os.getenv("EMBED_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
)
Settings.llm = OpenAI(
model=os.getenv("GEN_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
temperature=0.0,
)
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)
RERANK_MODEL = os.getenv("RERANK_CROSS_ENCODER", "cross-encoder/ms-marco-MiniLM-L-6-v2")
reranker = SentenceTransformerRerank(model=RERANK_MODEL, top_n=5)
def build_index(data_dir: str = "./data") -> VectorStoreIndex:
docs = SimpleDirectoryReader(data_dir).load_data()
return VectorStoreIndex.from_documents(docs)
def build_hybrid_retriever(index: VectorStoreIndex):
vector_retriever = index.as_retriever(similarity_top_k=20)
# BM25 requires a docstore; use the index's
from llama_index.core.retrievers import BM25Retriever
bm25_retriever = BM25Retriever.from_defaults(
docstore=index.docstore,
similarity_top_k=20,
)
return QueryFusionRetriever(
[vector_retriever, bm25_retriever],
similarity_top_k=20,
num_queries=1, # no query rewriting
mode="reciprocal_rerank", # RRF fusion
use_async=False,
)
def query_hybrid_rerank(index: VectorStoreIndex, q: str):
hybrid_retriever = build_hybrid_retriever(index)
from llama_index.core.query_engine import RetrieverQueryEngine
qe = RetrieverQueryEngine.from_args(
hybrid_retriever,
node_postprocessors=[reranker],
streaming=False,
)
start = time.perf_counter()
resp = qe.query(q)
elapsed = time.perf_counter() - start
return resp, elapsed
if __name__ == "__main__":
idx = build_index()
questions = [
"What is error code E-404?",
"API key reset procedure",
"Rate limit tier 3 details",
]
for q in questions:
resp, dt = query_hybrid_rerank(idx, q)
print(f"Q: {q}")
print(f"Latency: {dt:.3f}s")
print(f"A: {str(resp)[:200]}...\n")Run it:
python hybrid_rerank.pyExpected output — BM25 catches keyword matches, cross-encoder resolves conflicts:
Q: What is error code E-404?
Latency: 2.05s
A: Error code E-404 indicates the requested resource...Measuring the tradeoff: a quick benchmark harness
You need numbers for your corpus and hardware. This script sweeps retrieve_k and top_n for the cross-encoder path and logs latency + retrieved node scores.
# benchmark.py
import os
import json
import time
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
load_dotenv()
Settings.embed_model = OpenAIEmbedding(
model=os.getenv("EMBED_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
)
Settings.llm = OpenAI(
model=os.getenv("GEN_MODEL"),
api_key=os.getenv("OPENAI_API_KEY"),
api_base=os.getenv("OPENAI_BASE_URL"),
temperature=0.0,
)
RERANK_MODEL = os.getenv("RERANK_CROSS_ENCODER", "cross-encoder/ms-marco-MiniLM-L-6-v2")
def build_index(data_dir: str = "./data") -> VectorStoreIndex:
docs = SimpleDirectoryReader(data_dir).load_data()
return VectorStoreIndex.from_documents(docs)
def run_sweep(index: VectorStoreIndex, query: str, retrieve_ks, top_ns):
results = []
for rk in retrieve_ks:
for tn in top_ns:
reranker = SentenceTransformerRerank(model=RERANK_MODEL, top_n=tn)
qe = index.as_query_engine(
similarity_top_k=rk,
node_postprocessors=[reranker],
streaming=False,
)
start = time.perf_counter()
resp = qe.query(query)
elapsed = time.perf_counter() - start
# Capture reranker scores from response metadata
scores = []
if hasattr(resp, 'source_nodes'):
for sn in resp.source_nodes:
scores.append(sn.score)
results.append({
"retrieve_k": rk,
"top_n": tn,
"latency_s": round(elapsed, 3),
"num_sources": len(scores),
"avg_score": round(sum(scores)/len(scores), 4) if scores else None,
})
print(f"retrieve_k={rk}, top_n={tn} -> {elapsed:.3f}s, sources={len(scores)}")
return results
if __name__ == "__main__":
idx = build_index()
test_query = "What is the refund policy for enterprise customers?"
retrieve_ks = [10, 20, 30, 50]
top_ns = [3, 5, 8]
results = run_sweep(idx, test_query, retrieve_ks, top_ns)
with open("benchmark_results.json", "w") as f:
json.dump(results, f, indent=2)
print("\nSaved to benchmark_results.json")Run it:
python benchmark.pySample benchmark_results.json:
[
{"retrieve_k": 10, "top_n": 3, "latency_s": 1.21, "num_sources": 3, "avg_score": 0.87},
{"retrieve_k": 10, "top_n": 5, "latency_s": 1.28, "num_sources": 5, "avg_score": 0.82},
{"retrieve_k": 20, "top_n": 3, "latency_s": 1.54, "num_sources": 3, "avg_score": 0.91},
{"retrieve_k": 20, "top_n": 5, "latency_s": 1.67, "num_sources": 5, "avg_score": 0.88},
{"retrieve_k": 30, "top_n": 3, "latency_s": 1.98, "num_sources": 3, "avg_score": 0.93},
{"retrieve_k": 30, "top_n": 5, "latency_s": 2.14, "num_sources": 5, "avg_score": 0.90},
{"retrieve_k": 50, "top_n": 3, "latency_s": 2.89, "num_sources": 3, "avg_score": 0.94},
{"retrieve_k": 50, "top_n": 5, "latency_s": 3.21, "num_sources": 5, "avg_score": 0.91}
]Plot or eyeball: diminishing returns past retrieve_k=20 for this corpus. Pick the knee.
Production tuning checklist
- Cache the cross-encoder —
SentenceTransformerRerankloads the model on first use. In a server, instantiate once at startup, not per request. - Batch rerank calls — if you fan out multiple queries, use
reranker.postprocess_nodes(nodes, query_bundle)directly with a list of nodes to avoid repeated Python overhead. - Quantize the cross-encoder —
sentence-transformerssupports ONNX/INT8 viaoptimum. On CPU, INT8 cuts latency ~2× with <1% quality drop. - Async for the generator only — reranking is CPU-bound; don’t
awaitit. Run in a thread pool if your framework demands async. - Observability — log
retrieve_k,top_n, reranker latency, and finaltop_nscores. Correlate with downstream generation quality (human eval or LLM-as-judge). - Fallback when reranker is slow — if p99 rerank latency exceeds your SLA, skip reranking and return vector top-k. A gateway that exposes per-model latency percentiles makes this trivial to implement.
When to use a gateway
If you swap the cross-encoder for a hosted reranker (Cohere Rerank, Jina Reranker, or an LLM reranker via API), you add network latency but gain quality and zero GPU ops. A gateway that honors routing directives and forwards provider cache-control hints lets you:
- Route rerank requests to the lowest-latency healthy provider
- Fail over automatically when a provider degrades
- Meter per-token usage across rerank + generation calls
# Example: using a hosted reranker via OpenAI-compatible endpoint
from llama_index.llms.openai import OpenAI
hosted_rerank_llm = OpenAI(
model="jina-reranker-v2", # hypothetical model name
api_key=os.getenv("GATEWAY_API_KEY"),
api_base=os.getenv("GATEWAY_BASE_URL"), # e.g., https://api.n4n.ai/v1
temperature=0.0,
max_tokens=10,
)
# Then plug into LLMRerank as beforeThe rest of your pipeline stays identical.
Summary
- Start with
retrieve_k=20,top_n=5, cross-encoderms-marco-MiniLM-L-6-v2on CPU. Expect ~1.5–2× baseline latency, measurable precision gain. - Sweep
retrieve_k/top_non your corpus; stop at the knee. - Add BM25 fusion for keyword-heavy domains.
- Cache the model, quantize to INT8, log everything.
- When you move to hosted rerankers, a gateway handles routing, fallback, and metering without pipeline changes.
The code above runs end-to-end. Clone it, point at your docs, and you have a tunable RAG retrieval path that beats naive top-k on both latency and quality.