Retrieval quality makes or breaks a RAG system, and top-k is the single most impactful knob you can turn. This llamaindex top-k retrieval tuning tutorial walks through a repeatable process: measure a baseline, adjust k for your corpus, add reranking to shrink the effective candidate set, layer in hybrid retrieval, and automate evaluation so you stop guessing. The steps below assume you’re using LlamaIndex 0.10+ with a vector index backed by any supported embedding model.
Step 1: Understand what top-k actually controls
Top-k determines how many nodes the retriever returns to the query engine before synthesis. A larger k increases recall but feeds more noise to the LLM, raising latency and token cost. A smaller k speeds things up but risks dropping the needle in the haystack. The optimal value depends on corpus size, chunking strategy, embedding quality, and whether you rerank.
Start by inspecting your current retriever configuration:
from llama_index.core import VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever
index = VectorStoreIndex.from_vector_store(vector_store)
retriever = index.as_retriever(similarity_top_k=10) # default in many examples
print(f"Current top-k: {retriever.similarity_top_k}")If you’re using a custom query engine, the retriever may be buried inside. Check query_engine.retriever or query_engine._retriever.
Step 2: Set a baseline with the default retriever
Before tuning, capture baseline metrics on a representative query set. You need at least 20–50 questions that reflect real user intent, with ground-truth relevant document IDs or passages.
from llama_index.core.evaluation import RetrieverEvaluator
from llama_index.core.schema import QueryBundle
# Load your evaluation queries (list of dicts: {"query": str, "expected_ids": List[str]})
eval_queries = load_eval_set("eval_queries.jsonl")
retriever = index.as_retriever(similarity_top_k=10)
evaluator = RetrieverEvaluator.from_metric_names(
["hit_rate", "mrr"], retriever=retriever
)
results = await evaluator.aevaluate_dataset(eval_queries)
print(f"Baseline hit_rate@10: {results.metric_vals_dict['hit_rate']:.3f}")
print(f"Baseline MRR@10: {results.metric_vals_dict['mrr']:.3f}")Verify success: You have hit_rate and MRR numbers for k=10. Save these — they’re your comparison anchor.
Step 3: Tune top-k for your corpus size
Run a quick sweep over k values. Corpus size is the primary driver: small corpora (<10k chunks) often peak at k=5–10; large corpora (100k+) may need k=20–50 before reranking.
import asyncio
from llama_index.core.evaluation import RetrieverEvaluator
async def sweep_top_k(queries, k_values=[5, 10, 20, 30, 50]):
results = {}
for k in k_values:
retriever = index.as_retriever(similarity_top_k=k)
evaluator = RetrieverEvaluator.from_metric_names(
["hit_rate", "mrr"], retriever=retriever
)
res = await evaluator.aevaluate_dataset(queries)
results[k] = {
"hit_rate": res.metric_vals_dict["hit_rate"],
"mrr": res.metric_vals_dict["mrr"],
}
print(f"k={k}: hit_rate={results[k]['hit_rate']:.3f}, mrr={results[k]['mrr']:.3f}")
return results
k_results = await sweep_top_k(eval_queries)Plot hit_rate vs. k. Look for the elbow — diminishing returns after a certain point. That’s your candidate k before reranking.
Verify success: You’ve identified a k where hit_rate plateaus. Note it as k_prerank.
Step 4: Add reranking to shrink effective k
Reranking lets you retrieve broadly (high recall) then precision-filter with a cross-encoder. This is the highest-leverage improvement for most teams. LlamaIndex supports Cohere, Jina, BGE, and local sentence-transformer rerankers.
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.query_engine import RetrieverQueryEngine
# Retrieve 50, rerank to top 5
reranker = SentenceTransformerRerank(
model="cross-encoder/ms-marco-MiniLM-L-6-v2",
top_n=5,
)
retriever = index.as_retriever(similarity_top_k=50)
query_engine = RetrieverQueryEngine.from_args(
retriever=retriever,
node_postprocessors=[reranker],
)
# Evaluate the *reranked* output
from llama_index.core.evaluation import RetrieverEvaluator
evaluator = RetrieverEvaluator.from_metric_names(
["hit_rate", "mrr"], retriever=retriever
)
# Note: RetrieverEvaluator evaluates the retriever only.
# For end-to-end reranked evaluation, use a custom loop:
async def evaluate_with_rerank(queries, retriever, reranker, top_n):
hits = 0
rr_sum = 0.0
for q in queries:
nodes = await retriever.aretrieve(q.query)
reranked = reranker.postprocess_nodes(nodes, QueryBundle(q.query))
retrieved_ids = [n.node_id for n in reranked[:top_n]]
if any(eid in retrieved_ids for eid in q.expected_ids):
hits += 1
# MRR: rank of first relevant
for rank, nid in enumerate(retrieved_ids, 1):
if nid in q.expected_ids:
rr_sum += 1.0 / rank
break
n = len(queries)
return {"hit_rate": hits / n, "mrr": rr_sum / n}
reranked_metrics = await evaluate_with_rerank(eval_queries, retriever, reranker, 5)
print(f"Reranked hit_rate@5: {reranked_metrics['hit_rate']:.3f}")
print(f"Reranked MRR@5: {reranked_metrics['mrr']:.3f}")Verify success: Reranked hit_rate@5 exceeds or matches baseline hit_rate@10. If not, try a stronger reranker (e.g., bge-reranker-large) or increase similarity_top_k to 100.
Step 5: Layer in hybrid retrieval (vector + keyword)
Pure vector search misses exact-match terms (IDs, error codes, proper nouns). Hybrid retrieval combines dense embeddings with BM25 or SPLADE. LlamaIndex’s QueryFusionRetriever handles this.
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
from llama_index.core import Settings
# Build BM25 index over the same nodes
from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader("./data").load_data()
nodes = Settings.node_parser.get_nodes_from_documents(documents)
bm25_retriever = BM25Retriever.from_defaults(
nodes=nodes,
similarity_top_k=20,
)
vector_retriever = index.as_retriever(similarity_top_k=20)
fusion_retriever = QueryFusionRetriever(
retrievers=[vector_retriever, bm25_retriever],
similarity_top_k=20,
num_queries=1, # set >1 for query rewriting
mode="reciprocal_rerank", # or "relative_score"
use_async=True,
)
# Fuse, then rerank
query_engine = RetrieverQueryEngine.from_args(
retriever=fusion_retriever,
node_postprocessors=[reranker],
)Evaluate the same way as Step 4. Hybrid typically adds 3–8 points of hit_rate on keyword-heavy queries with minimal latency cost.
Verify success: Hybrid + rerank beats vector-only + rerank on your eval set, especially for queries containing entity names, codes, or acronyms.
Step 6: Evaluate with a retrieval metric that matches your downstream task
Hit_rate and MRR are retrieval proxies. The real test is answer quality. Use FaithfulnessEvaluator and RelevancyEvaluator on the full query engine output, or build a task-specific judge.
from llama_index.core.evaluation import (
FaithfulnessEvaluator,
RelevancyEvaluator,
CorrectnessEvaluator,
)
faithfulness = FaithfulnessEvaluator(llm=Settings.llm)
relevancy = RelevancyEvaluator(llm=Settings.llm)
async def evaluate_answers(queries, query_engine):
faith_scores = []
rel_scores = []
for q in queries:
response = await query_engine.aquery(q.query)
faith = await faithfulness.aevaluate_response(response=response)
rel = await relevancy.aevaluate_response(query=q.query, response=response)
faith_scores.append(faith.score)
rel_scores.append(rel.score)
return {
"faithfulness": sum(faith_scores) / len(faith_scores),
"relevancy": sum(rel_scores) / len(rel_scores),
}
answer_metrics = await evaluate_answers(eval_queries, query_engine)
print(f"Faithfulness: {answer_metrics['faithfulness']:.2f}")
print(f"Relevancy: {answer_metrics['relevancy']:.2f}")If faithfulness drops when you increase k, the LLM is getting distracted by irrelevant context. That’s your signal to tighten reranking or reduce top_n.
Verify success: Faithfulness ≥ 4.0/5.0 and relevancy ≥ 4.0/5.0 on your eval set. If not, iterate on reranker top_n or hybrid weights.
Step 7: Automate tuning with a parameter sweep
Don’t hand-tune once. Wrap the pipeline in a config-driven sweep so you can re-run when embeddings change, corpus grows, or you swap rerankers.
import itertools
from dataclasses import dataclass
from typing import List
@dataclass
class RetrievalConfig:
vector_top_k: int
bm25_top_k: int
fusion_mode: str
rerank_top_n: int
reranker_model: str
def build_pipeline(cfg: RetrievalConfig):
vector_ret = index.as_retriever(similarity_top_k=cfg.vector_top_k)
bm25_ret = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=cfg.bm25_top_k)
fusion = QueryFusionRetriever(
retrievers=[vector_ret, bm25_ret],
similarity_top_k=max(cfg.vector_top_k, cfg.bm25_top_k),
mode=cfg.fusion_mode,
use_async=True,
)
reranker = SentenceTransformerRerank(
model=cfg.reranker_model,
top_n=cfg.rerank_top_n,
)
return RetrieverQueryEngine.from_args(
retriever=fusion,
node_postprocessors=[reranker],
)
async def run_sweep(configs: List[RetrievalConfig], queries):
best = None
best_score = -1
for cfg in configs:
engine = build_pipeline(cfg)
metrics = await evaluate_answers(queries, engine)
composite = (metrics["faithfulness"] + metrics["relevancy"]) / 2
print(f"Config {cfg}: composite={composite:.3f}")
if composite > best_score:
best_score = composite
best = cfg
return best, best_score
# Define search space
configs = [
RetrievalConfig(v, b, m, r, "cross-encoder/ms-marco-MiniLM-L-6-v2")
for v in [20, 30, 50]
for b in [20, 30]
for m in ["reciprocal_rerank", "relative_score"]
for r in [3, 5, 8]
]
best_cfg, best_score = await run_sweep(configs, eval_queries)
print(f"Best config: {best_cfg} (score={best_score:.3f})")Persist best_cfg to a YAML file and load it in production. Re-run the sweep monthly or when you add >10% new documents.
Verify success: The sweep completes without errors and produces a config that beats your manual baseline by ≥2% composite score.
Step 8: Monitor retrieval quality in production
Tuning isn’t done at deploy. Log every query’s retrieved node IDs, reranker scores, and final answer. Set up a weekly job that samples 100 queries, runs them through your evaluators, and alerts on regression.
# Production logging middleware (simplified)
import json
import time
from contextvars import ContextVar
query_log_var: ContextVar[dict] = ContextVar("query_log")
async def logged_query(engine, query: str):
start = time.time()
response = await engine.aquery(query)
latency = time.time() - start
log_entry = {
"query": query,
"retrieved_nodes": [n.node_id for n in response.source_nodes],
"reranker_scores": [n.score for n in response.source_nodes],
"answer": str(response),
"latency_ms": latency * 1000,
"timestamp": time.time(),
}
# Ship to your observability stack (Datadog, Loki, ClickHouse, etc.)
log_to_observability(log_entry)
return responseVerify success: You have a dashboard showing weekly faithfulness/relevancy trends with alerts on >5% drop.
Verification checklist
Run through this list before considering the tuning cycle complete:
- Baseline hit_rate@10 and MRR@10 recorded
- Sweep over k identifies
k_prerankwhere hit_rate plateaus - Reranker added; reranked hit_rate@5 ≥ baseline hit_rate@10
- Hybrid retrieval evaluated; improves keyword-heavy queries
- Faithfulness and relevancy ≥ 4.0/5.0 on eval set
- Automated sweep produces a persisted best config
- Production logging captures retrieved nodes and reranker scores
- Weekly regression alert configured
Common pitfalls
Increasing k without reranking just feeds garbage to the LLM. The context window fills with low-similarity chunks, faithfulness tanks, and latency spikes. Always pair high k with a reranker.
Using the wrong reranker for your domain. General-domain cross-encoders (ms-marco) underperform on code, legal, or biomedical text. Fine-tune or pick a domain-specific model (bge-reranker-large, jina-reranker-v2-base-multilingual).
Ignoring chunking interaction. Top-k operates on chunks, not documents. If your chunks are 512 tokens with 50-token overlap, k=10 covers ~4.5k tokens. If you switch to 256-token chunks, you need ~2x k for equivalent coverage. Re-run the sweep after any chunking change.
Evaluating only retrieval. A retriever can have perfect hit_rate but the query engine still hallucinates if the synthesis prompt is weak. Always evaluate end-to-end with faithfulness/relevancy.
The loop is simple: measure, adjust, rerank, hybridize, evaluate, automate, monitor. Most teams stop at Step 2 and wonder why answer quality plateaus. Push through to Step 7 — the automated sweep — and you’ll have a system that improves every time you re-run it.