This llamaindex retriever benchmark recall at k tutorial shows how to measure whether your retriever surfaces the documents you expect before you ship a RAG pipeline. Recall at k is the fraction of relevant documents present in the top-k retrieved nodes, and it is the first metric you should track to catch silent retrieval regressions.
Step 1: Build a labeled evaluation set
Retrieval recall is meaningless without ground truth. You need a set of queries and the document IDs that are relevant to each query. In LlamaIndex, nodes carry node_id and metadata; tie your labels to those IDs.
I recommend hand-labeling at least 30–50 real queries from production logs or user sessions. Synthetic queries generated by an LLM inflate scores and hide failure modes because they tend to mirror the chunk boundaries you already have.
Define the dataset as a plain dictionary. Use id_ on Document so the resulting index nodes preserve that identifier:
from llama_index.core import Document
docs = [
Document(text="Postgres uses MVCC for concurrency control.", id_="doc1"),
Document(text="Redis is an in-memory key-value store.", id_="doc2"),
Document(text="Kafka partitions logs for horizontal scale.", id_="doc3"),
Document(text="SQLite is a serverless embedded database.", id_="doc4"),
]
# query -> list of relevant doc ids
ground_truth = {
"How does Postgres handle concurrency?": ["doc1"],
"What is Redis?": ["doc2"],
"Explain Kafka scaling.": ["doc3"],
"Which database is embedded and serverless?": ["doc4"],
}After indexing, verify the mapping by printing the ref doc ID from a retrieved node. If you chunk a document into multiple nodes, they share ref_doc_id; that is the correct key for document-level recall.
Chunk size affects labeling
Default LlamaIndex chunk size is 1024 tokens. For short factual corpora, that often places one sentence per chunk, which is fine. For long PDFs, use SentenceSplitter(chunk_size=256) and label at the chunk level using node.id_. Mixing document-level and chunk-level labels is the most common source of confusing benchmark numbers.
Step 2: Index documents and create the retriever
Use a local embedding model to avoid API keys during iteration. The HuggingFaceEmbedding class runs offline and is deterministic.
from llama_index.core import VectorStoreIndex, Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.node_parser import SentenceSplitter
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Settings.node_parser = SentenceSplitter(chunk_size=256)
index = VectorStoreIndex.from_documents(docs)
retriever = index.as_retriever(similarity_top_k=3)If you are benchmarking at scale and routing embeddings through an OpenAI-compatible gateway, n4n.ai addresses 240+ models with automatic fallback when a provider is rate-limited, which keeps a 10k-document index build from stalling mid-benchmark.
Step 3: Compute recall@k manually
Write a tight loop. For each query, retrieve top-k, collect the referenced doc IDs, and check intersection with ground truth. This explicit version beats a black-box call when you are debugging a new retriever.
def recall_at_k(retriever, ground_truth, k):
hits = 0
total = 0
per_query = {}
for query, relevant in ground_truth.items():
nodes = retriever.retrieve(query)
retrieved_ids = {
n.node.ref_doc_id or n.node.id_ for n in nodes[:k]
}
hit = len(set(relevant) & retrieved_ids)
hits += hit
total += len(relevant)
per_query[query] = hit / max(1, len(relevant))
return hits / max(1, total), per_query
overall, per_q = recall_at_k(retriever, ground_truth, k=3)
print(f"Recall@3: {overall:.2f}")ref_doc_id points to the source document. If you labeled chunk nodes, swap to n.node.id_. Always slice nodes[:k]; the retriever may return more than k nodes when hybrid fusion is involved.
Handling missing IDs
If ref_doc_id is None, fall back to n.node.id_. Log any query where retrieved IDs are empty—that indicates an embedding or index build failure, not a recall problem.
Step 4: Compare retrievers and parameters
A real llamaindex retriever benchmark recall at k tutorial must compare alternatives. Add a BM25 retriever as a lexical baseline; many corpora beat dense vectors on exact keyword matches.
from llama_index.retrievers.bm25 import BM25Retriever
bm25 = BM25Retriever.from_index(index, similarity_top_k=3)
vector_recall, _ = recall_at_k(retriever, ground_truth, 3)
bm25_recall, _ = recall_at_k(bm25, ground_truth, 3)
print(f"Vector Recall@3: {vector_recall:.2f}")
print(f"BM25 Recall@3: {bm25_recall:.2f}")Run the same loop across similarity_top_k values 1, 3, 5, 10 to plot the recall curve. Recall@k always increases with k; the slope tells you whether your embedding space is sparse. A flat curve after k=3 means additional context windows only add noise.
Add a reranker
A SentenceTransformerReranker can lift recall at small k by reordering. Benchmark it as a separate retriever wrapper:
from llama_index.core.postprocessor import SentenceTransformerRerank
reranked = index.as_retriever(similarity_top_k=10)
reranked.postprocessors = [SentenceTransformerRerank(top_n=3)]
recall_at_k(reranked, ground_truth, 3)Measure the reranked top-3 against the raw top-3. If the delta is under 0.05, skip the reranker in production.
Step 5: Use RetrieverEvaluator for reproducible runs
LlamaIndex ships RetrieverEvaluator with a RecallMetric. It standardizes the loop and plugs into CI.
from llama_index.core.evaluation import RetrieverEvaluator
from llama_index.core.evaluation.metrics import RecallMetric
evaluator = RetrieverEvaluator(
retriever=retriever,
metrics=[RecallMetric(k=3)],
)
eval_rows = []
for q, rel in ground_truth.items():
eval_rows.append({
"query": q,
"relevant_nodes": [index.docstore.get_node(doc_id) for doc_id in rel]
})
result = evaluator.evaluate(eval_rows)
print(result)Wrap this in a pytest fixture so CI fails when recall drops below a threshold:
def test_retriever_recall():
res = evaluator.evaluate(eval_rows)
assert res["recall@3"] >= 0.8Freeze the ground_truth dict in a versioned JSON file. Treat any edit to it as a new baseline, not a performance gain.
Step 6: Verify success
Success means you have a number you trust and a script you can rerun. After following this llamaindex retriever benchmark recall at k tutorial, you should:
- Have a
ground_truthdict with at least 30 queries. - Print
Recall@3andRecall@5for each retriever variant. - See the pytest assertion pass in CI on every commit.
If recall is below 0.7 on real queries, fix chunk size or add a keyword retriever before tuning embeddings. Recall@k will not improve by switching model names alone; it responds to retrieval architecture.
Common pitfalls
- Duplicate documents: same text under two IDs double-counts hits. Dedupe by content hash before indexing.
- Metadata filters: if your retriever applies
filters=, include them in the benchmark, or you measure a different system. - Top-k truncation: always slice
nodes[:k]. Passingsimilarity_top_k=10but computing @3 without slicing inflates nothing but your confusion. - Cold index: first retrieval after build may trigger lazy loading. Warm up with one dummy query before timing.
Extending the benchmark
Once the basic loop works, add latency measurement with time.perf_counter() around retrieve. Record p95 latency per query alongside recall. A retriever that needs k=20 to hit 0.9 recall but takes 2s is worse than one at k=5 with 0.85 recall and 80ms.
For larger corpora, generate the index once and persist to disk:
index.storage_context.persist(persist_dir="./storage")Load it back to avoid re-embedding on every benchmark run. The methodology in this llamaindex retriever benchmark recall at k tutorial stays valid as you swap vector stores, add rerankers, or move to hybrid search. The only variable is the retriever object passed to recall_at_k.
Keep the ground-truth set frozen. When you add new documents, create a new eval file with a date stamp and compare runs side by side, not as a replacement. That discipline is what turns a one-off script into a retrieval regression gate.