LlamaIndex retrieval evaluation: precision vs recall

If you’re running a llamaindex precision vs recall evaluation tutorial for the first time, you’ll quickly realize these metrics pull in opposite directions. Precision asks “of the chunks I retrieved, how many are actually relevant?” Recall asks “of all relevant chunks in the corpus, how many did I find?” Optimizing for one often degrades the other. This post breaks down how LlamaIndex measures both, where the trade-offs bite, and how to pick the right approach for your use case.

Understanding precision and recall in retrieval

Retrieval evaluation in LlamaIndex centers on the RetrieverEvaluator class, which compares a retriever’s output against a ground-truth dataset of (query, relevant_doc_ids) pairs. The mechanics are straightforward: for each query, you retrieve top-k nodes, then compute overlap with the expected set.

from llama_index.core.evaluation import RetrieverEvaluator
from llama_index.core.schema import NodeWithScore

evaluator = RetrieverEvaluator(
    retriever=my_retriever,
    metrics=["precision", "recall", "hit_rate", "mrr"],
)
results = await evaluator.aevaluate_dataset(dataset)

Precision@k = |retrieved ∩ relevant| / k. Recall@k = |retrieved ∩ relevant| / |relevant|. Hit rate@k = 1 if any relevant doc appears in top-k, else 0. MRR (mean reciprocal reward) weights the rank of the first relevant hit.

The key insight: these metrics assume your ground truth is complete. In practice, it rarely is. Annotators miss relevant chunks, corpora grow, and “relevance” is often subjective. Treat the numbers as directional, not absolute.

LlamaIndex evaluation framework overview

LlamaIndex provides two evaluation paths: the built-in RetrieverEvaluator for offline dataset-based evaluation, and ResponseEvaluator/FaithfulnessEvaluator for end-to-end RAG quality. For pure retrieval comparison, stick with RetrieverEvaluator.

You need a LabelledRetrieverDataset — essentially a JSONL file with queries and expected node IDs:

{"query": "How do I configure async in FastAPI?", "relevant_docs": ["node_12", "node_45"]}
{"query": "What's the default timeout for httpx?", "relevant_docs": ["node_7"]}

Generate this dataset manually for small corpora, or use LlamaIndex’s generate_question_context_pairs utility to bootstrap from documents (then human-review the output). The utility uses an LLM to synthesize questions from your chunks — useful for cold starts, but expect 30-50% noise.

from llama_index.core.evaluation import generate_question_context_pairs
from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("./docs").load_data()
dataset = generate_question_context_pairs(
    documents,
    llm=OpenAI(model="gpt-4o-mini"),
    num_questions_per_chunk=2,
)
dataset.save_json("eval_dataset.json")

Precision-focused evaluation

Precision optimization matters when false positives are costly: customer-facing search where irrelevant results erode trust, legal discovery where reviewing junk documents burns billable hours, or any pipeline where downstream LLM context windows are tight and you can’t afford noise.

LlamaIndex’s precision@k tells you how “clean” your top-k is. But precision alone is dangerous — a retriever that returns one perfect result and nine empty slots scores 10% precision@10 but 100% precision@1. Always pair it with hit rate or MRR.

# Precision-focused config: tight top-k, aggressive reranking
from llama_index.core.postprocessor import SentenceTransformerRerank

retriever = index.as_retriever(similarity_top_k=20)
reranker = SentenceTransformerRerank(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_n=5,
)

# Pipeline: retrieve 20, rerank to 5
nodes = retriever.retrieve(query)
filtered = reranker.postprocess_nodes(nodes, query_str=query)

Reranking is the primary precision lever. Cross-encoders (like the MS MARCO model above) score query-document pairs jointly, catching semantic mismatches that bi-encoders miss. Cost: ~50-200ms per query per 20 candidates on CPU, more on GPU. Budget accordingly.

Precision-focused teams should also track false positive rate by category. Tag your eval queries by type (factual, procedural, comparative) and see where precision collapses. In our experience, procedural queries (“how do I…”) suffer most from keyword overlap false positives — e.g., retrieving “how to deploy” docs for “how to rollback” queries.

Recall-focused evaluation

Recall optimization matters when missing relevant information is costly: research assistants, compliance search, debugging aids where the “needle in haystack” must be found. The trade-off: you’ll feed more noise to the generator, increasing token spend and hallucination risk.

LlamaIndex’s recall@k caps at 1.0 once all relevant docs are retrieved. But the denominator — total relevant docs — is only as good as your ground truth. If annotators found 3 relevant chunks but 5 actually exist, your max measurable recall is 60%.

# Recall-focused config: wide retrieval, hybrid search
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever

vector_retriever = index.as_retriever(similarity_top_k=50)
bm25_retriever = BM25Retriever.from_defaults(
    index=index,
    similarity_top_k=50,
)

fusion_retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    similarity_top_k=50,
    num_queries=4,  # query expansion
    mode="reciprocal_rerank",
)

Hybrid search (vector + BM25) is the main recall lever. BM25 catches exact keyword matches that embeddings miss — error codes, API names, version numbers. Query expansion (generating sub-queries via LLM) helps with underspecified queries. Both increase latency and token cost.

A practical pattern: retrieve 50-100 via hybrid, then rerank to 10-20 for the generator. This preserves recall while limiting context bloat. The reranker becomes your precision guardrail.

Comparison: precision vs recall configurations

Trade-offs in practice

The precision-recall curve isn’t theoretical — it shows up in three concrete places:

1. Chunking strategy. Smaller chunks (256 tokens) increase recall by isolating relevant passages, but hurt precision because context fragments across chunks. Larger chunks (1024+) improve precision per chunk but dilute relevance density. We’ve found 512 tokens with 50-token overlap to be a reasonable default for technical docs; tune per corpus.

2. Embedding model choice. General-purpose embeddings (text-embedding-3-small, bge-small-en) favor recall — they’re trained for broad semantic similarity. Domain-adapted embeddings (fine-tuned on your corpus) shift toward precision by learning your terminology. The cost: you need labeled pairs to fine-tune, and re-embedding the corpus on every model update.

3. Metadata filtering. Hard filters (date ranges, product versions, access control) improve both metrics by shrinking the search space before semantic matching. But they require reliable metadata extraction at ingestion time. If your metadata is 80% accurate, filters introduce 20% false negatives. Audit your metadata quality before relying on it.

Which to choose

Choose precision-optimized when:

• Building customer-facing search or chat where irrelevant answers damage credibility
• Downstream LLM calls are expensive (GPT-4, Claude Opus) and context budget is tight
• Your domain has high vocabulary overlap between distinct topics (e.g., medical specialties sharing terminology)
• You can invest in human-annotated eval sets and cross-encoder reranking

Choose recall-optimized when:

• Building internal research tools where missing a document is worse than skimming extras
• Queries are often underspecified or exploratory (“tell me about authentication”)
• You have cheap generator tokens (local models, GPT-4o-mini) and can absorb noise
• Your corpus has sparse keyword signals (error codes, config keys) that embeddings miss

Choose hybrid (the pragmatic default):

• Retrieve 50 via hybrid (vector + BM25)
• Rerank to 10 with a fast cross-encoder (ms-marco-MiniLM-L-6-v2)
• Evaluate both precision@10 and recall@50 on a representative dataset
• Adjust top-k and reranker threshold until the precision-recall trade-off matches your cost function

# The pragmatic default pipeline
from llama_index.core import QueryBundle
from llama_index.core.postprocessor import SentenceTransformerRerank

def retrieve_with_fallback(query: str, top_k: int = 10):
    # Wide hybrid retrieval
    nodes = fusion_retriever.retrieve(query)
    
    # Precision guardrail
    reranker = SentenceTransformerRerank(
        model="cross-encoder/ms-marco-MiniLM-L-6-v2",
        top_n=top_k,
    )
    return reranker.postprocess_nodes(nodes, query_str=query)

Run your eval dataset weekly. When precision@10 drops below 0.6 or recall@50 below 0.7, investigate: corpus drift, embedding degradation, or annotation gaps. The metrics are only as good as the discipline behind them.