Building a retrieval system is straightforward. Knowing whether it actually works is harder. Most teams skip evaluation because creating a labeled dataset feels like a separate research project. This llamaindex evaluation dataset generation tutorial shows you how to use an LLM to synthesize realistic question–context–answer triples from your existing documents, then validate the output before you run a single metric. The approach works with any document corpus and requires no human annotation to start.
Step 1: Prepare your document corpus
LlamaIndex expects a list of Document objects. If you already have a vector index, extract the underlying nodes. Otherwise, load raw files and chunk them to a size that matches your retrieval window — typically 512–1024 tokens for dense embeddings.
from llama_index.core import SimpleDirectoryReader, SentenceSplitter
from llama_index.core.schema import Document
reader = SimpleDirectoryReader(input_dir="./data", recursive=True)
raw_docs = reader.load_data()
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents(raw_docs)
# Optional: persist for reproducibility
import json
with open("corpus_nodes.json", "w") as f:
json.dump([n.to_dict() for n in nodes], f)Verify success: len(nodes) should be in the hundreds for a modest corpus. Inspect a few node.text values to confirm chunk boundaries make sense.
Step 2: Define the generation prompt
The quality of your synthetic dataset depends on the prompt. You want the LLM to produce questions that a real user would ask, grounded strictly in the provided context. Use a structured output schema so you can parse results reliably.
from pydantic import BaseModel, Field
from typing import List
class QAItem(BaseModel):
question: str = Field(..., description="A natural-language question answerable from the context.")
answer: str = Field(..., description="Concise answer derived only from the context.")
reasoning: str = Field(..., description="One-sentence justification linking question to context.")
class QASet(BaseModel):
items: List[QAItem] = Field(..., min_items=3, max_items=5)Prompt template (adjust num_questions per chunk):
from llama_index.core.prompts import PromptTemplate
QA_GEN_PROMPT = PromptTemplate(
"Context:\n{context_str}\n\n"
"Generate {num_questions} diverse question-answer pairs that a user might ask about this context. "
"Each question must be answerable *only* from the given text. "
"Return a JSON object matching this schema:\n{schema}\n\n"
"Do not include any extra commentary."
)Step 3: Generate questions per chunk
Iterate over your nodes, call the LLM, and collect structured outputs. Use a capable instruction-following model — GPT-4o, Claude 3.5 Sonnet, or a local Llama-3.1-70B-Instruct all work. If you route through a gateway that supports automatic fallback (for example, n4n.ai forwards to 240+ models and retries on provider degradation), you avoid halting the pipeline when one provider is rate-limited.
from llama_index.core.llms import OpenAI
from llama_index.core.output_parsers import PydanticOutputParser
from tqdm.auto import tqdm
llm = OpenAI(model="gpt-4o", temperature=0.3)
parser = PydanticOutputParser(output_cls=QASet)
all_qa = []
for node in tqdm(nodes, desc="Generating QA pairs"):
prompt = QA_GEN_PROMPT.format(
context_str=node.text,
num_questions=3,
schema=parser.schema()
)
try:
response = llm.complete(prompt)
qa_set = parser.parse(response.text)
for item in qa_set.items:
all_qa.append({
"question": item.question,
"answer": item.answer,
"reasoning": item.reasoning,
"source_node_id": node.node_id,
"source_text": node.text[:500] # truncate for storage
})
except Exception as e:
print(f"Failed on node {node.node_id}: {e}")
continue
import pandas as pd
df = pd.DataFrame(all_qa)
df.to_parquet("synthetic_qa.parquet", index=False)Verify success: df.shape[0] should equal roughly 3 * len(nodes). Spot-check 10 rows — questions should be specific, answers should appear verbatim or near-verbatim in source_text.
Step 4: Filter low-quality pairs
LLMs hallucinate or produce trivial questions (“What is the first word of this text?”). Apply automated filters before human review.
def is_trivial(q: str) -> bool:
trivial_patterns = [
r"^what is the first",
r"^what is the last",
r"^how many words",
r"^what is the length",
]
return any(re.search(p, q.lower()) for p in trivial_patterns)
def answer_in_context(row) -> bool:
# Loose containment check; tune threshold as needed
return row["answer"].lower() in row["source_text"].lower()
df["trivial"] = df["question"].apply(is_trivial)
df["grounded"] = df.apply(answer_in_context, axis=1)
clean = df[~df["trivial"] & df["grounded"]].copy()
clean.to_parquet("synthetic_qa_clean.parquet", index=False)
print(f"Kept {len(clean)} / {len(df)} pairs")Verify success: retention rate typically 70–90%. If it drops below 60%, tighten the prompt (add “non-trivial” instruction) or lower the temperature.
Step 5: Add negative samples for retrieval evaluation
Retrieval metrics (Recall@k, MRR, NDCG) need negatives — queries where the correct chunk is not the top result. The simplest approach: for each question, sample k other chunks as hard negatives using embedding similarity.
from llama_index.core.embeddings import OpenAIEmbedding
from llama_index.core import VectorStoreIndex
import numpy as np
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
index = VectorStoreIndex(nodes, embed_model=embed_model)
retriever = index.as_retriever(similarity_top_k=20)
negatives = []
for _, row in clean.iterrows():
retrieved = retriever.retrieve(row["question"])
# Exclude the true source node
neg_nodes = [n for n in retrieved if n.node_id != row["source_node_id"]][:5]
for n in neg_nodes:
negatives.append({
"question": row["question"],
"negative_node_id": n.node_id,
"negative_text": n.text[:500],
"score": n.score
})
neg_df = pd.DataFrame(negatives)
neg_df.to_parquet("synthetic_negatives.parquet", index=False)Verify success: each question should have 3–5 negatives with non-zero similarity scores. Inspect a few to confirm they are semantically related but not answer-bearing.
Step 6: Split into train / dev / test
Hold out a test set that never touches your prompt engineering or hyperparameter tuning. A 70/15/15 split by question (not by chunk) prevents leakage.
from sklearn.model_selection import train_test_split
questions = clean["question"].unique()
train_q, test_q = train_test_split(questions, test_size=0.15, random_state=42)
train_q, dev_q = train_test_split(train_q, test_size=0.176, random_state=42) # 0.15 of total
def split_df(df, q_list):
return df[df["question"].isin(q_list)].reset_index(drop=True)
train_set = split_df(clean, train_q)
dev_set = split_df(clean, dev_q)
test_set = split_df(clean, test_q)
for name, ds in [("train", train_set), ("dev", dev_set), ("test", test_set)]:
ds.to_parquet(f"eval_{name}.parquet", index=False)
print(f"{name}: {len(ds)} pairs, {ds['question'].nunique()} unique questions")Verify success: no question appears in more than one split. test_set should have at least 50 unique questions for stable metric confidence intervals.
Step 7: Run a baseline retrieval evaluation
Now you have a dataset. Plug it into LlamaIndex’s evaluation module (or your own metric code) to get a baseline before you change chunking, embedding model, or reranker.
from llama_index.core.evaluation import RetrieverEvaluator
evaluator = RetrieverEvaluator.from_metric_names(
["hit_rate", "mrr", "precision", "recall"],
retriever=retriever
)
# Evaluate on dev set
dev_questions = dev_set["question"].unique().tolist()
expected_ids = dev_set.groupby("question")["source_node_id"].apply(list).to_dict()
results = evaluator.evaluate(dev_questions, expected_ids)
print(results)Typical baseline on a clean corpus with text-embedding-3-small and 512-token chunks: Hit Rate@5 ~ 0.75, MRR ~ 0.65. Your numbers will differ — what matters is the delta after each experiment.
Step 8: Version and store the dataset
Treat the synthetic dataset as a first-class artifact. Commit the parquet files (or a manifest with hashes) to DVC or Git LFS. Record the generation metadata:
{
"generation_date": "2025-01-15",
"llm_model": "gpt-4o",
"embedding_model": "text-embedding-3-small",
"chunk_size": 512,
"chunk_overlap": 64,
"questions_per_chunk": 3,
"filter_retention_rate": 0.82,
"splits": {"train": 1240, "dev": 265, "test": 265}
}When you regenerate (new corpus, better prompt, different LLM), bump the version. This lets you bisect retrieval regressions to data changes vs. model changes.
Common pitfalls
Overfitting to synthetic style. LLMs generate questions with a distinct “textbook” flavor. Mitigate by adding a style-diversity instruction: “Vary question style: some direct, some conversational, some keyword-only, some multi-hop.”
Leakage via chunk metadata. If your nodes carry file_name or page_label, the LLM can cheat by asking “What is the title of document X?” Strip metadata from the context string passed to the generator.
Single-hop bias. Most generated questions answerable from one chunk. Explicitly request multi-hop: “Generate 1 question requiring synthesis across two non-adjacent sections.” Then pair those questions with both source node IDs.
No human spot-check. Automation gets you 80% of the way. Spend 30 minutes reading 50 random QA pairs. You will find systematic errors no filter catches.
Next steps
With a versioned evaluation set in hand, you can now:
- Sweep chunk sizes (256, 512, 1024) and measure Hit Rate@k delta
- Compare embedding models (OpenAI, Cohere, BGE, E5) on the same queries
- Test rerankers (Cohere Rerank, BGE-Reranker, cross-encoder) on the dev set
- Run ablation: remove negatives, add BM25 hybrid, swap retriever top-k
Each experiment is a single script run because the dataset is fixed. That is the point — evaluation becomes a fast feedback loop, not a quarterly project.