Haystack evaluation pipeline tutorial: dataset preparation

If you’re building a RAG system or LLM application with Haystack, you need an evaluation pipeline before you ship. The haystack evaluation pipeline dataset preparation phase is where most teams cut corners — they grab a few hand-crafted examples, call it a test set, and wonder why production behavior diverges. This tutorial walks through building a proper evaluation dataset from scratch: sourcing questions, generating ground truth, formatting for Haystack’s evaluators, and validating the whole thing runs.

Prerequisites

You need Python 3.10+ and a virtual environment. Install the core packages:

pip install haystack-ai==2.6.0 datasets==2.19.0 pandas==2.2.0 tqdm==4.66.0

If you’re evaluating a RAG pipeline, also install your document store and retriever dependencies (e.g., pip install weaviate-client==4.6.0 or pip install elasticsearch==8.13.0). This tutorial uses an in-memory document store for portability.

You should already have a Haystack pipeline you want to evaluate — either a Pipeline object or a RAG pipeline built with DocumentStore, Retriever, and Generator components. If you don’t have one yet, the code below includes a minimal RAG setup you can swap in.

What an evaluation dataset actually looks like

Haystack’s evaluators expect a Dataset object (from the datasets library) with specific columns. For a basic QA or RAG evaluation, you need at minimum:

Column	Purpose
`question`	The input query
`ground_truth_answer`	The expected answer (string or list of strings)
`context`	Retrieved documents (list of strings) — required for context-aware metrics
`predicted_answer`	Your pipeline’s output — filled in during evaluation run

For retrieval evaluation, you also need ground_truth_contexts (list of relevant document IDs or text). For generation-only evaluation, context is optional.

Let’s create a dataset from scratch.

Step 1: Source or write your questions

Start with a representative sample of real user queries. If you have production logs, extract 100-200 unique questions. If not, write them manually covering your domain’s edge cases: ambiguous phrasing, multi-hop reasoning, out-of-scope requests, and known failure modes.

# questions.py
QUESTIONS = [
    "What is the capital of France?",
    "Who wrote the novel '1984'?",
    "Explain the difference between SQL and NoSQL databases.",
    "How do I create a virtual environment in Python?",
    "What are the main causes of climate change?",
    "Summarize the plot of Hamlet in three sentences.",
    "What is the time complexity of quicksort?",
    "How does a transformer model differ from an RNN?",
    "What is the GDP of Germany in 2023?",
    "List the planets in our solar system in order from the sun.",
    # Add 90+ more covering your domain
]

Save this to questions.py. Aim for at least 50 questions for a meaningful signal; 200+ is better for statistical significance.

Step 2: Generate ground truth answers

You have two paths: human annotation (gold standard) or LLM-assisted with human review. For a tutorial, we’ll use an LLM to generate draft answers, then show how to validate them.

# generate_ground_truth.py
from haystack import Pipeline
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder
from haystack.utils import Secret
from tqdm import tqdm
import json

# Use a strong model for ground truth generation
generator = OpenAIGenerator(
    model="gpt-4o",
    api_key=Secret.from_env_var("OPENAI_API_KEY")
)

prompt_template = """
You are an expert creating ground truth answers for an evaluation dataset.
Answer the following question accurately and concisely.
If the question is ambiguous, state the ambiguity and provide the most likely answer.
If you don't know, say "I don't know" — do not hallucinate.

Question: {{question}}
Answer:
"""

prompt_builder = PromptBuilder(template=prompt_template)

pipeline = Pipeline()
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("generator", generator)
pipeline.connect("prompt_builder.prompt", "generator.prompt")

ground_truth = {}
for q in tqdm(QUESTIONS, desc="Generating ground truth"):
    result = pipeline.run({"prompt_builder": {"question": q}})
    answer = result["generator"]["replies"][0].strip()
    ground_truth[q] = answer

with open("ground_truth.json", "w") as f:
    json.dump(ground_truth, f, indent=2)

Run this once. Then review every answer manually. This step is non-negotiable — LLM-generated ground truth contains subtle errors that poison your evaluation. Fix inaccuracies, add nuance, and mark unanswerable questions explicitly.

Expected output (ground_truth.json):

{
  "What is the capital of France?": "Paris",
  "Who wrote the novel '1984'?": "George Orwell",
  "Explain the difference between SQL and NoSQL databases.": "SQL databases are relational, use structured schemas, and support ACID transactions. NoSQL databases are non-relational, schema-flexible, and optimize for horizontal scaling and specific data models (document, key-value, graph, column-family).",
  ...
}

Step 3: Retrieve contexts for each question (RAG only)

If you’re evaluating a RAG pipeline, you need the contexts your retriever would actually return. This lets you measure retrieval quality separately from generation quality.

# retrieve_contexts.py
from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders import PromptBuilder
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack import Document
from tqdm import tqdm
import json

# Load your actual document store (this is a toy example)
document_store = InMemoryDocumentStore()
documents = [
    Document(content="Paris is the capital and most populous city of France.", meta={"source": "wiki"}),
    Document(content="George Orwell, born Eric Arthur Blair, wrote 1984 in 1949.", meta={"source": "wiki"}),
    Document(content="SQL databases use structured query language and predefined schemas. NoSQL databases use flexible schemas...", meta={"source": "db-guide"}),
    # Add your full corpus here
]
document_store.write_documents(documents)

retriever = InMemoryBM25Retriever(document_store=document_store, top_k=5)

pipeline = Pipeline()
pipeline.add_component("retriever", retriever)

contexts = {}
for q in tqdm(QUESTIONS, desc="Retrieving contexts"):
    result = pipeline.run({"retriever": {"query": q}})
    retrieved_docs = result["retriever"]["documents"]
    contexts[q] = [doc.content for doc in retrieved_docs]

with open("contexts.json", "w") as f:
    json.dump(contexts, f, indent=2)

Run this against your production document store and retriever configuration. The contexts must match what your actual pipeline sees at inference time.

Expected output (contexts.json):

{
  "What is the capital of France?": ["Paris is the capital and most populous city of France."],
  "Who wrote the novel '1984'?": ["George Orwell, born Eric Arthur Blair, wrote 1984 in 1949."],
  ...
}

Step 4: Assemble the Haystack dataset

Now combine questions, ground truth, and contexts into a datasets.Dataset with the exact column names Haystack evaluators expect.

# build_dataset.py
from datasets import Dataset
import json

with open("ground_truth.json") as f:
    ground_truth = json.load(f)

with open("contexts.json") as f:
    contexts = json.load(f)

rows = []
for question in QUESTIONS:
    rows.append({
        "question": question,
        "ground_truth_answer": ground_truth[question],
        "context": contexts.get(question, []),
        # predicted_answer will be filled during evaluation
    })

dataset = Dataset.from_list(rows)
dataset.save_to_disk("eval_dataset")

print(f"Dataset size: {len(dataset)}")
print(f"Columns: {dataset.column_names}")
print(dataset[0])

Expected output:

Dataset size: 10
Columns: ['question', 'ground_truth_answer', 'context']
{'question': 'What is the capital of France?', 'ground_truth_answer': 'Paris', 'context': ['Paris is the capital and most populous city of France.']}

Step 5: Validate dataset quality

Before running evaluation, run automated checks. This catches missing contexts, empty answers, and format mismatches early.

# validate_dataset.py
from datasets import load_dataset

dataset = load_dataset("eval_dataset", split="train")

issues = []

for i, row in enumerate(dataset):
    if not row["question"].strip():
        issues.append(f"Row {i}: empty question")
    if not row["ground_truth_answer"].strip():
        issues.append(f"Row {i}: empty ground truth answer")
    if not row["context"]:
        issues.append(f"Row {i}: no retrieved contexts (retrieval failed?)")
    if not isinstance(row["context"], list):
        issues.append(f"Row {i}: context is not a list: {type(row['context'])}")
    for j, ctx in enumerate(row["context"]):
        if not isinstance(ctx, str):
            issues.append(f"Row {i}, context {j}: not a string: {type(ctx)}")

if issues:
    print(f"Found {len(issues)} issues:")
    for issue in issues[:20]:
        print(f"  - {issue}")
    if len(issues) > 20:
        print(f"  ... and {len(issues) - 20} more")
else:
    print("All checks passed.")

# Statistics
context_lengths = [len(ctx) for row in dataset for ctx in row["context"]]
print(f"\nContext stats: {len(context_lengths)} total contexts")
print(f"  Avg length: {sum(context_lengths)/len(context_lengths):.0f} chars")
print(f"  Min length: {min(context_lengths)} chars")
print(f"  Max length: {max(context_lengths)} chars")

Expected output (clean dataset):

All checks passed.

Context stats: 47 total contexts
  Avg length: 234 chars
  Min length: 42 chars
  Max length: 1,892 chars

If you see “no retrieved contexts”, your retriever configuration or document store has a problem — fix it before evaluating.

Step 6: Run a baseline evaluation

Now wire the dataset into Haystack’s evaluation framework. We’ll use FaithfulnessEvaluator and SASSEvaluator (semantic answer similarity) — the two most informative metrics for RAG.

# run_evaluation.py
from haystack import Pipeline
from haystack.components.evaluators import FaithfulnessEvaluator, SASSEvaluator
from haystack.components.generators import OpenAIGenerator
from haystack.utils import Secret
from datasets import load_dataset
import pandas as pd

# Load dataset
dataset = load_dataset("eval_dataset", split="train")

# Your RAG pipeline (replace with your actual pipeline)
from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders import PromptBuilder
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.generators import OpenAIGenerator
from haystack import Document

document_store = InMemoryDocumentStore()
documents = [
    Document(content="Paris is the capital and most populous city of France.", meta={"source": "wiki"}),
    Document(content="George Orwell, born Eric Arthur Blair, wrote 1984 in 1949.", meta={"source": "wiki"}),
    Document(content="SQL databases use structured query language and predefined schemas. NoSQL databases use flexible schemas...", meta={"source": "db-guide"}),
]
document_store.write_documents(documents)

retriever = InMemoryBM25Retriever(document_store=document_store, top_k=3)

prompt_template = """
Answer the question based only on the provided context.
If the context doesn't contain the answer, say "I don't know."

Context:
{% for doc in documents %}
{{doc.content}}
{% endfor %}

Question: {{question}}
Answer:
"""

prompt_builder = PromptBuilder(template=prompt_template)
generator = OpenAIGenerator(model="gpt-4o-mini", api_key=Secret.from_env_var("OPENAI_API_KEY"))

rag_pipeline = Pipeline()
rag_pipeline.add_component("retriever", retriever)
rag_pipeline.add_component("prompt_builder", prompt_builder)
rag_pipeline.add_component("generator", generator)
rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "generator.prompt")

# Run pipeline on all questions to get predicted answers
predicted_answers = []
retrieved_contexts = []

for row in tqdm(dataset, desc="Running RAG pipeline"):
    result = rag_pipeline.run({
        "retriever": {"query": row["question"]},
        "prompt_builder": {"question": row["question"]}
    })
    predicted_answers.append(result["generator"]["replies"][0].strip())
    retrieved_contexts.append([doc.content for doc in result["retriever"]["documents"]])

# Add predictions to dataset
dataset = dataset.add_column("predicted_answer", predicted_answers)
dataset = dataset.add_column("retrieved_contexts", retrieved_contexts)

# Evaluators
faithfulness = FaithfulnessEvaluator(
    api_key=Secret.from_env_var("OPENAI_API_KEY"),
    model="gpt-4o"
)
sas = SASSEvaluator(
    api_key=Secret.from_env_var("OPENAI_API_KEY"),
    model="gpt-4o"
)

# Run evaluation
faithfulness_results = faithfulness.run(
    questions=dataset["question"],
    contexts=dataset["retrieved_contexts"],
    predicted_answers=dataset["predicted_answer"]
)

sas_results = sas.run(
    questions=dataset["question"],
    ground_truth_answers=dataset["ground_truth_answer"],
    predicted_answers=dataset["predicted_answer"]
)

# Aggregate
faithfulness_scores = [r["score"] for r in faithfulness_results["results"]]
sas_scores = [r["score"] for r in sas_results["results"]]

print(f"Faithfulness: {sum(faithfulness_scores)/len(faithfulness_scores):.3f}")
print(f"SAS: {sum(sas_scores)/len(sas_scores):.3f}")

# Per-question breakdown
results_df = pd.DataFrame({
    "question": dataset["question"],
    "faithfulness": faithfulness_scores,
    "sas": sas_scores,
    "predicted": dataset["predicted_answer"],
    "ground_truth": dataset["ground_truth_answer"]
})
results_df.to_csv("evaluation_results.csv", index=False)
print("Saved detailed results to evaluation_results.csv")

Expected output:

Running RAG pipeline: 100%|██████████| 10/10 [00:12<00:00,  1.20s/it]
Faithfulness: 0.870
SAS: 0.820
Saved detailed results to evaluation_results.csv

Open evaluation_results.csv and sort by faithfulness or sas ascending. The bottom rows are your failure cases — investigate those first.

Step 7: Version and store your dataset

Treat evaluation datasets like code: version them, review changes, and track which dataset version produced which metric scores.

# Dataset versioning with DVC (recommended)
dvc init
dvc add eval_dataset
git add eval_dataset.dvc .gitignore
git commit -m "Add evaluation dataset v1.0 - 100 questions, BM25 retrieval"

# Tag the commit that produced your baseline metrics
git tag -a eval/v1.0-baseline -m "Baseline: Faithfulness=0.87, SAS=0.82"

When you modify the retriever, generator, or prompt, re-run Step 6 and tag the new results. This gives you a proper regression test suite for your LLM pipeline.

Common pitfalls

Using the same LLM for ground truth and evaluation. If you generate ground truth with GPT-4o and evaluate with GPT-4o, you measure self-consistency, not correctness. Use a stronger model for ground truth (or human annotation) and a cheaper model for evaluation.

Evaluating on training data. If your documents contain the exact Q&A pairs from your evaluation set, you’re measuring memorization. Hold out a true test set — ideally questions from real users that never appeared in your corpus.

Ignoring retrieval failures. A faithfulness score of 1.0 on an empty context is meaningless. Always check retrieval recall separately: what fraction of questions have any relevant document in the top-k?

Single-metric obsession. Faithfulness + SAS covers generation quality. Add ContextRelevanceEvaluator for retrieval quality. For classification tasks, use AccuracyEvaluator. Match metrics to your failure modes.

Next steps

This dataset preparation pipeline gives you a repeatable, versioned evaluation foundation. From here you can:

Add adversarial examples (typos, negations, out-of-domain) to stress-test robustness
Build a CI gate that fails if faithfulness drops below threshold
A/B test prompt variations against the same frozen dataset
Extend to multi-turn conversation evaluation with ConversationEvaluator

The haystack evaluation pipeline dataset preparation work you did here — sourcing real questions, generating reviewed ground truth, capturing actual retrieval contexts — is the difference between “it works on my machine” and “we know how it behaves in production.”