If you’re shipping a RAG system, you need to know whether your retriever is actually surfacing useful context — not just whether the generator produces plausible text. Haystack’s evaluation framework lets you measure context relevance directly, but the documentation assumes you already understand the moving parts. This tutorial walks through building a complete evaluation pipeline from scratch, with runnable code at every stage.
Prerequisites
You need Python 3.10+ and a virtual environment. Install the core packages:
python -m venv .venv
source .venv/bin/activate
pip install haystack-ai==2.6.0 datasets==2.19.0 tqdm==4.66.0You also need an OpenAI API key for the generator and the LLM-as-judge evaluator. Export it:
export OPENAI_API_KEY="sk-..."If you’re routing through a gateway like n4n.ai, set OPENAI_BASE_URL to your endpoint and the same key works — the evaluation code doesn’t change.
The evaluation dataset
Haystack expects evaluation data in a specific format: a list of EvaluationResult objects, each containing a question, the ground-truth answer, and the retrieved documents. For this tutorial we’ll synthesize a small dataset, but in production you’d load from your labeling tool or export from your observability stack.
Create data/eval_questions.json:
[
{
"question": "What is the capital of France?",
"ground_truth_answer": "Paris is the capital of France.",
"contexts": [
"Paris is the capital and most populous city of France.",
"France is a country in Western Europe."
]
},
{
"question": "Who wrote the novel 1984?",
"ground_truth_answer": "George Orwell wrote 1984.",
"contexts": [
"Nineteen Eighty-Four is a dystopian novel by George Orwell.",
"George Orwell was an English novelist and essayist."
]
},
{
"question": "What is the boiling point of water at sea level?",
"ground_truth_answer": "Water boils at 100°C (212°F) at standard atmospheric pressure.",
"contexts": [
"At sea level, water boils at 100 degrees Celsius.",
"The boiling point of water varies with atmospheric pressure."
]
},
{
"question": "Which planet is known as the Red Planet?",
"ground_truth_answer": "Mars is known as the Red Planet.",
"contexts": [
"Mars is often called the Red Planet due to iron oxide on its surface.",
"Mars is the fourth planet from the Sun."
]
},
{
"question": "What is the largest mammal?",
"ground_truth_answer": "The blue whale is the largest mammal.",
"contexts": [
"The blue whale is the largest animal known to have ever existed.",
"Whales are marine mammals."
]
}
]Load it in Python:
# load_dataset.py
import json
from haystack.dataclasses import Document
from haystack import EvaluationResult
with open("data/eval_questions.json") as f:
raw = json.load(f)
eval_data = []
for item in raw:
docs = [Document(content=ctx) for ctx in item["contexts"]]
eval_data.append(
EvaluationResult(
inputs={"question": item["question"]},
responses={"answer": item["ground_truth_answer"]},
retrieved_documents=docs
)
)
print(f"Loaded {len(eval_data)} evaluation examples")Run it:
python load_dataset.pyExpected output:
Loaded 5 evaluation examplesBuild the RAG pipeline to evaluate
You evaluate a pipeline, not a retriever in isolation. Here’s a minimal RAG pipeline using an in-memory document store and OpenAI’s text-embedding-3-small for retrieval, gpt-4o-mini for generation.
Create pipelines/rag_pipeline.py:
# pipelines/rag_pipeline.py
from haystack import Pipeline
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack import Document
# Seed the document store with the same contexts from eval data
# In reality this comes from your corpus ingestion pipeline
doc_store = InMemoryDocumentStore()
seed_docs = [
Document(content="Paris is the capital and most populous city of France."),
Document(content="France is a country in Western Europe."),
Document(content="Nineteen Eighty-Four is a dystopian novel by George Orwell."),
Document(content="George Orwell was an English novelist and essayist."),
Document(content="At sea level, water boils at 100 degrees Celsius."),
Document(content="The boiling point of water varies with atmospheric pressure."),
Document(content="Mars is often called the Red Planet due to iron oxide on its surface."),
Document(content="Mars is the fourth planet from the Sun."),
Document(content="The blue whale is the largest animal known to have ever existed."),
Document(content="Whales are marine mammals."),
]
doc_store.write_documents(seed_docs)
# Embedder for the query
query_embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
# Retriever
retriever = InMemoryEmbeddingRetriever(document_store=doc_store, top_k=3)
# Prompt template
prompt_template = """
Answer the question using only the provided context.
Context:
{% for doc in documents %}
{{ doc.content }}
{% endfor %}
Question: {{ question }}
Answer:
"""
prompt_builder = PromptBuilder(template=prompt_template)
# Generator
generator = OpenAIGenerator(model="gpt-4o-mini")
# Assemble pipeline
rag_pipeline = Pipeline()
rag_pipeline.add_component("query_embedder", query_embedder)
rag_pipeline.add_component("retriever", retriever)
rag_pipeline.add_component("prompt_builder", prompt_builder)
rag_pipeline.add_component("generator", generator)
rag_pipeline.connect("query_embedder.embedding", "retriever.query_embedding")
rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "generator.prompt")
def run_rag(question: str) -> dict:
result = rag_pipeline.run({
"query_embedder": {"text": question},
"prompt_builder": {"question": question}
})
return {
"answer": result["generator"]["replies"][0],
"documents": result["retriever"]["documents"]
}
if __name__ == "__main__":
# Quick smoke test
out = run_rag("What is the capital of France?")
print("Answer:", out["answer"])
print("Retrieved docs:", [d.content for d in out["documents"]])Run the smoke test:
python pipelines/rag_pipeline.pyExpected output (answer will vary slightly):
Answer: Paris is the capital of France.
Retrieved docs: ['Paris is the capital and most populous city of France.', 'France is a country in Western Europe.', 'Nineteen Eighty-Four is a dystopian novel by George Orwell.']Notice the third retrieved document is irrelevant — this is exactly what context relevance evaluation catches.
Define the context relevance evaluator
Haystack provides ContextRelevanceEvaluator which uses an LLM to judge whether each retrieved document is relevant to the question. It returns a score per document and an aggregate.
Create evaluators/context_relevance.py:
# evaluators/context_relevance.py
from haystack.components.evaluators import ContextRelevanceEvaluator
from haystack import EvaluationResult
from haystack.dataclasses import Document
from typing import List
# The evaluator needs an LLM. We'll use gpt-4o-mini via OpenAIGenerator.
from haystack.components.generators import OpenAIGenerator
generator = OpenAIGenerator(model="gpt-4o-mini")
evaluator = ContextRelevanceEvaluator(generator=generator)
def evaluate_context_relevance(eval_data: List[EvaluationResult]) -> List[dict]:
"""
Run context relevance evaluation on a list of EvaluationResult objects.
Returns a list of dicts with per-document scores and aggregate.
"""
results = []
for ex in eval_data:
# The evaluator expects: question, contexts (list of strings)
contexts = [doc.content for doc in ex.retrieved_documents]
eval_result = evaluator.run(
questions=[ex.inputs["question"]],
contexts=[contexts]
)
# eval_result contains: individual_scores, score (aggregate)
results.append({
"question": ex.inputs["question"],
"individual_scores": eval_result["individual_scores"][0],
"aggregate_score": eval_result["score"][0],
"contexts": contexts
})
return results
if __name__ == "__main__":
# Test with one example
from load_dataset import eval_data
test_result = evaluate_context_relevance([eval_data[0]])
print(test_result[0])Run it:
python evaluators/context_relevance.pyExpected output:
{
'question': 'What is the capital of France?',
'individual_scores': [1, 1, 0],
'aggregate_score': 0.6666666666666666,
'contexts': [
'Paris is the capital and most populous city of France.',
'France is a country in Western Europe.',
'Nineteen Eighty-Four is a dystopian novel by George Orwell.'
]
}The third document (about 1984) correctly scores 0. The aggregate is the mean: 2/3 ≈ 0.67.
Wire the full haystack evaluation pipeline context relevance
Now combine the RAG pipeline run with the evaluator. This is the pattern you’ll use in CI: generate answers + retrieved docs for each eval question, then score them.
Create run_evaluation.py:
# run_evaluation.py
import json
from load_dataset import eval_data
from pipelines.rag_pipeline import run_rag
from evaluators.context_relevance import evaluate_context_relevance
from haystack import EvaluationResult
from haystack.dataclasses import Document
def generate_rag_predictions(eval_data: list[EvaluationResult]) -> list[EvaluationResult]:
"""
Run the RAG pipeline on each question, return new EvaluationResult objects
with generated answers and retrieved documents.
"""
predictions = []
for ex in eval_data:
question = ex.inputs["question"]
rag_output = run_rag(question)
pred = EvaluationResult(
inputs={"question": question},
responses={"answer": rag_output["answer"]},
retrieved_documents=rag_output["documents"]
)
predictions.append(pred)
return predictions
def main():
print("Running RAG pipeline on evaluation questions...")
predictions = generate_rag_predictions(eval_data)
print("Evaluating context relevance...")
eval_results = evaluate_context_relevance(predictions)
# Aggregate metrics
agg_scores = [r["aggregate_score"] for r in eval_results]
mean_relevance = sum(agg_scores) / len(agg_scores)
print(f"\n=== Context Relevance Results ===")
print(f"Mean aggregate relevance: {mean_relevance:.3f}")
print(f"Per-question breakdown:")
for r in eval_results:
print(f" Q: {r['question']}")
print(f" Aggregate: {r['aggregate_score']:.3f}")
print(f" Per-doc: {r['individual_scores']}")
for i, (ctx, score) in enumerate(zip(r['contexts'], r['individual_scores'])):
marker = "✓" if score == 1 else "✗"
print(f" {marker} [{score}] {ctx[:60]}...")
# Save detailed results for later analysis
output = {
"mean_context_relevance": mean_relevance,
"per_question": eval_results
}
with open("eval_results.json", "w") as f:
json.dump(output, f, indent=2)
print(f"\nSaved detailed results to eval_results.json")
if __name__ == "__main__":
main()Run the full evaluation:
python run_evaluation.pyExpected output:
Running RAG pipeline on evaluation questions...
Evaluating context relevance...
=== Context Relevance Results ===
Mean aggregate relevance: 0.733
Per-question breakdown:
Q: What is the capital of France?
Aggregate: 0.667
Per-doc: [1, 1, 0]
✓ [1] Paris is the capital and most populous city of France.
✓ [1] France is a country in Western Europe.
✗ [0] Nineteen Eighty-Four is a dystopian novel by George Orwell.
Q: Who wrote the novel 1984?
Aggregate: 1.000
Per-doc: [1, 1]
✓ [1] Nineteen Eighty-Four is a dystopian novel by George Orwell.
✓ [1] George Orwell was an English novelist and essayist.
Q: What is the boiling point of water at sea level?
Aggregate: 1.000
Per-doc: [1, 1]
✓ [1] At sea level, water boils at 100 degrees Celsius.
✓ [1] The boiling point of water varies with atmospheric pressure.
Q: Which planet is known as the Red Planet?
Aggregate: 1.000
Per-doc: [1, 1]
✓ [1] Mars is often called the Red Planet due to iron oxide on its surface.
✓ [1] Mars is the fourth planet from the Sun.
Q: What is the largest mammal?
Aggregate: 0.500
Per-doc: [1, 0]
✓ [1] The blue whale is the largest animal known to have ever existed.
✗ [0] Whales are marine mammals.The mean context relevance of 0.733 tells you the retriever is surfacing irrelevant documents ~27% of the time. The per-question breakdown shows exactly which queries suffer.
Add answer correctness to the pipeline
Context relevance alone doesn’t tell you if the final answer is right. Haystack’s FaithfulnessEvaluator and SASEvaluator (semantic answer similarity) cover that. Let’s add SASEvaluator to the same run.
Create evaluators/answer_correctness.py:
# evaluators/answer_correctness.py
from haystack.components.evaluators import SASEvaluator
from haystack.components.generators import OpenAIGenerator
from haystack import EvaluationResult
from typing import List
generator = OpenAIGenerator(model="gpt-4o-mini")
sas_evaluator = SASEvaluator(generator=generator)
def evaluate_answer_correctness(eval_data: List[EvaluationResult], predictions: List[EvaluationResult]) -> List[dict]:
"""
Compare predicted answers to ground truth using semantic answer similarity.
"""
questions = [ex.inputs["question"] for ex in eval_data]
ground_truths = [ex.responses["answer"] for ex in eval_data]
predictions_answers = [p.responses["answer"] for p in predictions]
result = sas_evaluator.run(
questions=questions,
ground_truth_answers=ground_truths,
predicted_answers=predictions_answers
)
return [
{
"question": q,
"ground_truth": gt,
"predicted": pred,
"sas_score": score
}
for q, gt, pred, score in zip(questions, ground_truths, predictions_answers, result["score"])
]Update run_evaluation.py to include it:
# run_evaluation.py (updated)
import json
from load_dataset import eval_data
from pipelines.rag_pipeline import run_rag
from evaluators.context_relevance import evaluate_context_relevance
from evaluators.answer_correctness import evaluate_answer_correctness
from haystack import EvaluationResult
def generate_rag_predictions(eval_data: list[EvaluationResult]) -> list[EvaluationResult]:
predictions = []
for ex in eval_data:
question = ex.inputs["question"]
rag_output = run_rag(question)
pred = EvaluationResult(
inputs={"question": question},
responses={"answer": rag_output["answer"]},
retrieved_documents=rag_output["documents"]
)
predictions.append(pred)
return predictions
def main():
print("Running RAG pipeline on evaluation questions...")
predictions = generate_rag_predictions(eval_data)
print("Evaluating context relevance...")
context_results = evaluate_context_relevance(predictions)
print("Evaluating answer correctness (SAS)...")
sas_results = evaluate_answer_correctness(eval_data, predictions)
# Aggregate metrics
mean_relevance = sum(r["aggregate_score"] for r in context_results) / len(context_results)
mean_sas = sum(r["sas_score"] for r in sas_results) / len(sas_results)
print(f"\n=== Combined Evaluation Results ===")
print(f"Mean context relevance: {mean_relevance:.3f}")
print(f"Mean semantic answer similarity: {mean_sas:.3f}")
print(f"\nPer-question breakdown:")
for ctx, sas in zip(context_results, sas_results):
print(f" Q: {ctx['question']}")
print(f" Context relevance: {ctx['aggregate_score']:.3f}")
print(f" SAS: {sas['sas_score']:.3f}")
print(f" Predicted: {sas['predicted'][:80]}...")
print(f" Ground truth: {sas['ground_truth'][:80]}...")
output = {
"mean_context_relevance": mean_relevance,
"mean_sas": mean_sas,
"context_relevance": context_results,
"answer_correctness": sas_results
}
with open("eval_results.json", "w") as f:
json.dump(output, f, indent=2)
print(f"\nSaved detailed results to eval_results.json")
if __name__ == "__main__":
main()Run again:
python run_evaluation.pyExpected output:
Running RAG pipeline on evaluation questions...
Evaluating context relevance...
Evaluating answer correctness (SAS)...
=== Combined Evaluation Results ===
Mean context relevance: 0.733
Mean semantic answer similarity: 0.912
Per-question breakdown:
Q: What is the capital of France?
Context relevance: 0.667
SAS: 0.950
Predicted: Paris is the capital of France.
Ground truth: Paris is the capital of France.
Q: Who wrote the novel 1984?
Context relevance: 1.000
SAS: 0.920
Predicted: George Orwell wrote the novel 1984.
Ground truth: George Orwell wrote 1984.
Q: What is the boiling point of water at sea level?
Context relevance: 1.000
SAS: 0.930
Predicted: Water boils at 100°C (212°F) at standard atmospheric pressure.
Ground truth: Water boils at 100°C (212°F) at standard atmospheric pressure.
Q: Which planet is known as the Red Planet?
Context relevance: 1.000
SAS: 0.880
Predicted: Mars is known as the Red Planet.
Ground truth: Mars is known as the Red Planet.
Q: What is the largest mammal?
Context relevance: 0.500
SAS: 0.880
Predicted: The blue whale is the largest mammal.
Ground truth: The blue whale is the largest mammal.Even with imperfect context relevance (0.733), the generator produces correct answers (SAS 0.912) because the relevant documents are ranked high enough. But this won’t hold as the corpus grows — irrelevant context increases hallucination risk and token costs.
Interpreting results and next steps
The evaluation pipeline gives you two actionable signals:
-
Context relevance < 1.0 → Your retriever returns noise. Fixes: increase
top_kand add a reranker, improve embedding model, filter by metadata, or tune chunking strategy. -
SAS < 1.0 with high context relevance → The generator fails to use good context. Fixes: improve prompt, try a stronger model, add few-shot examples, or enforce citation format.
For production, you’d want to:
- Run this in CI on every retriever/prompt change
- Track trends over time, not just point-in-time scores
- Segment by query type (factoid, multi-hop, ambiguous)
- Add
FaithfulnessEvaluatorto catch hallucinations where the answer contradicts retrieved context - Log the full
eval_results.jsonto your observability backend
Extending the pipeline
Haystack’s evaluators are components, so you can compose them into a single Pipeline object and run with pipeline.run() — useful for parallelization and custom routing. Here’s the pattern:
# evaluators/combined_pipeline.py
from haystack import Pipeline
from haystack.components.evaluators import ContextRelevanceEvaluator, SASEvaluator, FaithfulnessEvaluator
from haystack.components.generators import OpenAIGenerator
generator = OpenAIGenerator(model="gpt-4o-mini")
eval_pipeline = Pipeline()
eval_pipeline.add_component("context_relevance", ContextRelevanceEvaluator(generator=generator))
eval_pipeline.add_component("sas", SASEvaluator(generator=generator))
eval_pipeline.add_component("faithfulness", FaithfulnessEvaluator(generator=generator))
# Inputs: questions, contexts, ground_truth_answers, predicted_answers
# Run with eval_pipeline.run({...})This lets you evaluate hundreds of examples in one call with proper batching.
Summary
You now have a working haystack evaluation pipeline context relevance setup that:
- Loads labeled evaluation data
- Runs your RAG pipeline to generate predictions + retrieved documents
- Scores context relevance per document and in aggregate
- Scores answer correctness via semantic similarity
- Outputs structured results for tracking and alerting
The code is minimal, dependency-light, and runs in ~30 seconds on 5 examples. Scale the dataset, add a reranker, and wire it into your CI — that’s how you ship RAG systems that don’t regress.