Haystack evaluation pipeline tutorial: faithfulness metrics

Haystack’s evaluation framework lets you measure whether your RAG system actually grounds answers in retrieved context — or hallucinates. This tutorial walks through building a faithfulness evaluation pipeline from scratch: preparing a labeled dataset, wiring the FaithfulnessEvaluator, running batch evaluation, and interpreting the results. You’ll end up with a reproducible script you can drop into CI.

Prerequisites

• Python 3.10+
• Haystack 2.0+ (pip install haystack-ai)
• An OpenAI-compatible endpoint for the judge LLM (GPT-4o-mini works well)
• A small labeled dataset: questions, retrieved contexts, generated answers, and ground-truth labels (faithful / unfaithful)

If you don’t have labeled data yet, the synthetic generation section below shows a quick way to bootstrap one.

Install dependencies

pip install haystack-ai datasets pandas tqdm

Prepare the evaluation dataset

Faithfulness evaluation needs four fields per sample: question, contexts (list of strings), predicted_answer, and ground_truth_answer. The ground truth here is a binary label — faithful or unfaithful — not a reference answer.

Create eval_data.jsonl:

{"question": "What is the capital of France?", "contexts": ["Paris is the capital city of France."], "predicted_answer": "Paris is the capital of France.", "ground_truth_answer": "faithful"}
{"question": "What is the capital of France?", "contexts": ["Paris is the capital city of France."], "predicted_answer": "Lyon is the capital of France.", "ground_truth_answer": "unfaithful"}
{"question": "When was the Eiffel Tower built?", "contexts": ["The Eiffel Tower was constructed from 1887 to 1889."], "predicted_answer": "It was built between 1887 and 1889.", "ground_truth_answer": "faithful"}
{"question": "When was the Eiffel Tower built?", "contexts": ["The Eiffel Tower was constructed from 1887 to 1889."], "predicted_answer": "Construction started in 1890.", "ground_truth_answer": "unfaithful"}

Load it into a Haystack EvaluationDataset:

from haystack import EvaluationDataset

dataset = EvaluationDataset.from_jsonl("eval_data.jsonl")
print(f"Loaded {len(dataset)} samples")

Expected output:

Loaded 4 samples

Configure the faithfulness evaluator

Haystack’s FaithfulnessEvaluator uses an LLM judge to score whether the predicted answer is supported by the provided contexts. It returns a score between 0 and 1 plus a reasoning string.

from haystack.components.evaluators import FaithfulnessEvaluator
from haystack.components.generators import OpenAIGenerator
from haystack.utils import Secret

# Judge LLM — use a cheap, fast model for evaluation
judge_llm = OpenAIGenerator(
    model="gpt-4o-mini",
    api_key=Secret.from_env_var("OPENAI_API_KEY"),
    generation_kwargs={"temperature": 0.0, "max_tokens": 512}
)

evaluator = FaithfulnessEvaluator(judge_llm=judge_llm)

The evaluator expects inputs named questions, contexts, and predicted_answers. We’ll wire those in the pipeline next.

Build the evaluation pipeline

Haystack pipelines connect components via typed inputs/outputs. For batch evaluation we need to:

1. Extract the three required fields from each dataset sample
2. Feed them to the evaluator
3. Collect scores and reasoning

from haystack import Pipeline
from haystack.components.others import Multiplexer

pipe = Pipeline()

# Multiplexer splits the dataset into parallel lists for the evaluator
pipe.add_component("mux", Multiplexer())
pipe.add_component("evaluator", evaluator)

# Connect dataset fields to multiplexer inputs
pipe.connect("mux.questions", "evaluator.questions")
pipe.connect("mux.contexts", "evaluator.contexts")
pipe.connect("mux.predicted_answers", "evaluator.predicted_answers")

The Multiplexer takes a list of dictionaries and emits separate lists for each key. We’ll pass the raw dataset samples to it.

Run batch evaluation

from haystack.dataclasses import EvaluationResult

# Convert dataset to list of dicts for the multiplexer
samples = [
    {
        "questions": sample["question"],
        "contexts": sample["contexts"],
        "predicted_answers": sample["predicted_answer"]
    }
    for sample in dataset
]

# Run pipeline
results = pipe.run({"mux": {"values": samples}})

# Extract evaluator outputs
eval_outputs = results["evaluator"]["results"]

eval_outputs is a list of EvaluationResult objects with score (float) and metadata["reasoning"] (str).

Inspect results

for i, (sample, result) in enumerate(zip(dataset, eval_outputs)):
    print(f"\nSample {i+1}")
    print(f"  Question: {sample['question']}")
    print(f"  Predicted: {sample['predicted_answer']}")
    print(f"  Ground truth label: {sample['ground_truth_answer']}")
    print(f"  Faithfulness score: {result.score:.2f}")
    print(f"  Reasoning: {result.metadata.get('reasoning', 'N/A')}")

Expected output (scores will vary slightly by judge model):

Sample 1
  Question: What is the capital of France?
  Predicted: Paris is the capital of France.
  Ground truth label: faithful
  Faithfulness score: 1.00
  Reasoning: The answer is fully supported by the context which states Paris is the capital of France.

Sample 2
  Question: What is the capital of France?
  Predicted: Lyon is the capital of France.
  Ground truth label: unfaithful
  Faithfulness score: 0.00
  Reasoning: The answer claims Lyon is the capital, but the context states Paris is the capital. Direct contradiction.

Sample 3
  Question: When was the Eiffel Tower built?
  Predicted: It was built between 1887 and 1889.
  Ground truth label: faithful
  Faithfulness score: 1.00
  Reasoning: The answer matches the context dates exactly.

Sample 4
  Question: When was the Eiffel Tower built?
  Predicted: Construction started in 1890.
  Ground truth label: unfaithful
  Faithfulness score: 0.00
  Reasoning: The context states construction was 1887-1889; 1890 contradicts this.

Aggregate metrics

Raw scores are useful, but you’ll want summary statistics for dashboards and regression checks.

from statistics import mean

scores = [r.score for r in eval_outputs]
labels = [s["ground_truth_answer"] for s in dataset]

# Binary classification metrics using 0.5 threshold
preds = ["faithful" if s >= 0.5 else "unfaithful" for s in scores]

tp = sum(1 for p, l in zip(preds, labels) if p == "faithful" and l == "faithful")
fp = sum(1 for p, l in zip(preds, labels) if p == "faithful" and l == "unfaithful")
tn = sum(1 for p, l in zip(preds, labels) if p == "unfaithful" and l == "unfaithful")
fn = sum(1 for p, l in zip(preds, labels) if p == "unfaithful" and l == "faithful")

precision = tp / (tp + fp) if (tp + fp) else 0
recall = tp / (tp + fn) if (tp + fn) else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0

print(f"\nAggregate metrics (threshold=0.5):")
print(f"  Mean faithfulness score: {mean(scores):.3f}")
print(f"  Precision: {precision:.3f}")
print(f"  Recall: {recall:.3f}")
print(f"  F1: {f1:.3f}")
print(f"  Confusion matrix: TP={tp} FP={fp} TN={tn} FN={fn}")

Expected output:

Aggregate metrics (threshold=0.5):
  Mean faithfulness score: 0.500
  Precision: 1.000
  Recall: 1.000
  F1: 1.000
  Confusion matrix: TP=2 FP=0 TN=2 FN=0

Perfect separation on this toy dataset. Real data will be noisier — use the confusion matrix to spot systematic failure modes (e.g., the judge consistently missing subtle hallucinations).

Threshold tuning

The 0.5 default threshold isn’t sacred. Plot score distributions by ground-truth label to pick a better operating point:

import matplotlib.pyplot as plt

faithful_scores = [s for s, l in zip(scores, labels) if l == "faithful"]
unfaithful_scores = [s for s, l in zip(scores, labels) if l == "unfaithful"]

plt.hist([faithful_scores, unfaithful_scores], bins=10, label=["faithful", "unfaithful"], alpha=0.7)
plt.xlabel("Faithfulness score")
plt.ylabel("Count")
plt.legend()
plt.title("Score distribution by ground truth")
plt.show()

On larger datasets you’ll see overlap. Choose a threshold that balances precision/recall for your use case — high precision if false alarms are costly, high recall if missing hallucinations is worse.

Synthetic dataset generation (optional)

If you lack labeled data, generate a starter set using a strong LLM. This isn’t a substitute for human review, but it accelerates iteration.

from haystack.components.generators import OpenAIGenerator
from haystack import Document

generator = OpenAIGenerator(
    model="gpt-4o",
    api_key=Secret.from_env_var("OPENAI_API_KEY"),
    generation_kwargs={"temperature": 0.3}
)

# Seed contexts
contexts = [
    ["The n4n.ai gateway routes requests to 240+ models via a single OpenAI-compatible endpoint."],
    ["Automatic fallback triggers when a provider returns 429 or 5xx errors."],
    ["Per-token usage metering is exposed via response headers."]
]

synthetic_samples = []
for ctx in contexts:
    # Generate a faithful answer
    prompt_faithful = f"Context: {ctx[0]}\nWrite a one-sentence answer grounded only in this context."
    faithful_answer = generator.run(prompt_faithful)["replies"][0]
    
    # Generate an unfaithful answer (hallucination)
    prompt_unfaithful = f"Context: {ctx[0]}\nWrite a one-sentence answer that SOUNDS plausible but contradicts or adds info not in the context."
    unfaithful_answer = generator.run(prompt_unfaithful)["replies"][0]
    
    synthetic_samples.append({
        "question": f"Question about: {ctx[0][:50]}...",
        "contexts": ctx,
        "predicted_answer": faithful_answer,
        "ground_truth_answer": "faithful"
    })
    synthetic_samples.append({
        "question": f"Question about: {ctx[0][:50]}...",
        "contexts": ctx,
        "predicted_answer": unfaithful_answer,
        "ground_truth_answer": "unfaithful"
    })

# Save
import json
with open("synthetic_eval.jsonl", "w") as f:
    for s in synthetic_samples:
        f.write(json.dumps(s) + "\n")

print(f"Generated {len(synthetic_samples)} synthetic samples")

Review every synthetic sample before using it — judges can be inconsistent, and you’ll catch obvious mislabels quickly.

Running in CI

Wrap the pipeline in a script that exits non-zero on regression:

# eval_faithfulness.py
import sys
from haystack import EvaluationDataset, Pipeline
from haystack.components.evaluators import FaithfulnessEvaluator
from haystack.components.generators import OpenAIGenerator
from haystack.components.others import Multiplexer
from haystack.utils import Secret
from statistics import mean

DATASET_PATH = "eval_data.jsonl"
THRESHOLD = 0.5
MIN_F1 = 0.85  # adjust per project

def main():
    dataset = EvaluationDataset.from_jsonl(DATASET_PATH)
    
    judge = OpenAIGenerator(
        model="gpt-4o-mini",
        api_key=Secret.from_env_var("OPENAI_API_KEY"),
        generation_kwargs={"temperature": 0.0}
    )
    evaluator = FaithfulnessEvaluator(judge_llm=judge)
    
    pipe = Pipeline()
    pipe.add_component("mux", Multiplexer())
    pipe.add_component("evaluator", evaluator)
    pipe.connect("mux.questions", "evaluator.questions")
    pipe.connect("mux.contexts", "evaluator.contexts")
    pipe.connect("mux.predicted_answers", "evaluator.predicted_answers")
    
    samples = [
        {"questions": s["question"], "contexts": s["contexts"], "predicted_answers": s["predicted_answer"]}
        for s in dataset
    ]
    
    results = pipe.run({"mux": {"values": samples}})
    eval_results = results["evaluator"]["results"]
    
    scores = [r.score for r in eval_results]
    labels = [s["ground_truth_answer"] for s in dataset]
    preds = ["faithful" if sc >= THRESHOLD else "unfaithful" for sc in scores]
    
    tp = sum(1 for p, l in zip(preds, labels) if p == "faithful" and l == "faithful")
    fp = sum(1 for p, l in zip(preds, labels) if p == "faithful" and l == "unfaithful")
    fn = sum(1 for p, l in zip(preds, labels) if p == "unfaithful" and l == "faithful")
    
    precision = tp / (tp + fp) if (tp + fp) else 0
    recall = tp / (tp + fn) if (tp + fn) else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0
    
    print(f"F1: {f1:.3f} (threshold={THRESHOLD})")
    print(f"Mean score: {mean(scores):.3f}")
    
    if f1 < MIN_F1:
        print(f"FAIL: F1 {f1:.3f} below minimum {MIN_F1}")
        sys.exit(1)
    print("PASS")
    sys.exit(0)

if __name__ == "__main__":
    main()

Add to your CI config:

# .github/workflows/eval.yml
name: Faithfulness Evaluation
on: [push, pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install haystack-ai datasets pandas tqdm
      - run: python eval_faithfulness.py
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Common pitfalls

Judge model variance. Different judges (GPT-4o vs. GPT-4o-mini vs. Llama-3-70B) produce different score distributions. Lock the judge model version in your eval config and re-baseline when you upgrade.

Context length. The evaluator sends all contexts to the judge. If your RAG retrieves 20 chunks, truncate or summarize before evaluation — otherwise you’ll hit token limits and the judge’s attention will dilute.

Binary vs. graded labels. FaithfulnessEvaluator returns a continuous score. If your ground truth is binary, you must choose a threshold. If you have graded human labels (1-5), compute Spearman correlation instead of F1.

Leakage. Don’t evaluate on the same contexts used to generate the answer during development — that measures self-consistency, not faithfulness. Use a held-out retrieval set or freeze the retriever.

Next steps

• Add AnswerRelevanceEvaluator and ContextRelevanceEvaluator to the same pipeline for a full RAG triad
• Log per-sample scores to a time-series DB (Prometheus, InfluxDB) for trend detection
• Build a small labeling UI so domain experts can correct judge errors — this improves both your ground truth and the judge via few-shot prompting
• Experiment with LLMEvaluator custom prompts for domain-specific faithfulness criteria (e.g., “does the answer cite the correct section of the regulatory doc?”)

The pipeline above is intentionally minimal. Extend it incrementally as your evaluation maturity grows.