Continuous evaluation haystack pipelines ci is the only way to catch RAG regressions before they hit production. Treating eval as a first-class CI job rather than a notebook afterthought turns fuzzy LLM quality into a merge gate. This guide lays out an ordered path to wire Haystack evaluation into your build, with concrete code and the tradeoffs you’ll hit.
1. Pick metrics that match your failure modes
Don’t measure everything; measure what breaks. For RAG pipelines, the three that catch most regressions are context relevance, faithfulness, and answer relevance. Haystack ships these as metric components that wrap an LLM judge.
from haystack.evaluation.metrics import (
ContextRelevanceMetric,
FaithfulnessMetric,
AnswerRelevanceMetric,
)
metrics = [
ContextRelevanceMetric(),
FaithfulnessMetric(),
AnswerRelevanceMetric(),
]Context relevance tells you if retrieval pulled junk. Faithfulness checks the generator didn’t hallucinate beyond the context. Answer relevance confirms the output actually addresses the query. If you’re doing summarization instead of QA, swap answer relevance for a coherence or coverage metric.
Avoid exact match or BLEU-style string overlap. They correlate poorly with human judgment for generative answers and will give you false confidence. Semantic and LLM-judge metrics are noisier but far more aligned with what users experience.
Pitfall: LLM-as-judge metrics are non-deterministic. Pin the judge model version and set temperature to 0. Expect ±2% score variance run-to-run; design thresholds around that noise band.
2. Build a golden eval set with structure
A 50–200 row dataset of (query, gold_answer, gold_context) triples is enough for a CI signal. Store it as JSON lines, not a pickle, so you can diff it in code review and compute a content hash for versioning.
{"query": "What is the refund policy?", "gold_answer": "Refunds within 30 days.", "gold_context": ["Policy: full refund within 30 days of purchase."]}
{"query": "How do I reset my password?", "gold_answer": "Use the forgot password link.", "gold_context": ["Password resets are handled via /forgot."]}Keep the set representative of real traffic, not edge cases only. If your product spans ten domains, stratify the set across them. Treat the golden set as code: changes require a PR and a rationale. Tag each row with a domain field so you can compute per-domain scores later.
Tradeoff: a small set is fast but blind to rare failures. A large set is thorough but slows CI and costs more in judge tokens. Start with 75 rows; grow only when a regression escapes the gate.
3. Wrap your pipeline in an EvaluationPipeline
Haystack’s EvaluationPipeline takes your existing Pipeline and runs it against the golden set, computing metrics per example. You don’t need to modify the pipeline under test.
from haystack import Pipeline
from haystack.evaluation import EvaluationPipeline
# Assume `rag_pipeline` is your built Haystack Pipeline
eval_pipeline = EvaluationPipeline(
pipeline=rag_pipeline,
metrics=metrics,
)
# inputs mirror your pipeline's input spec; gold_labels carry expected outputs
results = eval_pipeline.run(
inputs=[{"query": ex["query"]} for ex in eval_set],
gold_labels=[{
"answers": ex["gold_answer"],
"contexts": ex["gold_context"],
} for ex in eval_set],
)The results object exposes metrics as a dict of aggregated scores and inputs/outputs for per-example debugging. If your pipeline has multiple entry points, prefix the input keys with the component name as Haystack expects. Run this locally first to baseline your numbers before committing the gate.
If your pipeline calls multiple model providers, routing through a single OpenAI-compatible endpoint like n4n.ai (which fronts 240+ models with automatic fallback) keeps CI eval runs from flaking on provider rate limits.
4. Execute evaluation in CI without blowing the budget
A full eval run can cost dollars and minutes. Gate it: run unit tests always, run eval on a schedule or when RAG components change. Use a GitHub Actions job that triggers on changes to pipelines/, prompts/, or the eval set.
# .github/scripts/run_eval.sh
set -e
python -m pytest tests/eval/test_rag_eval.py --junitxml=eval-results.xmlIn the workflow, cache the Haystack document store and any local model weights. For cloud models, use a cheap judge model in CI and the production judge in nightly runs. The score delta between a small judge and a large one is usually correlated; you care about relative drops, not absolute parity.
Parallelize with pytest-xdist if your eval harness supports it. Haystack’s evaluation run is largely IO-bound on LLM calls, so -n 4 often cuts wall time by 3x. Watch concurrency limits on your inference gateway—bursting 200 judge calls at once will trip rate limits faster than a slow serial loop.
Tradeoff: cheap judges add noise. Mitigate by widening thresholds in CI (e.g., 5% below baseline) and tightening them in nightly (2% below baseline).
5. Set thresholds and fail the build
Evaluation that never fails is theater. Pick a baseline from your first green run, then set minimums below it per metric. Assert in the test and store the baseline as a committed JSON file.
def test_faithfulness_threshold(results):
score = results.metrics["faithfulness"]
assert score >= 0.85, f"Faithfulness {score} below gate"{"faithfulness": 0.90, "answer_relevance": 0.87, "context_relevance": 0.94}On intentional prompt changes, update the baseline via a separate pytest --update-baseline flag, not by silently editing the assert. This keeps history auditable.
Pitfall: a single bad golden example can drag a metric below threshold and fail CI for unrelated reasons. Add a per-example override list for known-flaky rows, but review it monthly. Never delete a failing row just to go green.
6. Track scores over time
CI gives a boolean; you need a trend. Upload the metrics dict as a CI artifact and plot it in your dashboard. A simple approach: write a JSON file per run with git SHA and scores.
{"sha": "a1b2c3", "faithfulness": 0.92, "answer_relevance": 0.88, "context_relevance": 0.95}Compare against main on every PR. If a PR drops any metric by more than 3%, block merge until the author explains it. This turns continuous evaluation haystack pipelines ci from a gate into a feedback loop. Post the scores as a PR comment using gh pr comment so reviewers see the delta without digging into logs.
Common pitfalls and tradeoffs
Judge model drift. The LLM you use for faithfulness scoring changes silently. Pin the model ID and version. If you must upgrade, re-baseline all metrics in one commit and note it in the changelog.
Context leakage. If your golden contexts are copied from the same source as the generator’s knowledge, metrics look artificially high. Use real retrieved contexts from a frozen index built at eval time.
Latency vs coverage. Running 200 examples with three judge calls each is 600 LLM requests. At 500 ms each, that’s five minutes serial. Parallelize, but cap concurrency to respect gateway limits.
Overfitting to the golden set. Engineers will tweak prompts to ace the 50 rows. Combat this by hiding 20% of the eval set as a held-out set that runs only nightly.
Metric saturation. Once faithfulness hits 0.98, it can’t show improvement and small drops look like noise. Track the absolute delta and the variance, not just the point value.
Cost control. Continuous evaluation haystack pipelines ci costs real money. Set a monthly token budget in your inference gateway, and fail the job with a clear “budget exceeded” message rather than letting it hang for 40 minutes.
The path is straightforward: define metrics, freeze a dataset, wrap the pipeline, run in CI with thresholds, and watch the trend. Do that and your RAG system stops regressing silently.