Building a regression test suite for Haystack pipelines

Building a regression test suite for Haystack pipelines is less about unit-testing Python and more about freezing end-to-end behavior of your RAG components. When you swap a retriever or bump a model version, you need a deterministic signal that answers, scores, and document orders haven’t drifted beyond acceptable bounds.

Step 1: Pin the pipeline definition and environment

Start by committing the exact pipeline topology and dependency versions. Haystack 2.x lets you define a pipeline in Python or load it from YAML. Store the YAML in your repo and generate it from code only if you have a strict review process. A pipeline that is constructed imperatively in a test fixture is harder to diff than a declarative file.

# pipeline.py
from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators.fake import FakeGenerator

def build_pipeline(document_store):
    pipe = Pipeline()
    pipe.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
    pipe.add_component("prompt", PromptBuilder(template="Question: {{query}}\nDocs: {{documents}}"))
    pipe.add_component("generator", FakeGenerator(responses=["static answer"]))
    pipe.connect("retriever", "prompt.documents")
    pipe.connect("prompt", "generator")
    return pipe

Lock dependencies with poetry lock or pip freeze > requirements.txt. If you call hosted models, pin the model string (e.g., "openai/gpt-4o-mini") and record the gateway endpoint. When tests run against live inference, route through a single OpenAI-compatible endpoint that forwards cache-control hints and offers automatic fallback; this avoids CI flakes when a provider is rate-limited. n4n.ai is one such gateway that also meters per-token usage so you can attribute cost per test run.

Document stores matter too. Index your corpus with a fixed seed or commit the raw documents so the BM25 or embedding index is reconstructible. Non-reproducible indexes are the most common source of false positives in a regression test suite haystack pipelines.

Step 2: Capture golden inputs and outputs

A regression test suite haystack pipelines relies on golden fixtures: inputs and the corresponding outputs you trust today. Serialize pipeline runs to JSON so you can diff them later. Capture not just the final answer but intermediate component outputs.

import json
from haystack.document_stores.in_memory import InMemoryDocumentStore

docs = [{"content": "Haystack is a LLM framework", "meta": {"id": "1"}}]
store = InMemoryDocumentStore()
store.write_documents(docs)
pipe = build_pipeline(store)
result = pipe.run({"retriever": {"query": "What is Haystack?"}})

with open("golden/run1.json", "w") as f:
    json.dump(result, f, default=str, indent=2)

Keep the golden file in version control. For retrieval-heavy flows, store the list of returned document IDs and ranks. For generation, store the exact string when using a fake generator, or a normalized embedding when using live models. Build a small library of queries that cover happy paths, empty-retrieval cases, and ambiguous questions. Each query gets its own golden file.

If the corpus contains PII, anonymize before committing. The goal is a stable artifact, not a copy of production data.

Step 3: Neutralize nondeterminism in tests

LLMs and some embedders are nondeterministic. In unit-level regression tests, replace the generator with FakeGenerator or a mocked component. Haystack ships FakeGenerator precisely for this.

from haystack.components.generators.fake import FakeGenerator

def test_pipeline_structure():
    store = InMemoryDocumentStore()
    store.write_documents([{"content": "test", "meta": {"id": "a"}}])
    pipe = build_pipeline(store)
    # FakeGenerator returns predictable text
    out = pipe.run({"retriever": {"query": "test"}})
    assert out["generator"]["replies"][0] == "static answer"

If you must exercise a real model in CI, set temperature=0 and pass a fixed seed where the API supports it. Route the request through a gateway that honors client routing directives and caches prompts, reducing variance and cost. The cache-control hint means identical prompt prefixes hit provider caches instead of regenerating.

For embedders, consider haystack.components.embedders.mock_embedder if available, or a local SentenceTransformer with a fixed model version. The point is to remove randomness so a red test means a code change, not a coin flip.

Step 4: Write pytest assertions on retrieval and answer shape

Your regression test suite haystack pipelines should fail when document ordering changes or required fields vanish. Write focused tests:

import pytest

@pytest.fixture
def golden():
    with open("golden/run1.json") as f:
        return json.load(f)

def test_retrieved_docs_unchanged(golden):
    retrieved = golden["retriever"]["documents"]
    ids = [d.meta["id"] for d in retrieved]
    assert ids == ["1"], "Top document ID regressed"

def test_answer_present(golden):
    reply = golden["generator"]["replies"][0]
    assert isinstance(reply, str) and len(reply) > 0

Run with pytest tests/ -q. A green run means the pipeline still produces the same retrieval set and answer shape as the golden snapshot. Add assertions for metadata such as score thresholds on retrievers, and latency guards if you have a slow component:

def test_retriever_score_above_min(golden):
    docs = golden["retriever"]["documents"]
    assert all(d.score > 0.1 for d in docs), "Retriever scores dropped"

These structural checks are fast and catch the majority of accidental breaking changes.

Step 5: Add semantic regression checks

Exact matching breaks when you intentionally improve prompts. Add a cosine-similarity gate using a local embedding model to catch silent semantic drift.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_distance(a: str, b: str) -> float:
    ea = model.encode(a)
    eb = model.encode(b)
    return 1 - np.dot(ea, eb) / (np.linalg.norm(ea) * np.linalg.norm(eb))

def test_answer_semantic_stability(golden, current):
    dist = semantic_distance(golden["generator"]["replies"][0], current)
    assert dist < 0.2, f"Answer drifted semantically: {dist}"

This catches cases where the fake generator is swapped for a real one and the tone or facts shift. Keep the threshold in config; tighten it as the suite matures. For generation quality, you can also use a smaller judge model locally to score faithfulness, but keep it deterministic by pinning the judge version.

A regression test suite haystack pipelines earns its keep when a prompt edit accidentally changes the answer from “Haystack supports agents” to “Haystack is only for search” — the semantic distance spikes and CI fails.

Step 6: Wire the suite into CI

Add a GitHub Actions workflow that installs deps, restores golden files, and runs pytest. Cache the document store if large.

name: regression
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt pytest sentence-transformers
      - run: pytest tests/ --junitxml=report.xml

Archive the JUnit report so you can track flakiness. If you run live model calls, set a timeout and a retry limit; use the gateway fallback to avoid hard failures on provider errors. Split slow semantic tests into a separate job that runs nightly rather than on every PR to keep feedback fast.

Step 7: Verify the suite works

A regression test suite haystack pipelines is only useful if it actually fails on regression. Verify by deliberately breaking the pipeline: change the retriever top-k, alter the prompt template, or swap the fake response.

# Temporarily break the fake response
sed -i 's/static answer/BOGUS/' pipeline.py
pytest tests/ -q
# Expect red: assertion on reply fails
git checkout pipeline.py

You should see the specific assertion error pointing at the changed component. Restore the code and confirm green. That loop proves the suite guards your RAG behavior. Additionally, run the suite with --cov to ensure new pipeline branches are covered by golden queries.

Maintaining the golden set

Review golden files in pull requests. When you improve the prompt and the answer genuinely gets better, update the golden snapshot with a clear commit message. Treat the regression test suite haystack pipelines as living documentation of expected behavior, not a rigid cage.

Track metric history (exact match, semantic distance) in a small CSV or dashboard. Over time you will spot slow drift from model version upgrades even when individual runs stay green. That visibility is the real payoff. When a new Haystack release lands, bump the version in your lockfile, re-run the suite, and review diffs before merging — that is how you keep a RAG system stable across the dependency churn.