Evaluating RAG pipelines in Haystack with n4n.ai models

To evaluate RAG pipeline Haystack n4n.ai models, you need a reproducible harness that separates retrieval quality from generation faithfulness. This guide walks through a concrete Haystack 2.x setup that uses an OpenAI-compatible gateway to test multiple LLMs without rewriting your pipeline.

1. Pin your stack and install

Haystack 2.x changed the evaluation API substantially from 1.x. Pin versions so your harness does not silently break.

pip install haystack-ai==2.2.0 openai==1.30.0

Use a virtualenv. The RagEvaluator component lives in haystack.evaluation and expects an LLM judge; we will point it at the same gateway as the generator to keep credentials in one place.

2. Build the baseline RAG pipeline

Start with an in-memory store. For evaluation you rarely need a vector DB until you are testing embedding models—BM25 isolates the generator from retriever noise.

from haystack import Document, Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIChatGenerator

store = InMemoryDocumentStore()
store.write_documents([
    Document(content="n4n.ai routes to 240+ models via one OpenAI-compatible endpoint.", meta={"src": "docs"}),
    Document(content="Haystack 2.x pipelines compose components via explicit connects.", meta={"src": "docs"}),
])

retriever = InMemoryBM25Retriever(store, top_k=2)
prompt = PromptBuilder(template="""
Answer strictly from context.
Context:
{% for d in documents %}{{ d.content }}
{% endfor %}
Question: {{ question }}
""")

generator = OpenAIChatGenerator(
    api_key="sk-your-key",
    api_base_url="https://api.n4n.ai/v1",
    model="gpt-4o-mini",
    generation_kwargs={"temperature": 0}
)

rag = Pipeline()
rag.add_component("retriever", retriever)
rag.add_component("prompt", prompt)
rag.add_component("generator", generator)
rag.connect("retriever", "prompt")
rag.connect("prompt", "generator")

The api_base_url is the only n4n.ai-specific line. Everything else is stock Haystack.

3. Configure the evaluator

RagEvaluator computes context relevance, faithfulness, and answer relevance using an LLM judge. Use a stronger model for judging than the one generating answers, or you will get inflated faithfulness scores.

import os
os.environ["OPENAI_API_KEY"] = "sk-your-key"
os.environ["OPENAI_API_BASE"] = "https://api.n4n.ai/v1"

from haystack.evaluation import RagEvaluator

evaluator = RagEvaluator(
    metrics=["context_relevance", "faithfulness", "answer_relevance"],
    judge_model="gpt-4o"
)

If you need to override the judge generator directly, pass an OpenAIChatGenerator instance instead of relying on env vars. The evaluator runs one prompt per metric per sample, so a 100-question set with three metrics is 300 judge calls—budget accordingly.

4. Run a single evaluation pass

Execute the pipeline, capture both the retrieved contexts and the generated answer, then hand them to the evaluator with a reference answer.

q = "What does n4n.ai route?"
out = rag.run({"retriever": {"query": q}, "prompt": {"question": q}})
answer = out["generator"]["replies"][0]
contexts = [d.content for d in out["retriever"]["documents"]]

result = evaluator.evaluate(
    questions=[q],
    contexts=[contexts],
    answers=[answer],
    reference_answers=["n4n.ai routes to 240+ models via one OpenAI-compatible endpoint."]
)
print(result.score)

Run this on a held-out set of 20–50 questions before touching model swaps. A baseline tells you whether poor faithfulness comes from the prompt or the model.

5. Scale across models without code changes

The most efficient way to evaluate RAG pipeline Haystack n4n.ai models across providers is to vary only the model string in the generator and judge. Because the gateway honors client routing directives and forwards provider cache-control hints, you can A/B test a Claude variant against a Mistral variant by changing one parameter. Its automatic fallback also keeps the harness running when a provider is rate-limited or degraded—no try/except sprawl in your eval script.

for model in ["gpt-4o-mini", "mistral-large", "claude-3-sonnet"]:
    generator.model = model
    # re-run section 4, append scores to a table

Keep the judge fixed. If you rotate the judge too, you introduce a confounder that makes cross-model comparisons meaningless.

Common pitfalls and tradeoffs

Judge bias and self-preference

LLMs favor text that looks like their own output. If the judge and generator are the same family, faithfulness scores drift upward by 5–15% in practice. Always use a separate judge model, ideally from a different provider.

Retrieval leakage

Context relevance metrics punish the retriever, not the generator. If your top_k is too high, irrelevant documents dilute the score even when the generator ignores them. Start with top_k=2 and increase only if answer coverage suffers.

Token cost and metering

Evaluation calls the judge once per metric per question. With three metrics and 200 questions, that is 600 judge completions plus generation. The gateway’s per-token usage metering lets you attribute cost per model run; export the metering JSON and sum usage.completion_tokens to find which model quietly burned your budget on long reasoning traces.

Metric false negatives

Answer relevance uses semantic similarity to the reference. If your reference answer is terse and the model adds a correct caveat, the metric may flag it as irrelevant. Treat scores below 0.8 as “investigate,” not “fail.” Pair automated eval with a weekly manual spot-check of 10 random traces.

Prompt template drift

Hard-coding the prompt in PromptBuilder is fine for a single run, but when you swap models, some expect system/user separation. Extract the template to a versioned file and load it, so a prompt change does not get mistaken for a model regression.

from haystack.components.builders import PromptBuilder
import yaml

with open("eval_prompt.yaml") as f:
    tmpl = yaml.safe_load(f)["template"]
prompt = PromptBuilder(template=tmpl)

Caching and reproducibility

Set temperature=0 for the generator during eval. Sampling variance will otherwise make two runs of the same model non-comparable. If you rely on provider caching, the gateway forwards cache-control hints, so identical prompt prefixes hit cache and reduce cost—but verify the eval set is static across runs, or cache hits will mask real differences.

Stick to this ordered path: baseline with BM25, fixed judge, single-model scores, then controlled model rotation. That gives you defensible numbers instead of a spreadsheet of vibes.