Comparing models with Haystack evaluation pipelines

When you need to compare models haystack evaluation pipeline offers a structured alternative to ad-hoc prompting. Instead of eyeballing outputs, you define a dataset, attach evaluators, and let the framework run the same prompts against each candidate. This post walks through a head-to-head of three representative LLMs using Haystack 2.x evaluation primitives, and shows exactly where the variables live in your code.

Why build an evaluation pipeline

Throwing a few prompts at a model and reading the answers does not scale past two candidates. You forget what you asked, the temperature drifts, and the second model gets a slightly different system prompt. Haystack’s evaluation pipeline fixes the inputs, wraps the model in a component, and emits scored results you can diff in a dataframe.

The core primitives are Pipeline, a generator component, and an evaluator component (LLMEvaluator, FaithfulnessEvaluator, AnswerExactMatch). You run the pipeline over a list of EvaluationExample objects and collect EvaluationRunResult. The compare models haystack evaluation pipeline approach removes every variable except the model name, which is the only thing you should be testing.

The contenders

We compare three models that cover distinct deployment styles:

• GPT-4o (OpenAI): proprietary flagship, multimodal, strong general reasoning, 128K context.
• Claude 3.5 Sonnet (Anthropic): proprietary, 200K context, strong coding and long-doc reasoning.
• Llama 3 70B (open weights, served via an OpenAI-compatible gateway): self-hostable or cheaply hosted, good enough for many RAG tasks.

If you want a single endpoint to address all three without juggling separate SDKs, an OpenAI-compatible gateway like n4n.ai lets you keep the same OpenAIChatGenerator code and only swap the model string and api_base_url. That keeps the setup honest: the only variable is the model behind the endpoint.

Head-to-head dimensions

Before writing code, lock the comparison axes. The ones that matter to engineers shipping production systems:

• Capabilities: what tasks the model handles reliably (multimodal, long context, function calling).
• Price/cost model: per-token cost, not just sticker price—include cached input discounts if the provider honors them.
• Latency/throughput: time to first token and tokens/sec under realistic concurrency.
• Ergonomics: how painful it is to call from your stack (SDKs, OpenAI compatibility, streaming).
• Ecosystem: first-class Haystack components, observability hooks, community eval recipes.
• Limits: rate limits, max context, regional availability, content filters.

Here is the summary table:

The table is a snapshot; verify current prices on provider sites. The point is to make those columns explicit in your own eval config so you are not surprised in production.

Building the Haystack evaluation pipeline

Install the framework and the eval extras:

pip install haystack-ai[eval]

Define a small QA dataset with known contexts so faithfulness can be measured. Keep it domain-specific; generic trivia hides real gaps.

[
  {
    "question": "What is the max context of GPT-4o?",
    "context": "GPT-4o supports a 128K token context window.",
    "reference": "128K tokens"
  },
  {
    "question": "Which model has 200K context?",
    "context": "Claude 3.5 Sonnet provides a 200K token context window.",
    "reference": "Claude 3.5 Sonnet"
  },
  {
    "question": "What is Llama 3 70B missing natively?",
    "context": "Llama 3 70B is text-only and does not support vision inputs.",
    "reference": "Vision support"
  }
]

Now build a pipeline that generates an answer and scores it with an LLM-based faithfulness evaluator. We loop over model configs to reuse the same harness.

from haystack import Pipeline
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.evaluators import FaithfulnessEvaluator
from haystack.evaluation import EvaluationHarness
import json

configs = {
    "gpt-4o": {"base": "https://api.openai.com/v1", "key": "sk-..."},
    "claude-3-5-sonnet": {"base": "https://api.openai.com/v1", "key": "sk-..."},
    "llama-3-70b": {"base": "https://gateway.n4n.ai/v1", "key": "n4n-..."},
}

with open("qa.json") as f:
    data = json.load(f)

for name, cfg in configs.items():
    gen = OpenAIChatGenerator(
        model=name,
        api_base_url=cfg["base"],
        api_key=cfg["key"],
        generation_kwargs={"temperature": 0}
    )
    eval_pipe = Pipeline()
    eval_pipe.add_component("gen", gen)
    eval_pipe.add_component("faith", FaithfulnessEvaluator(
        api_base_url=cfg["base"], api_key=cfg["key"], model=name
    ))
    eval_pipe.connect("gen.replies", "faith.responses")
    eval_pipe.connect("gen.meta", "faith.meta")
    harness = EvaluationHarness(pipeline=eval_pipe, data=data)
    result = harness.run()
    print(name, result.score)

The wiring step is standard Haystack; the snippet focuses on the model swap. Because temperature is pinned to 0, the compare models haystack evaluation pipeline is deterministic enough to trust deltas. If you use a gateway that honors provider cache-control hints, prefix caching kicks in automatically when your contexts repeat across examples.

Reading the results

EvaluationRunResult gives per-example scores and an aggregate. Export to CSV for offline analysis:

result.to_csv(f"{name}_scores.csv")

Look at faithfulness mean, but also variance. A model that scores 0.95 average but 0.6 on one question is riskier than one at 0.90 flat. Add a second evaluator (LLMEvaluator with a custom rubric) to capture style or correctness beyond faithfulness. For example, a rubric that penalizes verbose answers changes the ranking when throughput matters.

from haystack.components.evaluators import LLMEvaluator

rubric = {
    "score_points": [1, 2, 3, 4, 5],
    "template": "Rate conciseness from 1-5 for: {{ answer }}"
}
concise_eval = LLMEvaluator(model=name, api_base_url=cfg["base"], api_key=cfg["key"], rubric=rubric)

Run both evaluators in the same pipeline and merge the scores. The model that wins on faithfulness but loses on conciseness may cost more in token billing than you expect.

Gotchas when comparing models

First, never compare a model behind a gateway with one called directly unless the gateway adds zero transformation. A proxy that rewrites system prompts invalidates the test. Second, watch context window differences: if your dataset exceeds 8K tokens and you test Llama on a host capped at 8K, you are measuring truncation, not capability. Third, rate limits will skew latency numbers; run evaluations serially with a sleep, or parallelize within the documented tier.

Finally, evaluators themselves are LLMs. If you use GPT-4o to judge Llama, you import its biases. Rotate the judge or use a deterministic metric (AnswerExactMatch) on a subset to sanity-check.

Which to choose

Verdict by use case:

High-stakes document analysis with long inputs — Claude 3.5 Sonnet. The 200K context and strong reasoning on dense text win when you cannot chunk aggressively.

Multimodal product features (image + text) — GPT-4o. It is the only one in the trio with native vision, and the Haystack generator is battle-tested.

Cost-sensitive RAG at scale — Llama 3 70B served via a gateway. If your eval shows faithfulness within 2% of the proprietary models on your domain data, the per-token savings compound fast. Self-hosting removes per-call API cost entirely if you already have GPUs.

Fast prototyping with minimal code changes — Any of them behind one OpenAI-compatible base URL. The compare models haystack evaluation pipeline pattern means you change one string, not your integration.

Pick the model that wins on the dimensions you weighted, not the one with the best marketing. Run the pipeline on your own data before committing.