Building an LLM-as-judge evaluator in Haystack 2.0

Evaluating retrieval-augmented generation (RAG) outputs at scale demands automated scoring that correlates with human judgment. An llm-as-judge evaluator haystack 2.0 pipeline treats a capable language model as a critic that grades answers against a rubric, giving you per-sample scores without a labeled test set. Below is a complete, runnable path from environment setup to verified scores on your own data.

Step 1: Install Haystack 2.0 and dependencies

Use a clean virtual environment with Python 3.10+. Haystack 2.0 ships a redesigned component API built around typed run methods and declarative pipelines; pin to a recent 2.x release to avoid breaking changes.

pip install "haystack-ai>=2.0.0" openai

Import the pieces we need for a chat-based judge:

from haystack import Pipeline, component
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
import json, re, statistics

The OpenAIChatGenerator speaks the OpenAI chat protocol, so it works with any compatible endpoint. This is what makes the evaluator portable across model providers.

Step 2: Design the judge prompt and scoring schema

A judge is only as good as its instructions. Define a system prompt that enforces a 1–5 Likert scale and requires strict JSON. Keep the rubric narrow: faithfulness to context, answer helpfulness, or tone. Mixing dimensions in one call produces noisy scores.

SYSTEM_PROMPT = """You are a strict evaluation judge for RAG answers.
Score the answer on a single integer from 1 (broken) to 5 (excellent) for faithfulness to the provided context.
Do not use outside knowledge. Respond ONLY with JSON: {"score": int, "reason": str}"""

We will pass question, context, and answer as a user message. The judge must ignore prior training and grade only the triplet. If you later need multiple criteria, extend the JSON schema to {"faithfulness": int, "relevance": int, "reason": str} and adjust parsing accordingly.

Step 3: Configure the LLM judge generator

Instantiate the chat generator. Temperature zero reduces score variance; judge tasks need determinism, not creativity. If you want model diversity or resilience, point it at an OpenAI-compatible gateway like n4n.ai, which exposes 240+ models and fails over when a provider is rate-limited.

generator = OpenAIChatGenerator(
    model="gpt-4o-mini",
    api_key="your-key",
    api_base="https://api.n4n.ai/v1",  # swap for any compatible base
    generation_kwargs={"temperature": 0.0}
)

For self-hosted models, set api_base to your vLLM or TGI endpoint. The rest of the code stays identical.

Step 4: Build a custom judge component

Haystack 2.0 expects components decorated with @component and a run method with declared output_types. Wrap the generator so it batches over parallel lists of inputs.

@component
class LLMJudge:
    def __init__(self, generator: OpenAIChatGenerator):
        self.generator = generator

    @component.output_types(scores=list, reasons=list, raw=list)
    def run(self, questions: list, contexts: list, answers: list):
        scores, reasons, raw = [], [], []
        for q, c, a in zip(questions, contexts, answers):
            messages = [
                ChatMessage.from_system(SYSTEM_PROMPT),
                ChatMessage.from_user(
                    f"Question: {q}\nContext: {c}\nAnswer: {a}"
                ),
            ]
            res = self.generator.run(messages)
            text = res["replies"][0].text
            raw.append(text)
            try:
                parsed = json.loads(text)
                scores.append(int(parsed["score"]))
                reasons.append(parsed.get("reason", ""))
            except (json.JSONDecodeError, KeyError, ValueError):
                # Fallback: extract first integer with regex
                match = re.search(r"\d+", text)
                scores.append(int(match.group()) if match else None)
                reasons.append("parse_error")
        return {"scores": scores, "reasons": reasons, "raw": raw}

The component returns parallel lists. Malformed scores are captured as None so one bad sample doesn’t crash the run. The regex fallback keeps throughput high when the model occasionally emits markdown.

Step 5: Assemble the evaluation pipeline

Wire the judge into a Pipeline. In a real system you would connect a retriever and a generator-under-test upstream; here we feed precomputed triplets to keep the example focused.

pipeline = Pipeline()
pipeline.add_component("judge", LLMJudge(generator))

Run it on a tiny dataset:

data = {
    "judge": {
        "questions": ["What is the capital of France?"],
        "contexts": ["France is a country in Europe. Its capital is Paris."],
        "answers": ["The capital of France is Paris."],
    }
}
result = pipeline.run(data)
print(result["judge"]["scores"])
# => [5]

If you already have a RAG pipeline, connect its answers output to the judge’s answers input by name: pipeline.add_component("rag", rag_pipe); pipeline.connect("rag.answers", "judge.answers").

Step 6: Scale to a dataset and aggregate

Load examples from JSONL and compute mean score. Write results to CSV for offline analysis.

import csv

def evaluate_dataset(path):
    qs, cs, as_ = [], [], []
    with open(path) as f:
        for line in f:
            obj = json.loads(line)
            qs.append(obj["question"])
            cs.append(obj["context"])
            as_.append(obj["answer"])
    out = pipeline.run({"judge": {"questions": qs, "contexts": cs, "answers": as_}})
    valid = [s for s in out["judge"]["scores"] if s is not None]
    with open("judge_results.csv", "w", newline="") as cf:
        writer = csv.writer(cf)
        writer.writerow(["score", "reason"])
        for s, r in zip(out["judge"]["scores"], out["judge"]["reasons"]):
            writer.writerow([s, r])
    return statistics.mean(valid), len(valid)

mean_score, n = evaluate_dataset("samples.jsonl")
print(f"Mean faithfulness over {n} samples: {mean_score:.2f}")

A mean above 4.0 usually indicates the RAG system cites context correctly. Below 3.0, inspect the CSV reason column for recurring failure modes like hallucinated entities or ignored constraints.

Step 7: Verify success and harden the evaluator

Verification is two-fold: functional and qualitative. Functionally, assert the pipeline returns one score per input and that at least 95% parse as integers.

assert len(result["judge"]["scores"]) == 1
assert all(isinstance(s, int) for s in result["judge"]["scores"] if s is not None)

Add a pytest check in CI that runs the tiny example and fails on None rates above threshold:

def test_judge_runs():
    res = pipeline.run(data)
    assert res["judge"]["scores"] == [5]

Qualitatively, manually grade 10 samples yourself and compare. If your human scores diverge from the judge by more than one point on the scale, rewrite the system prompt or switch judge models. LLM judges are proxies, not ground truth.

For production, wrap generator.run in a retry loop with tenacity to handle transient 429s. Parallelize across samples with concurrent.futures.ThreadPoolExecutor if your endpoint supports concurrent requests. The LLMJudge component as written is sequential; refactor the loop to submit batches if latency becomes the bottleneck.

Caveats when using an llm-as-judge evaluator haystack 2.0 setup

Judge models exhibit position bias and verbosity preference. Mitigate by randomizing context order and capping answer length before scoring. Also, the same model family that generated the answer should not always be the judge; cross-model evaluation reduces self-preference. If you need to track cost, use a gateway that emits per-token usage metering so you can attribute spend per evaluation run without custom instrumentation.

Step 8: Extend to multi-dimensional scoring

The pattern stays identical when you need more than faithfulness. Change the system prompt to request multiple keys, update the parsing to extract a dict, and extend @component.output_types to emit faith_scores=list, rel_scores=list. Keep each dimension independently parseable so a failure in one doesn’t sink the others.

SYSTEM_PROMPT_MULTI = """You are a strict RAG judge. Output JSON with:
{"faithfulness": int 1-5, "relevance": int 1-5, "reason": str}"""

Then aggregate per dimension and build a composite score only after you’ve validated each axis against human ratings.

Wrapping up

You now have a working llm-as-judge evaluator haystack 2.0 pipeline that scores RAG answers with a configurable critic. The core moving parts are a strict prompt, a typed component, aggregated scores, and human spot-checks. Swap the generator’s model argument to compare judges, or point api_base at a routing gateway to avoid provider outages. From here, integrate the judge downstream of your production RAG pipeline to catch regressions before they reach users.