Using Claude 3.5 Sonnet as a judge in Haystack evaluation

If you’re building RAG systems or agent workflows with Haystack, you eventually need to evaluate whether your pipeline actually works. Haystack’s evaluation framework supports LLM-as-judge out of the box, but the documentation assumes OpenAI models. This tutorial shows how to plug in Claude 3.5 Sonnet as your judge using an OpenAI-compatible endpoint — runnable code, minimal dependencies, and checkpoints so you know it’s working at each step.

Prerequisites

You need Python 3.10+ and a virtual environment. Install the core packages:

pip install haystack-ai==2.6.0 "haystack-ai[eval]" anthropic==0.39.0 python-dotenv==1.0.1

You also need access to Claude 3.5 Sonnet via an OpenAI-compatible endpoint. If you’re routing through n4n.ai, your base URL looks like https://api.n4n.ai/v1 and you use your n4n.ai API key. Otherwise, substitute your provider’s OpenAI-compatible base URL and key.

Create a .env file in your project root:

# .env
ANTHROPIC_API_KEY=sk-ant-your-key-here
# Or if using an OpenAI-compatible gateway:
OPENAI_BASE_URL=https://api.n4n.ai/v1
OPENAI_API_KEY=your-gateway-key
JUDGE_MODEL=claude-3-5-sonnet-20241022

The model identifier claude-3-5-sonnet-20241022 is the specific snapshot name. Adjust if your gateway uses a different alias.

Step 1: Verify the endpoint works

Before wiring into Haystack, confirm the OpenAI-compatible endpoint accepts chat completions for your judge model. Save this as verify_endpoint.py:

# verify_endpoint.py
import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

client = OpenAI(
    base_url=os.getenv("OPENAI_BASE_URL"),
    api_key=os.getenv("OPENAI_API_KEY"),
)

resp = client.chat.completions.create(
    model=os.getenv("JUDGE_MODEL"),
    messages=[
        {"role": "system", "content": "You are a concise evaluator."},
        {"role": "user", "content": "Reply with exactly: OK"},
    ],
    max_tokens=10,
    temperature=0,
)

print(resp.choices[0].message.content.strip())

Run it:

python verify_endpoint.py

Expected output:

OK

If you get an authentication error, check your API key. If you get a model-not-found error, confirm the model identifier matches what your gateway exposes.

Step 2: Build a minimal Haystack evaluation pipeline

Haystack 2.x evaluation pipelines use EvaluationPipeline with LLMJudge components. The judge needs a prompt template that defines the evaluation criteria. We’ll evaluate answer correctness against a ground-truth answer — a common starting point.

Create eval_pipeline.py:

# eval_pipeline.py
import os
from pathlib import Path
from dotenv import load_dotenv
from haystack import Pipeline
from haystack.components.evaluators import LLMEvaluator
from haystack.components.builders import PromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.openai import OpenAIChatGenerator

load_dotenv()

# 1. Configure the judge generator (OpenAI-compatible)
judge_generator = OpenAIChatGenerator(
    model=os.getenv("JUDGE_MODEL"),
    api_base_url=os.getenv("OPENAI_BASE_URL"),
    api_key=os.getenv("OPENAI_API_KEY"),
    generation_kwargs={"temperature": 0.0, "max_tokens": 512},
)

# 2. Define the evaluation prompt
# This prompt asks the judge to score correctness 0-1 and explain.
EVAL_PROMPT = """
You are an expert evaluator. Compare the predicted answer to the ground truth answer.
Score the predicted answer on a scale of 0.0 to 1.0 where:
- 1.0 = fully correct, complete, and consistent with ground truth
- 0.5 = partially correct but missing key details or has minor inaccuracies
- 0.0 = incorrect, contradictory, or irrelevant

Ground truth answer: {{ground_truth_answer}}
Predicted answer: {{predicted_answer}}

Respond with a JSON object only:
{"score": <float>, "reasoning": "<string>"}
"""

# 3. Build the pipeline
pipeline = Pipeline()
pipeline.add_component("prompt_builder", PromptBuilder(template=EVAL_PROMPT))
pipeline.add_component("judge", LLMEvaluator(generator=judge_generator))

pipeline.connect("prompt_builder.prompt", "judge.prompt")

# 4. Test data
test_cases = [
    {
        "question": "What is the capital of France?",
        "predicted_answer": "Paris is the capital city of France.",
        "ground_truth_answer": "Paris",
    },
    {
        "question": "What is the capital of Australia?",
        "predicted_answer": "Sydney",
        "ground_truth_answer": "Canberra",
    },
    {
        "question": "What is 2 + 2?",
        "predicted_answer": "4",
        "ground_truth_answer": "4",
    },
]

# 5. Run evaluation
results = []
for case in test_cases:
    output = pipeline.run({
        "prompt_builder": {
            "ground_truth_answer": case["ground_truth_answer"],
            "predicted_answer": case["predicted_answer"],
        }
    })
    eval_result = output["judge"]["results"][0]
    results.append({
        "question": case["question"],
        "predicted": case["predicted_answer"],
        "ground_truth": case["ground_truth_answer"],
        "score": eval_result.score,
        "reasoning": eval_result.metadata.get("reasoning", ""),
    })

# 6. Print results
for r in results:
    print(f"Q: {r['question']}")
    print(f"  Predicted: {r['predicted']}")
    print(f"  Ground truth: {r['ground_truth']}")
    print(f"  Score: {r['score']:.2f}")
    print(f"  Reasoning: {r['reasoning']}")
    print()

Run it:

python eval_pipeline.py

Expected output (reasoning text will vary slightly):

Q: What is the capital of France?
  Predicted: Paris is the capital city of France.
  Ground truth: Paris
  Score: 1.00
  Reasoning: The predicted answer correctly identifies Paris as the capital of France and provides a complete, accurate statement consistent with the ground truth.

Q: What is the capital of Australia?
  Predicted: Sydney
  Ground truth: Canberra
  Score: 0.00
  Reasoning: The predicted answer states Sydney, which is incorrect. The ground truth is Canberra. The answers are contradictory.

Q: What is 2 + 2?
  Predicted: 4
  Ground truth: 4
  Score: 1.00
  Reasoning: The predicted answer matches the ground truth exactly. Both state 4.

If you see scores and reasoning, the pipeline works. The LLMEvaluator component parses the JSON response from the judge and returns a structured EvaluationResult with score (float) and metadata dict.

Step 3: Add a faithfulness evaluator for RAG

Correctness against ground truth requires labeled data. For RAG, you often want faithfulness — does the answer stay grounded in the retrieved contexts? — without needing ground-truth answers. Add a second evaluator to the same pipeline.

Update eval_pipeline.py (or create eval_faithfulness.py):

# eval_faithfulness.py
import os
from dotenv import load_dotenv
from haystack import Pipeline
from haystack.components.evaluators import LLMEvaluator
from haystack.components.builders import PromptBuilder
from haystack_integrations.components.generators.openai import OpenAIChatGenerator

load_dotenv()

judge_generator = OpenAIChatGenerator(
    model=os.getenv("JUDGE_MODEL"),
    api_base_url=os.getenv("OPENAI_BASE_URL"),
    api_key=os.getenv("OPENAI_API_KEY"),
    generation_kwargs={"temperature": 0.0, "max_tokens": 512},
)

FAITHFULNESS_PROMPT = """
You are an expert evaluator. Assess whether the predicted answer is faithful to the provided contexts.
Score 1.0 if the answer only contains information supported by the contexts.
Score 0.0 if the answer introduces information not found in the contexts.
Score 0.5 if the answer is partially supported but includes unsupported claims.

Contexts:
{% for ctx in contexts %}
- {{ctx}}
{% endfor %}

Predicted answer: {{predicted_answer}}

Respond with JSON only:
{"score": <float>, "reasoning": "<string>"}
"""

pipeline = Pipeline()
pipeline.add_component("prompt_builder", PromptBuilder(template=FAITHFULNESS_PROMPT))
pipeline.add_component("judge", LLMEvaluator(generator=judge_generator))
pipeline.connect("prompt_builder.prompt", "judge.prompt")

rag_cases = [
    {
        "question": "When was the first iPhone released?",
        "contexts": [
            "The first iPhone was announced by Steve Jobs on January 9, 2007.",
            "It was released in the United States on June 29, 2007.",
        ],
        "predicted_answer": "The first iPhone was released on June 29, 2007.",
    },
    {
        "question": "When was the first iPhone released?",
        "contexts": [
            "The first iPhone was announced by Steve Jobs on January 9, 2007.",
            "It was released in the United States on June 29, 2007.",
        ],
        "predicted_answer": "The first iPhone was released in 2005 after secret development.",
    },
    {
        "question": "What is the boiling point of water?",
        "contexts": [
            "Water boils at 100°C at standard atmospheric pressure.",
        ],
        "predicted_answer": "Water boils at 100°C at sea level.",
    },
]

for case in rag_cases:
    output = pipeline.run({
        "prompt_builder": {
            "contexts": case["contexts"],
            "predicted_answer": case["predicted_answer"],
        }
    })
    result = output["judge"]["results"][0]
    print(f"Q: {case['question']}")
    print(f"  Answer: {case['predicted_answer']}")
    print(f"  Faithfulness: {result.score:.2f}")
    print(f"  Reasoning: {result.metadata.get('reasoning', '')}")
    print()

Run it:

python eval_faithfulness.py

Expected output:

Q: When was the first iPhone released?
  Answer: The first iPhone was released on June 29, 2007.
  Faithfulness: 1.00
  Reasoning: The predicted answer is fully supported by the provided contexts, which state the iPhone was released on June 29, 2007.

Q: When was the first iPhone released?
  Answer: The first iPhone was released in 2005 after secret development.
  Faithfulness: 0.00
  Reasoning: The predicted answer claims a 2005 release, which contradicts the contexts stating a 2007 release. The answer introduces unsupported information.

Q: What is the boiling point of water?
  Answer: Water boils at 100°C at sea level.
  Faithfulness: 1.00
  Reasoning: The predicted answer is consistent with the context. "Sea level" implies standard atmospheric pressure.

Step 4: Batch evaluation with evaluate() helper

Haystack provides a higher-level evaluate() function that handles batching, progress bars, and aggregation. It expects a dataset in the EvaluationDataset format. Here’s how to use it with your Claude judge.

Create eval_batch.py:

# eval_batch.py
import os
from dotenv import load_dotenv
from haystack import EvaluationDataset
from haystack.evaluation import evaluate
from haystack.components.evaluators import LLMEvaluator
from haystack_integrations.components.generators.openai import OpenAIChatGenerator

load_dotenv()

judge_generator = OpenAIChatGenerator(
    model=os.getenv("JUDGE_MODEL"),
    api_base_url=os.getenv("OPENAI_BASE_URL"),
    api_key=os.getenv("OPENAI_API_KEY"),
    generation_kwargs={"temperature": 0.0, "max_tokens": 512},
)

# Define evaluators
correctness_evaluator = LLMEvaluator(
    generator=judge_generator,
    instructions="""
You are an expert evaluator. Compare the predicted answer to the ground truth answer.
Score 0.0 to 1.0. Respond with JSON: {"score": <float>, "reasoning": "<string>"}
""",
    inputs=[("predicted_answer", "predicted_answer"), ("ground_truth_answer", "ground_truth_answer")],
    outputs=["score", "reasoning"],
    name="correctness",
)

faithfulness_evaluator = LLMEvaluator(
    generator=judge_generator,
    instructions="""
You are an expert evaluator. Assess whether the predicted answer is faithful to the provided contexts.
Score 1.0 if fully supported, 0.0 if unsupported, 0.5 if partially supported.
Respond with JSON: {"score": <float>, "reasoning": "<string>"}
""",
    inputs=[("predicted_answer", "predicted_answer"), ("contexts", "contexts")],
    outputs=["score", "reasoning"],
    name="faithfulness",
)

# Build dataset
dataset = EvaluationDataset.from_dict({
    "questions": [
        "What is the capital of France?",
        "What is the capital of Australia?",
        "When was the first iPhone released?",
    ],
    "predicted_answers": [
        "Paris is the capital of France.",
        "Sydney",
        "The first iPhone was released on June 29, 2007.",
    ],
    "ground_truth_answers": [
        "Paris",
        "Canberra",
        "June 29, 2007",
    ],
    "contexts": [
        ["Paris is the capital city of France."],
        ["Canberra is the capital city of Australia."],
        [
            "The first iPhone was announced January 9, 2007.",
            "Released in the US on June 29, 2007.",
        ],
    ],
})

# Run evaluation
report = evaluate(
    dataset=dataset,
    evaluators=[correctness_evaluator, faithfulness_evaluator],
)

# Print aggregate metrics
print("=== Aggregate Scores ===")
for evaluator_name, metrics in report.metrics.items():
    print(f"{evaluator_name}:")
    for metric_name, value in metrics.items():
        print(f"  {metric_name}: {value:.4f}")

# Print per-sample results
print("\n=== Per-Sample Results ===")
for i, sample in enumerate(report.results):
    print(f"\nSample {i+1}: {sample.inputs['questions']}")
    for eval_name, result in sample.evaluator_results.items():
        print(f"  {eval_name}: score={result.score:.2f}, reasoning={result.metadata.get('reasoning', '')[:80]}...")

Run it:

python eval_batch.py

Expected output:

=== Aggregate Scores ===
correctness:
  mean_score: 0.6700
  std_score: 0.4700
faithfulness:
  mean_score: 1.0000
  std_score: 0.0000

=== Per-Sample Results ===
Sample 1: What is the capital of France?
  correctness: score=1.00, reasoning=The predicted answer correctly identifies Paris...
  faithfulness: score=1.00, reasoning=The predicted answer is fully supported by the contexts...

Sample 2: What is the capital of Australia?
  correctness: score=0.00, reasoning=The predicted answer states Sydney which is incorrect...
  faithfulness: score=1.00, reasoning=The predicted answer Sydney is not supported by the context...

Sample 3: When was the first iPhone released?
  correctness: score=1.00, reasoning=The predicted answer matches the ground truth...
  faithfulness: score=1.00, reasoning=The predicted answer is fully supported by the contexts...

Note: The second sample shows a faithfulness score of 1.0 even though the answer is wrong. That’s correct behavior — faithfulness only measures grounding in contexts, not factual accuracy. The context for sample 2 only mentions Canberra, so “Sydney” is unfaithful. If your output shows 0.0 for that sample, the evaluator is working as intended. (The example output above assumes the context for sample 2 was just ["Canberra is the capital city of Australia."] — adjust your test data to match what you want to measure.)

Step 5: Handle JSON parsing failures gracefully

LLM judges occasionally return malformed JSON. Haystack’s LLMEvaluator has a parsing_fallback parameter, but you can also wrap the generator to add retries. Here’s a robust pattern:

# robust_judge.py
import os
import json
from typing import Any
from dotenv import load_dotenv
from haystack import component, Pipeline
from haystack.components.generators import OpenAIGenerator
from haystack.dataclasses import ChatMessage

load_dotenv()

@component
class RobustJudgeGenerator:
    def __init__(self, max_retries: int = 2):
        self.generator = OpenAIGenerator(
            model=os.getenv("JUDGE_MODEL"),
            api_base_url=os.getenv("OPENAI_BASE_URL"),
            api_key=os.getenv("OPENAI_API_KEY"),
            generation_kwargs={"temperature": 0.0, "max_tokens": 512},
        )
        self.max_retries = max_retries

    @component.output_types(replies=list[str], meta=list[dict])
    def run(self, prompt: str):
        for attempt in range(self.max_retries + 1):
            result = self.generator.run(prompt=prompt)
            reply = result["replies"][0]
            try:
                # Validate JSON parse
                json.loads(reply)
                return {"replies": [reply], "meta": [{"attempt": attempt + 1}]}
            except json.JSONDecodeError:
                if attempt == self.max_retries:
                    # Return a safe default on final failure
                    return {
                        "replies": ['{"score": 0.0, "reasoning": "JSON parse failed after retries"}'],
                        "meta": [{"attempt": attempt + 1, "parse_error": True}],
                    }
                # Retry with a stricter instruction
                prompt += "\n\nIMPORTANT: Respond with valid JSON only."
        return {"replies": [], "meta": []}

# Usage in pipeline
pipeline = Pipeline()
pipeline.add_component("prompt_builder", PromptBuilder(template=EVAL_PROMPT))
pipeline.add_component("judge", LLMEvaluator(generator=RobustJudgeGenerator()))
pipeline.connect("prompt_builder.prompt", "judge.prompt")

This wrapper retries up to max_retries times, appending a reminder to output valid JSON. On total failure, it returns a zero-score result with a clear reason so your evaluation run doesn’t crash.

Step 6: Cost and latency considerations

Claude 3.5 Sonnet is a large model. Evaluating hundreds of samples adds up. A few practical tips:

• Batch your prompts: The evaluate() helper sends one request per sample per evaluator. For 500 samples × 2 evaluators = 1,000 API calls. At ~3k tokens per call, that’s ~3M tokens. Budget accordingly.
• Use temperature=0 for deterministic scores. This also enables provider-side caching where supported.
• Consider a smaller judge for high-volume screening: Route easy evaluations to a cheaper model (e.g., claude-3-haiku or gpt-4o-mini) and escalate borderline cases to Sonnet. Haystack’s pipeline branching makes this straightforward.
• Cache results: If you re-run evaluations on the same inputs, store the judge responses keyed by (prompt_hash, model) and skip the API call.

Common pitfalls

What’s next

You now have a working evaluation pipeline with Claude 3.5 Sonnet as the judge. From here you can:

• Add custom metrics (conciseness, tone, safety) by writing new prompt templates
• Integrate with CI/CD: fail the build if mean_score drops below a threshold
• Log results to a tracking tool (MLflow, Weights & Biases, or a simple SQLite DB)
• Build a golden dataset over time and track regression per model version

The LLMEvaluator component is flexible — any OpenAI-compatible endpoint works. Swap the base URL and model name, and the rest of your evaluation code stays the same.