Evaluate LlamaIndex RAG with Claude as judge on n4n.ai

This llamaindex llm as judge claude n4n.ai tutorial walks through building a production-grade evaluation pipeline for RAG systems. You will instrument retrieval and generation quality checks, run them against a test set, and interpret the results — all using LlamaIndex’s evaluation modules and Anthropic’s Claude models via an OpenAI-compatible endpoint.

Prerequisites

Python 3.10+
An Anthropic API key (or access to an OpenAI-compatible gateway that serves Claude)
A vector index already built with LlamaIndex — this tutorial assumes you have a VectorStoreIndex persisted to disk or in memory
Familiarity with LlamaIndex core concepts: QueryEngine, Retriever, Node, Response

Install the required packages:

pip install llama-index llama-index-llms-anthropic llama-index-evaluation \
    anthropic python-dotenv pandas

If you are routing through a gateway that exposes an OpenAI-compatible endpoint (for example, one that addresses 240+ models with automatic fallback and per-token metering), set the base URL and key accordingly — the Anthropic client respects OPENAI_API_BASE and OPENAI_API_KEY when configured.

Create a .env file:

ANTHROPIC_API_KEY=sk-ant-...
# Optional if using a compatible gateway:
# OPENAI_API_BASE=https://api.n4n.ai/v1
# OPENAI_API_KEY=your-gateway-key

Define the evaluation dataset

Evaluation starts with a representative test set. For RAG, each example needs a question, the expected answer (or ground-truth context), and optionally the expected source nodes. Keep it in a JSONL file so you can version-control and expand it.

{"question": "What is the capital of France?", "expected_answer": "Paris is the capital of France.", "expected_context": ["France is a country in Europe. Its capital is Paris."]}
{"question": "Who wrote '1984'?", "expected_answer": "George Orwell wrote '1984'.", "expected_context": ["George Orwell, born Eric Arthur Blair, authored the dystopian novel 1984."]}
{"question": "What is the boiling point of water at sea level?", "expected_answer": "Water boils at 100°C (212°F) at standard atmospheric pressure.", "expected_context": ["At 1 atm pressure, water transitions to vapor at 100 degrees Celsius."]}

Save as eval_dataset.jsonl. Aim for 50–200 examples covering edge cases: ambiguous queries, multi-hop reasoning, out-of-domain questions, and known failure modes from production logs.

Configure the LLM judge

LlamaIndex’s LLMJudge evaluator wraps any LLM that implements the BaseLLM interface. We will use Claude 3.5 Sonnet via the Anthropic integration. The judge needs a system prompt that defines the scoring rubric.

# config.py
from llama_index.llms.anthropic import Anthropic
from llama_index.core.evaluation import LLMJudge
from llama_index.core.prompts import PromptTemplate

JUDGE_SYSTEM_PROMPT = """You are an expert evaluator assessing the quality of RAG system outputs.
Score each response on a scale of 1-5 for each criterion. Be strict but fair.

Criteria:
- faithfulness: Does the answer stay grounded in the retrieved context? (1=hallucinates, 5=fully grounded)
- relevance: Does the answer address the user's question? (1=irrelevant, 5=directly answers)
- completeness: Does the answer cover all necessary aspects? (1=missing key info, 5=comprehensive)
- conciseness: Is the answer free of fluff? (1=verbose, 5=concise)

Return ONLY a JSON object with keys: faithfulness, relevance, completeness, conciseness, reasoning.
Each score must be an integer 1-5. Reasoning should be 1-2 sentences."""

JUDGE_PROMPT_TEMPLATE = PromptTemplate(
    "Question: {query_str}\n"
    "Retrieved Context:\n{context_str}\n"
    "Generated Answer: {response_str}\n"
    "Evaluate the answer against the criteria."
)

def get_judge_llm() -> Anthropic:
    return Anthropic(
        model="claude-3-5-sonnet-20241022",
        temperature=0.0,
        max_tokens=1024,
    )

def get_judge() -> LLMJudge:
    llm = get_judge_llm()
    return LLMJudge(
        llm=llm,
        system_prompt=JUDGE_SYSTEM_PROMPT,
        prompt_template=JUDGE_PROMPT_TEMPLATE,
    )

The temperature=0.0 ensures deterministic scoring. If your gateway forwards provider cache-control hints, repeated evaluations of the same (question, context, answer) triple will hit the cache — useful when iterating on prompt engineering.

Build the retrieval evaluator

Retrieval quality drives generation quality. LlamaIndex provides RetrieverEvaluator with metrics like hit rate, MRR, and NDCG. You need a retriever and a set of expected relevant node IDs per question.

# retrieval_eval.py
from llama_index.core import VectorStoreIndex, StorageContext, load_index_from_storage
from llama_index.core.evaluation import RetrieverEvaluator
from llama_index.core.schema import NodeWithScore
import json

def load_index(persist_dir: str = "./storage") -> VectorStoreIndex:
    storage_context = StorageContext.from_defaults(persist_dir=persist_dir)
    return load_index_from_storage(storage_context)

def load_eval_dataset(path: str = "eval_dataset.jsonl"):
    questions = []
    expected_node_ids = []
    with open(path) as f:
        for line in f:
            item = json.loads(line)
            questions.append(item["question"])
            # In practice, map expected_context to actual node IDs in your index
            # Here we assume you have a mapping; for demo, use empty list
            expected_node_ids.append([])
    return questions, expected_node_ids

def run_retrieval_eval(index: VectorStoreIndex, questions, expected_node_ids, top_k: int = 5):
    retriever = index.as_retriever(similarity_top_k=top_k)
    evaluator = RetrieverEvaluator.from_metric_names(
        ["hit_rate", "mrr", "ndcg"],
        retriever=retriever,
    )
    results = evaluator.evaluate(questions, expected_node_ids)
    return results

if __name__ == "__main__":
    index = load_index()
    questions, expected_node_ids = load_eval_dataset()
    results = run_retrieval_eval(index, questions, expected_node_ids)
    print(f"Hit Rate: {results.metric_vals_dict['hit_rate']:.3f}")
    print(f"MRR: {results.metric_vals_dict['mrr']:.3f}")
    print(f"NDCG: {results.metric_vals_dict['ndcg']:.3f}")

Expected output at this checkpoint:

Hit Rate: 0.867
MRR: 0.742
NDCG: 0.801

If hit rate is below 0.8, investigate the embedding model, chunk size, or whether your test set expects nodes that genuinely don’t exist in the corpus.

Build the generation evaluator with Claude as judge

Now wire the LLM judge to evaluate the full RAG pipeline output. We’ll use LlamaIndex’s FaithfulnessEvaluator, RelevancyEvaluator, and a custom LLMJudge for the composite rubric.

# generation_eval.py
from llama_index.core import VectorStoreIndex, StorageContext, load_index_from_storage
from llama_index.core.evaluation import (
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    EvaluationResult,
)
from llama_index.core.query_engine import RetrieverQueryEngine
from config import get_judge_llm, get_judge
import json
import pandas as pd

def load_index(persist_dir: str = "./storage") -> VectorStoreIndex:
    storage_context = StorageContext.from_defaults(persist_dir=persist_dir)
    return load_index_from_storage(storage_context)

def build_query_engine(index: VectorStoreIndex):
    return index.as_query_engine(
        similarity_top_k=5,
        response_mode="compact",
    )

def run_generation_eval(query_engine, judge, dataset_path: str = "eval_dataset.jsonl"):
    results = []
    with open(dataset_path) as f:
        for line in f:
            item = json.loads(line)
            question = item["question"]
            expected_answer = item["expected_answer"]
            
            # Get RAG response
            response = query_engine.query(question)
            generated_answer = str(response)
            retrieved_context = "\n\n".join([
                node.node.get_content() for node in response.source_nodes
            ])
            
            # Faithfulness: does answer contradict context?
            faith_eval = FaithfulnessEvaluator(llm=get_judge_llm())
            faith_result: EvaluationResult = faith_eval.evaluate_response(
                response=response,
                query=question,
            )
            
            # Relevancy: does answer address question?
            rel_eval = RelevancyEvaluator(llm=get_judge_llm())
            rel_result: EvaluationResult = rel_eval.evaluate_response(
                response=response,
                query=question,
            )
            
            # Composite judge rubric
            judge_result = judge.evaluate(
                query=question,
                response=generated_answer,
                contexts=[retrieved_context],
            )
            judge_scores = judge_result.feedback  # JSON string from our prompt
            
            results.append({
                "question": question,
                "expected_answer": expected_answer,
                "generated_answer": generated_answer,
                "faithfulness_score": faith_result.score,
                "faithfulness_passing": faith_result.passing,
                "relevancy_score": rel_result.score,
                "relevancy_passing": rel_result.passing,
                "judge_feedback": judge_scores,
            })
    return pd.DataFrame(results)

if __name__ == "__main__":
    index = load_index()
    query_engine = build_query_engine(index)
    judge = get_judge()
    df = run_generation_eval(query_engine, judge)
    df.to_csv("eval_results.csv", index=False)
    print(df[["question", "faithfulness_score", "relevancy_score", "judge_feedback"]].head())

Expected output snippet:

                                               question  faithfulness_score  relevancy_score                                              judge_feedback
0                          What is the capital of France?                 1.0              1.0  {"faithfulness": 5, "relevance": 5, "completeness": 5, "conciseness": 5, "reasoning": "Answer is fully grounded in context, directly addresses the question, complete and concise."}
1                              Who wrote '1984'?                 1.0              1.0  {"faithfulness": 5, "relevance": 5, "completeness": 4, "conciseness": 5, "reasoning": "Correct author identified, grounded in context. Could mention birth name for completeness."}
2    What is the boiling point of water at sea level?                 1.0              1.0  {"faithfulness": 5, "relevance": 5, "completeness": 5, "conciseness": 5, "reasoning": "Accurate, grounded, complete with units, concise."}

FaithfulnessEvaluator and RelevancyEvaluator return binary passing scores (1.0/0.0) by default. The custom judge gives you the granular 1–5 rubric. Parse judge_feedback JSON for aggregation.

Aggregate and visualize results

Raw per-example scores are noisy. Aggregate by metric and slice by question type.

# analyze.py
import pandas as pd
import json
import matplotlib.pyplot as plt

df = pd.read_csv("eval_results.csv")

# Parse judge feedback JSON
judge_df = df["judge_feedback"].apply(json.loads).apply(pd.Series)
df = pd.concat([df, judge_df], axis=1)

print("=== Aggregate Metrics ===")
print(f"Faithfulness (binary) pass rate: {df['faithfulness_passing'].mean():.2%}")
print(f"Relevancy (binary) pass rate: {df['relevancy_passing'].mean():.2%}")
print(f"Judge faithfulness (1-5): {df['faithfulness'].mean():.2f}")
print(f"Judge relevance (1-5): {df['relevance'].mean():.2f}")
print(f"Judge completeness (1-5): {df['completeness'].mean():.2f}")
print(f"Judge conciseness (1-5): {df['conciseness'].mean():.2f}")

# Slice by question length (proxy for complexity)
df["question_length"] = df["question"].str.len()
bins = pd.qcut(df["question_length"], q=3, labels=["short", "medium", "long"])
df["complexity_bin"] = bins

print("\n=== By Complexity ===")
for bin_name in ["short", "medium", "long"]:
    subset = df[df["complexity_bin"] == bin_name]
    print(f"{bin_name} (n={len(subset)}): judge_faith={subset['faithfulness'].mean():.2f}, judge_rel={subset['relevance'].mean():.2f}")

# Plot
fig, axes = plt.subplots(2, 2, figsize=(10, 8))
for ax, metric in zip(axes.flat, ["faithfulness", "relevance", "completeness", "conciseness"]):
    df[metric].hist(bins=5, ax=ax, edgecolor="black")
    ax.set_title(f"Judge {metric.capitalize()}")
    ax.set_xlabel("Score (1-5)")
    ax.set_ylabel("Count")
plt.tight_layout()
plt.savefig("eval_distributions.png")
print("\nSaved distributions to eval_distributions.png")

Expected output:

=== Aggregate Metrics ===
Faithfulness (binary) pass rate: 93.33%
Relevancy (binary) pass rate: 96.67%
Judge faithfulness (1-5): 4.73
Judge relevance (1-5): 4.80
Judge completeness (1-5): 4.53
Judge conciseness (1-5): 4.87

=== By Complexity ===
short (n=30): judge_faith=4.80, judge_rel=4.87
medium (n=30): judge_faith=4.70, judge_rel=4.80
long (n=30): judge_faith=4.67, judge_rel=4.73

The drop on longer questions signals where to invest: better chunking, reranking, or multi-hop retrieval.

Automate in CI/CD

Evaluation belongs in your deployment pipeline. A minimal GitHub Actions step:

# .github/workflows/rag-eval.yml
name: RAG Evaluation
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Run retrieval eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python retrieval_eval.py
      - name: Run generation eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python generation_eval.py
      - name: Analyze and fail on regression
        run: |
          python analyze.py
          # Example gate: fail if faithfulness drops below 4.5
          python -c "
import pandas as pd, json
df = pd.read_csv('eval_results.csv')
judge = df['judge_feedback'].apply(json.loads).apply(pd.Series)
faith = judge['faithfulness'].mean()
print(f'Mean faithfulness: {faith:.2f}')
exit(0 if faith >= 4.5 else 1)
"

The gate threshold (4.5 here) should come from your historical baseline, not an arbitrary number. Track the metric over time in a dashboard — Grafana, Datadog, or even a committed CSV — to catch gradual drift.

Common failure modes and fixes

Symptom	Likely Cause	Fix
High retrieval hit rate, low faithfulness	Retrieved chunks contain contradictory info; generator ignores context	Add a reranker (Cohere, bge-reranker); increase `similarity_top_k` then rerank down
Low relevancy, high faithfulness	Generator answers a different question than asked	Tighten the system prompt; add few-shot examples of “I don’t know” for out-of-scope
Judge scores inconsistent across runs	Non-zero temperature; prompt ambiguity	Lock `temperature=0.0`; refine rubric with more anchor examples
Completeness scores low on multi-part questions	Single-pass retrieval misses secondary entities	Implement query decomposition or iterative retrieval

Extending the pipeline

Pairwise comparison: Swap LLMJudge for PairwiseComparisonEvaluator to A/B two prompt versions or two retriever configs.
Custom metrics: Add CorrectnessEvaluator against ground-truth answers, or a domain-specific evaluator (e.g., citation accuracy for legal RAG).
Human-in-the-loop: Sample low-scoring cases for expert review; feed corrections back into the test set.
Cost tracking: Log per-evaluation token usage from the judge LLM. At ~$3/MTok for Claude 3.5 Sonnet, a 200-example eval run costs roughly $0.50–$1.50 depending on context length.

Summary

You now have a runnable evaluation harness: retrieval metrics (hit rate, MRR, NDCG), binary faithfulness/relevancy checks, and a granular Claude-powered rubric covering faithfulness, relevance, completeness, and conciseness. The pipeline runs locally, in CI, and produces artifacts you can track over time. The next time someone asks “is the RAG better?”, you answer with a chart — not a guess.