How to evaluate a RAG pipeline with an LLM judge

Evaluating a RAG pipeline with an LLM judge means replacing vibe checks with a repeatable, automated scoring system that measures retrieval relevance, answer faithfulness, and end-to-end correctness. Most teams start by eyeballing a few outputs, but that doesn’t scale. An LLM judge gives you a programmatic way to evaluate rag pipeline llm judge quality across hundreds of test cases, catch regressions in CI, and compare prompt or model changes objectively. This guide walks through building a minimal but production-grade evaluation harness you can run locally or in CI.

Step 1: Define your evaluation criteria and rubric

Before writing code, decide what you’re measuring. A RAG pipeline has three distinct failure modes, and each needs its own criterion:

Retrieval relevance — Did the retriever return the right chunks for the query?
Answer faithfulness — Does the generated answer stay grounded in the retrieved context, or does it hallucinate?
End-to-end correctness — Is the final answer actually correct and useful for the user’s intent?

For each criterion, write a 1-5 or pass/fail rubric with concrete examples. Avoid vague language like “helpful” or “accurate.” Instead, specify observable behaviors:

## Faithfulness rubric (1-5)
5 - Answer uses only information from provided context; cites sources inline
4 - Minor extrapolation but no factual contradictions
3 - Contains 1-2 claims not supported by context
2 - Significant hallucination; core claim unsupported
1 - Entirely fabricated; ignores context completely

Store rubrics as versioned markdown or YAML alongside your eval code. They become your ground truth for calibrating the judge.

Step 2: Build a representative test set

An LLM judge is only as good as the cases you feed it. Curate 50-200 examples that cover:

Happy path questions your pipeline should nail
Edge cases: ambiguous queries, multi-hop reasoning, out-of-domain requests
Known failures from production logs or user complaints
Adversarial inputs: prompt injections, contradictory context, empty retrieval

Structure each case as a JSON object with the query, expected answer (or key facts), and optionally the gold-standard retrieved chunks:

{
  "id": "rag-042",
  "query": "What was the Q3 2023 revenue for Acme Corp?",
  "expected_answer": "Acme Corp reported $47.2M revenue in Q3 2023.",
  "gold_chunks": [
    "Acme Corp Q3 2023 earnings: Revenue of $47.2M, up 12% YoY..."
  ],
  "tags": ["financial", "single-hop", "exact-match"]
}

Keep this dataset in version control. Add new cases whenever you discover a regression in production — this is your regression test suite.

Step 3: Implement the judge prompt

The judge prompt is the core of your evaluator. It takes the query, retrieved context, generated answer, and rubric, then outputs a structured score with reasoning. Use a strong instruction-following model (GPT-4o, Claude 3.5 Sonnet, or a capable open model like Llama-3.1-70B-Instruct).

Key design principles:

Force structured output — Use JSON schema or a parsing library so you can aggregate scores programmatically
Include few-shot examples — Show the judge 3-5 calibrated examples per score level
Separate criteria — Use distinct prompts for retrieval, faithfulness, and correctness; don’t ask one prompt to do everything

Here’s a faithfulness judge prompt template:

FAITHFULNESS_PROMPT = """You are an expert evaluator assessing whether an answer stays faithful to the provided context.

## Task
Score the answer on a 1-5 scale using the rubric below. Output ONLY valid JSON.

## Rubric
{faithfulness_rubric}

## Context
{context}

## Question
{query}

## Answer to evaluate
{answer}

## Few-shot examples
{examples}

## Output format
{{
  "score": <integer 1-5>,
  "reasoning": "<concise explanation citing specific claims and context spans>",
  "unsupported_claims": ["<list any claims in answer not found in context>"]
}}
"""

Load your rubric and few-shot examples from files so you can iterate on them independently:

import json
from pathlib import Path

def load_judge_prompt(criterion: str) -> str:
    rubric = Path(f"eval/rubrics/{criterion}.md").read_text()
    examples = json.loads(Path(f"eval/fewshots/{criterion}.json").read_text())
    return FAITHFULNESS_PROMPT.format(
        faithfulness_rubric=rubric,
        examples=json.dumps(examples, indent=2)
    )

Step 4: Wire the evaluation loop

Now connect your pipeline, test set, and judge. The loop: for each test case, run retrieval, generate an answer, then invoke the judge for each criterion. Store results in a structured format for analysis.

import asyncio
from dataclasses import dataclass, asdict
from typing import List
import openai  # or your preferred client

@dataclass
class EvalResult:
    case_id: str
    query: str
    retrieved_chunks: List[str]
    generated_answer: str
    retrieval_score: int
    faithfulness_score: int
    correctness_score: int
    retrieval_reasoning: str
    faithfulness_reasoning: str
    correctness_reasoning: str

async def evaluate_case(case: dict, pipeline, judge_client) -> EvalResult:
    # Run retrieval
    retrieved = await pipeline.retrieve(case["query"], k=5)
    context = "\n\n".join([c.text for c in retrieved])
    
    # Generate answer
    answer = await pipeline.generate(case["query"], context)
    
    # Judge each criterion in parallel
    retrieval_task = judge_retrieval(case["query"], retrieved, judge_client)
    faithfulness_task = judge_faithfulness(case["query"], context, answer, judge_client)
    correctness_task = judge_correctness(case["query"], case["expected_answer"], answer, judge_client)
    
    retrieval_result, faithfulness_result, correctness_result = await asyncio.gather(
        retrieval_task, faithfulness_task, correctness_task
    )
    
    return EvalResult(
        case_id=case["id"],
        query=case["query"],
        retrieved_chunks=[c.text for c in retrieved],
        generated_answer=answer,
        retrieval_score=retrieval_result["score"],
        faithfulness_score=faithfulness_result["score"],
        correctness_score=correctness_result["score"],
        retrieval_reasoning=retrieval_result["reasoning"],
        faithfulness_reasoning=faithfulness_result["reasoning"],
        correctness_reasoning=correctness_result["reasoning"],
    )

async def judge_faithfulness(query: str, context: str, answer: str, client) -> dict:
    prompt = load_judge_prompt("faithfulness").format(
        query=query, context=context, answer=answer
    )
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    return json.loads(response.choices[0].message.content)

Run the full suite:

async def run_eval_suite(test_cases: List[dict], pipeline, judge_client) -> List[EvalResult]:
    semaphore = asyncio.Semaphore(10)  # rate limit
    
    async def bounded_eval(case):
        async with semaphore:
            return await evaluate_case(case, pipeline, judge_client)
    
    tasks = [bounded_eval(case) for case in test_cases]
    return await asyncio.gather(*tasks)

Step 5: Calibrate the judge against human labels

An uncalibrated LLM judge drifts. Before trusting scores, measure inter-rater agreement between your judge and human annotators on a held-out calibration set (30-50 cases).

Process:

Have 2-3 domain experts independently score the calibration set using your rubrics
Run the same cases through your judge
Compute Cohen’s kappa or Krippendorff’s alpha for each criterion
If agreement < 0.7, revise rubric, add few-shots, or switch judge model

from sklearn.metrics import cohen_kappa_score
import numpy as np

def compute_agreement(human_scores: List[List[int]], judge_scores: List[int]) -> dict:
    """human_scores: list of [annotator1, annotator2, ...] per case"""
    results = {}
    for i, annotator in enumerate(zip(*human_scores)):
        kappa = cohen_kappa_score(annotator, judge_scores, weights="quadratic")
        results[f"annotator_{i+1}_vs_judge"] = kappa
    # Fleiss' kappa for multi-annotator
    return results

Log calibration results with timestamps and judge model version. Re-calibrate whenever you change the judge prompt, rubric, or model.

Step 6: Aggregate and visualize results

Raw scores per case are noisy. Aggregate by tags, score distributions, and trends over time.

import pandas as pd

def analyze_results(results: List[EvalResult]) -> dict:
    df = pd.DataFrame([asdict(r) for r in results])
    
    summary = {
        "overall": {
            "retrieval_mean": df["retrieval_score"].mean(),
            "faithfulness_mean": df["faithfulness_score"].mean(),
            "correctness_mean": df["correctness_score"].mean(),
            "n_cases": len(df),
        },
        "by_tag": {},
        "low_scoring_cases": df[
            (df["faithfulness_score"] <= 2) | (df["correctness_score"] <= 2)
        ]["case_id"].tolist(),
    }
    
    # Explode tags for per-tag analysis
    tagged = df.explode("tags")  # assumes tags column exists
    for tag in tagged["tags"].unique():
        subset = tagged[tagged["tags"] == tag]
        summary["by_tag"][tag] = {
            "retrieval_mean": subset["retrieval_score"].mean(),
            "faithfulness_mean": subset["faithfulness_score"].mean(),
            "correctness_mean": subset["correctness_score"].mean(),
            "count": len(subset),
        }
    
    return summary

Output a markdown report for PR comments or a dashboard. Track these metrics per commit:

Metric	Target	Alert threshold
Faithfulness mean	≥ 4.0	< 3.5
Correctness mean	≥ 4.0	< 3.5
Retrieval mean	≥ 4.0	< 3.5
% cases ≤ 2 on any criterion	< 5%	> 10%

Step 7: Integrate into CI/CD

Gate merges on evaluation thresholds. A minimal GitHub Actions workflow:

# .github/workflows/rag-eval.yml
name: RAG Evaluation
on:
  pull_request:
    paths:
      - 'rag/**'
      - 'eval/**'
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r eval/requirements.txt
      - name: Run evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          JUDGE_MODEL: gpt-4o
        run: python -m eval.run_suite --output eval_results.json
      - name: Check thresholds
        run: |
          python -c "
          import json, sys
          with open('eval_results.json') as f:
              data = json.load(f)
          faithfulness = data['summary']['overall']['faithfulness_mean']
          correctness = data['summary']['overall']['correctness_mean']
          if faithfulness < 3.5 or correctness < 3.5:
              print(f'FAIL: faithfulness={faithfulness:.2f}, correctness={correctness:.2f}')
              sys.exit(1)
          print('PASS')
          "
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: rag-eval-results
          path: eval_results.json

For faster feedback, run a smoke subset (10-20 critical cases) on every PR and the full suite nightly.

Step 8: Close the loop with error analysis

Scores tell you that something is wrong. Error analysis tells you why. Schedule a weekly 30-minute review:

Pull the bottom 10% of cases by faithfulness and correctness scores
Categorize each failure: retrieval miss, context truncation, prompt instruction gap, judge error
Assign each category an owner and a fix (retriever tuning, prompt rewrite, chunk size change, judge recalibration)
Track fix verification in the next eval run

Maintain a living failure taxonomy in a shared doc:

## Failure taxonomy (updated 2024-01-15)
- R1: Retriever misses key entity (12 cases) → investigate hybrid search
- R2: Retriever returns stale version (8 cases) → add date filtering
- F1: Answer adds plausible but unsupported detail (15 cases) → strengthen faithfulness prompt
- F2: Answer contradicts context (3 cases) → check context ordering
- C1: Correct but incomplete (20 cases) → expand expected answer criteria
- J1: Judge mis-scored (5 cases) → add few-shot, recalibrate

Step 9: Version and migrate judges

Your judge model and prompt will evolve. Treat them like model artifacts:

Pin the judge model version (e.g., gpt-4o-2024-08-06, not gpt-4o)
Store judge prompts, rubrics, and few-shots in git with semantic versioning (eval/v1.2.0/)
When upgrading, run the new judge side-by-side with the old on the full test set
Only promote if agreement on held-out calibration set improves or stays flat

# Compare judge versions
python -m eval.compare_judges \
  --old eval/v1.1.0/judge_prompt.md \
  --new eval/v1.2.0/judge_prompt.md \
  --cases eval/calibration_set.json \
  --output eval/judge_migration_report.md

Verification checklist

You’ve successfully wired an LLM judge for your RAG pipeline when:

Test set covers happy paths, edge cases, and known failures (≥ 50 cases)
Rubrics are written, versioned, and unambiguous enough for human annotators to agree (κ ≥ 0.7)
Judge outputs structured JSON with score + reasoning + specific evidence
Calibration shows judge-human agreement ≥ 0.7 on each criterion
Evaluation runs in < 10 minutes for full suite (parallelized, rate-limited)
CI gate blocks merges when faithfulness or correctness mean drops below threshold
Weekly error analysis produces actionable fixes that move metrics
Judge version migrations are tested side-by-side before promotion

Common pitfalls

Using the same model for generation and judging. If your pipeline uses for judging, you get correlated failures. Use a different model family (e.g., pipeline on Llama-3.1, judge on GPT-4o or Claude).

Asking one prompt to score everything. Retrieval, faithfulness, and correctness are orthogonal. Separate prompts let you optimize few-shots and rubrics per criterion.

Skipping calibration. An uncalibrated judge is a random number generator with better vocabulary. The 30-50 case calibration set pays for itself in prevented false alarms.

Treating scores as ground truth. Scores are signals. A drop from 4.2 to 3.8 warrants investigation, not panic. The weekly error analysis is where real improvement happens.

Ignoring retrieval evaluation. If retrieval fails, faithfulness and correctness will too — but for different reasons. Score retrieval separately so you know which component to fix.

An LLM judge doesn’t replace human evaluation — it makes human evaluation scalable. You still need domain experts to write rubrics, calibrate, and do error analysis. But once the harness exists, you can evaluate rag pipeline llm judge quality on every commit, catch regressions before they hit production, and make prompt or model changes with evidence instead of intuition. The upfront investment is roughly 2-3 engineering days; the ongoing cost is one weekly review and occasional recalibration. That’s a trade most teams should take.