Track RAG regression in LlamaIndex with evaluation metrics

Building a RAG system that works today is straightforward. Keeping it working as you swap embedding models, tweak chunk sizes, or upgrade LlamaIndex versions is where most teams struggle. This llamaindex rag regression testing evaluation tutorial walks through a practical approach: define your golden dataset, pick metrics that actually correlate with user experience, automate the loop, and catch regressions before they hit production.

Define what regression means for your pipeline

Regression in RAG isn’t a single number. It shows up as degraded retrieval precision, hallucinated answers, or latency spikes. Start by enumerating the failure modes you care about:

• Retrieval regression: Relevant documents drop out of top-k
• Generation regression: Answers become less grounded or more verbose
• Latency regression: p95 latency crosses your SLO threshold
• Cost regression: Token usage per query grows unexpectedly

Write these down as explicit assertions. “Recall@5 stays above 0.85” beats “retrieval feels good.” If you can’t express it as a threshold, you can’t automate the check.

Build a golden evaluation dataset

Your evaluation dataset is the contract between your pipeline and your users. It needs three properties:

1. Representative: Covers the query distribution you see in production (navigational, informational, comparative, adversarial)
2. Versioned: Stored alongside your code, not in a spreadsheet someone updates manually
3. Minimal but sufficient: 50-200 high-quality examples beat 10,000 noisy ones

Structure each example with the fields your evaluators need:

{
  "query": "How do I configure automatic fallback in n4n.ai?",
  "expected_answer": "Set the `fallback_models` parameter in your request body to an ordered list of model IDs. The gateway will try each in sequence until one succeeds.",
  "relevant_doc_ids": ["doc_fallback_config", "doc_model_routing"],
  "metadata": {
    "category": "configuration",
    "difficulty": "easy",
    "language": "en"
  }
}

Store this as JSONL or Parquet. Commit it to version control. Treat changes to this dataset like schema migrations — review them, don’t just append.

Pitfall: Using synthetic data generated by the same LLM you’re evaluating. This creates circular validation. Seed your golden set with real user queries (anonymized) and expert-written answers.

Choose metrics that map to user outcomes

LlamaIndex provides several evaluators out of the box. Don’t use all of them. Pick two or three that directly reflect your failure modes:

The FaithfulnessEvaluator and CorrectnessEvaluator use an LLM-as-judge. This introduces variance. Run each evaluation 3-5 times and take the median, or use a deterministic judge model (e.g., GPT-4 with temperature=0) and accept the cost.

from llama_index.core.evaluation import (
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    RetrieverEvaluator,
)
from llama_index.core import VectorStoreIndex, Settings
from llama_index.llms.openai import OpenAI

# Use a consistent judge model across runs
judge_llm = OpenAI(model="gpt-4", temperature=0.0)
Settings.llm = judge_llm

faithfulness = FaithfulnessEvaluator(llm=judge_llm)
relevancy = RelevancyEvaluator(llm=judge_llm)
retriever_eval = RetrieverEvaluator(
    retriever=index.as_retriever(similarity_top_k=5),
    metric_names=["hit_rate", "mrr", "recall"],
)

Wire evaluation into your CI pipeline

The evaluation must run on every PR that touches the RAG pipeline: embedding model changes, prompt templates, chunking logic, retriever configuration, LlamaIndex version bumps.

A minimal GitHub Actions workflow:

name: rag-regression-check
on:
  pull_request:
    paths:
      - 'rag/**'
      - 'eval/golden_set.jsonl'
      - 'requirements.txt'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - name: Run evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -m eval.run_evaluation \
            --golden-set eval/golden_set.jsonl \
            --output eval/results.json \
            --thresholds eval/thresholds.yaml
      - name: Comment PR with results
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('eval/results.json', 'utf8'));
            const summary = `## RAG Evaluation Results
            | Metric | Value | Threshold | Status |
            |--------|-------|-----------|--------|
            ${Object.entries(results.metrics).map(([k, v]) => 
              `| ${k} | ${v.value.toFixed(3)} | ${v.threshold} | ${v.passed ? '✅' : '❌'} |`
            ).join('\n')}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: summary
            });

The thresholds file keeps the contract explicit:

# eval/thresholds.yaml
hit_rate: 0.85
mrr: 0.72
faithfulness: 0.90
relevancy: 0.80
p95_latency_ms: 3000
tokens_per_query: 2500

Tradeoff: Strict thresholds catch regressions but create flaky failures when the judge model has variance. Start with warning-only thresholds (fail the job but don’t block merge) for the first two weeks. Tighten once you understand the noise floor.

Implement the evaluation runner

The runner loads your golden set, executes the pipeline, computes metrics, and compares against thresholds. Keep it deterministic: fix the random seed, pin the judge model version, disable any non-deterministic retriever behavior.

# eval/run_evaluation.py
import json
import yaml
import argparse
from pathlib import Path
from dataclasses import dataclass, asdict
from typing import List, Dict, Any

from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.evaluation import (
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    RetrieverEvaluator,
    EvaluationResult,
)
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

@dataclass
class MetricResult:
    name: str
    value: float
    threshold: float
    passed: bool

def load_golden_set(path: Path) -> List[Dict[str, Any]]:
    with open(path) as f:
        return [json.loads(line) for line in f]

def load_thresholds(path: Path) -> Dict[str, float]:
    with open(path) as f:
        return yaml.safe_load(f)

def build_pipeline() -> VectorStoreIndex:
    # Your actual pipeline construction logic
    # This should mirror production exactly
    Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
    Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.0)
    
    # Load your index from persistent storage
    # index = load_index_from_storage(...)
    # For demo, we'll create a dummy index
    from llama_index.core import Document
    docs = [Document(text="Sample document about fallback configuration.")]
    return VectorStoreIndex.from_documents(docs)

def evaluate_retrieval(index, golden_set: List[Dict]) -> Dict[str, float]:
    retriever = index.as_retriever(similarity_top_k=5)
    evaluator = RetrieverEvaluator(
        retriever=retriever,
        metric_names=["hit_rate", "mrr", "recall"],
    )
    
    # Format expected_ids for evaluator
    queries = [item["query"] for item in golden_set]
    expected_ids = [item["relevant_doc_ids"] for item in golden_set]
    
    results = evaluator.evaluate(queries=queries, expected_ids=expected_ids)
    return {
        "hit_rate": results.metric_vals_dict["hit_rate"],
        "mrr": results.metric_vals_dict["mrr"],
        "recall_at_5": results.metric_vals_dict["recall"],
    }

def evaluate_generation(index, golden_set: List[Dict]) -> Dict[str, float]:
    query_engine = index.as_query_engine(similarity_top_k=5)
    faithfulness = FaithfulnessEvaluator(llm=Settings.llm)
    relevancy = RelevancyEvaluator(llm=Settings.llm)
    
    faithfulness_scores = []
    relevancy_scores = []
    
    for item in golden_set:
        response = query_engine.query(item["query"])
        
        # Faithfulness: is the answer grounded in retrieved context?
        faith_result = faithfulness.evaluate_response(response=response)
        faithfulness_scores.append(1.0 if faith_result.passing else 0.0)
        
        # Relevancy: does the answer address the query?
        rel_result = relevancy.evaluate_response(
            query=item["query"], 
            response=response
        )
        relevancy_scores.append(1.0 if rel_result.passing else 0.0)
    
    return {
        "faithfulness": sum(faithfulness_scores) / len(faithfulness_scores),
        "relevancy": sum(relevancy_scores) / len(relevancy_scores),
    }

def measure_latency_and_cost(index, golden_set: List[Dict]) -> Dict[str, float]:
    import time
    query_engine = index.as_query_engine(similarity_top_k=5)
    
    latencies = []
    token_counts = []
    
    for item in golden_set:
        start = time.perf_counter()
        response = query_engine.query(item["query"])
        elapsed = (time.perf_counter() - start) * 1000  # ms
        
        latencies.append(elapsed)
        # Approximate token count from response metadata
        token_counts.append(
            response.metadata.get("token_count", len(str(response)) // 4)
        )
    
    latencies.sort()
    p95_idx = int(len(latencies) * 0.95)
    
    return {
        "p95_latency_ms": latencies[p95_idx],
        "avg_tokens_per_query": sum(token_counts) / len(token_counts),
    }

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--golden-set", required=True, type=Path)
    parser.add_argument("--thresholds", required=True, type=Path)
    parser.add_argument("--output", required=True, type=Path)
    args = parser.parse_args()
    
    golden_set = load_golden_set(args.golden_set)
    thresholds = load_thresholds(args.thresholds)
    
    index = build_pipeline()
    
    # Run all evaluations
    retrieval_metrics = evaluate_retrieval(index, golden_set)
    generation_metrics = evaluate_generation(index, golden_set)
    system_metrics = measure_latency_and_cost(index, golden_set)
    
    all_metrics = {**retrieval_metrics, **generation_metrics, **system_metrics}
    
    # Compare against thresholds
    results = []
    for name, value in all_metrics.items():
        threshold = thresholds.get(name, 0.0)
        passed = value >= threshold if name != "p95_latency_ms" else value <= threshold
        results.append(MetricResult(name, value, threshold, passed))
    
    # Write results
    output = {
        "metrics": {r.name: asdict(r) for r in results},
        "overall_passed": all(r.passed for r in results),
    }
    
    with open(args.output, "w") as f:
        json.dump(output, f, indent=2)
    
    # Exit code for CI
    if not output["overall_passed"]:
        print("REGRESSION DETECTED")
        for r in results:
            if not r.passed:
                print(f"  FAIL: {r.name}={r.value:.3f} (threshold: {r.threshold})")
        exit(1)
    
    print("ALL CHECKS PASSED")

if __name__ == "__main__":
    main()

Handle judge variance with statistical rigor

LLM-as-judge evaluators are noisy. A single run might show faithfulness at 0.88, the next at 0.92. This causes flaky CI failures. Two practical mitigations:

Run multiple times, take the median

def evaluate_with_repeats(evaluator, query, response, n=5):
    scores = []
    for _ in range(n):
        result = evaluator.evaluate_response(query=query, response=response)
        scores.append(1.0 if result.passing else 0.0)
    return sorted(scores)[n // 2]  # median

Use a cheaper, deterministic judge for CI, expensive judge for release

# CI: fast, deterministic
ci_judge = OpenAI(model="gpt-4o-mini", temperature=0.0, seed=42)

# Release: more accurate, higher cost
release_judge = OpenAI(model="gpt-4", temperature=0.0, seed=42)

Pitfall: Don’t set temperature=0 and assume determinism. OpenAI models still exhibit variance at temperature 0 due to batching and hardware non-determinism. The seed parameter helps but isn’t a guarantee. Design your thresholds with a 2-3% margin.

Track trends, not just pass/fail

A binary pass/fail tells you something broke. A trend tells you when and why. Store every evaluation run in a time-series database or even a committed JSON file:

{
  "timestamp": "2024-01-15T14:32:00Z",
  "commit": "a1b2c3d",
  "branch": "main",
  "metrics": {
    "hit_rate": 0.87,
    "mrr": 0.74,
    "faithfulness": 0.91,
    "relevancy": 0.83,
    "p95_latency_ms": 2100,
    "tokens_per_query": 1840
  },
  "pipeline_config": {
    "embedding_model": "text-embedding-3-small",
    "chunk_size": 512,
    "chunk_overlap": 50,
    "top_k": 5,
    "llm_model": "gpt-4o-mini"
  }
}

This lets you answer questions like: “Did hit_rate drop when we upgraded LlamaIndex from 0.10 to 0.11?” or “Which commit introduced the latency regression?”

**ToolStrip outlier detection on the time series catches gradual drift that threshold checks miss.

Automate dataset evolution

Your golden set will rot if you never update it. New product features generate new query patterns. User feedback reveals blind spots. Build a lightweight process:

1. Monthly review: Sample 20 production queries that got negative feedback. Add 5-10 to the golden set with expert answers.
2. Adversarial augmentation: When you find a failure mode (e.g., “queries with negations fail”), write 5-10 targeted examples.
3. Version the dataset: Tag releases (eval/v1.2.0.jsonl). This lets you bisect: “Did performance drop because the pipeline changed or because the dataset got harder?”

# Simple dataset versioning
git tag -a eval/v1.2.0 -m "Added 15 negation queries, updated 3 answers per user feedback"
git push origin eval/v1.2.0

Common pitfalls and how to avoid them

When to run what

Closing the loop

Regression testing only works if someone acts on the signal. Assign ownership: each metric has a named owner who gets paged on sustained regression. The evaluation dashboard should be visible to the whole team, not buried in CI logs.

Most importantly, treat the evaluation pipeline itself as production code. It needs tests, monitoring, and on-call rotation. If your evaluation is broken, you’re flying blind — and that’s a SEV-1 incident.