Correctness evaluation is the difference between a RAG system that looks impressive in demos and one you can trust in production. LlamaIndex provides a structured evaluation framework, but the documentation leaves the hard parts — designing criteria, handling edge cases, interpreting scores — as exercises for the reader. This guide walks through a practical evaluation pipeline you can drop into a CI job or run ad-hoc against a golden dataset.
What correctness means for RAG
In a retrieval-augmented system, correctness has two dimensions: faithfulness (does the answer only use information from the retrieved context?) and accuracy (is the answer factually correct given that context?). LlamaIndex’s CorrectnessEvaluator focuses on accuracy by comparing a generated answer against a reference answer or expected facts. It does not measure whether the retriever found the right documents — that’s a separate retrieval evaluation problem.
The evaluator uses an LLM as a judge. You provide the query, the generated response, and a reference answer (or a set of expected facts). The judge returns a score and reasoning. This sounds simple. The complexity lives in designing reference answers that don’t bake in brittleness, choosing a judge model that’s actually capable of the comparison, and interpreting the output in a way that drives iteration.
Setting up the evaluation environment
Install the evaluation extras and configure your judge model. I recommend using a different model for evaluation than for generation — ideally a stronger model — to avoid the generator grading its own homework.
pip install "llama-index-core" "llama-index-llms-openai" "llama-index-evaluation"from llama_index.core.evaluation import CorrectnessEvaluator
from llama_index.llms.openai import OpenAI
# Use a capable model for judging. GPT-4o or Claude 3.5 Sonnet work well.
judge_llm = OpenAI(model="gpt-4o", temperature=0.0)
evaluator = CorrectnessEvaluator(llm=judge_llm)If you’re routing through a gateway that supports multiple providers, you can swap the judge model without changing evaluation logic. For example, n4n.ai exposes 240+ models behind one OpenAI-compatible endpoint, so you can test whether a cheaper judge model correlates with your preferred one before committing budget.
Building a golden dataset
Evaluation is only as good as your test cases. A golden dataset for correctness needs three columns: the query, the expected answer (or key facts), and optionally the retrieved context if you want to evaluate faithfulness separately.
from dataclasses import dataclass
from typing import List
@dataclass
class GoldenCase:
query: str
expected_answer: str
# Optional: key facts that must appear for correctness
required_facts: List[str] = None
# Optional: context to check faithfulness against
context: str = None
GOLDEN_SET = [
GoldenCase(
query="What is the capital of France?",
expected_answer="Paris is the capital of France.",
required_facts=["Paris", "capital of France"]
),
GoldenCase(
query="How does the RAG pipeline handle conflicting sources?",
expected_answer=(
"The pipeline ranks sources by recency and authority, "
"then synthesizes a response that acknowledges conflicts "
"rather than picking a single winner."
),
required_facts=["recency", "authority", "acknowledges conflicts"]
),
# Add 50-200 cases covering your domain
]Pitfall: Writing reference answers that are too specific. If your expected answer says “The pipeline uses a cross-encoder reranker with a threshold of 0.7” but the system changes to 0.75, the evaluator will mark a correct answer as wrong. Prefer required facts over full reference answers, or use semantic similarity as a secondary signal.
Running the evaluator
LlamaIndex’s CorrectnessEvaluator accepts a query, response, and reference answer. It returns an EvaluationResult with a score (1-5 by default), a passing boolean, and feedback.
from llama_index.core.evaluation import EvaluationResult
def evaluate_case(case: GoldenCase, generated_response: str) -> EvaluationResult:
# Use required_facts as the reference if provided, else expected_answer
reference = case.expected_answer
if case.required_facts:
reference = "The answer must contain: " + "; ".join(case.required_facts)
result = evaluator.evaluate(
query=case.query,
response=generated_response,
reference=reference
)
return result
# Example usage
generated = "Paris is the capital city of France, located in the north-central part of the country."
result = evaluate_case(GOLDEN_SET[0], generated)
print(f"Score: {result.score}") # 1-5
print(f"Passing: {result.passing}") # True/False (threshold >= 4 by default)
print(f"Feedback: {result.feedback}") # Judge's reasoningThe default threshold (score >= 4) is aggressive. For production gating, I lower it to 3 and treat the score as a continuous signal rather than a binary gate.
Customizing the evaluation prompt
The default prompt asks the judge to score 1-5 based on “correctness and completeness.” That’s vague. You can inject domain-specific criteria by subclassing or passing a custom prompt template.
from llama_index.core.evaluation import CorrectnessEvaluator
from llama_index.core.prompts import PromptTemplate
CUSTOM_CORRECTNESS_TEMPLATE = PromptTemplate(
"You are an expert evaluator for a technical documentation RAG system.\n"
"Query: {query_str}\n"
"Generated Answer: {response_str}\n"
"Reference Answer: {reference_str}\n\n"
"Evaluate the generated answer on these criteria:\n"
"1. Factual accuracy: Does it match the reference?\n"
"2. Completeness: Does it cover all key points from the reference?\n"
"3. No hallucination: Does it avoid adding unsupported claims?\n"
"4. Conciseness: Is it free of fluff?\n\n"
"Score 1-5 where:\n"
"5 = Perfect on all criteria\n"
"4 = Minor omission or verbosity\n"
"3 = Missing one key point or minor inaccuracy\n"
"2 = Multiple inaccuracies or major omission\n"
"1 = Fundamentally wrong\n\n"
"Provide your score and brief reasoning.\n"
"Score: {{score}}\nReasoning: {{reasoning}}"
)
custom_evaluator = CorrectnessEvaluator(
llm=judge_llm,
eval_template=CUSTOM_CORRECTNESS_TEMPLATE,
score_threshold=3.0
)Tradeoff: Custom prompts improve alignment with your definition of correctness but make it harder to compare scores across projects or teams. Keep a baseline evaluator with the default prompt for cross-project benchmarking.
Batch evaluation and aggregation
Run the evaluator across your golden set and aggregate results. This is where you find systemic issues — consistent failures on a query type, or a judge that’s too lenient/harsh.
from concurrent.futures import ThreadPoolExecutor, as_completed
from statistics import mean
from typing import Dict, List
def run_batch_evaluation(
cases: List[GoldenCase],
generate_fn, # Your RAG pipeline: query -> response
evaluator: CorrectnessEvaluator,
max_workers: int = 5
) -> Dict:
results = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_case = {
executor.submit(evaluate_case, case, generate_fn(case.query)): case
for case in cases
}
for future in as_completed(future_to_case):
case = future_to_case[future]
try:
result = future.result()
results.append({
"query": case.query,
"score": result.score,
"passing": result.passing,
"feedback": result.feedback
})
except Exception as e:
results.append({
"query": case.query,
"error": str(e)
})
scores = [r["score"] for r in results if "score" in r]
passing_rate = sum(1 for r in results if r.get("passing")) / len(results)
return {
"mean_score": mean(scores) if scores else 0,
"passing_rate": passing_rate,
"per_case": results
}Pitfall: Running evaluation sequentially is slow. The thread pool above helps, but beware of rate limits on your judge model. If you hit 429s, add exponential backoff or use a gateway with automatic fallback — n4n.ai handles provider degradation transparently, which keeps evaluation runs from stalling mid-batch.
Interpreting results: beyond the mean score
A mean score of 4.2 tells you little. Look at the distribution and the failures.
def analyze_results(batch_result: Dict):
per_case = batch_result["per_case"]
# Score histogram
from collections import Counter
score_dist = Counter(r["score"] for r in per_case if "score" in r)
print("Score distribution:", dict(sorted(score_dist.items())))
# Worst cases — investigate these first
worst = sorted(
[r for r in per_case if "score" in r],
key=lambda x: x["score"]
)[:5]
print("\nWorst cases:")
for r in worst:
print(f" Score {r['score']}: {r['query'][:80]}...")
print(f" Feedback: {r['feedback'][:200]}")
# Check for judge inconsistency: same query type, wildly different scores
# (requires tagging cases with categories in your golden set)Common patterns to watch for:
- Consistent 3s on procedural queries: Your reference answers may be too terse, or the generator omits steps the judge considers essential.
- Bimodal distribution (1s and 5s): The judge is confused by the prompt, or your test cases mix trivial and impossible queries.
- High passing rate but low mean: The threshold is too low. Raise it or fix the generator.
Evaluating faithfulness alongside correctness
Correctness without faithfulness is hallucination with a citation. LlamaIndex provides FaithfulnessEvaluator to check whether the answer stays grounded in the retrieved context.
from llama_index.core.evaluation import FaithfulnessEvaluator
faithfulness_evaluator = FaithfulnessEvaluator(llm=judge_llm)
def evaluate_faithfulness(case: GoldenCase, generated_response: str) -> EvaluationResult:
if not case.context:
return None # Skip if no context captured
return faithfulness_evaluator.evaluate(
query=case.query,
response=generated_response,
contexts=[case.context]
)Run both evaluators in your batch job. A case that passes correctness but fails faithfulness means your generator is “lucky” — it got the right answer for the wrong reasons. That’s a retrieval or prompt issue, not a generation issue.
CI integration: gating on regression
Add a step that fails the build if correctness drops below a baseline. Store the baseline in a file committed to the repo.
# eval_gate.py
import json
import sys
BASELINE_FILE = "eval_baseline.json"
def load_baseline():
with open(BASELINE_FILE) as f:
return json.load(f)
def save_baseline(metrics: Dict):
with open(BASELINE_FILE, "w") as f:
json.dump(metrics, f, indent=2)
def main():
# ... run batch_evaluation ...
current = run_batch_evaluation(GOLDEN_SET, rag_pipeline.query, custom_evaluator)
baseline = load_baseline()
# Gate: mean score must not drop more than 0.2
if current["mean_score"] < baseline["mean_score"] - 0.2:
print(f"REGRESSION: mean score {current['mean_score']:.2f} < baseline {baseline['mean_score']:.2f} - 0.2")
sys.exit(1)
# Gate: passing rate must not drop more than 5%
if current["passing_rate"] < baseline["passing_rate"] - 0.05:
print(f"REGRESSION: passing rate {current['passing_rate']:.2%} < baseline {baseline['passing_rate']:.2%} - 5%")
sys.exit(1)
print("Evaluation gates passed.")
# Optionally update baseline on green builds
# save_baseline(current)
if __name__ == "__main__":
main()Tradeoff: Hard gates catch regressions but create flakiness if the judge model is non-deterministic (temperature > 0) or if the golden set has ambiguous cases. Run the gate on a deterministic judge (temperature=0) and audit failures manually before blocking merges.
Common pitfalls and how to avoid them
1. Judge model too weak: A 7B model cannot reliably evaluate nuanced technical answers. Use a frontier model for evaluation even if you serve a smaller model in production. The cost is negligible — evaluation runs are infrequent.
2. Reference answers encode implementation details: “The system uses Pinecone with cosine similarity” will fail when you switch to Weaviate. Reference behavior, not implementation.
3. No negative cases: Your golden set should include queries where the correct answer is “I don’t know” or “The context doesn’t contain this.” Otherwise you only measure precision, not recall of uncertainty.
4. Single-judge dependency: LLM judges have biases (verbosity, hedging, style). Periodically run a second judge model and check correlation. If they disagree systematically, your prompt is underspecified.
5. Evaluating stale context: If your golden set captures context at test creation time, but the underlying documents change, faithfulness scores become meaningless. Version your context snapshots or regenerate them before each evaluation run.
Scaling the evaluation loop
Once the basic pipeline works, you’ll want:
- Slice analysis: Tag cases by category (factual, procedural, comparative, adversarial) and track metrics per slice.
- Adversarial generation: Use an LLM to generate edge-case queries from your corpus, then human-verify a subset for the golden set.
- Online evaluation: Log production queries and responses, sample for human annotation, and feed back into the golden set. This closes the loop between offline eval and real user satisfaction.
The evaluation code you write today will outlive the RAG pipeline it tests. Invest in the golden set, keep the evaluator configurable, and treat the judge model as a dependency you can swap. That’s how you build a system that gets better instead of just getting bigger.