LlamaIndex RelevancyEvaluator explained

Retrieval quality silently dictates the ceiling of any RAG system. This llamaindex relevancyevaluator tutorial walks through how LlamaIndex’s RelevancyEvaluator uses an LLM judge to score whether retrieved passages actually match a query, and how to wire it into a real pipeline without guesswork.

What RelevancyEvaluator Actually Measures

RelevancyEvaluator is not a semantic similarity metric. It prompts an LLM with the user query and one or more retrieved context strings, then asks the model to decide if the context is relevant enough to answer the query. The output is a binary pass/fail plus a short rationale.

It evaluates the retriever, not the generator. If you feed it the final response text instead of source nodes, you are using the wrong tool. Use ResponseEvaluator or FaithfulnessEvaluator for answer-level checks.

The judge model receives a fixed prompt template. In LlamaIndex the default asks: “Given the query and the context, is the context relevant? Answer with yes or no.” The model’s completion is parsed for a boolean.

Install and Import

Use LlamaIndex core plus an LLM provider. The evaluator itself ships inside the core package.

pip install llama-index-core llama-index-llms-openai

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai import OpenAI
from llama_index.evaluation.relevancy import RelevancyEvaluator

If you run an older version (pre-0.10), the import path was llama_index.evaluation.RelevancyEvaluator. The namespace moved; the class behavior is stable.

Build a Retrieval Pipeline

Load documents, build an index, and expose a retriever. The evaluator needs raw text contexts, so pull nodes from the retriever directly.

docs = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(docs)
retriever = index.as_retriever(similarity_top_k=3)

query = "What is the evacuation procedure for building 4?"
nodes = retriever.retrieve(query)
contexts = [n.get_content() for n in nodes]

Keep similarity_top_k small for evaluation. Measuring relevancy on ten passages dilutes signal and multiplies judge calls.

Run the Evaluator

Instantiate the evaluator with a judge LLM. A smaller model like gpt-4o-mini is usually sufficient and cuts cost.

judge_llm = OpenAI(model="gpt-4o-mini")
evaluator = RelevancyEvaluator(llm=judge_llm)

result = evaluator.evaluate(query=query, contexts=contexts)
print(result.score, result.feedback, result.passing)

evaluate is synchronous. For datasets, use the batch runner (covered below). The call returns an EvaluationResult object.

Output Schema and Interpretation

The EvaluationResult has three fields you care about:

class EvaluationResult:
    query: str
    contexts: List[str]
    score: float        # 1.0 if relevant, 0.0 otherwise
    feedback: str       # free-text rationale from judge
    passing: bool       # score >= threshold (default 1.0)

A single score of 1.0 means the judge deemed all provided contexts relevant. If any context is irrelevant, the default parser flips the whole sample to 0.0. That all-or-nothing behavior surprises engineers who expect per-context scores.

To get per-node signal, loop and call evaluate on each context individually:

for ctx in contexts:
    r = evaluator.evaluate(query=query, contexts=[ctx])
    print(r.score, r.feedback[:80])

Choose a Judge Model

The judge is itself an LLM, so its biases propagate. A weak judge mislabels borderline contexts. A frontier model improves accuracy but adds latency and cost per evaluation.

If you want to avoid vendor lock or handle rate limits, point the judge LLM at an OpenAI-compatible gateway such as n4n.ai, which exposes one endpoint for 240+ models with automatic fallback when a provider degrades. The LlamaIndex OpenAI class accepts a base_url and api_key, so the swap is one line.

judge_llm = OpenAI(
    model="anthropic/claude-3-haiku",
    base_url="https://api.n4n.ai/v1",
    api_key="sk-...",
)

Tradeoff: larger context windows help when your retrieved passages are long, but token cost scales linearly. Start with a haiku-class model and only upgrade when manual spot-checks show systematic false negatives.

Batch Evaluation with BatchEvalRunner

Running evaluations one query at a time is fine for debugging. For regression testing, use BatchEvalRunner.

from llama_index.evaluation import BatchEvalRunner

runner = BatchEvalRunner(
    {"relevancy": evaluator},
    workers=4,
)

eval_dataset = [
    {"query": "What is the evacuation procedure for building 4?",
     "contexts": contexts},
    {"query": "Who approves external vendor contracts?",
     "contexts": retriever.retrieve("Who approves external vendor contracts?")},
]

results = runner.evaluate(eval_dataset)

workers controls concurrency. Set it to match your provider’s RPM limit; exceeding it triggers 429s that the runner does not retry by default.

Common Pitfalls

Binary Scoring Hides Nuance

A context that partially answers the query but misses a key constraint gets the same 0.0 as pure noise. If your product needs graded relevance, subclass RelevancyEvaluator and modify the prompt to return a 1–5 score, then parse accordingly.

Context Truncation

The judge prompt concatenates all contexts. With similarity_top_k=5 and 1k-token chunks, you can blow past the judge’s context window. LlamaIndex does not auto-truncate; the API call fails or the model ignores later contexts. Cap max_tokens on retrieval or slice contexts before evaluation.

Judge Bias and Prompt Leakage

The default template is terse. Judges sometimes say “yes” when the context merely contains the same keywords as the query but no actual answer. Mitigate by appending “The context must contain information that directly answers the query, not just related terms” to the evaluator’s prompt via the eval_template argument.

Cost and Latency

Every evaluated query is at least one LLM call. A 10k-query regression set at $0.0001 per call is still $1 and minutes of runtime. Cache judge responses keyed by (model, query, hash(contexts)) to avoid recomputation during iterative prompt tuning.

Integrating Into CI

Store a golden query set as JSON. Run the batch evaluator in a GitHub Action step. Fail the build if mean relevancy drops below a threshold.

[
  {"query": "VPN setup for contractors", "expected_min_score": 1.0},
  {"query": "Payroll cut-off dates", "expected_min_score": 1.0}
]

import json
with open("golden.json") as f:
    golden = json.load(f)

dataset = []
for item in golden:
    ctxs = [n.get_content() for n in retriever.retrieve(item["query"])]
    dataset.append({"query": item["query"], "contexts": ctxs})

out = runner.evaluate(dataset)
mean_score = sum(r["relevancy"].score for r in out) / len(out)
assert mean_score >= 0.9, f"Relevancy regression: {mean_score}"

This catches retriever drift when you change chunk size or embedding model.

Complementary Metrics

Relevancy alone misses recall. Pair it with:

• Hit Rate / Recall@k: fraction of queries where at least one relevant node appears in top-k. Computable without an LLM using labeled data.
• FaithfulnessEvaluator: checks if the generated answer is grounded in retrieved contexts.
• Context Relevance via Embeddings: cheap cosine similarity as a pre-filter before the LLM judge.

Use the LLM judge where labeled data is sparse; use deterministic metrics where you have ground truth.

Final Recommendations

Start with gpt-4o-mini as judge, similarity_top_k=3, and per-context evaluation to get clean signal. Log the feedback field—it is the fastest way to spot prompt leakage. Wire the batch runner into CI before you tune embeddings, not after. The llamaindex relevancyevaluator tutorial pattern above is enough to block most retriever regressions in a afternoon.

When you outgrow binary scoring, fork the prompt and parse graded output. The evaluator is a thin wrapper around an LLM call; treat it as configurable infrastructure, not a black box.