LlamaIndex FaithfulnessEvaluator vs RelevancyEvaluator

When you search for llamaindex faithfulnessevaluator vs relevancyevaluator, you’re usually trying to decide which metric tells you whether your RAG pipeline is actually working. The short answer: they measure orthogonal things. FaithfulnessEvaluator catches hallucinations in the generated answer. RelevancyEvaluator catches garbage in the retrieved context. You need both, but at different stages of your eval pipeline.

What each evaluator actually measures

FaithfulnessEvaluator answers: “Did the model make up facts not in the provided context?” It takes a query, retrieved context, and generated response, then asks an LLM judge to verify every claim in the response against the context. A score of 1.0 means every statement is grounded; 0.0 means pure hallucination.

RelevancyEvaluator answers: “Is the retrieved context actually useful for answering the query?” It takes the query and retrieved context (no response needed), then asks an LLM judge whether the context contains information relevant to the query. A score of 1.0 means the context directly addresses the question; 0.0 means the retriever returned noise.

These are not interchangeable. A pipeline can have perfect faithfulness (the generator only says what’s in context) but zero relevancy (the context is irrelevant, so the answer is useless). Conversely, highly relevant context with low faithfulness means your generator is ignoring good evidence and hallucinating instead.

How they work under the hood

Both evaluators use the LLM-as-judge pattern. You provide a judge model (default: gpt-4 or gpt-3.5-turbo via OpenAI), and the evaluator constructs a structured prompt asking the judge to score the input.

FaithfulnessEvaluator’s prompt roughly decomposes the response into atomic claims, then checks each claim against the context. The implementation in llama_index.core.evaluation.faithfulness uses a FaithfulnessPromptTemplate that instructs the judge to output a score and reasoning.

from llama_index.core.evaluation import FaithfulnessEvaluator

evaluator = FaithfulnessEvaluator(llm=judge_llm)
result = evaluator.evaluate(
    query="What is the capital of France?",
    response="The capital of France is Paris.",
    contexts=["Paris is the capital city of France."]
)
print(result.score)  # 1.0
print(result.feedback)  # "All claims supported by context."

RelevancyEvaluator’s prompt asks the judge to assess whether the context contains information that helps answer the query. It lives in llama_index.core.evaluation.relevancy.

from llama_index.core.evaluation import RelevancyEvaluator

evaluator = RelevancyEvaluator(llm=judge_llm)
result = evaluator.evaluate(
    query="What is the capital of France?",
    contexts=["Paris is the capital city of France.", "Lyon is a city in France."]
)
print(result.score)  # 1.0
print(result.feedback)  # "Context directly answers the query."

Both return an EvaluationResult with score (float 0-1), passing (bool against threshold), and feedback (string reasoning). You can customize the threshold via threshold parameter (default 0.5 for both).

Comparison table

Cost and latency in practice

Cost is dominated by the judge model, not the evaluator class. With gpt-4o-mini as judge, expect roughly 1-2k input tokens per faithfulness eval (response + context + prompt) and 500-1k for relevancy. At $0.15/1M input tokens, that’s ~$0.0002-0.0003 per faithfulness eval and ~$0.0001 per relevancy eval. gpt-4o runs 10-15x more expensive.

Latency scales with context length and judge model. FaithfulnessEvaluator takes longer because the prompt includes the full response and the judge must decompose claims. In my experience with gpt-4o-mini, faithfulness evals average 1.2s; relevancy averages 0.7s. Local judges (Ollama, vLLM) add 2-5x latency but eliminate per-token cost.

If you’re evaluating thousands of samples, batch and async are essential:

# Batch sync
results = evaluator.evaluate_batch(queries, responses, contexts_list)

# Async (recommended for production eval pipelines)
results = await evaluator.aevaluate_batch(queries, responses, contexts_list)

Ergonomics and API differences

The APIs are nearly identical by design — both inherit from BaseEvaluator. The only required difference is the input signature. FaithfulnessEvaluator requires response; RelevancyEvaluator does not.

Both support custom prompt templates via the prompt_template parameter. This matters when the default criteria don’t match your domain. For example, medical RAG might need stricter faithfulness (no paraphrasing allowed) or domain-specific relevancy (context must cite specific guidelines).

from llama_index.core.prompts import PromptTemplate

custom_faithfulness = PromptTemplate(
    "You are a medical fact-checker. "
    "A claim passes ONLY if it appears verbatim in context. "
    "Context: {context_str}\n"
    "Response: {response_str}\n"
    "Score (0-1): "
)
evaluator = FaithfulnessEvaluator(llm=judge_llm, prompt_template=custom_faithfulness)

Both evaluators integrate with llama_index.core.evaluation.eval_utils for dataset-level aggregation (mean score, pass rate, confidence intervals). They also work with LlamaIndexCallbackHandler for tracing.

Common failure modes

FaithfulnessEvaluator struggles with:

• Paraphrase sensitivity: Default prompt often penalizes valid paraphrasing. If context says “Paris is France’s capital” and response says “France’s capital city is Paris,” some judges score <1.0.
• Implicit knowledge: The judge may flag “The Eiffel Tower is in Paris” as unfaithful if context only mentions “Paris is France’s capital,” even though this is world knowledge the model reasonably knows.
• Claim decomposition errors: Long responses with compound sentences sometimes get scored as a single claim, masking partial hallucination.

RelevancyEvaluator struggles with:

• Topical overlap ≠ utility: Context about “Paris tourism” scores high for “capital of France” query because of entity overlap, but doesn’t answer the question.
• Length bias: Longer contexts tend to score higher simply by covering more tokens, even if the relevant snippet is buried.
• No negative signal: It doesn’t penalize missing critical information — only rewards presence of relevant info.

Mitigation: run both evaluators on a held-out set, inspect failures manually, then tune prompts or thresholds. Don’t trust raw scores blindly.

Which to choose

Use FaithfulnessEvaluator when:

• You’re tuning generator prompts, temperature, or model selection. It tells you whether the generator respects its context.
• You’re comparing RAG vs. non-RAG outputs. Faithfulness isolates the grounding property.
• You’re doing regression testing on generator changes. A drop in faithfulness means your new prompt breaks grounding.

Use RelevancyEvaluator when:

• You’re tuning retrieval: chunk size, top-k, embedding model, reranker, hybrid search weights. It tells you whether the retriever finds useful evidence.
• You’re debugging “I don’t know” responses. Low relevancy explains why the generator abstained.
• You’re comparing retriever architectures (vector vs. BM25 vs. hybrid). Relevancy is the direct quality signal.

Use both in CI/CD:

• Gate merges on faithfulness ≥ 0.9 AND relevancy ≥ 0.7 (adjust thresholds per domain).
• Track them separately. A faithfulness regression with stable relevancy = generator issue. A relevancy regression with stable faithfulness = retriever issue.
• Log the feedback strings. They’re more actionable than scores for root-cause analysis.

Skip both when:

• You’re evaluating end-to-end answer correctness against ground truth. Use CorrectnessEvaluator or SemanticSimilarityEvaluator with reference answers instead.
• You need token-level attribution. Use ContextRelevancyEvaluator (different class) or citation-based metrics.

The llamaindex faithfulnessevaluator vs relevancyevaluator distinction matters because they catch different bugs. Run both. Automate them. Read the feedback. That’s how you ship RAG that doesn’t embarrass you in production.