Retrieval-augmented generation fails quietly: the model answers fluently while ignoring or contradicting the retrieved context. LlamaIndex faithfulness evaluation metrics let you catch that drift programmatically by scoring whether a response is grounded in the source nodes. This guide shows how to wire those metrics into an existing pipeline and enforce them in CI.
Step 1: Install the evaluation dependencies
LlamaIndex ships the evaluation classes in the core package, but you still need an LLM client to act as the judge. Use the OpenAI LLM integration as the default path; it is the most tested backend for the evaluator.
pip install llama-index llama-index-llms-openaiConfirm the imports resolve before building anything else:
from llama_index import VectorStoreIndex, Document
from llama_index.evaluation import FaithfulnessEvaluator
from llama_index.llms.openai import OpenAIIf you are on a newer version where llama_index.evaluation is split, install llama-index-evaluation and import from llama_index.evaluation.faithfulness. The API surface is identical.
Step 2: Index a small corpus and create a query engine
You need a working RAG pipeline before you can evaluate it. The snippet below builds an in-memory vector index from two sentences and exposes a query engine with a top-k of 2.
docs = [
Document(text="RAG systems retrieve context before generation. Faithfulness measures grounding in that context."),
Document(text="LlamaIndex provides evaluators that use an LLM judge to score responses."),
]
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(similarity_top_k=2)Keep the corpus tiny during local development. A small set makes it obvious when the evaluator flags a hallucination, because you can read both the source nodes and the generated answer.
Step 3: Configure the judge model for the evaluator
The FaithfulnessEvaluator does not compute a heuristic; it prompts an LLM to compare the response against the retrieved nodes. Instantiate it with an explicit LLM to avoid silent defaults.
judge_llm = OpenAI(model="gpt-4o")
evaluator = FaithfulnessEvaluator(llm=judge_llm)If you route the judge model through an OpenAI-compatible gateway such as n4n.ai, you get automatic fallback across providers and per-token metering without changing the LlamaIndex call site. Point the underlying client at the gateway:
from openai import OpenAI as OpenAIClient
judge_llm = OpenAI(
model="gpt-4o",
client=OpenAIClient(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
)
evaluator = FaithfulnessEvaluator(llm=judge_llm)The evaluator only cares about the llm interface, so any compliant endpoint works.
Step 4: Run a single faithfulness evaluation
Query the engine and pass the entire response object to the evaluator. The response carries the source nodes, which the judge needs to check grounding.
response = query_engine.query("What does faithfulness measure?")
eval_result = evaluator.evaluate_response(response=response)
print("Passing:", eval_result.passing)
print("Score:", eval_result.score)
print("Feedback:", eval_result.feedback)The LlamaIndex faithfulness evaluation metrics returned here are simple: score is a float (typically 1.0 or 0.0), passing is a boolean derived from a threshold (default 1.0), and feedback is a natural-language explanation from the judge. A score of 1.0 means every claim in the answer is supported by the retrieved context.
Step 5: Batch-evaluate a question set
A single check proves the wiring; a batch run gives you a distribution. Define a list of representative questions and loop:
eval_questions = [
"What does faithfulness measure?",
"Does LlamaIndex provide evaluators?",
"Why use a judge model?",
]
records = []
for q in eval_questions:
resp = query_engine.query(q)
res = evaluator.evaluate_response(response=resp)
records.append({"question": q, "score": res.score, "passing": res.passing})
mean_score = sum(r["score"] for r in records) / len(records)
print(f"Mean faithfulness: {mean_score}")For larger corpora, use evaluate_response_async inside an asyncio event loop to parallelize judge calls. The metrics themselves do not change; only the throughput does.
Step 6: Interpret LlamaIndex faithfulness evaluation metrics
A high score does not mean the answer is correct, only that it stays within the retrieved text. If the retriever pulls the wrong nodes, a faithful answer can still be useless. Treat faithfulness as a necessary but insufficient condition.
Common failure modes
- Score 0 with plausible answer: The judge found a phrase not in the nodes. Often the LLM rephrased a date or number. Read
feedbackto decide if it is a false positive. - Score 1 but user unhappy: Retrieval missed the relevant node. Faithfulness cannot catch missing context, only extraneous claims.
- Empty source nodes: If you build the response manually without
response.source_nodes, the evaluator has nothing to check and will error or default to 0.
When LlamaIndex faithfulness evaluation metrics drop after a prompt change, the prompt is likely encouraging the model to elaborate beyond the context. Tighten the system prompt with “answer only from the provided context.”
Step 7: Enforce faithfulness in continuous integration
Run the batch evaluation inside a pytest session and fail the build if the mean score drops below a threshold you trust. Start conservative at 0.9; raise it as the pipeline matures.
def test_mean_faithfulness():
scores = [r["score"] for r in records]
mean = sum(scores) / len(scores)
assert mean >= 0.9, f"Mean faithfulness {mean} below threshold"Commit the question set alongside the test. When a teammate changes the index or the prompt, CI will surface regressions before they reach production.
Verifying success
After Step 4, you should see Passing: True and Score: 1.0 for the grounded query about faithfulness. Running the Step 5 loop should print a mean faithfulness of 1.0 on the toy corpus. If you execute the pytest from Step 7, the test passes. Any deviation means either the judge model is misconfigured or the response object lacks source nodes—inspect response.source_nodes first.