Catching hallucinations in LlamaIndex responses

RAG systems built with LlamaIndex fail silently when the retriever returns weak context and the LLM fills gaps with confident fiction. Detecting hallucinations in LlamaIndex RAG pipelines requires more than eyeballing responses—you need programmatic checks at multiple stages. This guide gives an ordered path from instrumentation to automated evaluation that you can drop into a CI loop.

1. Instrument the pipeline before you can catch anything

You cannot measure what you do not log. LlamaIndex exposes a callback system that captures every retriever call, prompt, and completion. Wire it up in your test harness:

from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager, LlamaDebugHandler

debug_handler = LlamaDebugHandler()
Settings.callback_manager = CallbackManager([debug_handler])

# after a query
response = query_engine.query("What is the refund policy?")
events = debug_handler.get_events()

The events object holds the exact nodes retrieved and the final prompt sent to the model. Run this only in development and CI. Attaching a debug handler on a production hot path adds latency and memory overhead you do not want in a live request.

A common mistake is to log only the final answer. The retrieved source_nodes are the ground truth you will later compare against, so make sure they are persisted with a correlation ID.

2. Separate retrieval quality from generation quality

Detecting hallucinations in LlamaIndex RAG often starts with confirming the answer is grounded in retrieved nodes. The Response object gives you direct access:

response = query_engine.query("Does plan X include VPN?")
context_text = "\n".join(n.get_content() for n in response.source_nodes)
answer = str(response)

import re
answer_tokens = set(re.findall(r"\b\w+\b", answer.lower()))
context_tokens = set(re.findall(r"\b\w+\b", context_text.lower()))
missing = answer_tokens - context_tokens

A naive token-difference check flags blatant fabrications (e.g., the model says “Yes, plan X includes VPN” but no node mentions VPN). It will produce false positives on synonyms and paraphrases, so treat it as a smoke test, not a verdict.

For a stronger signal, embed each answer sentence and each source node, then compute cosine similarity. If the best match for any answer sentence is below a threshold (say 0.65 with a decent embedding model), that sentence is likely ungrounded.

3. Add a faithfulness evaluator

LlamaIndex ships a FaithfulnessEvaluator that uses an LLM to judge whether the response is faithful to the provided context. This is the core of automated detecting hallucinations in LlamaIndex RAG:

from llama_index.core.evaluation import FaithfulnessEvaluator
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini", temperature=0)
evaluator = FaithfulnessEvaluator(llm=llm)

result = evaluator.evaluate_response(
    query="Does plan X include VPN?",
    response=response,
)
print(result.passing, result.feedback)

The evaluator returns a boolean passing and a natural-language feedback string explaining its reasoning. Use a smaller, cheaper model for the judge to keep cost down; faithfulness grading does not require frontier reasoning.

Tradeoff: the judge is itself an LLM and can hallucinate its judgment. Mitigate by setting temperature to 0 and by spot-checking its feedback during the first weeks of use. If the judge contradicts the retrieved text, trust the text.

4. Use answer relevancy and correctness metrics

Faithfulness only tells you the answer is grounded—not that it answers the question. Add an AnswerRelevancyEvaluator:

from llama_index.core.evaluation import AnswerRelevancyEvaluator

relevancy_eval = AnswerRelevancyEvaluator(llm=llm)
res = relevancy_eval.evaluate_response(query=query, response=response)

For reference-based testing, maintain a small gold set of question/answer pairs. Compare the generated answer to the expected one with semantic similarity (embedding distance) or a strict substring assertion for non-negotiable facts like IDs or dates.

Do not skip human-labeled correctness entirely. Faithfulness and relevancy are reference-free and will both pass on a confidently wrong but well-grounded answer if your retriever pulled the wrong document.

5. Build a regression suite with fixed queries

Turn the checks above into a runnable suite. Store cases as JSONL:

{"query": "Refund policy?", "must_mention": ["30 days", "original payment"]}
{"query": "Plan X VPN?", "must_mention": ["VPN"], "forbid": ["unlimited bandwidth"]}

Then iterate:

import json

with open("eval_set.jsonl") as f:
    cases = [json.loads(l) for l in f]

for c in cases:
    resp = query_engine.query(c["query"])
    ev = evaluator.evaluate_response(query=c["query"], response=resp)
    text = str(resp).lower()
    for fact in c.get("must_mention", []):
        assert fact.lower() in text, f"Missing expected fact: {fact}"
    for bad in c.get("forbid", []):
        assert bad.lower() not in text, f"Forbidden term present: {bad}"
    assert ev.passing, ev.feedback

Run this in CI on every index change. The must_mention and forbid lists catch regressions that a fuzzy similarity metric misses.

6. Scale eval without blowing up cost or latency

Running a faithfulness judge on every query in a suite of 500 cases means 500 extra LLM calls per run. When you run detecting hallucinations in LlamaIndex RAG at scale, each judge call adds tokens and API surface. Routing eval traffic through a gateway like n4n.ai that provides per-token usage metering and automatic fallback keeps the suite resilient when a provider is rate-limited or degraded, and lets you attribute cost per test run.

Cache embeddings for source nodes between runs so the grounding similarity step does not re-embed the same corpus nightly. Use gpt-4o-mini or a local model for the judge; reserve larger models for the application itself.

7. Common pitfalls and tradeoffs

• Strict string matching false positives. Synonyms (“VPN” vs “virtual private network”) trip must_mention asserts. Prefer semantic checks for open text, reserve exact match for codes and dates.
• Judge bias toward verbosity. LLM judges often rate longer answers as more faithful. Keep answer length bounded in the app.
• Context truncation. If your retriever returns nodes that exceed the context window after packing, the model may answer from truncated text and the evaluator will wrongly flag hallucination. Log token counts of the final prompt.
• Latency and cost. Evaluation roughly doubles inference spend. Run full suites nightly, not on every commit.
• Retrieval recall gap. Faithfulness passing does not mean the right document was retrieved—only that the model stuck to what it was given. Pair with offline recall metrics on a labeled set.

8. Ordered checklist

1. Enable LlamaDebugHandler in test/CI environments and persist source_nodes.
2. Compute a grounding heuristic (token overlap or embedding similarity) as a smoke test.
3. Wrap queries with FaithfulnessEvaluator using a cheap, zero-temp judge model.
4. Add AnswerRelevancyEvaluator and a human-labeled gold set for correctness.
5. Encode fixed queries with must_mention/forbid constraints in a JSONL regression file.
6. Run the suite nightly, track token cost per run, and alert on any assertion failure.

Follow this order and you will catch most hallucinations before they reach users, instead of discovering them in support tickets.