Measure faithfulness in LlamaIndex RAG with n4n.ai

Faithfulness evaluation tells you whether your RAG system hallucinates or sticks to retrieved context. This llamaindex faithfulness evaluation n4n.ai tutorial walks through building a minimal RAG pipeline, generating test questions, running the faithfulness metric, and interpreting scores so you can iterate with confidence.

Step 1: Set up environment and dependencies

Create a fresh virtual environment and install the core packages. You need LlamaIndex, its evaluation module, and an LLM provider for both generation and judging.

python -m venv .venv
source .venv/bin/activate
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai \
    llama-index-readers-file pandas tqdm

Set your API key. If you route through n4n.ai, point the base URL at the gateway and use your n4n.ai key; the OpenAI-compatible endpoint works without code changes.

export OPENAI_API_KEY="sk-..."
# Optional: route via n4n.ai
# export OPENAI_BASE_URL="https://api.n4n.ai/v1"

Verify the install works:

# verify_setup.py
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

response = Settings.llm.complete("Say ok")
print(response.text.strip())  # should print "ok"

Run it. You should see ok printed. If you get an authentication error, check your key and base URL.

Step 2: Build a minimal RAG pipeline

Use a small, controllable corpus so you know the ground truth. Create a data/ directory with two text files.

mkdir -p data
cat > data/company_policy.txt << 'EOF'
Acme Corp Remote Work Policy
Effective: January 1, 2024

1. Eligibility: Full-time employees with 90+ days tenure may work remotely up to 3 days per week.
2. Core hours: 10:00–15:00 ET, regardless of location.
3. Equipment: Company provides laptop and monitor. Internet stipend: $75/month.
4. Security: VPN required for all production access. No public Wi-Fi without approved tethering.
5. Office days: Teams coordinate in-office days via shared calendar. Minimum 1 day/week in office.
EOF

cat > data/benefits.txt << 'EOF'
Acme Corp Benefits Summary 2024

Health: Three PPO plans, one HSA-eligible. Company covers 85% of employee premium.
Dental/Vision: Fully covered for employee, 50% for dependents.
401(k): 4% match on first 6% contributed. Immediate vesting.
PTO: 15 days vacation + 10 holidays + 5 sick days. Unlimited carryover up to 30 days.
Parental leave: 12 weeks paid at 100% for primary caregiver, 4 weeks for secondary.
EOF

Now build the index and query engine:

# build_rag.py
from pathlib import Path
from llama_index.core import (
    VectorStoreIndex, SimpleDirectoryReader, Settings, StorageContext
)
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=2)

# Sanity check
response = query_engine.query("What is the internet stipend for remote work?")
print(response.response)

Run it. The answer should reference $75/month from the policy doc. If it hallucinates a different number, your retrieval or prompt needs work — but that’s exactly what faithfulness evaluation will catch.

Step 3: Generate evaluation questions

Faithfulness needs question-context-answer triples. LlamaIndex can synthesize questions from your documents, but hand-curating a few ensures coverage of edge cases.

# generate_questions.py
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.evaluation import generate_question_context_pairs
import json

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.3)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Auto-generate 8 questions from the corpus
qa_pairs = generate_question_context_pairs(
    documents,
    llm=Settings.llm,
    num_questions_per_chunk=2,
)

# Save for inspection and reuse
output = []
for q, ctx in qa_pairs:
    output.append({
        "question": q,
        "contexts": [c.get_content() for c in ctx],
    })

with open("eval_questions.json", "w") as f:
    json.dump(output, f, indent=2)

print(f"Generated {len(output)} question-context pairs")
for item in output[:3]:
    print(f"Q: {item['question']}")
    print(f"Contexts: {len(item['contexts'])} chunks")
    print()

Run this. Open eval_questions.json and verify questions make sense. Add or edit manually if the generator misses key policies (e.g., parental leave details, VPN requirement). Aim for 10–15 questions covering each document section.

Step 4: Run faithfulness evaluation

LlamaIndex’s FaithfulnessEvaluator uses an LLM judge to score whether each answer sentence is supported by the retrieved context. It returns a score (0–1) and a per-sentence breakdown.

# evaluate_faithfulness.py
import json
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.evaluation import FaithfulnessEvaluator
from llama_index.core.query_engine import RetrieverQueryEngine
from tqdm import tqdm

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# Load index and questions
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=2)

with open("eval_questions.json") as f:
    eval_data = json.load(f)

evaluator = FaithfulnessEvaluator(llm=Settings.llm)

results = []
for item in tqdm(eval_data, desc="Evaluating"):
    question = item["question"]
    response = query_engine.query(question)
    
    # Evaluate: pass question, response, and retrieved contexts
    eval_result = evaluator.evaluate_response(
        query=question,
        response=response,
    )
    
    results.append({
        "question": question,
        "answer": str(response),
        "score": eval_result.score,  # 0.0 to 1.0
        "passing": eval_result.passing,  # True/False at default threshold 0.5
        "feedback": eval_result.feedback,  # per-sentence reasoning
    })

# Save detailed results
with open("faithfulness_results.json", "w") as f:
    json.dump(results, f, indent=2)

# Summary
passing = sum(1 for r in results if r["passing"])
avg_score = sum(r["score"] for r in results) / len(results)
print(f"Passing: {passing}/{len(results)} ({passing/len(results)*100:.0f}%)")
print(f"Average score: {avg_score:.3f}")

Run it. Typical output on this corpus:

Evaluating: 100%|██████████| 12/12 [00:15<00:00,  1.25s/it]
Passing: 10/12 (83%)
Average score: 0.87

Open faithfulness_results.json. Each entry shows the question, generated answer, numeric score, boolean pass/fail, and the judge’s feedback explaining which sentences lacked support.

Step 5: Interpret results and iterate

A score of 1.0 means every sentence in the answer traces to retrieved context. Below 0.5 fails. The feedback field tells you exactly what to fix.

Common failure patterns

Hallucinated specifics — The model invents numbers not in context.

{
  "question": "What is the 401(k) match percentage?",
  "answer": "Acme Corp offers a 6% 401(k) match on the first 8% contributed.",
  "score": 0.33,
  "feedback": "Sentence 1: '6% match on first 8%' not supported. Context states '4% match on first 6%'."
}

Fix: Check retrieval — did the right chunk arrive? If yes, the prompt may need stricter “only use context” instructions.

Over-generalization — The model states something plausible but not in the docs.

{
  "question": "Is there a wellness stipend?",
  "answer": "Yes, Acme Corp provides a $200 annual wellness stipend for gym memberships.",
  "score": 0.0,
  "feedback": "No mention of wellness stipend in any retrieved context."
}

Fix: This is a true gap. Either add the info to your corpus or accept that the system correctly says “not mentioned.”

Partial support — Some sentences grounded, others not.

{
  "question": "Describe the parental leave policy.",
  "answer": "Primary caregivers get 12 weeks paid at 100%. Secondary caregivers get 4 weeks. Leave must be taken within 6 months of birth.",
  "score": 0.66,
  "feedback": "Sentence 1-2 supported. Sentence 3: 'within 6 months' not in context."
}

Fix: The last sentence is a reasonable inference but not in the doc. Decide if your use case allows synthesis or requires strict extraction.

Adjust retrieval first, then prompt

Before rewriting prompts, verify retrieval quality. Log the retrieved nodes for failing questions:

# debug_retrieval.py
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
retriever = index.as_retriever(similarity_top_k=2)

question = "What is the 401(k) match percentage?"
nodes = retriever.retrieve(question)
for i, n in enumerate(nodes):
    print(f"--- Node {i} (score: {n.score:.3f}) ---")
    print(n.get_content()[:300])
    print()

If the relevant chunk isn’t in top-2, increase similarity_top_k or switch to a hybrid retriever (vector + BM25). Only after retrieval is solid should you tune the system prompt.

Prompt template for stricter faithfulness

from llama_index.core import PromptTemplate

strict_qa_prompt = PromptTemplate(
    "You are a precise QA system. Answer ONLY using the provided context. "
    "If the context does not contain the answer, say 'The provided context does not contain this information.' "
    "Do not use external knowledge. Do not infer. Cite specific sentences.\n\n"
    "Context:\n{context_str}\n\n"
    "Question: {query_str}\n\n"
    "Answer:"
)

query_engine = index.as_query_engine(
    similarity_top_k=3,
    text_qa_template=strict_qa_prompt,
)

Re-run evaluation. You’ll typically see scores rise, but also more “context does not contain” answers — which is correct behavior for unanswerable questions.

Step 6: Automate in CI/CD

Faithfulness regresses when you swap models, change chunking, or update the corpus. Run evaluation on every PR.

# .github/workflows/rag-eval.yml
name: RAG Faithfulness Evaluation
on: [pull_request]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python evaluate_faithfulness.py
      - name: Check threshold
        run: |
          python -c "
import json, sys
with open('faithfulness_results.json') as f:
    data = json.load(f)
avg = sum(r['score'] for r in data) / len(data)
passing = sum(1 for r in data if r['passing']) / len(data)
print(f'Average: {avg:.3f}, Pass rate: {passing:.1%}')
if avg < 0.8 or passing < 0.75:
    sys.exit('Faithfulness below threshold')
"

Set thresholds that match your risk tolerance. A customer-facing support bot might need 0.9 average; an internal research assistant can tolerate 0.75.

Verification checklist

Before considering the evaluation complete, confirm:

1. Corpus coverage — Every document section has at least 2 test questions.
2. Judge consistency — Re-run evaluation twice; scores should vary < 0.05 per question. If not, lower judge temperature to 0.0 or use a stronger model (gpt-4o) for evaluation only.
3. Failure analysis — You’ve categorized each failing case as retrieval gap, prompt issue, or genuine corpus gap.
4. Regression baseline — Current scores committed as faithfulness_baseline.json for CI comparison.

What to do next

• Add context relevance evaluation — Faithfulness only checks answer-to-context. Use RelevancyEvaluator to verify retrieval quality independently.
• Test adversarial questions — Add questions designed to trigger hallucination (e.g., “What is the CEO’s home address?”).
• Compare judges — Run the same evaluation with gpt-4o and a local model (via Ollama) to see if a cheaper judge suffices.
• Track per-segment — Tag questions by topic (benefits, remote, security) and monitor segment-level scores.

Faithfulness is the metric that catches the hallucinations your users will notice first. Build the eval once, run it often, and treat any regression as a blocker.