Shipping a retrieval-augmented assistant without an eval set catch llm hallucinations is how teams learn about broken outputs from angry users. This tutorial builds a minimal but effective offline evaluation harness that flags unfaithful generations before they reach production. You will leave with a runnable Python pipeline and a CI hook.
Prerequisites
- Python 3.11+ installed locally
openaiPython SDK (pip install openai)- A dataset of past queries with retrieved context and model answers (
logs.jsonl, even 30 rows works) - An API key for an OpenAI-compatible inference endpoint
Set the key in your shell:
export LLM_API_KEY="sk-..."Step 1: Define hallucination classes for your app
Hallucinations are not monolithic. For a support bot we separate three operational classes:
HALLUCINATION_CLASSES = {
"contradiction": "Answer states a fact that conflicts with provided context.",
"fabrication": "Answer introduces entities, numbers, or claims absent from context.",
"leakage": "Answer reveals system prompt internals or ignores scoped instructions.",
}Write these down. They become the contract for your grader and keep scoring objective.
Step 2: Extract seed examples from production logs
Assume a JSONL file where each line is {"query": str, "context": str, "answer": str, "tag": str}. Sample a stable set so results are comparable across runs.
import json, random
def load_logs(path, n=30, seed=42):
random.seed(seed)
with open(path) as f:
rows = [json.loads(l) for l in f if l.strip()]
return random.sample(rows, min(n, len(rows)))
seeds = load_logs("logs.jsonl")
print(f"Loaded {len(seeds)} seed examples")Expected output:
Loaded 30 seed examplesManually label a handful to validate the grader later. Store the raw logs; do not discard the context field—it is the ground truth for faithfulness.
Step 3: Build a faithfulness scorer
We use an LLM-as-judge with strict JSON output. The judge sees only context and answer, never the query, to avoid relevance bias.
from openai import OpenAI
import os, json
client = OpenAI(
base_url="https://api.n4n.ai/v1", # OpenAI-compatible, fronts 240+ models
api_key=os.environ["LLM_API_KEY"]
)
JUDGE_PROMPT = """You are a strict eval grader. Given CONTEXT and ANSWER, output JSON:
{
"contradiction": bool,
"fabrication": bool,
"leakage": bool,
"reason": str
}
Mark true only on unambiguous violation. No partial credit."""
def score_example(context: str, answer: str, model="anthropic/claude-3.5-sonnet") -> dict:
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": JUDGE_PROMPT},
{"role": "user", "content": f"CONTEXT:\n{context}\n\nANSWER:\n{answer}"}
],
response_format={"type": "json_object"}
)
return json.loads(resp.choices[0].message.content)
ex = seeds[0]
result = score_example(ex["context"], ex["answer"])
print(result)Expected shape:
{"contradiction": false, "fabrication": false, "leakage": false, "reason": "Answer grounded in context."}The n4n.ai endpoint forwards provider cache-control hints, so repeated context blocks across batch calls reduce token spend. Use a strong judge model; the grader is not the product, so accuracy matters more than latency.
Step 4: Run batch eval and aggregate
The batch run is the core of your eval set catch llm hallucinations effort. Iterate, compute per-class rates, and fail the build on breach.
def run_eval(dataset, threshold=0.1, model="anthropic/claude-3.5-sonnet"):
hits = {"contradiction": 0, "fabrication": 0, "leakage": 0}
for row in dataset:
sc = score_example(row["context"], row["answer"], model)
for k in hits:
if sc.get(k):
hits[k] += 1
total = len(dataset)
rates = {k: round(v / total, 3) for k, v in hits.items()}
print("Hallucination rates:", rates)
failed = any(r > threshold for r in rates.values())
return failed, rates
failed, rates = run_eval(seeds)
print("FAILED CI:", failed)Expected output:
Hallucination rates: {'contradiction': 0.033, 'fabrication': 0.067, 'leakage': 0.0}
FAILED CI: FalseIf you run large batches, the automatic fallback in n4n.ai prevents rate-limit errors from breaking the eval job when a provider is degraded.
Step 5: Wire into CI
Create eval.py that exits non-zero on failure. Add a GitHub Actions workflow:
# .github/workflows/eval.yml
name: llm-eval
on: [push]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.11" }
- run: pip install openai
- run: python eval.py
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}In eval.py, end with:
import sys
sys.exit(1 if failed else 0)You can also wrap run_eval in a pytest test for parallel local execution:
def test_hallucination_rate():
failed, _ = run_eval(seeds, threshold=0.1)
assert not failedStep 6: Expand the eval set catch llm hallucinations continuously
A static set decays. When a user reports a bad output, append it to logs.jsonl with its context and label. Over time the eval set catch llm hallucinations becomes a regression suite. Target 200+ examples sliced by domain.
def slice_by_tag(dataset, tag):
return [r for r in dataset if r.get("tag") == tag]
for tag in ["refund", "tech_spec", "onboarding"]:
sub = slice_by_tag(seeds, tag)
if sub:
_, r = run_eval(sub, threshold=0.15)
print(tag, r)Synthetic generation helps cover edge cases: prompt a model to produce contradictory answers from clean context, then store them as positive hallucination samples. This hardens the judge.
Expected output at key checkpoints
- After Step 2:
Loaded 30 seed examples - After Step 3: single JSON with three booleans and a reason string
- After Step 4: aggregated rates and
FAILED CI: False(when under threshold) - In CI: green build when rates stay below threshold; red on regression
Recommendations
Treat the eval set catch llm hallucinations as code: review diffs in PRs, bump thresholds only with written justification. Run nightly against new model versions to catch silent regressions. Keep the judge prompt frozen when comparing providers so the only variable is the generation model. Store scores per commit to track trend lines, not just pass/fail.