Fact-checking AI outputs tools and techniques have become essential infrastructure for any team shipping LLM features to production. The core problem is straightforward: models hallucinate, and users trust them anyway. This guide walks through an ordered path from quick wins to production-grade verification pipelines, with code you can adapt immediately.
Start with retrieval-augmented generation
Before you build verification layers, reduce the surface area for hallucinations. RAG grounds responses in source documents you control. The pattern is simple: retrieve relevant chunks, stuff them into context, generate with citations.
from openai import OpenAI
import numpy as np
client = OpenAI()
def retrieve_chunks(query: str, index, top_k: int = 5) -> list[str]:
"""Retrieve top-k chunks from a vector index."""
query_embedding = client.embeddings.create(
input=query, model="text-embedding-3-small"
).data[0].embedding
scores, indices = index.search(np.array([query_embedding]), top_k)
return [chunks[i] for i in indices[0]]
def generate_with_citations(query: str, chunks: list[str]) -> str:
context = "\n\n".join(f"[Doc {i+1}] {c}" for i, c in enumerate(chunks))
prompt = f"""Answer the question using only the provided documents.
Cite sources inline like [Doc 1]. If the answer isn't in the docs, say so.
Documents:
{context}
Question: {query}
Answer:"""
return client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0
).choices[0].message.contentThis alone eliminates entire classes of fabrication. But RAG has limits: retrieval misses, context window pressure, and the model can still misread cited sources. You need verification on top.
Programmatic fact extraction and verification
For structured claims — numbers, dates, entity relationships — extract assertions programmatically and verify against authoritative sources. Don’t ask an LLM to fact-check itself; use deterministic lookups where possible.
import re
from datetime import datetime
import requests
CLAIM_PATTERNS = [
(r"(\$[\d,]+(?:\.\d{2})?)", "currency"),
(r"(\d{1,2}/\d{1,2}/\d{4})", "date_us"),
(r"(\d{4}-\d{2}-\d{2})", "date_iso"),
(r"(\d+(?:\.\d+)?%)", "percentage"),
]
def extract_claims(text: str) -> list[dict]:
"""Extract verifiable claims from generated text."""
claims = []
for pattern, claim_type in CLAIM_PATTERNS:
for match in re.finditer(pattern, text):
claims.append({
"text": match.group(1),
"type": claim_type,
"position": match.span(),
"context": text[max(0, match.start()-50):match.end()+50]
})
return claims
def verify_currency_claim(claim: str, source_api: str = "https://api.exchangerate.host/latest") -> dict:
"""Example: verify a currency amount against live rates."""
# Parse amount and currency from claim like "$1,234.56"
match = re.match(r"\$([\d,]+(?:\.\d{2})?)", claim)
if not match:
return {"verified": False, "reason": "unparsable"}
amount = float(match.group(1).replace(",", ""))
# In practice, you'd check against your business logic or external API
return {"verified": True, "amount_usd": amount, "source": "parsed"}
def verify_date_claim(claim: str) -> dict:
"""Validate date format and reasonableness."""
for fmt in ("%m/%d/%Y", "%Y-%m-%d"):
try:
parsed = datetime.strptime(claim, fmt)
# Business rule: dates shouldn't be in the far future
if parsed > datetime.now().replace(year=datetime.now().year + 2):
return {"verified": False, "reason": "future_date"}
return {"verified": True, "parsed": parsed.isoformat()}
except ValueError:
continue
return {"verified": False, "reason": "invalid_format"}Run this as a post-processing step on every generation. Flag unverified claims for review or suppress them entirely.
LLM-as-judge for semantic verification
Some claims resist programmatic verification: “The new policy improves customer satisfaction” or “Our competitor launched a similar feature last quarter.” For these, use a smaller, cheaper model as a critic — but structure the task carefully.
from pydantic import BaseModel, Field
from typing import Literal
import instructor
client = instructor.from_openai(OpenAI())
class VerificationResult(BaseModel):
claim: str
verdict: Literal["supported", "contradicted", "unverifiable"]
confidence: float = Field(ge=0.0, le=1.0)
evidence: str
reasoning: str
VERIFICATION_PROMPT = """You are a fact-checker. Given a claim and source documents, determine if the claim is supported.
Rules:
- "supported": claim is directly stated or logically entailed by the sources
- "contradicted": sources explicitly disagree with the claim
- "unverifiable": sources don't contain relevant information
- Be conservative. Default to "unverifiable" when uncertain.
- Quote the exact text that supports your verdict.
Claim: {claim}
Sources:
{sources}
Return JSON matching the VerificationResult schema."""
def verify_claim_semantic(claim: str, source_chunks: list[str]) -> VerificationResult:
sources_text = "\n\n".join(f"[Source {i+1}] {c}" for i, c in enumerate(source_chunks))
return client.chat.completions.create(
model="gpt-4o-mini",
response_model=VerificationResult,
messages=[{"role": "user", "content": VERIFICATION_PROMPT.format(
claim=claim, sources=sources_text
)}],
temperature=0
)Key insight: the verifier model should be different from the generator. Use a smaller model (gpt-4o-mini, Claude Haiku) for verification — it’s cheaper, faster, and less prone to sycophancy. Pass only the specific claim and relevant sources, not the full conversation.
Build a verification pipeline
Chain these layers together. Each layer catches what the previous missed.
from dataclasses import dataclass
from enum import Enum
from typing import Optional
class VerificationStatus(Enum):
VERIFIED = "verified"
FLAGGED = "flagged"
REJECTED = "rejected"
@dataclass
class VerifiedClaim:
claim: str
claim_type: str
status: VerificationStatus
programmatic_result: Optional[dict] = None
semantic_result: Optional[VerificationResult] = None
action: str = "allow" # allow, flag, reject
def verify_generation(text: str, source_chunks: list[str]) -> list[VerifiedClaim]:
claims = extract_claims(text)
results = []
for claim in claims:
# Layer 1: programmatic verification
prog_result = None
if claim["type"] == "currency":
prog_result = verify_currency_claim(claim["text"])
elif claim["type"] in ("date_us", "date_iso"):
prog_result = verify_date_claim(claim["text"])
if prog_result and prog_result.get("verified"):
results.append(VerifiedClaim(
claim=claim["text"],
claim_type=claim["type"],
status=VerificationStatus.VERIFIED,
programmatic_result=prog_result
))
continue
# Layer 2: semantic verification
sem_result = verify_claim_semantic(claim["text"], source_chunks)
if sem_result.verdict == "supported" and sem_result.confidence > 0.8:
status = VerificationStatus.VERIFIED
action = "allow"
elif sem_result.verdict == "contradicted":
status = VerificationStatus.REJECTED
action = "reject"
else:
status = VerificationStatus.FLAGGED
action = "flag"
results.append(VerifiedClaim(
claim=claim["text"],
claim_type=claim["type"],
status=status,
programmatic_result=prog_result,
semantic_result=sem_result,
action=action
))
return resultsThis pipeline runs in ~200-500ms per generation. Cache verification results by claim hash to avoid re-checking identical assertions across requests.
Human-in-the-loop for high-stakes outputs
Automated verification has false negatives. For medical, legal, financial, or customer-facing outputs, route flagged claims to human reviewers.
import uuid
from datetime import datetime, timezone
from typing import Callable
import json
class ReviewQueue:
def __init__(self, storage_backend):
self.storage = storage_backend # Redis, Postgres, etc.
def enqueue(self, generation_id: str, claims: list[VerifiedClaim],
original_text: str, metadata: dict) -> str:
review_id = str(uuid.uuid4())
flagged = [c for c in claims if c.status == VerificationStatus.FLAGGED]
rejected = [c for c in claims if c.status == VerificationStatus.REJECTED]
if not flagged and not rejected:
return None # Nothing to review
record = {
"review_id": review_id,
"generation_id": generation_id,
"original_text": original_text,
"flagged_claims": [
{"claim": c.claim, "type": c.claim_type, "reason": c.semantic_result.reasoning if c.semantic_result else "programmatic_failure"}
for c in flagged
],
"rejected_claims": [
{"claim": c.claim, "type": c.claim_type, "evidence": c.semantic_result.evidence if c.semantic_result else str(c.programmatic_result)}
for c in rejected
],
"metadata": metadata,
"status": "pending",
"created_at": datetime.now(timezone.utc).isoformat()
}
self.storage.set(f"review:{review_id}", json.dumps(record))
return review_id
def resolve(self, review_id: str, reviewer_id: str, decision: str, notes: str = "") -> bool:
"""decision: 'approve' | 'reject' | 'modify'"""
record = json.loads(self.storage.get(f"review:{review_id}"))
record.update({
"status": "resolved",
"reviewer_id": reviewer_id,
"decision": decision,
"notes": notes,
"resolved_at": datetime.now(timezone.utc).isoformat()
})
self.storage.set(f"review:{review_id}", json.dumps(record))
return TrueIntegrate this with your existing ticketing system (Linear, Jira, GitHub Issues) so reviewers work in familiar tools. The key metric: time-to-resolution for flagged claims. Target < 4 hours for customer-facing content.
Observability: measure what matters
You can’t improve what you don’t measure. Instrument every layer.
from dataclasses import dataclass, asdict
import time
import logging
@dataclass
class VerificationMetrics:
generation_id: str
total_claims: int
verified_programmatic: int
verified_semantic: int
flagged: int
rejected: int
latency_ms: int
model_used: str
def record_verification(metrics: VerificationMetrics):
# Ship to your observability stack (Datadog, Honeycomb, etc.)
logging.info("verification_complete", extra=asdict(metrics))
# Alert on anomalies
if metrics.flagged / max(metrics.total_claims, 1) > 0.3:
logging.warning("high_flag_rate", extra={"generation_id": metrics.generation_id})
if metrics.latency_ms > 1000:
logging.warning("verification_latency_spike", extra={"generation_id": metrics.generation_id})
# Wrap your pipeline
def verified_generate(query: str, index, metadata: dict) -> tuple[str, list[VerifiedClaim]]:
start = time.time()
generation_id = str(uuid.uuid4())
chunks = retrieve_chunks(query, index)
raw_answer = generate_with_citations(query, chunks)
claims = verify_generation(raw_answer, chunks)
# Apply actions
final_answer = raw_answer
for claim in claims:
if claim.action == "reject":
final_answer = final_answer.replace(claim.claim, f"[REMOVED: {claim.claim}]")
elif claim.action == "flag":
final_answer = final_answer.replace(claim.claim, f"⚠️ {claim.claim}")
latency_ms = int((time.time() - start) * 1000)
metrics = VerificationMetrics(
generation_id=generation_id,
total_claims=len(claims),
verified_programmatic=sum(1 for c in claims if c.programmatic_result and c.programmatic_result.get("verified")),
verified_semantic=sum(1 for c in claims if c.semantic_result and c.semantic_result.verdict == "supported"),
flagged=sum(1 for c in claims if c.status == VerificationStatus.FLAGGED),
rejected=sum(1 for c in claims if c.status == VerificationStatus.REJECTED),
latency_ms=latency_ms,
model_used="gpt-4o-mini"
)
record_verification(metrics)
return final_answer, claimsTrack these dashboards:
- Claim verification rate by type (currency, date, percentage, semantic)
- False positive rate — claims flagged by automation but approved by humans
- False negative rate — claims that passed automation but caught in human review or user reports
- End-to-end latency p50/p95/p99
Common pitfalls and tradeoffs
Over-verifying kills latency. Every verification layer adds 100-300ms. Profile your pipeline. If p95 latency exceeds your budget, drop semantic verification for low-risk claim types (percentages in marketing copy) and keep it for high-risk (dosage amounts in medical summaries).
The verifier can hallucinate too. A study by Min et al. (2023) found LLM judges agree with human annotators only ~80% of the time on factual consistency. Mitigate: use few-shot examples in your verification prompt, require quoted evidence, and set confidence thresholds conservatively.
Citation format matters. Models cite inconsistently: [1], [Doc 1], (source: doc1). Normalize citations during generation with a strict format instruction, then parse them programmatically to map claims to source chunks.
Context contamination. When you feed the full generation to the verifier, it may “remember” the model’s reasoning and rationalize errors. Pass only the extracted claim + source chunks to the verifier. Keep contexts isolated.
Cost scales with claim density. A 500-token response might contain 20 verifiable claims. At $0.15/1M tokens for gpt-4o-mini, that’s ~$0.003 per verification pass. At 10K generations/day, budget ~$90/month for the verification layer alone. Cache aggressively.
n4n.ai handles routing fallbacks automatically when a provider degrades, which matters for verification pipelines that call multiple models in sequence — if your primary verifier model goes down, the gateway fails over without your code changing.
What to build next
Start with programmatic extraction on your highest-volume endpoint. Add semantic verification for the claim types that matter most to your domain. Instrument everything. Route flagged claims to your existing review workflow. Iterate thresholds based on false positive/negative rates.
The goal isn’t perfect accuracy — it’s a measurable, improvable system where hallucinations become visible, auditable, and rare enough that users trust the output without blind faith.