Grounding LLM outputs with RAG to cut hallucinations

RAG grounding reduce hallucinations by binding model outputs to retrieved evidence, but most pipelines fail because they treat retrieval as a black box. You need deliberate chunking, retrieval tuning, and output validation to see real gains.

1. Define a grounding contract

Decide what “grounded” means before writing code. If you need verifiability, mandate inline citations with chunk IDs. If you only care about internal error rates, softer constraints suffice.

{
  "require_citations": true,
  "max_tokens": 512,
  "on_missing_context": "abstain"
}

Without a contract, you cannot measure success. Write it down and enforce it in the prompt and the parser.

2. Chunk for retrieval, not for storage

Chunk size drives recall. Too large, and embeddings blur topics; too small, and you fragment context. Start with 512–1024 tokens and 10–20% overlap.

def chunk_text(text: str, size: int = 800, overlap: int = 100) -> list[str]:
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i+size]) for i in range(0, len(words), step)]

Attach metadata: source, timestamp, section. Prefix the chunk with its title so the embedding captures context:

chunk_with_meta = f"[{doc_title}] {chunk}"

Pitfall: embedding the body without metadata loses provenance. The retriever will surface anonymous text that the model cannot cite.

3. Retrieve with hybrid search

Pure vector search misses exact keywords. Combine BM25 with cosine similarity, then rerank with a cross-encoder.

def hybrid_score(vec_score: float, bm25_score: float, alpha: float = 0.7) -> float:
    return alpha * vec_score + (1 - alpha) * bm25_score

Take top-20 from hybrid, rerank to top-5. Tradeoff: reranking adds latency but strips irrelevant context that causes hallucinations.

Tune k aggressively

Feeding 10 chunks often hurts more than helps. The model anchors to early context; if it’s noisy, it invents. Start with k=3 and increase only if an ablation shows gain.

4. Prompt to compel grounding

The model follows explicit instructions. Use a system prompt that forbids guessing.

{
  "role": "system",
  "content": "Answer using ONLY the provided context. Cite the chunk ID for each claim as [cid]. If the context lacks the answer, respond 'UNKNOWN'."
}

Place context before the question. Models attend to recent tokens, but putting the ask last keeps the constraint salient.

const userMsg = `Context:
${chunks.map(c => `[${c.id}] ${c.text}`).join("\n\n")}

Question: ${query}
Answer with citations:`;

Common pitfall: putting context after the question. The model sees the query first and commits to an answer before reading evidence.

5. Validate outputs programmatically

After generation, verify cited IDs exist in the retrieved set.

import re

def validate_citations(answer: str, retrieved_ids: set[str]) -> bool:
    cites = set(re.findall(r"\[([^\]]+)\]", answer))
    return cites.issubset(retrieved_ids)

For higher stakes, run a critic model: “Given context X, is claim Y supported?” Use a small model to keep cost low.

If you front your RAG pipeline with an OpenAI-compatible gateway, n4n.ai lets you honor client routing directives and forward provider cache-control hints, so repeated retrieval-augmented calls hit cached context across 240+ models without rewriting your client.

6. Measure hallucination rate

Build a golden set of 100 queries with known answers and source docs. Run monthly. Track:

• Unsupported claim rate (validator flags)
• Abstain rate when context absent
• User correction feedback

eval_results = {
  "total": 100,
  "cited_unknown": 12,
  "unsupported": 5,
  "grounded": 83
}

If unsupported > 5%, revisit chunking or reranking.

Tradeoffs: latency vs quality

Adding reranking and validation adds 200–400ms. For chat, that’s acceptable; for inline autocomplete, it isn’t. Decide based on UX, not dogma.

7. Keep the index fresh

Stale knowledge is a silent hallucination source. Schedule incremental updates. If you change embedding models, reindex everything—embedding drift breaks old vectors.

Common pitfall: using one collection for disparate domains. Split by tenant or topic to improve signal-to-noise.

Common pitfalls summary

• Context stuffing: more text ≠ better answers.
• No abstain path: model fills gaps with fiction.
• Ignoring metadata filters: date filters prevent outdated citations.
• Single-vector retrieval: misses rare terms and proper nouns.

Final workflow

1. Define grounding contract (cite or abstain).
2. Chunk with metadata, size ~800 tokens.
3. Hybrid retrieve, rerank, k=3–5.
4. Prompt context-first, cite-or-abstain.
5. Validate citations, run critic if needed.
6. Eval monthly, tune chunk size and k.

RAG grounding reduce hallucinations only when each stage is owned and measured. Treat it as a system, not a library call.