How RAG grounds LLM responses in real data

Retrieval-augmented generation (RAG) grounds LLM responses by fetching relevant documents from an external knowledge base and injecting them into the model’s context window before generation. The model then answers using only the provided context, not its parametric memory. This shifts the burden of factual accuracy from model weights to a retrievable, updatable data store.

How RAG works

The pipeline has three stages: index, retrieve, and generate. Each stage has engineering decisions that determine whether the system actually works in production.

Index

You chunk source documents, embed each chunk with a dense vector model, and store vectors in a search index. Chunking strategy matters more than most teams admit. Fixed-size chunks (512 tokens, 100 overlap) are a reasonable default, but semantic chunking — splitting on headings, paragraphs, or logical boundaries — preserves context better for technical docs and legal text.

# Naive fixed-size chunking
def chunk_text(text: str, chunk_size: int = 512, overlap: int = 100) -> list[str]:
    tokens = tokenizer.encode(text)
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[i:i + chunk_size]
        chunks.append(tokenizer.decode(chunk_tokens))
    return chunks

Embedding model choice determines retrieval quality. text-embedding-3-large (3072 dims) outperforms text-embedding-ada-002 (1536 dims) on most benchmarks, but costs more and increases index size. For latency-sensitive paths, consider smaller models like bge-small-en-v1.5 (384 dims) with a reranker.

Store vectors in a purpose-built index: pgvector for Postgres-native teams, Pinecone or Weaviate for managed scale, Qdrant for self-hosted with filtering. Metadata filtering (tenant, version, doc_type) at query time is non-negotiable for multi-tenant systems.

Retrieve

At query time, embed the user question, search the index for top-k similar chunks, then optionally rerank. A typical configuration:

async def retrieve(query: str, k: int = 20, rerank_k: int = 5) -> list[Document]:
    query_vec = await embed(query)
    candidates = await index.search(query_vec, top_k=k, filter={"tenant_id": tenant})
    
    # Cross-encoder rerank for precision
    pairs = [(query, doc.text) for doc in candidates]
    scores = await reranker.score(pairs)
    reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    
    return [doc for doc, _ in reranked[:rerank_k]]

Two retrieval patterns deserve attention:

Hybrid search combines dense vectors with sparse (BM25) for exact keyword matches — model numbers, error codes, proper nouns. Most vector databases now support this natively.

Query rewriting expands or decomposes the user question before search. A simple approach: use an LLM to generate 3-5 sub-queries, retrieve for each, deduplicate, then rerank. This handles multi-hop questions (“Compare the 2023 and 2024 pricing for enterprise tiers”) that single embeddings miss.

Generate

Inject retrieved chunks into the prompt with clear delimiters and instructions. The prompt template is where grounding succeeds or fails.

SYSTEM_PROMPT = """You are a precise technical assistant. Answer using ONLY the provided context. 
If the context doesn't contain the answer, say "I don't have enough information." 
Cite sources inline using [doc_id] format."""

def build_prompt(query: str, docs: list[Document]) -> list[dict]:
    context_blocks = []
    for i, doc in enumerate(docs):
        context_blocks.append(f"[doc_{i}] {doc.text}\nSource: {doc.metadata.get('source', 'unknown')}")
    
    context = "\n\n---\n\n".join(context_blocks)
    
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}
    ]

Key prompt details: explicit “only use context” instruction, citation format, and a refusal pathway. Without the refusal pathway, models hallucinate confidently when retrieval misses.

Why grounding matters

Parametric knowledge has three problems RAG solves directly.

Training cutoff. Models know nothing after their training data ends. Your product docs, API changelogs, and internal runbooks change weekly. Retraining or fine-tuning on every update is impractical; updating a vector index takes seconds.

Hallucination on niche domains. Models confidently invent function signatures, configuration flags, and error messages for your proprietary stack. Grounding forces the model to either find the answer in your docs or admit ignorance.

Attribution and audit. Regulated industries (fintech, healthcare, legal) require traceable answers. RAG provides a citation chain: question → retrieved chunks → generated answer. You can log the full chain for compliance review.

The tradeoff: latency. A naive RAG call adds 200-800ms (embedding + vector search + rerank + generation). Cache frequent queries, stream the generation, and consider a smaller generator model (e.g., gpt-4o-mini or llama-3.1-8b) for the final step.

Concrete example: internal developer portal

A platform team runs an internal developer portal with 50k markdown files: service specs, runbooks, incident postmortems, API references. Engineers ask questions like “What’s the retry policy for the payments service?” or “How do I roll back a canary deployment?”

Before RAG

Engineers search the wiki, open 3-4 tabs, piece together answers. New hires ask seniors on Slack. Knowledge lives in heads and stale Confluence pages.

After RAG

Index all markdown with semantic chunking (split on H2/H3). Embed with text-embedding-3-large. Store in pgvector with metadata: service, doc_type, last_updated.

Query flow:

1. Engineer asks: “What’s the circuit breaker config for auth-service?”
2. System embeds query, retrieves top 20 chunks filtered to service=auth-service
3. Reranks with cross-encoder, keeps top 5
4. Generates answer with citations: “The circuit breaker uses a failure threshold of 5 errors in 10 seconds, with a 30-second half-open period [doc_2]. See auth-service/config/resilience.yaml [doc_2].”
5. Engineer gets answer in 1.2s with a link to the source file.

Measurable outcomes

• Median time-to-answer dropped from ~8 minutes (search + read) to ~1.5 seconds
• Onboarding questions in #platform-help Slack channel decreased 60% in month one
• Stale doc detection: chunks with last_updated > 180 days flagged for review during retrieval

Common misconceptions

“RAG replaces fine-tuning”

They solve different problems. Fine-tuning teaches behavior (style, format, reasoning patterns). RAG provides knowledge (facts, docs, current state). Use both: fine-tune a smaller model on your response format, then RAG for knowledge. A 7B fine-tuned model with RAG often outperforms a 70B base model without it.

“Long context windows make RAG obsolete”

A 128k or 1M token context window lets you stuff entire codebases or doc sets into the prompt. Three problems remain:

1. Cost. Input tokens cost money. Retrieving 5 relevant chunks (2k tokens) vs. dumping 100k tokens is 50x cheaper per query.
2. Distraction. Models lose the needle in the haystack. Irrelevant context degrades reasoning, especially on multi-step tasks.
3. Freshness. You still need to update the context. RAG’s index update is surgical; re-uploading a 500k token context on every doc change is not.

Long context complements RAG for synthesis tasks (summarize this entire repo, find all references to X). Use RAG for lookup tasks.

“Vector search is enough”

Dense vectors capture semantic similarity but miss exact matches. A query for “error code 429” needs keyword search. Hybrid search (dense + sparse) is the production baseline. Add a reranker — cross-encoders like bge-reranker-v2-m3 or cohere-rerank-3.5 — to recover precision after the initial recall pass.

“Chunking doesn’t matter”

Bad chunking breaks retrieval. A 2000-token chunk about “authentication” that mixes OAuth flows, API key rotation, and SSO config will match queries for any of those but retrieve noise for all of them. Semantic chunking aligned to document structure (headings, sections) keeps related content together and unrelated content separate.

Test your chunking: sample 50 queries, retrieve top-5, manually label relevance. If precision@5 < 0.6, fix chunking before tuning embeddings or prompts.

“One index fits all”

Multi-tenant systems need tenant isolation at the index level (separate namespaces or filtered search). Versioned documentation needs version-aware retrieval: a question about “v2 API” must not return v3 chunks. Build metadata filters into every query, not as an afterthought.

# Version-aware retrieval filter
filter = {
    "tenant_id": tenant_id,
    "version": {"$in": [requested_version, "latest"]},  # fallback to latest
    "doc_type": {"$in": ["spec", "guide", "reference"]}
}

Evaluation you can run tomorrow

Don’t ship RAG without an eval harness. Start with three metrics:

Retrieval precision@k: Of the top-k chunks, how many are actually relevant? Label 100 query-chunk pairs manually. Target > 0.7 at k=5.

Answer faithfulness: Does the generated answer contradict the retrieved context? Use an LLM judge (cheap model, strict prompt) to flag hallucinations. Target < 5% contradiction rate.

Answer relevance: Does the answer address the user question? Separate from faithfulness — a faithful but irrelevant answer (correctly quoting the wrong doc) is still a failure.

# Lightweight eval loop
async def evaluate(test_set: list[TestCase]) -> EvalResults:
    results = []
    for case in test_set:
        docs = await retrieve(case.query)
        answer = await generate(case.query, docs)
        
        faithfulness = await judge_faithfulness(answer, docs)
        relevance = await judge_relevance(answer, case.query)
        precision = retrieval_precision(docs, case.relevant_doc_ids)
        
        results.append(EvalResult(
            query=case.query,
            faithfulness=faithfulness,
            relevance=relevance,
            precision=precision
        ))
    return aggregate(results)

Run this weekly. Track regressions when you change embedding models, chunking, or prompts.

Operational considerations

Index updates. Re-embed changed documents incrementally. Watch for embedding model version drift — if you switch from ada-002 to text-embedding-3-large, re-index everything or maintain separate indices per model version.

PII and secrets. Scrub API keys, passwords, and PII before indexing. Use a regex/ML pipeline in the ingestion path. Log retrieval hits on sensitive chunks for audit.

Cost control. Cache embeddings for repeated queries. Route simple factual queries to a smaller generator. Set a max context token budget (e.g., 4k tokens) and truncate retrieved chunks proportionally.

Observability. Log every query with: retrieval latency, rerank latency, generation latency, tokens in/out, retrieved doc IDs, and user feedback (thumbs up/down). This is your debugging data when answers go wrong.

Summary

RAG grounds LLM responses by making external knowledge retrievable and citable. The engineering work lives in chunking strategy, hybrid retrieval with reranking, prompt discipline, and continuous evaluation. Long context windows don’t eliminate the need for retrieval — they change the economics of when to retrieve vs. when to stuff. Build the eval harness first, then iterate the pipeline. The difference between a demo and a production system is almost entirely in the retrieval quality and the refusal behavior when retrieval fails.