Retrieval-augmented generation (RAG) is a technique that grounds large language model outputs in external, verifiable data by retrieving relevant documents before generation. Instead of relying solely on parametric knowledge frozen at training time, the model conditions its response on fetched context — typically from a vector database, search index, or structured store. This shifts the burden of factual accuracy from model weights to a controllable retrieval pipeline.
How retrieval-augmented generation works
A RAG system has three moving parts: an indexer, a retriever, and a generator. The indexer chunks source documents, embeds them, and stores the vectors alongside metadata. The retriever takes a user query, embeds it, and runs a nearest-neighbor search (or hybrid lexical + semantic search) to return the top-k most relevant chunks. The generator receives the original query plus the retrieved chunks as context and produces the final answer.
# Minimal RAG pipeline using sentence-transformers and a vector store
from sentence_transformers import SentenceTransformer
import numpy as np
class SimpleRAG:
def __init__(self, model_name="sentence-transformers/all-MiniLM-L6-v2"):
self.encoder = SentenceTransformer(model_name)
self.documents = []
self.embeddings = None
def index(self, docs: list[str], metadatas: list[dict] | None = None):
self.documents = list(zip(docs, metadatas or [{}] * len(docs)))
self.embeddings = self.encoder.encode(docs, normalize_embeddings=True)
def retrieve(self, query: str, k: int = 4) -> list[tuple[str, dict, float]]:
q_emb = self.encoder.encode([query], normalize_embeddings=True)
scores = (self.embeddings @ q_emb.T).flatten()
top_idx = np.argpartition(scores, -k)[-k:]
top_idx = top_idx[np.argsort(scores[top_idx])[::-1]]
return [(self.documents[i][0], self.documents[i][1], float(scores[i])) for i in top_idx]
def generate(self, query: str, k: int = 4) -> str:
context = self.retrieve(query, k)
context_str = "\n\n".join(f"[Score: {s:.3f}] {doc}" for doc, _, s in context)
prompt = f"""Answer the question using only the context below.
If the answer isn't in the context, say you don't know.
Context:
{context_str}
Question: {query}
Answer:"""
# In production, call your LLM endpoint here
return prompt # Placeholder for demonstrationThe retriever is the quality gate. Dense retrieval (embeddings + ANN) handles semantic matching well but misses exact keywords, identifiers, and numbers. Hybrid search — combining BM25 or a learned sparse encoder with dense vectors — covers both. Re-ranking with a cross-encoder (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) on the top 50 candidates before feeding the top 5–10 to the generator typically adds 5–15 points on recall@k benchmarks.
Chunking strategy matters more than most teams admit. Fixed-size chunks (512 tokens, 100 overlap) are a reasonable default, but semantic chunking — splitting on headings, paragraphs, or logical boundaries — preserves coherence. For code, chunk by function or class. For legal or financial docs, preserve clause boundaries. Store the parent document ID and chunk index in metadata so you can reconstruct full sections at generation time if needed.
Why retrieval-augmented generation matters
Parametric knowledge has three hard limits: it cuts off at the training date, it hallucinates confidently on low-frequency facts, and it cannot reflect proprietary or user-specific data. RAG addresses all three.
First, freshness. A model trained in early 2024 knows nothing about API changes shipped last week. By indexing your changelog, docs, and support tickets, the same model answers correctly today and next month without retraining.
Second, attribution. When the generator cites [Score: 0.847] chunk_12, you can trace the answer to a source. This is non-negotiable for regulated domains — healthcare, finance, legal — where “the model said so” fails audit.
Third, access control. You can filter retrieval by user permissions before the generator ever sees the context. A support agent sees public docs plus internal runbooks; a customer sees only public docs. The model weights never touch restricted data.
Fourth, cost control. Stuffing 100k tokens of context into a long-context model costs more and degrades quality (lost-in-the-middle effect). Retrieving 3–5 relevant chunks (2–4k tokens) is cheaper and often more accurate.
A concrete example: internal developer assistant
Imagine an internal tool that answers questions about your platform’s APIs, deprecation policies, and incident runbooks. The corpus: 12k markdown files (API reference, architecture decision records, postmortems), 3k Jira tickets, and a Confluence space.
# Production-grade retrieval with hybrid search and re-ranking
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder
import numpy as np
class HybridRetriever:
def __init__(self, docs: list[str], metadatas: list[dict]):
self.docs = docs
self.metadatas = metadatas
self.dense_encoder = SentenceTransformer("sentence-transformers/all-mpnet-base-v2")
self.sparse_encoder = BM25Okapi([d.split() for d in docs])
self.dense_embeddings = self.dense_encoder.encode(docs, normalize_embeddings=True)
self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def retrieve(self, query: str, k: int = 5, candidate_k: int = 50) -> list[dict]:
# Dense candidates
q_dense = self.dense_encoder.encode([query], normalize_embeddings=True)
dense_scores = (self.dense_embeddings @ q_dense.T).flatten()
# Sparse candidates
sparse_scores = self.sparse_encoder.get_scores(query.split())
# Reciprocal rank fusion
dense_rank = np.argsort(dense_scores)[::-1]
sparse_rank = np.argsort(sparse_scores)[::-1]
rrf_scores = np.zeros(len(self.docs))
for rank, idx in enumerate(dense_rank[:candidate_k]):
rrf_scores[idx] += 1.0 / (rank + 60)
for rank, idx in enumerate(sparse_rank[:candidate_k]):
rrf_scores[idx] += 1.0 / (rank + 60)
# Top candidates for re-ranking
candidate_idx = np.argpartition(rrf_scores, -candidate_k)[-candidate_k:]
pairs = [[query, self.docs[i]] for i in candidate_idx]
rerank_scores = self.reranker.predict(pairs)
final_idx = candidate_idx[np.argsort(rerank_scores)[::-1][:k]]
return [
{"text": self.docs[i], "metadata": self.metadatas[i], "score": float(rerank_scores[list(candidate_idx).index(i)])}
for i in final_idx
]The generator prompt enforces citation discipline:
GENERATOR_PROMPT = """You are a platform engineer answering developer questions.
Use ONLY the provided context. Cite sources inline like [doc-123, chunk-4].
If context is insufficient, say "I don't have enough information to answer."
Context:
{context}
Question: {query}
Answer:"""A query like “How do I roll back a canary deployment for the payments service?” retrieves the deployment runbook, the payments service ADR, and the last incident postmortem. The answer cites each source. If the runbook was updated yesterday, the answer reflects that — no model update required.
Common misconceptions
“RAG is just vector search”
Vector search is the retriever. RAG is the full loop: indexing strategy, chunking, retrieval, re-ranking, context construction, prompt engineering, generation, citation, and evaluation. Teams that treat it as “embed, top-k, stuff into prompt” get poor results and blame the technique.
“Long-context models make RAG obsolete”
Models with 128k–1M token windows change the economics but not the fundamentals. Lost-in-the-middle degradation is real: models attend poorly to information buried in large contexts. Retrieval concentrates signal. Long context is useful for post-retrieval expansion — feeding full parent sections instead of chunks — not for replacing retrieval.
“One embedding model fits all domains”
General-purpose embeddings (e.g., all-MiniLM-L6-v2, text-embedding-3-small) work adequately for broad English text. They fail on code, multilingual corpora, legal language, and domain-specific jargon. Fine-tune a bi-encoder on in-domain pairs (query, relevant doc) or use a domain-adapted model like jina-embeddings-v2-base-code for codebases. The retrieval gap between generic and domain-tuned embeddings is often 15–30% recall@10.
“Chunk size doesn’t matter much”
It matters enormously. Too small (128 tokens): fragments lose context, retrieval returns orphaned sentences. Too large (2048 tokens): noise dilutes the signal, generator gets distracted, token budget wastes. Start with 512 tokens / 100 overlap for prose, 256 / 50 for code. Measure recall@k on a labeled eval set and tune.
“RAG eliminates hallucinations”
RAG reduces hallucinations by constraining the generator to retrieved evidence. It does not eliminate them. The generator can still misread a chunk, conflate two sources, or ignore the context and fall back on parametric knowledge. Guardrails help: strict citation requirements, a verifier pass that checks claims against sources, and a “I don’t know” trigger when retrieval scores fall below a threshold.
“You need a vector database”
You need a vector index. A dedicated vector database (Pinecone, Weaviate, Qdrant, Milvus) adds managed scaling, filtering, hybrid search, and multi-tenancy. For prototypes and moderate scale (< 5M vectors), faiss + sqlite or pgvector in Postgres works fine. Choose based on operational maturity, not hype.
Evaluation you can actually run
Don’t ship without an eval set. Build 50–200 representative queries with ground-truth answer spans (doc ID + chunk ID). Measure:
- Retrieval recall@k: fraction of queries where the ground-truth chunk appears in top-k
- Answer correctness: LLM-as-judge or human eval on generated answers
- Citation accuracy: do cited sources actually support the claim?
- Latency: p50/p95 end-to-end, broken down by retrieve / rerank / generate
def evaluate_retrieval(retriever, eval_set: list[dict], k: int = 10) -> float:
"""eval_set: [{'query': str, 'relevant_doc_ids': set[str]}]"""
hits = 0
for item in eval_set:
results = retriever.retrieve(item["query"], k=k)
retrieved_ids = {r["metadata"]["doc_id"] for r in results}
if item["relevant_doc_ids"] & retrieved_ids:
hits += 1
return hits / len(eval_set)Run this on every index rebuild, every embedding model swap, every chunking change. Treat retrieval metrics as regression tests.
Where this fits in a gateway architecture
If you operate an inference gateway that routes to multiple providers, RAG becomes a preprocessing step before the model call. The gateway receives the request, runs retrieval (or calls a retrieval service), constructs the augmented prompt, and forwards to the selected model. This keeps retrieval logic out of client code and lets you swap generators — OpenAI, Anthropic, open weights on self-hosted GPUs — without rewriting the grounding layer. The gateway can also enforce citation formats, truncate context to fit the model’s window, and log retrieval metadata for observability.
Start small: index your docs, build a 50-query eval set, measure recall@10. If it’s below 0.7, fix chunking or add hybrid search before touching the generator. The generator is rarely the bottleneck; the retriever is.