What is RAG? Retrieval-augmented generation explained

Retrieval-augmented generation (RAG) is a technique that grounds large language model outputs in external knowledge by retrieving relevant documents before generation. Instead of relying solely on parametric knowledge baked into model weights during training, RAG systems fetch context from a knowledge base at inference time and feed it to the model alongside the user query. This architecture separates knowledge storage from reasoning, letting you update facts without retraining.

How RAG works

A minimal RAG pipeline has three stages: indexing, retrieval, and generation. Each stage introduces design decisions that affect latency, cost, and answer quality.

Indexing

You chunk source documents (PDFs, markdown, Confluence pages, code) into segments small enough to fit in a context window but large enough to preserve semantic coherence. Common chunk sizes range from 256 to 1024 tokens with 10–20% overlap. Each chunk gets embedded into a vector using a model like text-embedding-3-small or bge-large-en-v1.5, then stored in a vector database alongside metadata (source URL, page number, document ID).

# Minimal indexing pipeline
from langchain.text_splitter import RecursiveCharacterTextSplitter
from openai import OpenAI

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)

def index_documents(docs: list[dict]) -> list[dict]:
    """Return list of {text, embedding, metadata} ready for upsert."""
    client = OpenAI()
    chunks = []
    for doc in docs:
        for chunk in splitter.split_text(doc["content"]):
            resp = client.embeddings.create(
                model="text-embedding-3-small",
                input=chunk
            )
            chunks.append({
                "text": chunk,
                "embedding": resp.data[0].embedding,
                "metadata": {
                    "source_id": doc["id"],
                    "source_url": doc.get("url"),
                }
            })
    return chunks

Choose your embedding model based on the retrieval task. General-purpose models work for most English text. Domain-specific models (legal, biomedical, code) improve recall on specialized corpora. Multilingual models like intfloat/multilingual-e5-large handle mixed-language collections.

Retrieval

At query time, you embed the user question with the same model, then search the vector index for top-k nearest neighbors. Cosine similarity is the default metric; dot product works when vectors are normalized. Typical k values range from 4 to 20 depending on context budget.

def retrieve(query: str, index, k: int = 8) -> list[dict]:
    client = OpenAI()
    q_emb = client.embeddings.create(
        model="text-embedding-3-small",
        input=query
    ).data[0].embedding
    
    # Pseudocode — replace with your vector DB client
    results = index.query(vector=q_emb, top_k=k, include_metadata=True)
    return [
        {"text": m.metadata["text"], "score": m.score, "source": m.metadata["source_id"]}
        for m in results.matches
    ]

Hybrid search — combining vector similarity with keyword (BM25) scores — often outperforms pure vector search on entity-heavy queries (product codes, error messages, proper nouns). Reranking with a cross-encoder (e.g., cross-encoder/ms-marco-MiniLM-L-6-v2) further improves precision by scoring query-document pairs jointly rather than independently.

Generation

Retrieved chunks are formatted into a prompt template with the user query. The template should instruct the model to cite sources, refuse when context is insufficient, and avoid hallucination. A typical system prompt:

You are a precise technical assistant. Answer the user's question using ONLY the provided context.
Cite sources inline like [doc-12], [doc-45]. If the context doesn't contain the answer, say so.

def generate_answer(query: str, context_chunks: list[dict], model: str = "gpt-4o-mini") -> str:
    context = "\n\n".join(
        f"[Source: {c['source']}]\n{c['text']}" for c in context_chunks
    )
    
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}
    ]
    
    client = OpenAI()
    resp = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0.1,
        max_tokens=1024
    )
    return resp.choices[0].message.content

Token budget management matters. If retrieved context exceeds the model’s context window (minus room for the query and response), truncate lowest-score chunks first. Some teams summarize long chunks before inclusion; others use hierarchical retrieval (retrieve section summaries, then expand only the relevant sections).

Why RAG matters for production systems

Parametric knowledge has three hard limits: it freezes at training cutoff, it cannot access private data, and it hallucinates confidently on low-probability tokens. RAG addresses all three.

Freshness without retraining. When your API docs change, you re-index the affected pages — minutes, not weeks. No GPU hours, no distillation pipeline, no eval suite re-run.

Access control. Retrieval runs in your infrastructure with your permissions. The LLM only sees chunks the user is authorized to read. This is harder with fine-tuning, where data bakes into weights.

Attribution. Citations let users verify answers. In regulated domains (medical, legal, financial), this is often a compliance requirement, not a nice-to-have.

Cost control. A 7B parameter model with good retrieval often beats a 70B model without it on domain tasks. Smaller models mean lower latency and lower per-token cost.

Concrete example: Internal developer assistant

Imagine a 200-engineer organization with 3 years of Notion docs, GitHub wikis, and Slack exports. Engineers ask: “How do I rotate the staging database credentials?” “What’s the retry policy for the payments webhook?” “Which team owns the user-events Kafka topic?”

Naive approach (fails)

Feed the entire corpus into a long-context model. Problems: 500K+ tokens exceeds most context windows; irrelevant context degrades reasoning; no access control; cost scales linearly with corpus size.

RAG approach (works)

1. Ingest: Nightly pipeline pulls Notion pages, wiki markdown, and curated Slack threads. Chunk by heading hierarchy. Embed with text-embedding-3-large. Upsert to Pinecone/Weaviate/Qdrant with metadata: team, last_updated, sensitivity_level.

2. Query: Engineer asks in Slack bot. Bot embeds query, retrieves top-12 chunks hybrid (vector + BM25), reranks with cross-encoder, keeps top-6.

3. Filter: Drop chunks where sensitivity_level > user_clearance.

4. Generate: Feed filtered chunks to gpt-4o-mini with citation-enforced prompt. Return answer with source links.

5. Log: Store query, retrieved chunk IDs, generated answer, user feedback (thumbs up/down) for eval.

Latency: ~800ms p50 (embedding + retrieval + generation). Cost: ~$0.002/query. Accuracy: measurable via labeled eval set — target 90%+ correct on “how-to” and “ownership” queries.

Common misconceptions

“RAG is just vector search”

Vector search is the retrieval component. RAG includes the generation component with its own failure modes: context stuffing, lost-in-the-middle, citation hallucination, and prompt sensitivity. Treating them as identical leads to under-investing in generation-side eval.

“Long-context models make RAG obsolete”

Models with 1M+ token contexts (Gemini 1.5, Claude 3 Opus) change the tradeoffs but don’t eliminate RAG. Three reasons: (1) Private data still shouldn’t leave your VPC. (2) Full-corpus attention is O(n²) — expensive and slow at scale. (3) Irrelevant context actively harms reasoning on complex tasks (the “lost in the middle” effect persists even at 100K tokens).

“Chunk size doesn’t matter much”

Chunk size is a primary quality lever. Too small (128 tokens): fragments lose context, retrieval returns disjointed snippets. Too large (2048 tokens): relevant signal dilutes in noise, context budget wastes on boilerplate. Optimal size depends on document structure — code wants smaller chunks (function-level), prose wants larger (section-level). Test with your eval set.

“Embedding model choice is set-and-forget”

Embedding models age. Newer models (text-embedding-3-large, bge-m3, nomic-embed-text-v1.5) improve on older ones (text-embedding-ada-002) significantly on MTEB benchmarks. Re-embedding a 10M-chunk corpus is non-trivial but worthwhile annually. Version your embeddings: store embedding_model_version in metadata so you can migrate incrementally.

“RAG eliminates hallucination”

RAG reduces hallucination by constraining the generation space. It doesn’t eliminate it. Models still hallucinate citations (citing a real doc for a claim it doesn’t support), conflate similar entities, and over-extrapolate from partial context. Guardrails: citation verification pass, confidence thresholds, and “I don’t know” training examples in your few-shot prompt.

“One retrieval strategy fits all queries”

Fact-seeking queries (“What’s the SSL cert expiry?”) need precise retrieval — high k, reranking, maybe SQL lookup. Exploratory queries (“How do we handle idempotency?”) need breadth — diverse retrieval, maybe query expansion. Code generation needs repository-aware retrieval (import graph, call hierarchy). Build a query classifier that routes to specialized retrievers.

Evaluation you can actually run

Don’t ship RAG without an eval harness. Minimum viable eval:

# eval_set.jsonl — one per line
{"query": "How do I rotate staging DB creds?", "expected_sources": ["doc-infra-44", "doc-security-12"], "must_contain": ["vault", "rotate", "staging"]}
{"query": "Who owns user-events topic?", "expected_sources": ["doc-kafka-07"], "must_contain": ["data-platform", "team"]}

def evaluate(rag_pipeline, eval_path: str) -> dict:
    import json
    from difflib import SequenceMatcher
    
    hits = 0
    citation_recall = 0
    total = 0
    
    with open(eval_path) as f:
        for line in f:
            case = json.loads(line)
            result = rag_pipeline(case["query"])
            answer = result["answer"].lower()
            sources = {c["source"] for c in result["citations"]}
            
            # Source recall
            expected = set(case["expected_sources"])
            if expected & sources:
                citation_recall += 1
            
            # Content match (fuzzy)
            content_ok = all(term.lower() in answer for term in case["must_contain"])
            if content_ok:
                hits += 1
            total += 1
    
    return {
        "answer_accuracy": hits / total,
        "citation_recall": citation_recall / total,
        "total_cases": total
    }

Run this nightly. Track regressions when you change chunking, embedding model, prompt template, or LLM version. Add adversarial cases: out-of-domain queries, ambiguous pronouns, multi-hop questions.

Operational concerns

Index freshness. Stale answers erode trust faster than no answer. Automate re-indexing on document change events (webhooks from Notion, GitHub, Confluence). For batch sources, schedule nightly with change detection (hash comparison).

Latency budgets. Retrieval should be <200ms p99. Generation dominates. Stream tokens to the user; don’t wait for full completion. Cache frequent queries (exact match or semantic similarity >0.95) with TTL.

Cost attribution. Tag each request with team, project, environment. Roll up embedding tokens, retrieval calls, and generation tokens separately. The embedding cost is often invisible but significant at scale.

PII scrubbing. Run a detector (Presidio, AWS Comprehend, or regex rules) on chunks before indexing. Redact or drop. Once PII hits the vector DB, it’s in the ANN graph — hard to fully purge.

Multi-tenancy. If you serve multiple customers from one deployment, namespace your vector index by tenant_id. Never rely on metadata filtering alone for isolation — bugs leak data. Separate indexes or separate collections per tenant.

When not to use RAG

• Static, public knowledge that the model already knows well (capital cities, syntax for popular languages). Retrieval adds latency for no gain.
• Ultra-low-latency requirements (<100ms) where even cached retrieval misses the budget. Consider distilled classifiers or rule-based routing instead.
• Highly structured analytical queries (“Sum of Q3 revenue by region”). Text-to-SQL or a semantic layer beats unstructured retrieval.
• Creative tasks where grounding constrains the desired output (brainstorming, fiction, open-ended design).

What to build next

Start with a 50-doc eval set, a hybrid retriever (vector + BM25), a cross-encoder reranker, and a citation-enforced prompt. Measure. Then iterate on the component that hurts most: retrieval recall, generation faithfulness, or latency. The architecture is simple; the quality comes from disciplined eval and incremental improvement on real queries.