RAG explained for non-technical readers

RAG explained simply: Retrieval-Augmented Generation is a technique that gives large language models access to external knowledge at inference time by retrieving relevant documents and feeding them into the model’s context window. Instead of relying solely on parametric knowledge baked into weights during training, the model grounds its responses in specific, up-to-date source material you control. This shifts the problem from “what does the model know?” to “what documents should the model see?”

How RAG works

The RAG pipeline has three stages: indexing, retrieval, and generation. Each stage has design decisions that materially affect quality.

Indexing

You chunk source documents (PDFs, markdown, Confluence pages, database rows) into passages, embed each passage with an embedding model, and store the vectors in a vector database alongside metadata (source URL, section title, timestamps, access controls).

# Minimal indexing sketch
from sentence_transformers import SentenceTransformer
import chromadb

embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
client = chromadb.PersistentClient(path="./chroma")
collection = client.get_or_create_collection("docs")

def index_document(doc_id: str, text: str, metadata: dict):
    chunks = chunk_text(text, max_tokens=512, overlap=50)
    embeddings = embedder.encode(chunks).tolist()
    ids = [f"{doc_id}-{i}" for i in range(len(chunks))]
    metadatas = [{**metadata, "chunk_index": i, "text": c} for i, c in enumerate(chunks)]
    collection.add(ids=ids, embeddings=embeddings, metadatas=metadatas)

Chunking strategy matters. Fixed-size token windows with overlap work for many cases, but semantic chunking (splitting on headings, paragraphs, or using an LLM to identify logical boundaries) often preserves context better. Store the raw text in metadata so you can return it verbatim at retrieval time without a second lookup.

Retrieval

At query time, you embed the user’s question with the same embedding model, run a nearest-neighbor search in the vector store, and return the top-k passages. Several refinements improve recall and precision:

• Hybrid search: Combine vector similarity with BM25/keyword search. Vector search catches semantic matches; keyword search catches exact terminology, model numbers, error codes.
• Query rewriting: Use a small LLM to expand or decompose the user query into multiple search queries (e.g., “How do I configure retries?” → [“retry configuration”, “backoff policy”, “max attempts”]).
• Reranking: Feed the top 20–50 candidates through a cross-encoder reranker (e.g., BAAI/bge-reranker-base) to reorder by true relevance before passing to the generator.
• Metadata filtering: Restrict search by tenant, document version, date range, or permission tags.

def retrieve(query: str, k: int = 8, filters: dict = None):
    query_emb = embedder.encode([query]).tolist()
    results = collection.query(
        query_embeddings=query_emb,
        n_results=k * 3,  # fetch more for reranking
        where=filters
    )
    passages = results["metadatas"][0]
    # optional: rerank here
    return passages[:k]

Generation

You construct a prompt that includes the retrieved passages, the user’s question, and instructions for the model. The prompt template is where you enforce citation behavior, tone, and refusal policies.

SYSTEM_PROMPT = """You are a precise technical assistant. Answer the user's question using ONLY the provided context.
Cite sources inline using [doc_id] format. If the context does not contain the answer, say you don't know."""

def generate_answer(query: str, passages: list[dict]) -> str:
    context_blocks = []
    for p in passages:
        context_blocks.append(f"[{p['doc_id']}] {p['text']}")
    context = "\n\n".join(context_blocks)
    
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"}
    ]
    return llm.chat(messages)

Why RAG matters

RAG solves three problems that parametric models cannot:

Knowledge freshness. Model training cuts off at a point in time. Your API docs, pricing pages, and compliance policies change weekly. RAG lets you update the index without retraining.

Source attribution. Regulated industries (finance, healthcare, legal) require traceability. RAG makes citations a first-class output, not a post-hoc rationalization.

Access control. Different users see different documents. RAG enforces this at retrieval time — the model never sees passages the user isn’t authorized to read.

Cost control. Stuffing an entire knowledge base into a 128k or 1M token context window is expensive and degrades reasoning. Retrieval brings only what’s relevant.

A concrete example: internal developer portal

Imagine a platform team maintains 2,000 markdown files covering service templates, deployment procedures, incident runbooks, and API contracts. Engineers ask questions in Slack: “How do I add a new environment to the staging cluster?” “What’s the current rate limit for the payments API?” “Walk me through the rollback procedure for the auth service.”

Without RAG, the model hallucinates plausible but wrong runbook steps. With RAG:

1. Indexing: Nightly job chunks all markdown, embeds with bge-small-en-v1.5, stores in Chroma with metadata: service, doc_type (runbook, api, template), last_updated, owner_team.
2. Retrieval: User asks about “staging cluster environment”. Query rewritten to [“staging cluster environment setup”, “add environment staging”, “cluster configuration staging”]. Hybrid search + rerank returns 5 passages from the cluster ops runbook and the environment template.
3. Generation: Prompt includes passages, user question, and a system instruction to cite doc_id. Model responds with steps and inline citations like [cluster-ops-runbook-v3] and [env-template-v12].
4. Feedback loop: Engineers upvote/downvote answers. Downvoted queries trigger a review queue to fix missing or stale docs.

This runs in production at companies using n4n.ai to route generation calls across providers while keeping retrieval logic in their own infrastructure.

Common misconceptions

“RAG is just vector search”

Vector search is the retrieval component. RAG includes the full loop: indexing strategy, query understanding, retrieval, reranking, prompt construction, generation, citation enforcement, and evaluation. Treating it as “embed and search” produces fragile demos that fail on ambiguous queries, stale docs, or permission boundaries.

“Larger context windows make RAG obsolete”

Context windows grow, but three constraints remain: cost (input tokens are billed), latency (more tokens = slower first token), and attention dilution (models lose focus over long contexts). Retrieval concentrates the model’s attention on the 2–10 passages that actually answer the question. Even with 1M token contexts, you still need retrieval for cost, speed, and precision.

“One embedding model fits all use cases”

General-purpose embeddings (text-embedding-3-small, bge-base) work for broad semantic similarity. They struggle with:

• Domain-specific jargon (medical codes, legal citations, internal acronyms)
• Exact-match requirements (error codes, model numbers, config keys)
• Multilingual collections without language-specific tuning

Fine-tuning an embedding model on your own query-passage pairs (even a few thousand) often yields 10–20% recall improvement. Alternatively, use a hybrid BM25 + vector approach to cover exact matches without retraining.

“Chunk size doesn’t matter much”

Chunk size directly controls the granularity of retrieval and the context fed to the generator. Too small (128 tokens): passages lack context, model can’t synthesize across sections. Too large (2048 tokens): noise drowns signal, citation precision drops, token waste increases. Start with 512 tokens and 10–15% overlap; evaluate against a held-out QA set; adjust.

“RAG eliminates hallucination”

RAG grounds generation in retrieved evidence, but the model can still:

• Misread a passage (especially tables, lists, or negated statements)
• Conflate two similar passages
• Ignore the context and fall back to parametric knowledge
• Hallucinate citations that look plausible but don’t match the source

Mitigations: strict system prompts, citation verification post-processing, and an evaluation harness that measures groundedness (e.g., using llm-as-judge or NLI models to check entailment between answer and cited passages).

“You need a vector database”

Vector databases (Pinecone, Weaviate, Qdrant, Chroma, Milvus) add scaling, filtering, and managed infrastructure. For prototypes, internal tools, or datasets under ~100k vectors, sqlite-vec, pgvector, or even FAISS on disk with a metadata JSONL file work fine. Choose the simplest thing that meets your latency and scale requirements.

Evaluation: the part everyone skips

You cannot improve what you don’t measure. Build a small evaluation set (50–200 realistic questions with gold-standard answers and expected source doc IDs). Track:

• Retrieval recall@k: Does the correct passage appear in the top-k?
• Answer correctness: Does the generated answer match the gold answer? (Use LLM-as-judge with a rubric.)
• Citation accuracy: Do cited doc IDs match the passages that actually support the answer?
• Latency: p50/p95 end-to-end, broken down by retrieval vs. generation.

Run this evaluation on every indexing pipeline change, embedding model swap, prompt tweak, and chunking strategy change. CI integration catches regressions before they hit users.

Where to start

1. Collect 20 real questions from your team or support logs. Write ideal answers with source references.
2. Index a representative corpus (100–500 docs) with a default chunking strategy and bge-small-en-v1.5.
3. Build the minimal pipeline: embed query → vector search → top-5 → prompt → generate.
4. Run your 20 questions. Measure recall@5 and answer quality.
5. Iterate on the biggest gap: if retrieval misses, try hybrid search or query rewriting. If generation hallucinates, tighten the system prompt and add citation verification. If latency is high, reduce k or switch to a smaller generator.

RAG is not a single component — it’s a system. The teams that ship reliable RAG products treat retrieval, generation, and evaluation as equally important engineering surfaces.