Embeddings and RAG: how retrieval actually works

Embeddings rag retrieval is the backbone of every production RAG system, yet most tutorials stop at “chunk, embed, cosine similarity.” That works for demos. In production, you need to understand why retrieval fails, how to measure it, and which knobs actually move the needle. This guide walks through the retrieval pipeline end to end with code you can adapt.

The retrieval pipeline in five stages

Every RAG system runs the same logical pipeline, whether you build it yourself or use a framework:

1. Ingestion — split documents into chunks, compute embeddings, store vectors + metadata
2. Indexing — build a search structure (HNSW, IVF, flat) over those vectors
3. Query embedding — embed the user question with the same model
4. Search — find top-k nearest neighbors, optionally filtered by metadata
5. Reranking — score candidates with a cross-encoder or LLM judge before passing to the generator

Skipping or weakening any stage degrades answer quality. The most common failure mode is treating stage 1 as “done” after a one-time embed-and-index job.

Chunking: the silent quality killer

Chunk size and overlap directly control what the retriever can see. Too small — you lose context. Too large — you dilute signal and waste context window.

# Naive fixed-size chunking (avoid)
def chunk_fixed(text: str, size: int = 512, overlap: int = 50) -> list[str]:
    return [text[i:i+size] for i in range(0, len(text), size - overlap)]

# Better: recursive character splitting respects boundaries
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
)
chunks = splitter.split_text(document_text)

Rule of thumb: match chunk size to your embedding model’s training context. For text-embedding-3-small (8192 tokens), 512–1024 token chunks work well. For older models with 512-token limits, stay under 256.

Pitfall: chunking by token count without respecting document structure (headings, tables, code blocks) splits related content across chunks. Use a structure-aware splitter for Markdown, HTML, or PDFs.

Embedding model selection

You have three tiers, each with different tradeoffs:

Don’t mix models. The query and document embeddings must come from the same model. If you re-embed your corpus, re-embed your queries too.

# OpenAI-compatible embedding call (works with n4n.ai, OpenAI, local proxies)
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

def embed(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
        encoding_format="float",
    )
    return [d.embedding for d in resp.data]

Vector indexing: HNSW is the default

For most workloads under 10M vectors, HNSW (Hierarchical Navigable Small World) gives the best recall/latency tradeoff. Key parameters:

• M (bi-directional links per node): 16–48. Higher = better recall, more memory.
• ef_construction (index build exploration): 100–400. Higher = better graph, slower build.
• ef_search (query-time exploration): 50–200. Tune at query time for recall/latency.

import faiss
import numpy as np

dim = 1536  # text-embedding-3-small
index = faiss.IndexHNSWFlat(dim, 32)  # M=32
index.hnsw.efConstruction = 200
index.hnsw.efSearch = 128  # adjust per query

# Add vectors (float32, L2-normalized for cosine similarity)
vectors = np.array(embeddings, dtype=np.float32)
faiss.normalize_L2(vectors)
index.add(vectors)

# Search
query_vec = np.array([query_embedding], dtype=np.float32)
faiss.normalize_L2(query_vec)
distances, indices = index.search(query_vec, k=10)

Pitfall: FAISS defaults to L2 distance. For cosine similarity, you must L2-normalize vectors before adding and searching, or use IndexFlatIP / IndexHNSWFlat with inner product.

Metadata filtering: do it before or during search

Post-filtering (retrieve 100, filter to 10) wastes compute and misses relevant results that fell outside top-100. Pre-filtering (filter index, then search) requires a vector DB that supports it.

# Pinecone-style filter at query time
results = index.query(
    vector=query_embedding,
    top_k=10,
    filter={"source": "confluence", "team": "platform"},
    include_metadata=True,
)

# Weaviate hybrid: vector + BM25 + filter
results = client.query.get("Document", ["text", "source"]).with_near_vector({
    "vector": query_embedding,
    "certainty": 0.7,
}).with_where({
    "path": ["source"],
    "operator": "Equal",
    "valueText": "confluence"
}).with_limit(10).do()

If your vector DB doesn’t support pre-filtering, consider a two-stage approach: filter metadata in your primary DB (PostgreSQL, Elasticsearch), fetch candidate IDs, then re-rank with vector scores.

Hybrid retrieval: vector + keyword

Pure vector search misses exact matches (error codes, product names, acronyms). Hybrid search combines dense vectors with sparse lexical scores (BM25, SPLADE).

# Reciprocal Rank Fusion (RRF) — simple, no training needed
def rrf_fuse(dense_results: list[dict], sparse_results: list[dict], k: int = 60) -> list[dict]:
    """
    dense_results: [{"id": "doc1", "score": 0.9}, ...] sorted by score desc
    sparse_results: same format
    """
    scores = {}
    for rank, r in enumerate(dense_results):
        scores.setdefault(r["id"], 0)
        scores[r["id"]] += 1 / (k + rank + 1)
    for rank, r in enumerate(sparse_results):
        scores.setdefault(r["id"], 0)
        scores[r["id"]] += 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

When to use hybrid: any domain with precise terminology (legal, medical, code, support tickets). Pure vector is fine for semantic QA over general prose.

Reranking: the highest ROI stage

A cross-encoder (reranker) scores (query, doc) pairs jointly. It’s slower than bi-encoder retrieval but dramatically improves precision@k.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)
    for c, s in zip(candidates, scores):
        c["rerank_score"] = float(s)
    return sorted(candidates, key=lambda x: x["rerank_score"], reverse=True)[:top_n]

Tradeoff: Cross-encoders process ~100–500 pairs/second on GPU. At k=50 candidates, that’s 100–500ms added latency. Typical production pattern: retrieve 50–100 with bi-encoder, rerank to 5–10.

Model choices: BGE-Reranker-v2-M3 (multilingual, 512 tokens), Cohere Rerank 3.5 (API, 4k tokens), Jina Reranker v2 (open, 8k tokens). Match context window to your chunk size.

Measuring retrieval quality

You cannot improve what you don’t measure. Build an eval set of (query, relevant_doc_ids) pairs — 50–200 is enough to start.

def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    return len(set(retrieved_ids[:k]) & relevant_ids) / len(relevant_ids)

def ndcg_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
    # Binary relevance NDCG
    dcg = sum(1 / np.log2(i + 2) for i, doc_id in enumerate(retrieved_ids[:k]) if doc_id in relevant_ids)
    ideal = sum(1 / np.log2(i + 2) for i in range(min(len(relevant_ids), k)))
    return dcg / ideal if ideal > 0 else 0.0

Track these per query type (fact lookup, summarization, code generation). A single aggregate number hides regressions on specific categories.

Common failure modes and fixes

Production checklist

Before shipping:

• Embedding model version pinned in config (not “latest”)
• Chunking strategy documented with rationale
• Eval set covering top 10 query categories
• Recall@10 > 0.8 on eval set (adjust threshold per domain)
• Reranker latency budgeted (p99 < 500ms added)
• Metadata filter pushed to vector DB pre-search
• Index rebuild pipeline tested (blue/green or rolling)
• Fallback to keyword search when vector returns low-confidence results

Scaling past single-node

Above ~5M vectors or 100 QPS, single-node FAISS hits memory and throughput limits. Options:

1. Sharded FAISS — split by tenant, time, or random hash. Query all shards, merge top-k.
2. Managed vector DB — Pinecone, Weaviate, Qdrant Cloud, Milvus/Zilliz. Handle replication, filtering, hybrid search.
3. Quantization — Product Quantization (PQ) or Scalar Quantization (SQ) reduces memory 4–8x with <2% recall loss.

# FAISS IVF+PQ for large scale (disk-backed possible)
nlist = 4096  # ~sqrt(N) clusters
quantizer = faiss.IndexFlatIP(dim)
index = faiss.IndexIVFPQ(quantizer, dim, nlist, 64, 8)  # 64 subvecs, 8 bits each
index.train(vectors)
index.add(vectors)
index.nprobe = 32  # search 32 clusters at query time

Closing thought

Retrieval is not “solved” by picking a vector DB. It’s a pipeline where each stage compounds. The teams shipping reliable RAG invest in eval-driven iteration: change one thing, measure, deploy, repeat. Start with a 50-query eval set, a fixed embedding model, and hybrid search + rerank. That gets you 80% of the way. The last 20% is domain-specific tuning — and that’s where the real work begins.