How embeddings power semantic search

Embeddings for semantic search turn text into dense vectors that capture meaning, not just keywords. This guide walks through the full pipeline — from model selection to production indexing — with concrete code and the tradeoffs you’ll hit at each step.

What embeddings actually are

An embedding model maps variable-length text to a fixed-dimensional vector where semantic similarity corresponds to geometric proximity. Cosine similarity between vectors approximates semantic relatedness. The math is straightforward: normalize vectors to unit length, then compute dot product.

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

But the model choice determines everything downstream. A 384-dimension MiniLM runs in 10ms on CPU. A 4096-dimension E5-mistral needs GPU and 100ms. Both return “similar” vectors for “car” and “automobile,” but the larger model distinguishes “bank loan” from “river bank” more reliably.

Choosing a model

Start with the MTEB leaderboard, but filter for your constraints. Three dimensions matter:

Latency budget: If you embed at query time, 50ms is your ceiling. That rules out most 7B+ parameter models unless you batch aggressively.

Domain match: General models (E5, BGE, GTE) handle broad queries. For legal, biomedical, or code, fine-tuned variants (Legal-BERT, BioBERT, CodeBERT) win by 10-15 nDCG points.

Multilingual needs: If you serve non-English traffic, pick a model trained on 50+ languages (multilingual-e5-large, jina-embeddings-v2-base-en). Single-language models degrade sharply on accented or transliterated text.

# Quick local benchmark
from sentence_transformers import SentenceTransformer
import time

models = [
    "sentence-transformers/all-MiniLM-L6-v2",      # 384 dim, ~10ms CPU
    "BAAI/bge-small-en-v1.5",                       # 384 dim, ~15ms CPU
    "intfloat/multilingual-e5-large",               # 1024 dim, ~80ms GPU
]

sentences = ["How do I reset my password?", "Password reset instructions"] * 100

for name in models:
    model = SentenceTransformer(name)
    start = time.perf_counter()
    _ = model.encode(sentences, batch_size=32, show_progress_bar=False)
    elapsed = (time.perf_counter() - start) * 1000 / len(sentences)
    print(f"{name}: {elapsed:.1f}ms per sentence")

Run this on your target hardware. Published benchmarks use batch sizes and hardware you don’t have.

Building the index

Vector search libraries fall into three tiers:

For most teams, FAISS IVF+PQ hits the sweet spot. Index build is offline; query latency is sub-millisecond on CPU.

import faiss
import numpy as np

def build_ivf_pq_index(vectors: np.ndarray, nlist: int = 1024, m: int = 16) -> faiss.Index:
    """
    vectors: (N, d) float32, already L2-normalized
    nlist: number of Voronoi cells (sqrt(N) is a good start)
    m: number of PQ subquantizers (d must be divisible by m)
    """
    d = vectors.shape[1]
    quantizer = faiss.IndexFlatIP(d)  # inner product = cosine on normalized vectors
    index = faiss.IndexIVFPQ(quantizer, d, nlist, m, 8)  # 8 bits per subquantizer
    
    index.train(vectors)
    index.add(vectors)
    index.nprobe = 32  # search 32 cells at query time
    return index

Pitfall: Forgetting to normalize before IndexFlatIP. Inner product on unnormalized vectors ranks by magnitude, not angle. Always faiss.normalize_L2(vectors) after encoding.

Pitfall: Setting nprobe too low. Default is 1. At nprobe=1, IVF recalls 60-70%. At nprobe=32, you recover 95%+ with 2-3x latency. Tune this per workload.

Query processing and retrieval

Query embedding is the same model, same preprocessing. But three practical details change results:

Instruction tuning: E5 and BGE models expect a prefix: "query: " for queries, "passage: " for documents. Omitting this drops recall 5-10%.

def embed_query(model, text: str) -> np.ndarray:
    return model.encode(f"query: {text}", normalize_embeddings=True)

def embed_passages(model, texts: list[str]) -> np.ndarray:
    return model.encode([f"passage: {t}" for t in texts], normalize_embeddings=True)

Query expansion: For short queries (“error 500”), expand with hyponyms or rewrite via LLM before embedding. A single rewrite call adds 200ms but can lift recall@10 by 15%.

# Simple synonym expansion — replace with LLM rewrite for production
SYNONYMS = {
    "error": ["exception", "failure", "bug", "issue"],
    "login": ["sign in", "authenticate", "auth"],
    "reset": ["recover", "restore", "change"],
}

def expand_query(query: str) -> list[str]:
    words = query.lower().split()
    expanded = [query]
    for w in words:
        if w in SYNONYMS:
            for syn in SYNONYMS[w]:
                expanded.append(query.replace(w, syn))
    return expanded

Filtering before search: If you have hard filters (tenant_id, date_range, permission), apply them after vector search, not before. Pre-filtering requires a separate index per filter combination. Post-filtering with IndexIDMap or payload filtering (in Qdrant, Weaviate, Pinecone) keeps one index.

# FAISS post-filter pattern
ids = np.arange(len(vectors), dtype=np.int64)
index = faiss.IndexIDMap(index)
index.add_with_ids(vectors, ids)

# At query time: retrieve 100, filter in Python, return top 10
D, I = index.search(query_vec, k=100)
filtered = [(d, i) for d, i in zip(D[0], I[0]) if metadata[i]["tenant_id"] == current_tenant]
results = filtered[:10]

Reranking and hybrid search

Embeddings retrieve candidates. A cross-encoder reranker scores (query, doc) pairs with full attention. This is expensive — 50-200ms per pair — so rerank only the top 50-100.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, candidates: list[dict], top_k: int = 10) -> list[dict]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)
    for c, s in zip(candidates, scores):
        c["rerank_score"] = float(s)
    candidates.sort(key=lambda x: x["rerank_score"], reverse=True)
    return candidates[:top_k]

Hybrid search: Combine vector scores with BM25. Lexical matching catches exact terms (error codes, product IDs) that embeddings blur. Weighted reciprocal rank fusion (RRF) is simple and effective:

def rrf_fusion(vector_results: list[dict], bm25_results: list[dict], k: int = 60) -> list[dict]:
    """
    RRF: score = sum(1 / (k + rank)) across result sets
    """
    scores = {}
    for rank, doc in enumerate(vector_results):
        scores[doc["id"]] = scores.get(doc["id"], 0) + 1.0 / (k + rank + 1)
    for rank, doc in enumerate(bm25_results):
        scores[doc["id"]] = scores.get(doc["id"], 0) + 1.0 / (k + rank + 1)
    
    # Merge and sort
    all_docs = {d["id"]: d for d in vector_results + bm25_results}
    return sorted(all_docs.values(), key=lambda d: scores[d["id"]], reverse=True)

Set k=60 as default. Lower k favors top-ranked items; higher k smooths across deeper results.

Common pitfalls

Chunking strategy: Embedding whole documents loses granularity. Embedding sentences loses context. The sweet spot: 256-512 token chunks with 50-token overlap. For code, chunk by function/class. For legal, chunk by section.

def chunk_text(text: str, tokenizer, max_tokens: int = 512, overlap: int = 50) -> list[str]:
    tokens = tokenizer.encode(text)
    chunks = []
    for i in range(0, len(tokens), max_tokens - overlap):
        chunk_tokens = tokens[i:i + max_tokens]
        chunks.append(tokenizer.decode(chunk_tokens))
    return chunks

Stale embeddings: When content updates, re-embed. A nightly batch job works for most. For real-time needs, use a write-behind queue: write to primary DB, push embedding job to queue, update vector index asynchronously. Accept eventual consistency.

Dimension mismatch: Changing models changes dimensions. Your index is tied to a specific model version. Version your index: index_v1_minilm, index_v2_bge. Keep both during migration. Route queries by model version header.

Cold start: New content has no embeddings. Either embed synchronously on write (adds latency) or accept a visibility window. For user-generated content, synchronous embed with a 384-dim model adds ~15ms — usually acceptable.

Production considerations

Monitoring: Track three metrics per query: embedding latency, vector search latency, rerank latency. Alert on p99 > 200ms total. Log recall@k via sampled human evaluation — automated metrics drift.

Capacity planning: FAISS IVF+PQ stores ~1 byte per dimension per vector (PQ compression). 10M vectors × 768 dims ≈ 7.5 GB RAM. HNSW stores full vectors: 10M × 768 × 4 bytes ≈ 30 GB. Plan for 3x headroom.

Rolling updates: To update an IVF index without downtime, build a new index, swap the reference, retire the old. HNSW supports incremental add/remove but degrades over time — rebuild weekly.

Fallback: When your vector index is unavailable, fall back to BM25 or keyword search. The degradation is graceful, not catastrophic. n4n.ai’s routing layer can redirect embedding calls to a backup provider if your primary embedding endpoint fails, but the same principle applies at the application layer: always have a lexical backup.

# Simple fallback pattern
def search(query: str, top_k: int = 10) -> list[dict]:
    try:
        return vector_search(query, top_k)
    except (TimeoutError, ConnectionError, faiss.FaissError) as e:
        logger.warning(f"Vector search failed: {e}, falling back to BM25")
        return bm25_search(query, top_k)

Scaling beyond single-node

When query volume exceeds one machine, shard by tenant or by embedding namespace. Each shard gets its own index. A router directs queries to the correct shard. For multi-tenant systems, per-tenant indexes isolate noisy neighbors but increase memory overhead. Shared index with tenant_id filtering is more efficient until a single tenant dominates traffic.

Replication: run 3+ replicas per shard behind a load balancer. Health checks must verify index responsiveness, not just process liveness — a stuck GC pause on a 30 GB HNSW index looks healthy to kubelet but returns 5s latencies.

Start small: MiniLM + FAISS IVF+PQ + BM25 fallback gets you 80% of the way. Add reranking, hybrid fusion, and cross-encoder when metrics demand it. The embedding model is the easiest component to swap; the indexing and serving infrastructure is where you invest.