Semantic search retrieves documents by meaning rather than exact token overlap. It encodes queries and corpus passages into dense vectors using a trained encoder, then ranks results by vector similarity — typically cosine or dot product — so “how do I reset my password” matches “account recovery steps” even when they share no keywords. The approach trades exact-match precision for recall on paraphrases, synonyms, and conceptual relationships.
How semantic search works end to end
The pipeline has three stages: indexing, query encoding, and retrieval. Each stage has engineering decisions that materially affect latency, cost, and quality.
Indexing: from text to vectors
You pass every searchable chunk through an embedding model — BERT-family (e.g., sentence-transformers/all-MiniLM-L6-v2), E5, BGE, or a commercial API like OpenAI’s text-embedding-3-large. The model outputs a fixed-length vector (384–3072 dimensions depending on the model). You store these vectors alongside metadata (doc_id, chunk_id, timestamps, ACLs) in a vector index.
# Minimal indexing loop with sentence-transformers
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
def embed_chunks(chunks: list[str]) -> np.ndarray:
# Batch for throughput; normalize for cosine similarity
embeddings = model.encode(
chunks,
batch_size=256,
normalize_embeddings=True,
show_progress_bar=True,
)
return embeddings.astype("float32") # (N, 384)Chunking strategy matters more than model choice for many workloads. Fixed-size windows (256–512 tokens with 10–20% overlap) work for generic corpora. For code, use AST-aware splitters. For legal or technical docs, preserve section boundaries. Bad chunking — splitting mid-sentence or merging unrelated sections — degrades retrieval more than a weaker embedding model.
Store vectors in a proper ANN index: HNSW (pgvector, Weaviate, Qdrant, Milvus), IVF-PQ (Faiss), or DiskANN for billion-scale. Brute-force numpy.dot tops out around 100k vectors on a single machine. HNSW gives sub-millisecond latency at 10M+ vectors with 95%+ recall if you tune ef_construction and M for your recall target.
Query encoding: same model, same distribution
Encode the user query with the exact same model and preprocessing used at index time. Distribution shift between query and passage embeddings kills recall. If you lowercased and stripped punctuation at index time, do it at query time. If you prefixed passages with “passage: “ and queries with “query: “ (E5 style), keep the prefixes.
def search(query: str, index, top_k: int = 10) -> list[tuple[int, float]]:
q_vec = model.encode([query], normalize_embeddings=True).astype("float32")
# index.search returns (indices, distances) for HNSW/IVF
ids, scores = index.search(q_vec, top_k)
return list(zip(ids[0], scores[0]))Retrieval: ANN search + optional rerank
ANN search returns top_k candidates (typically 50–200) in milliseconds. For higher precision, rerank candidates with a cross-encoder — a model that attends over query + passage jointly. Cross-encoders are 10–100x slower than bi-encoders but significantly more accurate on hard negatives.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, candidates: list[str], top_n: int = 10) -> list[tuple[int, float]]:
pairs = [[query, c] for c in candidates]
scores = reranker.predict(pairs, batch_size=32)
ranked = sorted(zip(range(len(candidates)), scores), key=lambda x: x[1], reverse=True)
return ranked[:top_n]Hybrid search — combining BM25 (lexical) scores with vector scores — often beats pure semantic search on entity-heavy queries (product codes, error numbers, proper nouns). Reciprocal rank fusion (RRF) is a simple, parameter-light way to merge:
def rrf_fuse(ranked_lists: list[list[int]], k: int = 60) -> list[int]:
"""ranked_lists: each list is doc_ids in rank order"""
scores = {}
for ranked in ranked_lists:
for rank, doc_id in enumerate(ranked):
scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
return [doc_id for doc_id, _ in sorted(scores.items(), key=lambda x: -x[1])]Why semantic search matters for engineers
Keyword search (BM25, TF-IDF) fails on three classes of queries that dominate real traffic:
Synonymy and paraphrase. “Cancel subscription” vs “stop auto-renewal” vs “turn off recurring billing.” BM25 sees zero overlap. Semantic search places these in the same neighborhood because the training objective (contrastive learning on question-answer or query-passage pairs) forces paraphrases together.
Implicit intent. “Why is my build failing?” matches “CI pipeline exit code 137: OOM killer” even though the query contains none of the diagnostic terms. The encoder has learned that “build failing” co-occurs with CI error patterns in its training data.
Cross-lingual retrieval. Multilingual models (mE5, LaBSE, BGE-M3) let an English query retrieve Korean docs without translation. This is not perfect — performance drops 10–20% vs monolingual — but it works well enough for many product search use cases.
The trade-off: semantic search returns false positives on polysemous terms. “Apple” the fruit and “Apple” the company occupy nearby regions in many embedding spaces. Hybrid search or metadata filtering (category: “technology”) mitigates this.
Concrete example: technical documentation search
Consider a developer platform with 50k markdown pages — API reference, tutorials, troubleshooting guides, changelogs. Users search with queries like:
- “auth token expired what do i do”
- “rate limit 429 handling”
- “python sdk install fails”
BM25 baseline
BM25 matches “token” → OAuth reference page (mentions “access_token” 40 times). Misses “refresh token rotation” guide because it uses “refresh_token” not “token.” Misses “429 Too Many Requests” page because query says “rate limit” not “429.”
Semantic search with bge-small-en-v1.5
Embed all chunks (512 tokens, 50 overlap). Index in Qdrant with HNSW (m=16, ef_construct=200). Query encoding + ANN search ~15ms p99.
Results for “auth token expired what do i do”:
- “Refreshing access tokens” (cosine 0.82) — explains refresh flow
- “Error codes: invalid_grant” (0.78) — covers expired refresh tokens
- “OAuth 2.0 quickstart” (0.71) — mentions token lifecycle
- “Migrating from v1 API” (0.65) — mentions token format changes
The top result uses “refresh” not “expired,” “access token” not “auth token.” BM25 would rank it 200+.
Adding hybrid + rerank
Fuse BM25 (weight 0.5) + vector (weight 0.5) via RRF. Rerank top 50 with cross-encoder/ms-marco-MiniLM-L-6-v2. Latency: +40ms for rerank. NDCG@10 improves ~12% on internal eval set.
# Production-style hybrid search
def hybrid_search(query: str, top_k: int = 10) -> list[SearchResult]:
# 1. Vector candidates (oversample for rerank)
vec_ids, vec_scores = vector_index.search(embed(query), top_k=100)
# 2. BM25 candidates
bm25_ids, bm25_scores = bm25_index.search(query, top_k=100)
# 3. RRF fusion
fused_ids = rrf_fuse([vec_ids[0].tolist(), bm25_ids[0].tolist()])
# 4. Fetch text for rerank
candidates = [doc_store[doc_id].text for doc_id in fused_ids[:50]]
# 5. Cross-encoder rerank
reranked = rerank(query, candidates, top_n=top_k)
return [SearchResult(doc_id=fused_ids[idx], score=score) for idx, score in reranked]Common misconceptions
“Semantic search replaces keyword search”
False. Hybrid beats pure semantic on entity queries, exact error codes, version numbers, and proper nouns. BM25 is also cheaper — no GPU for encoding, smaller index footprint. Run both and fuse. The exception: pure semantic works for exploratory, natural-language queries where users don’t know the terminology.
“Larger embedding model = better retrieval”
Diminishing returns past ~768 dimensions for most corpora. bge-large-en-v1.5 (1024-dim) beats bge-small (384-dim) by 3–5% nDCG on BEIR but costs 3x latency and 2.7x index size. Quantization (int8, binary) recovers most latency but adds complexity. Profile on your data before defaulting to the largest model.
“Cosine similarity is the only metric”
Dot product equals cosine when vectors are L2-normalized. If you skip normalization (some APIs don’t normalize by default), dot product favors longer vectors — which correlates with nothing useful. Always normalize or use cosine explicitly. For inner-product indexes (Faiss IndexFlatIP), normalize at insert and query time.
# Correct: normalize for cosine via inner product
embeddings = model.encode(texts, normalize_embeddings=True) # L2 norm = 1
faiss_index.add(embeddings) # IndexFlatIP
# Query: also normalized
scores, ids = faiss_index.search(query_vec, k)“Embeddings capture all semantic nuance”
They capture distributional similarity — words appearing in similar contexts. They don’t inherently understand logic, negation, or temporal ordering. “Not working” and “working” can have high cosine similarity because they appear in similar troubleshooting contexts. Cross-encoders handle this better because they see the full query-passage interaction. For hard negation cases, consider training a custom reranker on your domain’s hard negatives.
“One embedding model works for everything”
Domain mismatch hurts. A model trained on MS MARCO (web search) underperforms on legal contracts, code, or biomedical text. Domain-adapted models (CodeBERT, Legal-BERT, PubMedBERT) or continued pretraining on your corpus + contrastive fine-tuning can add 10–20% recall. If you can’t fine-tune, at least evaluate on a labeled sample from your domain before committing.
Operational considerations
Index freshness. Re-embedding the full corpus nightly is feasible to ~10M docs on a single GPU. For larger or real-time needs, use a write-ahead log: new/updated docs go to a small “delta” index (rebuilt hourly), merged into main index nightly. Query searches both and merges results.
Multitenancy. If tenants have isolated data, don’t share an HNSW graph — it leaks vectors across tenants via graph connections. Use per-tenant indexes or a filtered ANN index (Qdrant, Weaviate support payload filters on HNSW). Filter-first-then-ANN is slower but secure; ANN-then-filter is faster but can miss results if the filter is selective.
Observability. Log query, top-10 doc_ids, scores, latency, and whether the user clicked a result. Without clickthrough or explicit relevance labels, you cannot measure drift or evaluate model upgrades. Sample 1–5% of traffic for human eval if you lack implicit signals.
Cost at scale. For 100M vectors, HNSW in RAM needs ~200GB (float32, 768-dim). DiskANN or product quantization (PQ) reduces to ~30GB with <5% recall loss. Cloud managed services (Pinecone, Weaviate Cloud, Qdrant Cloud) charge per million vectors + query volume. Self-hosted on GPU instances is cheaper past ~50M vectors if you have ops capacity.
When to reach for something else
- Exact-match requirements (legal citation lookup, SKU search): keyword/BM25 or trigram index (pg_trgm)
- Structured filters dominate (date range + category + status): Postgres + BM25 or Elasticsearch
- Sub-10ms p99 at 10k QPS: consider learned sparse retrieval (SPLADE) or a cached BM25 layer in front of semantic
- Explainability required: semantic search is a black box; BM25 scores are interpretable
TL;DR
Semantic search uses dense vectors from a bi-encoder to match queries by meaning. Index with HNSW, query with the same encoder, fuse with BM25 via RRF, rerank with a cross-encoder for precision. It wins on synonymy, paraphrase, and implicit intent. It loses on exact entities, negation, and logic. Hybrid is the default production choice. Evaluate on your data — BEIR benchmarks don’t predict your domain.