Semantic search retrieves documents by meaning rather than exact keyword overlap. It encodes queries and documents into dense vector embeddings, then ranks results by proximity in that embedding space. This lets a search for “cheap wireless headphones” surface “budget bluetooth earbuds” even when they share no tokens.
How semantic search works under the hood
Traditional keyword search (BM25, TF-IDF) treats documents as bags of words. It scores matches by term frequency and inverse document frequency. Semantic search replaces this with a two-stage pipeline: an embedding model maps text to fixed-length vectors, then an approximate nearest neighbor (ANN) index retrieves the closest vectors to the query embedding.
# Minimal semantic search pipeline
from sentence_transformers import SentenceTransformer
import faiss
import numpy as np
# 1. Load embedding model (384-dim, fast, decent quality)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
# 2. Encode corpus
documents = [
"Wireless noise-canceling headphones with 30hr battery",
"Budget bluetooth earbuds, sweat resistant",
"Over-ear studio monitoring headphones",
"True wireless earbuds with charging case",
]
doc_embeddings = model.encode(documents, normalize_embeddings=True)
# 3. Build ANN index (HNSW for production, flat for demo)
dim = doc_embeddings.shape[1]
index = faiss.IndexFlatIP(dim) # inner product == cosine on normalized vectors
index.add(doc_embeddings.astype(np.float32))
# 4. Query
query = "cheap wireless headphones"
query_embedding = model.encode([query], normalize_embeddings=True)
scores, indices = index.search(query_embedding.astype(np.float32), k=3)
for idx, score in zip(indices[0], scores[0]):
print(f"{score:.3f} | {documents[idx]}")Output:
0.742 | Budget bluetooth earbuds, sweat resistant
0.718 | True wireless earbuds with charging case
0.691 | Wireless noise-canceling headphones with 30hr batteryThe embedding model does the heavy lifting. all-MiniLM-L6-v2 is a 22M parameter transformer trained with contrastive learning on 1B+ sentence pairs. It learns to pull semantically similar texts together and push dissimilar ones apart in the 384-dimensional space.
Why embedding quality matters more than index choice
Engineers often obsess over ANN algorithms (HNSW vs IVF vs DiskANN) while neglecting the embedding model. The index only approximates distances in the space the model created. If the model maps “Apple stock price” and “fruit nutrition facts” close together, no index fixes that.
Three factors determine embedding quality for your domain:
Training objective. Contrastive learning (pull positive pairs together, push negatives apart) works better than masked language modeling for retrieval. Models like bge-large-en-v1.5 or e5-large-v2 use hard negative mining — they explicitly train on confusing pairs like “how to bake chicken” vs “how to bake salmon.”
Domain adaptation. General-purpose embeddings degrade on specialized vocabulary. Legal contracts, medical notes, and code repositories each benefit from continued pretraining on in-domain text. A 2023 study showed fine-tuning bge-base on 50k legal clauses improved nDCG@10 by 18 points over the base model.
Dimensionality vs latency tradeoff. Larger embeddings (1024-dim, 1536-dim) capture more nuance but increase index size and query latency linearly. For most product search, 384-768 dimensions hits the sweet spot. Quantization (int8, binary) can compress vectors 4-32x with <2% recall loss.
# Quantized index for production scale
import faiss
# Train product quantizer on sample data
pq = faiss.IndexPQ(dim, 64, 8) # 64 subquantizers, 8 bits each
pq.train(doc_embeddings.astype(np.float32))
pq.add(doc_embeddings.astype(np.float32))
# Search with quantized vectors
scores, indices = pq.search(query_embedding.astype(np.float32), k=3)Concrete example: E-commerce product search
Consider a catalog of 500k SKUs. Keyword search fails on:
- Synonyms: “sneakers” vs “running shoes” vs “trainers”
- Attributes: “waterproof hiking boots size 11” matches “waterproof” and “hiking” but misses “Gore-Tex” and “men’s 11”
- Intent: “gift for mom who gardens” has zero token overlap with “ergonomic pruning shears”
Semantic search handles all three. But pure vector search has its own failure modes:
# Failure case: vector search ignores exact matches
query = "iPhone 15 Pro Max 256GB natural titanium"
# Embedding may rank "iPhone 15 Pro 128GB blue" higher than
# "iPhone 15 Pro Max 256GB natural titanium" if the model
# emphasizes form factor over storage/color specificsThe production solution is hybrid search: combine BM25 and vector scores with reciprocal rank fusion (RRF) or a learned reranker.
# Hybrid search with reciprocal rank fusion
def hybrid_search(query, bm25_index, vector_index, model, k=10, rrf_k=60):
# BM25 results
bm25_scores, bm25_indices = bm25_index.search(query, k * 2)
# Vector results
query_emb = model.encode([query], normalize_embeddings=True)
vec_scores, vec_indices = vector_index.search(
query_emb.astype(np.float32), k * 2
)
# RRF fusion
fused_scores = {}
for rank, idx in enumerate(bm25_indices[0]):
fused_scores[idx] = fused_scores.get(idx, 0) + 1 / (rrf_k + rank + 1)
for rank, idx in enumerate(vec_indices[0]):
fused_scores[idx] = fused_scores.get(idx, 0) + 1 / (rrf_k + rank + 1)
# Sort and return top-k
sorted_results = sorted(fused_scores.items(), key=lambda x: -x[1])
return [idx for idx, _ in sorted_results[:k]]RRF requires no training data and outperforms weighted linear combination in most benchmarks. The constant k=60 is the standard default from the original paper.
When to use semantic search (and when not to)
Use semantic search when:
- Queries are natural language questions or descriptive phrases
- Synonym coverage matters (legal, medical, e-commerce)
- Zero-shot generalization to unseen query phrasing is required
- You can tolerate ~50-200ms latency per query
Stick with keyword search when:
- Queries are exact identifiers (SKU, ISBN, error codes, function names)
- Users expect literal matching (grep-style log search, code search)
- Latency budget is <10ms at high QPS
- Domain vocabulary is highly specialized with no training data
Hybrid is the default for production. Most real systems blend both. The mixing ratio depends on query distribution — analyze your query logs. If 70% of queries are navigational (“reset password”, “billing page”), keyword dominates. If 70% are informational (“how to reduce churn”, “best practices for onboarding”), semantic dominates.
Common misconceptions
“Semantic search understands meaning like a human.”
Embeddings capture statistical co-occurrence patterns from training data. They don’t “understand” — they approximate semantic similarity based on distributional hypothesis. They fail on negation (“not expensive”), compositionality (“red car” ≠ “car red” in some models), and logic (“all mammals are animals” doesn’t entail “all animals are mammals”).
“Larger embeddings are always better.”
Beyond 1024 dimensions, returns diminish sharply for general domains. The Johnson-Lindenstrauss lemma guarantees you can project to O(log n / ε²) dimensions while preserving pairwise distances. For 1M documents, 384 dimensions with ε=0.1 is theoretically sufficient. Larger models help only when the extra capacity captures domain-specific distinctions.
“You need GPUs for embedding inference.”
all-MiniLM-L6-v2 runs at ~2000 queries/second on a modern CPU (AVX2). Batch inference with ONNX Runtime or sentence-transformers + torch.compile pushes this higher. Reserve GPUs for training/fine-tuning and large-model inference (7B+ parameter embeddings like SFR-Embedding-Mistral).
“Vector databases are required.”
FAISS, HNSWLib, and ScaNN run in-process with zero infrastructure. They handle 10M+ vectors on a single machine with enough RAM. Vector databases (Pinecone, Weaviate, Qdrant, Milvus) add distributed scaling, filtering, multi-tenancy, and managed operations. Choose based on operational capacity, not hype.
Evaluation: measure what matters
Don’t rely on cosine similarity distributions. Build a labeled eval set — even 100 queries with graded relevance judgments (0: irrelevant, 1: partially relevant, 2: highly relevant) beats zero eval.
# Minimal evaluation framework
def evaluate_search(search_fn, eval_queries, k=10):
"""eval_queries: list of (query, {doc_id: relevance_grade})"""
ndcg_scores = []
recall_scores = []
for query, relevance in eval_queries:
results = search_fn(query, k)
# nDCG@k
dcg = sum(
(2**relevance.get(doc_id, 0) - 1) / np.log2(rank + 2)
for rank, doc_id in enumerate(results)
)
ideal = sorted(relevance.values(), reverse=True)[:k]
idcg = sum(
(2**rel - 1) / np.log2(i + 2)
for i, rel in enumerate(ideal)
)
ndcg_scores.append(dcg / idcg if idcg > 0 else 0)
# Recall@k
relevant = {doc_id for doc_id, rel in relevance.items() if rel > 0}
recall_scores.append(len(set(results) & relevant) / len(relevant) if relevant else 0)
return {
"ndcg@k": np.mean(ndcg_scores),
"recall@k": np.mean(recall_scores),
}Track nDCG@10 and Recall@100 weekly. A 0.02 nDCG drop often precedes user complaints by weeks.
Reranking: the last mile
First-stage retrieval (BM25 + ANN) optimizes for recall. A cross-encoder reranker optimizes for precision. Cross-encoders attend jointly over query and document — they’re slower but far more accurate.
# Cross-encoder reranking
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query, candidates, top_k=5):
pairs = [[query, doc] for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
return [doc for doc, _ in ranked[:top_k]]Typical latency: 5-15ms per query-document pair on CPU. Rerank top-50 from first stage to final top-10. This two-stage architecture (retrieve → rerank) is standard at Google, Amazon, and every serious search team.
Putting it in production
A minimal production stack:
- Embedding service:
sentence-transformers+ ONNX Runtime, batched, behind a load balancer - Vector index: FAISS HNSW on SSD (memory-mapped) or in RAM if <5M vectors
- Keyword index: Elasticsearch/OpenSearch or Tantivy for BM25
- Reranker: Cross-encoder service, GPU optional
- Fusion logic: RRF in application code, configurable weights
Monitor: p99 latency, recall@100 (via sampled human eval), embedding drift (track cosine similarity of frequent query embeddings over time).
Semantic search isn’t magic — it’s a well-understood pipeline with clear tradeoffs. Start with hybrid BM25 + MiniLM + RRF. Add a cross-encoder reranker when precision matters. Fine-tune embeddings only when eval proves the base model fails on your domain. Skip the vector database until you have a scaling problem that FAISS can’t solve.