Semantic search vs keyword search: what's the difference

Semantic search and keyword search solve the same problem — finding relevant documents — but they operate on fundamentally different principles. Keyword search matches tokens; semantic search matches meaning. That distinction cascades into every dimension that matters to engineers: index size, query latency, relevance quality, and operational complexity. Understanding where each excels lets you pick the right tool or combine them effectively.

How they work

Keyword search builds an inverted index mapping terms to document IDs. When a query arrives, the engine tokenizes it, looks up posting lists for each term, scores documents using TF-IDF or BM25, and returns the top-k results. The process is deterministic, well-understood, and has been optimized for decades.

Semantic search embeds both documents and queries into a shared vector space using a transformer model. Relevance becomes proximity in that space — typically cosine similarity or dot product between query and document vectors. The engine then performs approximate nearest neighbor (ANN) search over the vector index.

# Keyword search (BM25) - conceptual
def bm25_score(query_terms, doc, avg_dl, k1=1.2, b=0.75):
    score = 0.0
    for term in query_terms:
        tf = doc.term_freq(term)
        idf = math.log((N - df[term] + 0.5) / (df[term] + 0.5) + 1)
        score += idf * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * len(doc) / avg_dl))
    return score

# Semantic search - conceptual
def semantic_search(query, index, embedder, k=10):
    q_vec = embedder.encode(query)
    return index.ann_search(q_vec, k=k)  # HNSW, IVF, DiskANN, etc.

The embedding model is the semantic engine’s core dependency. Models like BGE-M3, E5-Mistral, or OpenAI’s text-embedding-3-large produce 768–3072 dimensional vectors. Quality varies significantly by model, domain, and language.

Capabilities compared

Keyword search wins on precision for exact matches: part numbers, error codes, legal citations, usernames. Semantic search wins on recall for conceptual queries: “how do I optimize database writes”, “symptoms of memory leak”, “best practices for API versioning”.

Latency and throughput

Keyword search is fast. A well-tuned inverted index on SSD serves millions of QPS with sub-millisecond p99 latency. The index is compact — typically 10–30% of raw text size — and fits in memory for most workloads.

Semantic search adds two latency components: embedding inference and ANN search.

# Rough latency breakdown (single query, batch=1)
# Embedding (CPU, BGE-base):     15–40 ms
# Embedding (GPU, batched):       2–8 ms
# ANN search (HNSW, 1M vecs):     1–5 ms
# Total semantic:                3–45 ms
# Keyword (BM25, 1M docs):       0.5–3 ms

ANN index choice dominates vector search latency. HNSW offers the best recall/latency tradeoff but consumes significant memory (1.5–2x raw vectors). IVF-PQ and DiskANN trade recall for memory efficiency. Quantization (int8, int4, binary) reduces memory and speeds up distance computation at modest recall cost.

For high-throughput semantic search, you need GPU embedding inference with request batching. CPU-only embedding becomes a bottleneck above ~50 QPS. Keyword search scales horizontally with simple sharding; semantic search requires coordinated sharding of both the embedding service and the vector index.

Index construction and maintenance

Keyword indexes update incrementally. Adding a document means tokenizing and appending to posting lists — milliseconds per document. Deletes are lazy (tombstones) or merged during segment compaction.

Semantic indexes are heavier. Each new document requires embedding inference, then ANN index insertion. HNSW insertion is O(log n) but with high constant factors; bulk loading is preferred. Many teams rebuild vector indexes nightly or hourly rather than streaming updates.

# Incremental update pattern for semantic search
class VectorIndex:
    def __init__(self, embedder, index_path):
        self.embedder = embedder
        self.index = hnswlib.Index(space='cosine', dim=768)
        self.index.load_index(index_path)
        self.pending = []
    
    def add_documents(self, docs):
        vectors = self.embedder.encode([d.text for d in docs])
        for i, vec in enumerate(vectors):
            self.pending.append((self.next_id, vec, docs[i].metadata))
            self.next_id += 1
        
        if len(self.pending) >= 10000:
            self.flush()
    
    def flush(self):
        ids = [p[0] for p in self.pending]
        vecs = np.array([p[1] for p in self.pending])
        self.index.add_items(vecs, ids)
        self.pending.clear()
        self.index.save_index(self.index_path)

Index size: a 1M document corpus with 768-dim float32 vectors is ~3 GB raw. HNSW overhead pushes this to 5–6 GB. Quantized (int8) drops to ~1.5 GB. Keyword index for the same corpus: 200–500 MB.

Cost model

Keyword search runs on CPU. A single r6g.xlarge (4 vCPU, 32 GB) handles ~10M documents and thousands of QPS. Cost is predictable and linear with corpus size.

Semantic search has two cost centers: embedding inference and vector index hosting.

• Embedding inference: GPU instances (g5.xlarge, A10G) at $0.50–1.50/hr for ~1000–3000 embeddings/sec batched. Or CPU with batching at lower throughput.
• Vector database: Managed services (Pinecone, Weaviate Cloud, Qdrant Cloud) charge per million vectors + query volume. Self-hosted on EC2: storage + memory for HNSW.

At 10M documents, semantic search typically costs 5–20x more than keyword search for equivalent throughput. The gap narrows with quantization and efficient ANN indexes, but rarely closes completely.

Hybrid search: the practical answer

Production systems rarely choose one. They combine both.

def hybrid_search(query, keyword_index, vector_index, embedder, k=20, alpha=0.5):
    # Retrieve more candidates from each
    kw_results = keyword_index.search(query, k=k*3)
    vec_results = vector_index.search(embedder.encode(query), k=k*3)
    
    # Normalize scores to [0, 1]
    kw_scores = normalize([r.score for r in kw_results])
    vec_scores = normalize([r.score for r in vec_results])
    
    # Fuse with reciprocal rank fusion or weighted sum
    fused = {}
    for i, r in enumerate(kw_results):
        fused[r.doc_id] = fused.get(r.doc_id, 0) + alpha * kw_scores[i] + (1-alpha) * (1/(i+1))
    for i, r in enumerate(vec_results):
        fused[r.doc_id] = fused.get(r.doc_id, 0) + (1-alpha) * vec_scores[i] + alpha * (1/(i+1))
    
    # Re-rank top candidates with cross-encoder (optional)
    top_ids = sorted(fused, key=fused.get, reverse=True)[:k]
    return rerank_with_cross_encoder(query, top_ids)

Reciprocal Rank Fusion (RRF) is the most robust fusion method — no score normalization needed, works across heterogeneous scorers. Weighted sum requires careful calibration. Cross-encoder reranking (e.g., BGE-reranker-v2, monoT5) adds 20–50 ms but significantly improves precision@k.

Operational ergonomics

Keyword search engines (Elasticsearch, OpenSearch, Meilisearch, Typesense) offer mature ecosystems: SQL-like query DSLs, per-field boosting, synonym graphs, analyzer chains, highlight snippets, faceted aggregation, and robust monitoring. Operations teams know them.

Vector databases (Qdrant, Weaviate, Milvus, Pinecone, Chroma) are younger. They excel at vector operations but often lack: full-text search, complex filtering, aggregations, and mature tooling. Many teams run both: Elasticsearch for keyword + metadata filtering, Qdrant for vectors, joined at application layer.

Some newer engines unify both: Elasticsearch 8.x with dense vector support, OpenSearch k-NN, Meilisearch hybrid search, Typesense vector search. These reduce infrastructure sprawl but may not match dedicated vector DBs on ANN recall/latency at scale.

When to choose which

Choose keyword search when:

• Queries are exact-match heavy: IDs, codes, SKUs, error messages, log search
• You need transparent debugging: “why did this document rank #1?”
• Latency budget is <5 ms p99 at high QPS
• Team has existing Elasticsearch/OpenSearch expertise
• Corpus is small (<100K docs) — semantic overhead isn’t justified
• Strong negation and boolean logic requirements

Choose semantic search when:

• Queries are natural language: questions, descriptions, symptoms
• Synonyms and paraphrases are common and unbounded
• Multilingual search without per-language analyzers
• You can tolerate 20–100 ms latency for better recall
• Domain-specific embedding models exist (biomedical, legal, code)
• You need “similar document” recommendations, not just query matching

Choose hybrid (default for most production systems):

• General-purpose search with mixed query types
• E-commerce: exact SKU + conceptual “red running shoes for marathon”
• Documentation: error code + “how to fix connection timeout”
• RAG pipelines: retrieve diverse candidates, rerank for generator
• Any system where recall matters and you can afford the complexity

Implementation checklist

If you’re adding semantic search to an existing keyword stack:

1. Start with a strong embedding model — BGE-M3 or E5-Mistral-7B-Instruct for general purpose; domain-specific if available. Avoid random Hugging Face models without MTEB benchmarks.
2. Quantize aggressively — int8 HNSW loses <1% recall, cuts memory 4x. Binary quantization (BGE-M3 supports this natively) cuts 32x with ~3–5% recall drop.
3. Use RRF for fusion — no hyperparameter tuning, robust across score distributions.
4. Add cross-encoder reranking — 50 ms for 50 candidates, huge precision gain. Cache reranker scores for repeated queries.
5. Monitor recall@k, not just latency — semantic search can return plausible-looking garbage. Evaluate with labeled queries.
6. Plan for re-indexing — embedding model upgrades require full corpus re-embedding. Version your vectors.

The semantic vs keyword distinction isn’t binary — it’s a spectrum of matching granularity. Keyword matches surface form; semantic matches latent intent. Most production systems need both. Start with keyword, add semantic when query analysis shows conceptual gaps, fuse with RRF, rerank with a cross-encoder, and measure recall@k against labeled data. That’s the architecture that ships.