Hybrid search explained: vector plus keyword search

Hybrid search explained simply: it runs vector search and keyword search in parallel, then merges the results so you get both semantic matching and exact-term precision. Pure vector search misses rare tokens and exact phrases; pure keyword search misses synonyms and conceptual overlap. Hybrid search gives you both without choosing.

How hybrid search works

At its core, hybrid search executes two independent retrieval paths against the same corpus:

Vector path: Embed the query with a dense encoder (for example, text-embedding-3-large, bge-large-en-v1.5, or a domain-specific model), then run approximate nearest neighbor search (HNSW, IVF, DiskANN) over the vector index. This surfaces documents that are semantically related even when they share no vocabulary with the query.

Keyword path: Tokenize the query, apply the same analyzer used at index time (stemming, lowercasing, stop-word removal, n-grams), and run a traditional inverted-index lookup — typically BM25 or a learned sparse retrieval model like SPLADE. This surfaces documents containing the exact terms, IDs, error codes, or proper nouns the user typed.

Fusion: Combine the two ranked lists into a single result set. The fusion strategy determines whether you get the best of both worlds or the worst.

Fusion strategies

RRF is the default starting point for most teams. It requires no score normalization — vector cosine similarities and BM25 scores live on completely different scales — because it operates on ranks, not raw scores. Set k=60 (the original paper’s recommendation) and adjust only if you have a strong reason.

def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """
    ranked_lists: list of [doc_id, doc_id, ...] from each retriever (vector, keyword, ...)
    Returns: [(doc_id, rrf_score), ...] sorted descending
    """
    scores: dict[str, float] = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

If you prefer weighted interpolation, you must normalize scores first. Min-max per query works but is unstable with small result sets. Quantile normalization against a held-out query set is more robust:

def normalize_scores(scores: list[float], reference_quantiles: list[float]) -> list[float]:
    """Map scores to [0,1] using precomputed reference quantiles (e.g., from 10k queries)."""
    import numpy as np
    arr = np.array(scores)
    if arr.size == 0:
        return []
    # Clip to reference range, then linear interpolate
    lo, hi = reference_quantiles[0], reference_quantiles[-1]
    clipped = np.clip(arr, lo, hi)
    return ((clipped - lo) / (hi - lo)).tolist()

Why hybrid search matters

Three failure modes of single-path retrieval drive adoption of hybrid search:

1. Vocabulary mismatch — A user searches “how to rotate credentials in k8s”. Vector search finds “rotating service account tokens in Kubernetes” (good). Keyword search finds nothing because “k8s” ≠ “Kubernetes” in the index. Hybrid catches both.

2. Rare tokens and identifiers — Error codes (ORA-01652), model names (claude-3-opus-20240229), internal project codenames. Dense embeddings smear these into near-meaningless vectors. Keyword search nails them. Hybrid ensures they surface.

3. Negation and constraint queries — “vector database without Pinecone” or “Python async not threading”. Vector search treats negation as noise; keyword search can handle it with boolean operators or learned sparse representations that preserve term-level signals.

Hybrid search also reduces the “embedding drift” problem. When your embedding model updates, vector rankings shift — sometimes dramatically. Keyword rankings stay stable. The fused result set is more stable across model versions, which matters for production systems where users notice regressions.

Concrete example: technical documentation search

Consider a developer portal for a cloud platform. Corpus: 500k pages (API reference, tutorials, troubleshooting guides, release notes). Query: “timeout configuring gRPC keepalive in Go client v1.4”.

Vector-only top-5:

1. “gRPC keepalive best practices” (generic, not Go-specific)
2. “Configuring timeouts in the Python client” (wrong language)
3. “Understanding gRPC connection lifecycle” (conceptual, no code)
4. “Client library versioning policy” (irrelevant)
5. “Troubleshooting network latency” (too broad)

Keyword-only top-5 (BM25, standard analyzer):

1. “Go client v1.4 release notes” (exact version match)
2. “gRPC keepalive configuration reference” (exact term match)
3. “Timeout settings reference” (exact term match)
4. “Go client v1.3 migration guide” (version adjacency)
5. “gRPC keepalive troubleshooting” (term overlap)

Hybrid (RRF, k=60) top-5:

1. “gRPC keepalive configuration reference — Go client v1.4” (both paths agree)
2. “Timeout settings reference — Go client v1.4” (both paths agree)
3. “Go client v1.4 release notes” (keyword strong, vector moderate)
4. “gRPC keepalive troubleshooting” (keyword strong, vector moderate)
5. “Configuring gRPC keepalive in Go client v1.4” (vector strong, keyword moderate — the exact doc the user wanted)

The hybrid result puts the exact version-specific configuration page at position 5 instead of position 50+ (vector) or position 12 (keyword). The top-2 are pages that both retrievers agree are relevant — a strong signal.

Indexing pipeline for this example

from sentence_transformers import SentenceTransformer
from rank_bm25 import BM25Okapi
import json

# Corpus: list of {"id": str, "title": str, "content": str, "metadata": dict}
corpus = load_corpus()

# Vector index
encoder = SentenceTransformer("BAAI/bge-large-en-v1.5")
embeddings = encoder.encode(
    [f"{doc['title']}\n{doc['content']}" for doc in corpus],
    batch_size=64,
    show_progress_bar=True,
    normalize_embeddings=True  # cosine similarity = dot product
)
# Upsert to vector DB (Qdrant, Weaviate, Pinecone, etc.)
vector_client.upsert(collection="docs", points=[
    {"id": doc["id"], "vector": emb.tolist(), "payload": doc["metadata"]}
    for doc, emb in zip(corpus, embeddings)
])

# Keyword index (BM25)
tokenized_corpus = [
    simple_tokenize(doc["title"] + " " + doc["content"]) for doc in corpus
]
bm25 = BM25Okapi(tokenized_corpus)
# Persist BM25 model (idf, doc_len, etc.) for query-time scoring
save_bm25(bm25, "bm25_index.pkl")

At query time, you run both retrievers in parallel (async, thread pool, or separate services), fuse with RRF, and optionally rerank the top-50 with a cross-encoder:

async def hybrid_search(query: str, top_k: int = 10, rerank_top_n: int = 50) -> list[SearchResult]:
    # Parallel retrieval
    vector_task = vector_client.search(query=query, limit=rerank_top_n)
    keyword_task = run_in_threadpool(bm25_search, query, rerank_top_n)
    vector_results, keyword_results = await asyncio.gather(vector_task, keyword_task)

    # Fuse
    fused_ids = reciprocal_rank_fusion([
        [r.id for r in vector_results],
        [r.id for r in keyword_results]
    ])[:rerank_top_n]

    # Fetch full docs for reranking
    docs = await doc_store.mget([doc_id for doc_id, _ in fused_ids])

    # Cross-encoder rerank (optional but recommended)
    if rerank_top_n > 0:
        pairs = [(query, doc["content"][:2000]) for doc in docs]  # truncate for length
        rerank_scores = cross_encoder.predict(pairs, batch_size=32)
        reranked = sorted(zip(docs, rerank_scores), key=lambda x: x[1], reverse=True)
        return [SearchResult(id=d["id"], score=float(s), content=d["content"]) for d, s in reranked[:top_k]]

    return [SearchResult(id=doc_id, score=score, content=doc_store.get(doc_id)["content"]) for doc_id, score in fused_ids[:top_k]]

Common misconceptions

“Hybrid search is just running two queries and concatenating results”

Concatenation (vector top-K + keyword top-K, deduplicated) loses the ranking signal. A doc at position 1 in vector and position 50 in keyword is far more relevant than a doc at position 50 in both. RRF and learned fusion preserve this signal; concatenation throws it away.

“You need a specialized hybrid database”

You don’t. Any vector database + any keyword index (Elasticsearch, OpenSearch, Tantivy, SQLite FTS5, PostgreSQL tsvector, even a custom BM25 implementation) works. The fusion logic lives in your application layer or a thin gateway. Some managed platforms (Elasticsearch, OpenSearch, Vespa, Weaviate, Qdrant) now offer native hybrid APIs — convenient, but not required.

“BM25 is obsolete; learned sparse retrieval (SPLADE, uniCOIL) is strictly better”

Learned sparse models expand queries with weighted expansion terms, which helps recall on short queries. But they add model-serving complexity, latency, and index size. BM25 with a well-tuned analyzer (custom stop words, synonym graphs, n-grams for codes/IDs) remains a strong baseline. Start with BM25. Move to learned sparse only when you have labeled data proving it wins on your corpus.

“Reranking makes hybrid search unnecessary”

A cross-encoder reranker improves precision on the top-N, but it only sees what the first-stage retrievers brought back. If neither vector nor keyword retrieval surfaces a relevant doc in the top-100, the reranker never sees it. Hybrid first-stage retrieval expands recall; reranking improves precision on that expanded set. They’re complementary, not substitutes.

“Hybrid search always improves nDCG”

Not automatically. If your vector and keyword retrievers are highly correlated (e.g., both dominated by the same high-frequency terms), fusion adds noise. If one retriever is significantly worse than the other on your domain, naive fusion can degrade results. Measure per-retriever recall@K and nDCG@K on a held-out judgment set before committing to fusion weights.

Production considerations

Latency budget: Vector search (HNSW) typically 10–50ms p99. Keyword search (inverted index) typically 5–20ms p99. Parallel execution means hybrid latency ≈ max(vector, keyword) + fusion overhead (sub-millisecond). Reranking adds 20–100ms depending on model size and batch size. Budget accordingly.

Score calibration: Vector cosine similarity ∈ [-1, 1] (typically [0.2, 0.8] for relevant docs). BM25 scores are unbounded positive, heavily dependent on document length and query term rarity. Never interpolate raw scores. Use RRF, or normalize against query-level statistics collected offline.

Index freshness: Vector indexes (especially HNSW) support incremental upserts but degrade recall over many mutations without periodic rebuilds. Keyword indexes handle incremental updates natively. Schedule vector index rebuilds (or use a database that handles this automatically) to prevent drift.

Query routing: Not every query needs hybrid. Short queries with rare tokens (“ORA-01652”) benefit heavily from keyword. Long natural-language questions (“how do I optimize connection pooling for high-throughput gRPC services?”) benefit heavily from vector. A lightweight classifier or heuristic (query length, presence of code-like tokens, entropy) can route to single-path retrieval and save compute.

def should_use_hybrid(query: str) -> bool:
    tokens = simple_tokenize(query)
    has_code_token = any(re.match(r'^[A-Z]+-\d+$', t) or t.isupper() for t in tokens)  # error codes, IDs
    is_short = len(tokens) <= 4
    return has_code_token or is_short  # simple heuristic; replace with learned classifier

Evaluation: Build a judgment set (query → relevant doc IDs) — even 200–500 queries is enough to start. Measure recall@10, nDCG@10, and latency for: vector-only, keyword-only, hybrid (RRF), hybrid + rerank. Track these per query category (navigational, informational, troubleshooting, code). You’ll find hybrid wins on some categories, loses on others. That’s the signal to invest in routing or per-category fusion weights.

When to skip hybrid search

• Pure semantic product search (e.g., “show me dresses like this one”) where vocabulary mismatch is the dominant challenge and exact-term matches are noise.
• Pure navigational lookup (e.g., “stripe api reference”) where users know the exact title and keyword search achieves near-perfect recall@1.
• Extreme latency constraints (<10ms p99) where even parallel retrieval blows the budget. In this regime, a well-tuned single retriever (usually keyword for navigational, vector for exploratory) beats a rushed hybrid.
• Prototyping — start with one retriever, measure, add the second only when the data shows a gap.

Summary

Hybrid search explained in one paragraph: run vector and keyword retrieval in parallel, fuse with RRF (start here) or learned fusion (when you have data), optionally rerank with a cross-encoder. It solves vocabulary mismatch, rare-token recall, and negation — three failure modes that single-path retrieval cannot. The implementation is straightforward: two indexes, one fusion function, optional reranker. The hard part is evaluation — build a judgment set, measure per-category, and iterate on fusion weights or routing logic. Most teams over-engineer the fusion and under-invest in evaluation. Don’t.