Hybrid search in LangChain: combining BM25 and embeddings

Hybrid search combines the precision of keyword matching with the semantic understanding of embeddings. In LangChain, this means running BM25 and vector search in parallel, then fusing the results. The approach handles queries where exact terminology matters (error codes, product SKUs, function names) while still catching conceptually relevant documents that don’t share vocabulary. This guide walks through a production-ready implementation using LangChain’s built-in retrievers and a few lines of custom fusion logic.

Why hybrid search beats pure vector or pure keyword

Vector search excels at semantic similarity but fails on exact matches. A query for “HTTP 429 rate limit” might retrieve documents about “throttling” and “backoff strategies” while missing the one page that literally contains “429”. BM25 catches the exact term but misses “rate limiting” when the query says “throttling”. Hybrid search gets both.

The tradeoff is complexity: you need two indexes, a fusion strategy, and tuning parameters. For most RAG systems, the recall improvement justifies the cost. If your corpus is small (<10k docs) or your queries are purely conversational, pure vector search may suffice. If you’re building enterprise search over technical documentation, codebases, or legal contracts, hybrid is table stakes.

Architecture overview

Query → [BM25 Retriever] ──┐
                           ├──→ [Reciprocal Rank Fusion] → [Re-ranker (optional)] → Final Results
Query → [Vector Retriever] ┘

LangChain provides BM25Retriever and vector store retrievers out of the box. The fusion step is where most implementations diverge. Reciprocal Rank Fusion (RRF) is the standard choice: it’s parameter-free, theoretically grounded, and works well without training data. The formula for a document d across k retrievers:

score(d) = Σ 1 / (k + rank_i(d))

Where rank_i(d) is the position of d in retriever i’s results (1-indexed). The constant k (typically 60) dampens the impact of high ranks from any single retriever.

Setting up the indexes

You need two indexes over the same documents. For BM25, LangChain’s BM25Retriever builds an in-memory index from a list of Document objects. For vectors, pick a vector store — Chroma, FAISS, Pinecone, Weaviate, etc. The documents must be identical across both indexes, including metadata.

from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_core.documents import Document

# Sample documents — replace with your loader
docs = [
    Document(page_content="HTTP 429 Too Many Requests indicates rate limiting", metadata={"source": "http-codes.md"}),
    Document(page_content="Implement exponential backoff when hitting rate limits", metadata={"source": "retry-strategies.md"}),
    Document(page_content="The 429 status code means the user has sent too many requests in a given timeframe", metadata={"source": "api-docs.md"}),
    Document(page_content="Throttling protects downstream services from overload", metadata={"source": "architecture.md"}),
]

# BM25 index (in-memory, good for <100k docs)
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 10  # candidates for fusion

# Vector index (persistent, scales)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(docs, embeddings, persist_directory="./chroma_db")
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 10})

Pitfall: BM25Retriever loads everything into memory. For corpora above ~100k documents, use a proper search engine (Elasticsearch, OpenSearch, Typesense) with a LangChain wrapper instead. The in-memory index also doesn’t persist — rebuild on startup or serialize with pickle.

Implementing reciprocal rank fusion

LangChain doesn’t ship an RRF retriever yet (as of 0.2.x), so you’ll write a thin wrapper. The logic: run both retrievers, collect ranked document lists, apply RRF, return top-k.

from typing import List, Dict, Any
from langchain_core.retrievers import BaseRetriever
from langchain_core.documents import Document
from langchain_core.callbacks import CallbackManagerForRetrieverRun

class HybridRetriever(BaseRetriever):
    bm25_retriever: BaseRetriever
    vector_retriever: BaseRetriever
    k: int = 60  # RRF constant
    top_k: int = 5  # final results

    def _get_relevant_documents(
        self, query: str, *, run_manager: CallbackManagerForRetrieverRun
    ) -> List[Document]:
        # Retrieve from both sources
        bm25_docs = self.bm25_retriever.invoke(query)
        vector_docs = self.vector_retriever.invoke(query)

        # Score with RRF
        scores: Dict[str, float] = {}
        doc_map: Dict[str, Document] = {}

        for rank, doc in enumerate(bm25_docs, 1):
            key = self._doc_key(doc)
            scores[key] = scores.get(key, 0) + 1.0 / (self.k + rank)
            doc_map[key] = doc

        for rank, doc in enumerate(vector_docs, 1):
            key = self._doc_key(doc)
            scores[key] = scores.get(key, 0) + 1.0 / (self.k + rank)
            doc_map[key] = doc

        # Sort by fused score
        ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
        return [doc_map[key] for key, _ in ranked[:self.top_k]]

    def _doc_key(self, doc: Document) -> str:
        # Stable identifier — use content hash or metadata ID
        return doc.metadata.get("id", hash(doc.page_content))

Pitfall: The _doc_key must uniquely identify documents across both retrievers. If your vector store assigns new IDs on each ingestion, use a content hash or a stable doc_id field you control. Mismatched keys silently break fusion.

Adding a cross-encoder re-ranker

RRF gives you a fused ranking, but the top candidates still benefit from a cross-encoder. Cross-encoders score query-document pairs jointly (unlike bi-encoders that embed independently), capturing finer relevance signals. The standard pattern: retrieve 20-50 candidates via hybrid, re-rank with a cross-encoder, return top 5.

from langchain_community.cross_encoders import HuggingFaceCrossEncoder
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker

cross_encoder = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")
reranker = CrossEncoderReranker(model=cross_encoder, top_n=5)

# Wrap the hybrid retriever
hybrid = HybridRetriever(
    bm25_retriever=bm25_retriever,
    vector_retriever=vector_retriever,
    top_k=20  # retrieve more for re-ranking
)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=hybrid
)

# Usage
results = compression_retriever.invoke("HTTP 429 rate limit handling")

Tradeoff: Cross-encoders add latency (50-200ms per query on CPU). For high-throughput systems, consider:

Caching re-ranker scores for repeated queries
Using a smaller model (ms-marco-TinyBERT-L-2)
Running re-ranking asynchronously and returning hybrid results immediately for streaming UIs
Skipping re-ranking entirely if hybrid recall@5 is already sufficient

Tuning the retrieval parameters

Three knobs matter most:

Parameter	Typical range	Effect
`bm25_retriever.k`	10-50	More candidates → better fusion coverage, slower
`vector_retriever.k`	10-50	Same tradeoff
`HybridRetriever.k` (RRF constant)	40-100	Lower = more weight to top ranks; 60 is the standard default

Start with k=20 for both retrievers, RRF k=60, final top_k=5. Evaluate on a held-out query set with ground truth. If recall@5 is low, increase retriever k. If precision drops, decrease k or add re-ranking.

Pitfall: Don’t tune on the same queries you use for development. Build a small eval set (50-100 queries) with labeled relevant documents. Use langchain.evaluation or ragas for automated metrics.

Handling metadata filters

Real systems filter by tenant, document type, date range, or access control. Both retrievers must honor the same filters. With Chroma, pass filter to search_kwargs. With BM25, you’ll need to pre-filter the document list before building the index, or implement a filtered BM25 wrapper.

# Vector store with metadata filter
vector_retriever = vectorstore.as_retriever(
    search_kwargs={"k": 20, "filter": {"tenant_id": "acme-corp"}}
)

# BM25: filter at index build time (for static filters) or wrap
class FilteredBM25Retriever(BM25Retriever):
    def __init__(self, *args, filter_fn=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.filter_fn = filter_fn or (lambda _: True)

    def _get_relevant_documents(self, query: str, *, run_manager) -> List[Document]:
        docs = super()._get_relevant_documents(query, run_manager=run_manager)
        return [d for d in docs if self.filter_fn(d)]

# Usage
filtered_bm25 = FilteredBM25Retriever.from_documents(
    docs, filter_fn=lambda d: d.metadata.get("tenant_id") == "acme-corp"
)
filtered_bm25.k = 20

Pitfall: If filters are dynamic (per-request), rebuilding the BM25 index per query defeats the purpose. For dynamic filters at scale, move BM25 to Elasticsearch/OpenSearch where filtered search is native.

Evaluating hybrid vs. single retrievers

Run a quick A/B comparison before committing:

from langchain.evaluation import load_evaluator
from langchain.evaluation.schema import StringEvaluator

# Assume you have ground truth: List[Tuple[query, relevant_doc_ids]]
eval_data = [
    ("HTTP 429 rate limit", {"doc-1", "doc-3"}),
    ("exponential backoff retry", {"doc-2"}),
    ("throttling architecture", {"doc-4"}),
]

def evaluate_retriever(retriever, name: str):
    hits = 0
    total = 0
    for query, relevant_ids in eval_data:
        results = retriever.invoke(query)
        retrieved_ids = {r.metadata.get("id") for r in results[:5]}
        hits += len(relevant_ids & retrieved_ids)
        total += len(relevant_ids)
    recall = hits / total if total else 0
    print(f"{name}: recall@5 = {recall:.2f}")

evaluate_retriever(bm25_retriever, "BM25 only")
evaluate_retriever(vector_retriever, "Vector only")
evaluate_retriever(hybrid, "Hybrid (RRF)")
evaluate_retriever(compression_retriever, "Hybrid + Rerank")

Typical outcome: hybrid beats both single retrievers by 10-25% recall@5; re-ranking adds another 5-15%. If the gap is small, question whether the complexity pays off.

Production considerations

Index freshness: Vector stores support incremental upserts. BM25Retriever does not — you rebuild the entire index. For frequently updated corpora, either:

Rebuild BM25 on a schedule (cron, Airflow, etc.)
Use a search engine with real-time indexing
Accept slight staleness in keyword search

Latency budget: Hybrid search runs two retrievers sequentially by default. Parallelize with asyncio.gather or ThreadPoolExecutor:

import asyncio
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=2)

async def aget_relevant_documents(self, query: str) -> List[Document]:
    loop = asyncio.get_event_loop()
    bm25_task = loop.run_in_executor(executor, self.bm25_retriever.invoke, query)
    vector_task = loop.run_in_executor(executor, self.vector_retriever.invoke, query)
    bm25_docs, vector_docs = await asyncio.gather(bm25_task, vector_task)
    # ... RRF fusion ...

Observability: Log the retriever breakdown — how many results came from BM25 only, vector only, both. This tells you if one retriever is dead weight. Track fusion score distributions to detect query types where hybrid helps or hurts.

When to skip hybrid

Corpus < 5k documents: vector search alone often suffices
Queries are purely natural language (“how do I reset my password?”) with no domain terminology
Team lacks capacity to maintain dual indexes and eval pipeline
Latency budget < 100ms p99 and you can’t parallelize

In these cases, invest in better embeddings (fine-tuned, larger model) or query rewriting instead.

Summary checklist

Build BM25 and vector indexes over identical document sets with stable IDs
Implement RRF fusion with k=60 as starting point
Retrieve 20-50 candidates from each retriever
Add cross-encoder re-ranking if latency budget allows
Create eval set with ground truth; measure recall@5
Tune retriever k and RRF constant based on eval
Ensure metadata filters apply to both retrievers consistently
Parallelize retriever calls in production
Log retriever contribution breakdown for ongoing monitoring

Hybrid search isn’t magic — it’s two imperfect retrievers covering each other’s blind spots. The engineering work is in the eval loop, not the fusion math. Start simple, measure, iterate.