n4nAI
Open console →
Framework tutorialsHow-toHaystack Document Stores & Retrievers

Building a hybrid retriever in Haystack 2.0

Hands-on hybrid retriever Haystack 2.0 tutorial: wire BM25 and embedding retrievers into one pipeline with rank fusion, then verify results locally.

n4n Team3 min read651 words

Audio narration

Coming soon — every post will get a voice note here.

Most production search stacks fail on either keyword precision or semantic recall, not both. This hybrid retriever Haystack 2.0 tutorial shows how to combine a sparse BM25 retriever with a dense embedding retriever in a single pipeline, using reciprocal rank fusion to get the best of both. You will end up with a runnable Python script that queries local documents and returns merged, re-ranked results.

Step 1: Install Haystack 2.0 and dependencies

Haystack 2.0 changed the component API significantly from 1.x. Install the core package plus a local embedding model backend. We use sentence-transformers for dense vectors to keep the example self-contained.

pip install "haystack-ai>=2.0.0" sentence-transformers

If you plan to use a hosted embedding model instead of local inference, also have an API key ready. The rest of this hybrid retriever Haystack 2.0 tutorial assumes a Python 3.9+ environment.

Step 2: Initialize the document store and load data

We use InMemoryDocumentStore to avoid external infrastructure. In a real system you would swap this for ElasticsearchDocumentStore or PgvectorDocumentStore, but the retriever interfaces stay identical.

from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()

raw_docs = [
    Document(content="The Federal Reserve raised interest rates by 25 basis points to combat inflation."),
    Document(content="Solar panel efficiency has improved due to perovskite tandem cell research."),
    Document(content="Inflation metrics in the Eurozone showed a slight cooling in core prices."),
    Document(content="Central banks use rate policy as a lever against sustained price increases."),
    Document(content="Battery density improvements are critical for long-haul electric trucks."),
]

# BM25 needs no vectors; write raw docs first
document_store.write_documents(raw_docs)

BM25 operates on token statistics, so it works immediately. Dense retrieval requires vectors stored alongside the documents.

Step 3: Generate and store embeddings

Use SentenceTransformersDocumentEmbedder to compute dense vectors locally. Warm up the model once, then embed the documents and overwrite them in the store with vectors attached.

from haystack.components.embedders import SentenceTransformersDocumentEmbedder

doc_embedder = SentenceTransformersDocumentEmbedder(
    model="sentence-transformers/all-MiniLM-L6-v2"
)
doc_embedder.warm_up()
embedded_docs = doc_embedder.run(raw_docs)["documents"]
document_store.write_documents(embedded_docs, policy="overwrite")

If you would rather not run models locally, point Haystack’s OpenAI-compatible embedder at an inference gateway. For example, n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and automatically falls back when a provider is degraded, so you can swap SentenceTransformersDocumentEmbedder for OpenAIDocumentEmbedder(api_base_url="https://api.n4n.ai/v1", model="text-embedding-3-small") without changing the pipeline shape.

Step 4: Configure sparse and dense retrievers

Both retrievers target the same store but use different indexes. InMemoryBM25Retriever uses the text field; InMemoryEmbeddingRetriever uses the stored embedding. Set top_k per retriever to control candidate volume before fusion.

from haystack.components.retrievers.in_memory import (
    InMemoryBM25Retriever,
    InMemoryEmbeddingRetriever,
)

bm25_retriever = InMemoryBM25Retriever(document_store, top_k=5)
embedding_retriever = InMemoryEmbeddingRetriever(document_store, top_k=5)

A common mistake is setting top_k too low on one retriever. Hybrid search only helps if each side contributes candidates; keep at least 5–10 per branch.

Step 5: Build the hybrid pipeline with DocumentJoiner

Haystack 2.0 provides DocumentJoiner with join_mode="reciprocal_rank_fusion" (RRF). RRF scores each document as sum(1 / (rank + k)) across retrievers, which is robust to score scale differences between BM25 and cosine similarity.

from haystack import Pipeline
from haystack.components.embedders import SentenceTransformersTextEmbedder
from haystack.components.joiners import DocumentJoiner

text_embedder = SentenceTransformersTextEmbedder(
    model="sentence-transformers/all-MiniLM-L6-v2"
)
text_embedder.warm_up()

joiner = DocumentJoiner(join_mode="reciprocal_rank_fusion", top_k=10)

pipeline = Pipeline()
pipeline.add_component("text_embedder", text_embedder)
pipeline.add_component("bm25_retriever", bm25_retriever)
pipeline.add_component("embedding_retriever", embedding_retriever)
pipeline.add_component("joiner", joiner)

pipeline.connect("text_embedder.embedding", "embedding_retriever.query_embedding")
pipeline.connect("bm25_retriever.documents", "joiner.documents")
pipeline.connect("embedding_retriever.documents", "joiner.documents")

Note the connection: the query text is embedded by text_embedder, and its vector feeds the dense retriever. The BM25 retriever takes the raw query string directly. Both output lists flow into the joiner.

Step 6: Run queries and verify success

Execute the pipeline with a query that has both a lexical match and a semantic paraphrase. “rate hikes to control prices” should trigger BM25 via “rate”/“prices” and dense match via “control prices” ~ “combat inflation”.

query = "rate hikes to control prices"
result = pipeline.run({
    "bm25_retriever": {"query": query},
    "text_embedder": {"text": query},
})

merged = result["joiner"]["documents"]
for i, doc in enumerate(merged, 1):
    print(f"{i}. score={doc.score:.4f} :: {doc.content}")

Verification criteria:

  • The output list contains documents from both retrievers (e.g., the Fed rate doc and the central bank doc).
  • doc.score is a fused RRF value, not a raw BM25 or cosine score.
  • No document appears twice; DocumentJoiner deduplicates by id.

If you see only one retriever’s results, check that embeddings were actually written in Step 3 and that top_k is non-zero.

Step 7: Tuning and production notes

The default RRF constant k is 61 in Haystack; you can pass weights to DocumentJoiner if you want to bias toward dense or sparse. For example, DocumentJoiner(join_mode="reciprocal_rank_fusion", weights=[0.3, 0.7]) favors the embedding retriever.

In this hybrid retriever Haystack 2.0 tutorial we used an in-memory store, but the same code works against persistent stores. When you move to Elasticsearch, switch the store class and use ElasticsearchBM25Retriever and ElasticsearchEmbeddingRetriever; the joiner and pipeline wiring are unchanged.

Two operational caveats:

  • Embedding inference is the latency bottleneck. Batch embed at index time, and consider async embedding for streaming writes.
  • BM25 benefits from proper preprocessing. Haystack’s default tokenizer is basic; add a DocumentCleaner or custom PreProcessor before writing if your corpus has HTML or noisy text.

Hybrid retrieval is not a silver bullet, but for knowledge bases where users issue both precise error-code lookups and vague conceptual questions, it removes the single-retriever failure mode with minimal code. The finished pipeline here is ~30 lines and ready to drop into a RAG service.

Tagshaystackretrieverhybrid-searchtutorial

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsHaystack retriever tutorial: filtering by metadataA hands-on Haystack retriever metadata filtering tutorial showing how to filter documents by custom metadata fields with runnable code examples and expected outputs.Framework tutorialsHaystack document store tutorial: in-memory store basicsLearn Haystack's InMemoryDocumentStore basics with runnable code — write documents, retrieve with BM25 and embeddings, filter by metadata, and understand when to graduate to persistent stores.Framework tutorialsHaystack document store tutorial: OpenSearch integrationStep-by-step tutorial for integrating OpenSearch as a Haystack document store with runnable code and expected outputs at each checkpoint.Framework tutorialsUsing Elasticsearch as a Haystack document storeA complete elasticsearch haystack document store tutorial with step-by-step setup, indexing, retrieval, and production hardening for engineers building RAG pipelines.Framework tutorialsHaystack retriever tutorial: BM25 vs embedding retrievalCompare BM25 and embedding retrievers in Haystack across latency, cost, accuracy, and operational complexity with code examples and a verdict by use case.Framework tutorialsConnecting Haystack to Pinecone for vector retrievalStep-by-step haystack pinecone document store tutorial: install deps, configure Pinecone, embed docs, build a retriever, and verify vector search in Haystack 2.x.

All haystack document stores & retrievers posts →