Haystack retriever tutorial: BM25 vs embedding retrieval

If you’re running a haystack retriever bm25 vs embedding tutorial search, you’re likely deciding which retrieval strategy to put in front of your LLM. BM25 is lexical, fast, and cheap. Embedding retrieval is semantic, slower, and costs more at query time. Both ship as first-class retrievers in Haystack, and the right choice — or combination — depends on your corpus, latency budget, and whether you can afford a vector database.

How BM25 retrieval works in Haystack

BM25 scores documents by term frequency and inverse document frequency, with a saturation function that prevents common terms from dominating. Haystack’s BM25Retriever wraps the implementation from your document store (OpenSearch, Elasticsearch, Weaviate, or the in-memory InMemoryDocumentStore). No external model, no GPU, no embedding API calls.

from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever

docstore = InMemoryDocumentStore()
docstore.write_documents([
    Document(content="Haystack BM25 retriever uses Okapi BM25 scoring"),
    Document(content="Embedding retrievers need a vector database"),
    Document(content="BM25 works on exact term matches"),
])

retriever = InMemoryBM25Retriever(document_store=docstore, top_k=3)
results = retriever.run(query="BM25 scoring")

The retriever returns documents ranked by lexical overlap. It shines when queries share vocabulary with your corpus — error codes, product SKUs, function names, legal citations. It fails on synonyms (“car” vs “automobile”) and conceptual matches (“how to optimize latency” vs “speed up inference”).

How embedding retrieval works in Haystack

Embedding retrieval encodes queries and documents into the same vector space, then ranks by cosine similarity (or dot product). Haystack provides SentenceTransformersDocumentEmbedder and SentenceTransformersTextEmbedder for local models, plus OpenAIDocumentEmbedder and OpenAITextEmbedder for hosted APIs. You need a document store with vector search: WeaviateDocumentStore, QdrantDocumentStore, PineconeDocumentStore, MilvusDocumentStore, or OpenSearchDocumentStore with a k-NN index.

from haystack import Document
from haystack.document_stores.weaviate import WeaviateDocumentStore
from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder
from haystack.components.retrievers.weaviate import WeaviateEmbeddingRetriever

docstore = WeaviateDocumentStore(url="http://localhost:8080", index="Docs")

embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
docs_with_embeddings = embedder.run(documents=[
    Document(content="Haystack embedding retriever uses vector similarity"),
    Document(content="Semantic search finds conceptually related documents"),
    Document(content="Vector databases index embeddings for fast ANN search"),
])["documents"]

docstore.write_documents(docs_with_embeddings)

retriever = WeaviateEmbeddingRetriever(document_store=docstore, top_k=3)
query_embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
query_embedding = query_embedder.run(text="vector search")["embedding"]
results = retriever.run(query_embedding=query_embedding)

The same model must embed both corpus and queries. Mismatched models produce garbage rankings. Embedding retrieval handles synonyms, paraphrases, and conceptual similarity — but it introduces model dependency, GPU costs (or API latency), and vector index maintenance.

Head-to-head comparison

Latency and throughput in practice

BM25 on InMemoryDocumentStore serves ~5,000 qps on a single core for 100k documents. On OpenSearch with SSD, expect 500–1,000 qps per node. Latency is predictable — inverted index lookup plus scoring.

Embedding retrieval adds two variable stages: query embedding (10–100 ms local, 50–300 ms API) and ANN search (HNSW: 5–30 ms on CPU, faster on GPU). Total p99 often lands at 200–500 ms. Throughput drops to 50–200 qps per GPU for local models, or whatever your embedding API quota allows.

If your SLA is <100 ms p99, BM25 wins unless you provision heavily. If you can tolerate 300–500 ms, embedding retrieval opens semantic matching.

Cost model differences

BM25 costs are infrastructure: document store nodes, storage, CPU. No per-query marginal cost. A 10M document OpenSearch cluster on r6g.xlarge instances runs ~$2,000/month for three nodes.

Embedding retrieval adds per-query embedding cost. OpenAI text-embedding-3-small at $0.02/1M tokens means $0.20 per 10k queries (assuming 100 tokens/query). Local inference on an A10G ($1.50/hr) serves ~500 qps, so ~$0.01 per 10k queries — but you pay for the GPU 24/7. Vector index storage adds 2–10 GB per million 384-dim vectors.

For low-volume prototypes, API embeddings are simpler. For production scale, local embedding models on spot GPUs or dedicated inference endpoints (like n4n.ai’s /embeddings route) reduce marginal cost by 10–50x.

Ergonomics and debugging

BM25 is transparent. You can print the query terms, see which documents matched which terms, and tune analyzers. Haystack’s BM25Retriever exposes score breakdowns when the underlying store supports it.

# Debug BM25: see term contributions
results = retriever.run(query="optimize inference latency", top_k=5)
for doc in results["documents"]:
    print(f"score={doc.score:.3f} content={doc.content[:80]}")

Embedding retrieval is a black box. You get a cosine similarity score. Debugging means checking: did the embedder run? Is the vector dimension correct? Is the ANN index built? Did the model change between index and query time? Haystack pipelines help — you can log embeddings at each stage — but root-causing a bad ranking often means retraining or swapping models.

Ecosystem and document store support

If you need hybrid search (BM25 + embedding fused via reciprocal rank fusion), choose OpenSearch, Weaviate, Milvus, or Elasticsearch. Qdrant and Pinecone are embedding-only. InMemoryDocumentStore is BM25-only — fine for prototypes, not production.

Limits and failure modes

BM25 fails on:

• Synonyms and paraphrases (“CPU” vs “processor”)
• Conceptual queries (“documents about scaling” matching “horizontal pod autoscaler”)
• Short queries with rare terms (idf spikes)
• Languages without analyzers (you must configure one per language)

Embedding retrieval fails on:

• Exact-match requirements (error codes, IDs, citations) — vectors blur boundaries
• Domain-specific jargon not in training data (legal, medical, proprietary APIs)
• Adversarial queries (embedding collapse, dimensionality curse)
• Model drift — re-embedding 10M docs takes hours
• Cold-start latency on first query (model load, index warmup)

Both retrievers benefit from a cross-encoder reranker (TransformersSimilarityRanker or CohereRanker) as a second stage. This corrects BM25’s precision gaps and embedding retrieval’s false positives.

Which to choose

Choose BM25 when:

• Queries share vocabulary with documents (logs, code, legal, support tickets)
• Latency budget <100 ms p99
• Zero marginal cost per query matters
• You need explainable rankings for compliance or debugging
• Corpus is <1M docs and fits in OpenSearch/Elasticsearch
• Team lacks GPU infrastructure or embedding expertise

Choose embedding retrieval when:

• Users query in natural language with synonyms/paraphrases
• Conceptual similarity matters more than keyword overlap
• Multilingual corpus with a single model
• You can tolerate 200–500 ms latency
• You have GPU budget or API budget for embeddings
• Corpus benefits from semantic clustering (RAG over docs, not keywords)

Choose hybrid (BM25 + embedding + RRF) when:

• You need both exact-match and semantic recall
• Document store supports it (OpenSearch, Weaviate, Milvus, Elasticsearch)
• You can afford the index complexity and dual write path
• Reranker latency budget allows two retrievers + cross-encoder

Default starting point for most teams: BM25 on OpenSearch with a cross-encoder reranker. It covers 70–80% of RAG use cases, costs less, and is easier to operate. Add embedding retrieval when BM25 recall plateaus and you have evidence that semantic gaps are hurting answer quality. Measure recall@k on your eval set before adding complexity.