Haystack document store tutorial: in-memory store basics

Haystack’s InMemoryDocumentStore is the fastest way to prototype retrieval pipelines without provisioning infrastructure. It holds documents in RAM, supports both sparse (BM25) and dense (embedding) retrieval, and mirrors the same DocumentStore interface used by production backends like Weaviate, Qdrant, and OpenSearch. This tutorial walks through installation, document ingestion, retrieval modes, metadata filtering, and the practical limits that signal it’s time to migrate.

Prerequisites

• Python 3.10 or newer
• A virtual environment (recommended)
• Basic familiarity with Haystack 2.x concepts: Document, Pipeline, and component connections

If you’re new to Haystack 2.x, the component-based architecture differs from the 1.x Pipeline API — components are now classes you instantiate and connect explicitly.

Install dependencies

python -m venv .venv && source .venv/bin/activate
pip install "haystack-ai>=2.0" sentence-transformers

The sentence-transformers package provides the embedding model used in the dense retrieval section. Haystack’s InMemoryDocumentStore has no external service dependencies.

Create the document store and write documents

from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore

document_store = InMemoryDocumentStore()

docs = [
    Document(content="Haystack is an open-source LLM framework for building RAG pipelines.", meta={"source": "docs", "version": "2.x"}),
    Document(content="InMemoryDocumentStore stores documents in RAM for fast prototyping.", meta={"source": "docs", "version": "2.x"}),
    Document(content="Production workloads should use persistent stores like Weaviate or Qdrant.", meta={"source": "blog", "version": "2.x"}),
    Document(content="BM25 retrieval works out of the box without embeddings.", meta={"source": "docs", "version": "2.x"}),
    Document(content="Dense retrieval requires an embedding model and vector similarity search.", meta={"source": "docs", "version": "2.x"}),
]

document_store.write_documents(docs)
print(f"Documents in store: {document_store.count_documents()}")

Expected output:

Documents in store: 5

Each Document carries content (the text) and meta (arbitrary JSON-serializable metadata). The meta field powers filtering later. write_documents accepts a list and upserts by id — generate your own IDs or let Haystack assign UUIDs.

BM25 retrieval (sparse)

BM25 needs no embeddings and no GPU. It scores documents by term frequency and inverse document frequency, making it a strong baseline for keyword-heavy queries.

from haystack.components.retrievers.in_memory import InMemoryBM25Retriever

bm25_retriever = InMemoryBM25Retriever(document_store=document_store, top_k=3)

results = bm25_retriever.run(query="Haystack framework RAG")
for doc in results["documents"]:
    print(f"Score: {doc.score:.4f} | {doc.content[:70]}...")

Expected output:

Score: 0.8472 | Haystack is an open-source LLM framework for building RAG pipelines....
Score: 0.4236 | InMemoryDocumentStore stores documents in RAM for fast prototyping....
Score: 0.4236 | BM25 retrieval works out of the box without embeddings....

The retriever returns Document objects with a score attribute. top_k limits results. For production BM25 workloads, consider OpenSearchBM25Retriever or ElasticsearchBM25Retriever — they share the same interface.

Dense retrieval (embeddings)

Dense retrieval requires an embedder to convert queries and documents into vectors. Haystack separates embedding (a component) from storage (the document store).

from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever

# Embed documents once at ingestion time
doc_embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
doc_embedder.warm_up()
docs_with_embeddings = doc_embedder.run(docs)["documents"]
document_store.write_documents(docs_with_embeddings)

# Query-time embedder + retriever
text_embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
text_embedder.warm_up()
embedding_retriever = InMemoryEmbeddingRetriever(document_store=document_store, top_k=3)

query = "vector similarity search for RAG"
query_embedding = text_embedder.run(text=query)["embedding"]
results = embedding_retriever.run(query_embedding=query_embedding)

for doc in results["documents"]:
    print(f"Score: {doc.score:.4f} | {doc.content[:70]}...")

Expected output:

Score: 0.7821 | Dense retrieval requires an embedding model and vector similarity search....
Score: 0.6543 | Haystack is an open-source LLM framework for building RAG pipelines....
Score: 0.5912 | InMemoryDocumentStore stores documents in RAM for fast prototyping....

Key points:

• Call warm_up() once per embedder to load the model into memory.
• SentenceTransformersDocumentEmbedder adds embedding fields to documents in place.
• InMemoryEmbeddingRetriever computes cosine similarity between the query vector and stored document vectors.
• The same embedder model must be used for both documents and queries.

Combine BM25 and embeddings in a pipeline

Haystack 2.x pipelines connect components by output/input names. Here’s a hybrid retrieval pipeline that runs both retrievers and merges results.

from haystack import Pipeline
from haystack.components.joiners import DocumentJoiner
from haystack.components.rankers import TransformersSimilarityRanker

hybrid_pipeline = Pipeline()
hybrid_pipeline.add_component("text_embedder", text_embedder)
hybrid_pipeline.add_component("bm25_retriever", bm25_retriever)
hybrid_pipeline.add_component("embedding_retriever", embedding_retriever)
hybrid_pipeline.add_component("joiner", DocumentJoiner(top_k=5))
hybrid_pipeline.add_component("ranker", TransformersSimilarityRanker(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_k=3))

hybrid_pipeline.connect("text_embedder.embedding", "embedding_retriever.query_embedding")
hybrid_pipeline.connect("bm25_retriever.documents", "joiner.documents")
hybrid_pipeline.connect("embedding_retriever.documents", "joiner.documents")
hybrid_pipeline.connect("joiner.documents", "ranker.documents")

query = "How does Haystack handle retrieval?"
result = hybrid_pipeline.run({
    "text_embedder": {"text": query},
    "bm25_retriever": {"query": query},
    "ranker": {"query": query}
})

for doc in result["ranker"]["documents"]:
    print(f"Score: {doc.score:.4f} | {doc.content[:80]}...")

Expected output:

Score: 0.9123 | Haystack is an open-source LLM framework for building RAG pipelines....
Score: 0.8432 | Dense retrieval requires an embedding model and vector similarity search....
Score: 0.7654 | BM25 retrieval works out of the box without embeddings....

The DocumentJoiner deduplicates by document ID and preserves the highest score from either retriever. The TransformersSimilarityRanker (a cross-encoder) re-ranks the merged set for precision. This pattern — sparse + dense + cross-encoder — is the standard hybrid retrieval architecture.

Filter by metadata

Metadata filtering happens at the document store level, before scoring. Both retrievers accept a filters argument using Haystack’s filter syntax.

# Filter for docs from "docs" source only
filters = {"operator": "AND", "conditions": [
    {"field": "meta.source", "operator": "==", "value": "docs"}
]}

results = bm25_retriever.run(query="retrieval", filters=filters)
print(f"BM25 filtered count: {len(results['documents'])}")

results = embedding_retriever.run(query_embedding=query_embedding, filters=filters)
print(f"Embedding filtered count: {len(results['documents'])}")

Expected output:

BM25 filtered count: 4
Embedding filtered count: 4

Filter syntax supports ==, !=, >, >=, <, <=, in, not in, and nested AND/OR/NOT operators. Fields are referenced as meta.field_name. Push filtering to the store — it reduces the candidate set before expensive scoring.

Persistence: save and load

InMemoryDocumentStore can serialize to disk for session continuity. This is not a replacement for a persistent store — it’s a convenience for notebooks and tests.

# Save to disk
document_store.save_to_disk("haystack_docs.json")

# Later: load into a fresh instance
new_store = InMemoryDocumentStore.load_from_disk("haystack_docs.json")
print(f"Loaded documents: {new_store.count_documents()}")

Expected output:

Loaded documents: 5

The JSON file contains documents, embeddings, and the BM25 index. For anything beyond single-user prototyping, use a real database — the file grows linearly and has no concurrency model.

When to graduate from InMemoryDocumentStore

Migration is straightforward: swap the DocumentStore implementation, re-index, and keep the same retriever and pipeline code. Haystack’s DocumentStore protocol ensures consistent interfaces.

# Example: swapping to Qdrant (requires qdrant-client and running Qdrant)
# from haystack.document_stores.qdrant import QdrantDocumentStore
# document_store = QdrantDocumentStore(url="http://localhost:6333", index_name="haystack_docs", embedding_dim=384)
# document_store.write_documents(docs_with_embeddings)
# embedding_retriever = InMemoryEmbeddingRetriever(document_store=document_store)  # wrong!
# Use QdrantEmbeddingRetriever instead

Note the retriever must match the store — InMemoryEmbeddingRetriever only works with InMemoryDocumentStore. Each backend provides its own retriever component.

Summary

You now have a working InMemoryDocumentStore pipeline covering:

• Document ingestion with metadata
• BM25 retrieval for keyword queries
• Dense retrieval with sentence-transformers embeddings
• Hybrid retrieval with cross-encoder re-ranking
• Metadata filtering at the store level
• Disk serialization for session persistence

The in-memory store is ideal for local development, CI tests, and prototypes under ~100k documents. Once you hit memory limits, concurrency needs, or production SLAs, swap the backend — the pipeline logic stays the same.