Haystack setup tutorial: document stores explained

A document store in Haystack is the persistence layer that holds your indexed documents and their embeddings, enabling fast semantic retrieval for retrieval-augmented generation pipelines. It abstracts the underlying database — whether that’s Elasticsearch, Weaviate, Pinecone, or a local FAISS index — behind a consistent interface so your retrieval logic stays portable. This haystack document store setup tutorial walks through the core concepts, the major backends, and the practical decisions that shape production RAG systems.

What a document store actually does

At its core, a Haystack document store implements two contracts: write_documents() for ingestion and query() or query_by_embedding() for retrieval. Everything else — filtering, hybrid search, metadata management — builds on those primitives.

When you call write_documents(), Haystack serializes each Document object (content, metadata, optional embedding) into the backend’s native format. If you pass pre-computed embeddings, the store indexes them directly. If you don’t, and the backend supports it (like Weaviate or Elasticsearch with ELSER), the store can generate embeddings on write via an attached embedder.

On the read path, query_by_embedding() takes a vector and returns the top-k nearest neighbors using the backend’s ANN index. query() with a string delegates to the backend’s full-text engine (BM25, Lucene, etc.). Hybrid search combines both scores, typically with reciprocal rank fusion or a weighted sum.

The document store also owns the schema: index mappings, vector dimensions, metadata field types, and filterable attributes. Getting this right upfront prevents painful re-indexing later.

Major backends and when to use them

Haystack ships first-party integrations for over a dozen stores. The right choice depends on scale, latency requirements, operational maturity, and whether you need managed or self-hosted.

Elasticsearch / OpenSearch

The default for teams already running the ELK stack. Mature, battle-tested, supports dense vectors (via k-NN plugin), sparse vectors (ELSER), and full-text in one cluster. Filtering on metadata is expressive — range, geo, nested objects, script queries.

from haystack.document_stores import ElasticsearchDocumentStore

document_store = ElasticsearchDocumentStore(
    hosts="https://es-cluster:9200",
    index="documents",
    embedding_dim=768,
    similarity="cosine",
    verify_certs=True,
    ca_certs="/path/to/ca.crt",
)

Operational cost is real: you manage shards, replicas, JVM tuning, and upgrade cycles. For teams without dedicated ops, a managed offering (Elastic Cloud, AWS OpenSearch) shifts burden but adds cost.

Weaviate

Purpose-built for vector search with a GraphQL-native API. Stores vectors and objects together, supports hybrid search out of the box, and includes a built-in vectorizer module (OpenAI, Cohere, Hugging Face) so you can skip a separate embedding step.

from haystack.document_stores import WeaviateDocumentStore

document_store = WeaviateDocumentStore(
    url="https://your-cluster.weaviate.network",
    index_name="Document",
    embedding_dim=1536,
    similarity="cosine",
    auth_client_secret=weaviate.AuthApiKey(api_key="YOUR_KEY"),
)

Weaviate’s filter syntax is proprietary (GraphQL where clauses), which can leak into your retrieval code if you’re not careful. The managed cloud tier is generous for prototyping; self-hosted requires Kubernetes and etcd.

Pinecone

Fully managed, serverless, pay-per-request. Zero infrastructure. Latency is consistently low (p99 < 50ms) because the control plane handles sharding and replication invisibly. Metadata filtering is limited to exact match, range, and $in — no nested objects or geo.

from haystack.document_stores import PineconeDocumentStore

document_store = PineconeDocumentStore(
    api_key="YOUR_API_KEY",
    environment="us-east-1-aws",
    index="documents",
    namespace="default",
    embedding_dim=1536,
    metric="cosine",
)

You trade query flexibility for operational simplicity. Pinecone’s namespace feature maps cleanly to multi-tenancy — one index, many isolated namespaces.

FAISS / InMemoryDocumentStore

Local, file-based, zero dependencies. FAISS uses Facebook’s ANN library; InMemoryDocumentStore keeps everything in Python dicts. Both are single-node, no persistence across processes (FAISS writes index files on save()).

from haystack.document_stores import FAISSDocumentStore

document_store = FAISSDocumentStore(
    sql_url="sqlite:///faiss_document_store.db",
    faiss_index_factory_str="Flat",
    embedding_dim=768,
)

Use these for local development, CI pipelines, or prototypes under 100k documents. They don’t scale horizontally, and FAISS’s Flat index is exact search (O(n)) — switch to IVF or HNSW for larger corpuses.

Qdrant

Rust-based, supports both managed cloud and self-hosted (Docker, Kubernetes). Rich payload filtering (nested, geo, range), payload indexing for fast filtered ANN, and on-disk storage with memory mapping. Hybrid search via sparse vectors is in beta.

from haystack.document_stores import QdrantDocumentStore

document_store = QdrantDocumentStore(
    url="http://localhost:6333",
    index_name="documents",
    embedding_dim=768,
    similarity="cosine",
    hnsw_config={"m": 16, "ef_construct": 128},
)

Qdrant hits a sweet spot: more query expressiveness than Pinecone, less operational weight than Elasticsearch. The Haystack integration is maintained by the Qdrant team.

Milvus / Zilliz

Built for billion-scale vectors. Distributed architecture with separate query, data, and index nodes. Supports multiple index types (HNSW, IVF, DiskANN), partitioning, and role-based access control. Zilliz is the managed cloud variant.

from haystack.document_stores import MilvusDocumentStore

document_store = MilvusDocumentStore(
    host="localhost",
    port=19530,
    collection_name="documents",
    embedding_dim=768,
    similarity="IP",
    index_params={"index_type": "HNSW", "params": {"M": 16, "efConstruction": 200}},
)

Overkill for most RAG workloads under 10M documents. The operational surface area is large — consider only when you need multi-tenancy at scale or have a Milvus-savvy platform team.

Why the document store choice matters

The document store isn’t plug-and-play invisible infrastructure. It shapes three things you’ll feel in production: retrieval quality, latency tail, and operational friction.

Retrieval quality depends on index type and filter expressiveness. A flat FAISS index gives exact cosine similarity but dies past 50k vectors. HNSW approximates with tunable recall — ef_search at query time trades latency for accuracy. If your metadata filters are complex (nested arrays, geo-polygons), Pinecone’s flat filter model forces post-filtering, which kills recall on filtered queries.

Latency tail is dominated by network hops and cold starts. Self-hosted Elasticsearch on the same VPC as your Haystack service typically sees p99 < 100ms. Pinecone and Weaviate Cloud add cross-region latency unless you pin to the same cloud region. FAISS in-process is microseconds but doesn’t survive a pod restart.

Operational friction compounds. Schema changes (adding a filterable field, changing vector dimension) require re-indexing in most stores. Elasticsearch and OpenSearch support dynamic mapping updates; Pinecone requires deleting and recreating the index. Weaviate allows schema evolution but with migration caveats. Plan for at least one full re-index in your first year.

Concrete example: building a multi-tenant RAG pipeline

Imagine a SaaS product where each customer uploads their own docs and queries only their data. You need isolation, per-tenant quotas, and shared infrastructure.

Schema design

Use a single index with a tenant_id field that’s filterable on every query. In Pinecone, this maps to a namespace per tenant. In Elasticsearch, Weaviate, Qdrant, and Milvus, it’s a keyword field with a filter clause.

from haystack import Document
from haystack.document_stores import QdrantDocumentStore

document_store = QdrantDocumentStore(
    url="http://qdrant:6333",
    index_name="tenant_docs",
    embedding_dim=1024,
    similarity="cosine",
)

# Ingestion: tag every document with tenant_id
docs = [
    Document(content="...", meta={"tenant_id": "acme-corp", "source": "pdf", "page": 3}),
    Document(content="...", meta={"tenant_id": "globex", "source": "html", "url": "..."}),
]
document_store.write_documents(docs)

Retrieval with tenant isolation

from haystack.nodes import EmbeddingRetriever

retriever = EmbeddingRetriever(
    document_store=document_store,
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    top_k=10,
    filters={"tenant_id": ["acme-corp"]},  # enforced at query time
)

results = retriever.retrieve(query="refund policy", filters={"tenant_id": ["acme-corp"]})

The filter pushes down to Qdrant’s payload index — no post-filtering, no leakage. Same pattern works across backends; only the filter syntax changes.

Quota enforcement

Track document counts per tenant in a sidecar table (PostgreSQL, Redis) and gate write_documents() calls in your ingestion service. The document store itself doesn’t enforce quotas.

async def ingest_documents(tenant_id: str, documents: list[Document]) -> int:
    current_count = await quota_store.get_count(tenant_id)
    if current_count + len(documents) > TENANT_QUOTA:
        raise QuotaExceeded(f"Tenant {tenant_id} at limit")
    document_store.write_documents(documents)
    await quota_store.increment(tenant_id, len(documents))
    return len(documents)

This pattern — document store for vectors, relational store for metadata and quotas — scales cleanly.

Common misconceptions

“The document store handles chunking”

It doesn’t. Haystack’s PreProcessor splits documents into chunks before they reach the document store. The store only sees the final Document objects. If you change chunk size or overlap, you re-run the preprocessor and re-index — the store has no concept of parent-child relationships unless you model them in metadata.

“Hybrid search is free quality”

Hybrid search (BM25 + dense) helps when queries mix keywords and semantics — product codes, error messages, proper nouns. But it adds latency (two index scans + fusion) and complexity (score normalization, weight tuning). Start with dense-only. Add hybrid only when evaluation shows a measurable gap on your specific query distribution.

“Managed means no ops”

Managed removes server patching and scaling decisions. It doesn’t remove: index design, dimension mismatches, quota monitoring, backup/restore testing, or vendor deprecation risk. Pinecone’s serverless model abstracts shards but you still choose metric and embedding_dim at index creation — immutable decisions.

“FAISS is fine for production if we’re small”

FAISS has no replication, no HA, no concurrent write safety. A single pod restart loses the in-memory index unless you’ve configured persistent volume mounts and save() calls. For any user-facing service, use a backend with built-in replication (Qdrant, Weaviate, Elasticsearch) even at low scale.

“All document stores support the same filter syntax”

They don’t. Elasticsearch uses Query DSL. Weaviate uses GraphQL where. Pinecone uses a flat dict with $in, $nin, $gt, $lt. Qdrant uses a Filter object with must, should, must_not. Haystack’s filters parameter normalizes simple equality filters ({"field": "value"}), but complex filters leak backend specifics. If you need portable complex filters, build a thin translation layer or accept backend lock-in.

Choosing a backend: a decision checklist

Start with the simplest backend that satisfies your current hard constraints (filter complexity, scale, ops capacity). Migrate when a constraint becomes binding — not before.

Wiring it into a Haystack pipeline

The document store plugs into EmbeddingRetriever, BM25Retriever, or custom retrievers. A minimal RAG pipeline:

from haystack import Pipeline
from haystack.nodes import EmbeddingRetriever, PromptNode, PromptTemplate
from haystack.document_stores import QdrantDocumentStore

document_store = QdrantDocumentStore(
    url="http://qdrant:6333",
    index_name="docs",
    embedding_dim=768,
)

retriever = EmbeddingRetriever(
    document_store=document_store,
    embedding_model="sentence-transformers/all-mpnet-base-v2",
    top_k=5,
)

prompt_template = PromptTemplate(
    prompt="""Answer the question based on the context.
Context: {join(documents)}
Question: {query}
Answer:""",
    output_parser=lambda x: x.split("Answer:")[-1].strip(),
)

prompt_node = PromptNode(
    model_name_or_path="gpt-4o-mini",
    default_prompt_template=prompt_template,
    max_length=2048,
)

pipeline = Pipeline()
pipeline.add_node(component=retriever, name="Retriever", inputs=["Query"])
pipeline.add_node(component=prompt_node, name="PromptNode", inputs=["Retriever"])

result = pipeline.run(query="How do I reset my API key?")
print(result["answers"][0].answer)

Swap QdrantDocumentStore for PineconeDocumentStore or ElasticsearchDocumentStore — the pipeline code doesn’t change. That’s the point of the abstraction.

One final consideration: routing and fallback

If you run multiple providers (e.g., Pinecone primary, Qdrant backup), you need a routing layer that directs traffic based on health, latency, or cost. n4n.ai handles this at the inference layer — one OpenAI-compatible endpoint that addresses 240+ models with automatic fallback when a provider is rate-limited or degraded — but the document store routing lives in your application logic. A simple pattern: wrap retriever calls in a circuit breaker that fails over to a secondary store on timeout or error.

import asyncio
from haystack.nodes import EmbeddingRetriever

class FallbackRetriever:
    def __init__(self, primary: EmbeddingRetriever, secondary: EmbeddingRetriever):
        self.primary = primary
        self.secondary = secondary

    def retrieve(self, query: str, filters: dict = None, top_k: int = 10):
        try:
            return asyncio.wait_for(
                self.primary.retrieve(query=query, filters=filters, top_k=top_k),
                timeout=2.0,
            )
        except (asyncio.TimeoutError, Exception):
            return self.secondary.retrieve(query=query, filters=filters, top_k=top_k)

This keeps your RAG pipeline available even when a vector store has a bad day.

The document store is the foundation of your RAG system. Choose it like you’d choose a primary database: evaluate the query patterns, scale targets, and operational reality — not just the benchmark numbers. Start simple, measure, migrate when the data tells you to.