Chroma as a Haystack document store: setup tutorial

This chroma haystack document store setup tutorial walks through wiring Chroma into a Haystack pipeline from scratch. You’ll create a local Chroma instance, configure the Haystack document store, index documents with embeddings, and run retrieval queries. The code is minimal, runnable, and shows expected output at each checkpoint so you can verify things work before moving on.

Prerequisites

• Python 3.10+
• A virtual environment (recommended)
• An OpenAI API key for embeddings (or any embedding provider Haystack supports)

Install the required packages:

pip install haystack-ai chromadb openai

Haystack 2.x uses the haystack-ai package name. The chromadb package provides the local Chroma server and client. We’ll use OpenAI’s text-embedding-3-small for embeddings, but you can swap in any Haystack-compatible embedder.

Start a local Chroma instance

Chroma can run in-memory, as a local persistent directory, or as a separate server process. For development, the persistent local mode is simplest — no separate process to manage, data survives restarts.

# chroma_setup.py
import chromadb

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(name="haystack_docs")
print(f"Collection '{collection.name}' created. Count: {collection.count()}")

Run it:

python chroma_setup.py

Expected output:

Collection 'haystack_docs' created. Count: 0

The ./chroma_db directory now contains your vector data. If you prefer an in-memory instance for throwaway experiments, replace PersistentClient with chromadb.Client().

Configure the Haystack Chroma document store

Haystack’s ChromaDocumentStore wraps the Chroma client. You need to pass the same persistence path and collection name so Haystack talks to the same data.

# document_store.py
from haystack_integrations.document_stores.chroma import ChromaDocumentStore

document_store = ChromaDocumentStore(
    persist_path="./chroma_db",
    collection_name="haystack_docs",
    # Optional: distance metric. Chroma defaults to cosine (hnsw:space=cosine).
    # For dot product or l2, pass: hnsw_space="ip" or hnsw_space="l2"
)
print(f"Document store ready. Docs in store: {document_store.count_documents()}")

Run it:

python document_store.py

Expected output:

Document store ready. Docs in store: 0

If you see a non-zero count, you already have documents from a previous run — that’s fine.

Create an indexing pipeline

An indexing pipeline embeds documents and writes them to the document store. We’ll use Haystack’s Document class, the OpenAI embedder, and the DocumentWriter.

# indexing_pipeline.py
import os
from haystack import Document, Pipeline
from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack.components.writers import DocumentWriter
from haystack_integrations.document_stores.chroma import ChromaDocumentStore

# 1. Document store
document_store = ChromaDocumentStore(
    persist_path="./chroma_db",
    collection_name="haystack_docs"
)

# 2. Embedder — requires OPENAI_API_KEY in env
embedder = OpenAIDocumentEmbedder(
    model="text-embedding-3-small",
    api_key=os.getenv("OPENAI_API_KEY")
)

# 3. Writer
writer = DocumentWriter(document_store=document_store)

# 4. Pipeline
indexing_pipeline = Pipeline()
indexing_pipeline.add_component("embedder", embedder)
indexing_pipeline.add_component("writer", writer)
indexing_pipeline.connect("embedder.documents", "writer.documents")

# 5. Sample documents
docs = [
    Document(content="Haystack is an open-source LLM framework for building RAG pipelines."),
    Document(content="Chroma is a vector database designed for AI applications."),
    Document(content="Embeddings convert text into high-dimensional vectors for semantic search."),
    Document(content="RAG stands for Retrieval-Augmented Generation."),
]

# 6. Run
result = indexing_pipeline.run({"embedder": {"documents": docs}})
print(f"Indexed {result['writer']['documents_written']} documents")
print(f"Store now has {document_store.count_documents()} documents")

Set your API key and run:

export OPENAI_API_KEY="sk-..."
python indexing_pipeline.py

Expected output:

Indexed 4 documents
Store now has 4 documents

At this point, Chroma holds four vectors with associated metadata. You can inspect the raw collection directly if needed:

# inspect.py
import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("haystack_docs")
print(collection.peek(limit=4))

Create a retrieval pipeline

Now build a query pipeline: embed the query, retrieve top-k documents from Chroma, and optionally pass them to a generator. We’ll stop at retrieval for this tutorial.

# retrieval_pipeline.py
import os
from haystack import Pipeline
from haystack.components.embedders import OpenAITextEmbedder
from haystack_integrations.components.retrievers.chroma import ChromaEmbeddingRetriever
from haystack_integrations.document_stores.chroma import ChromaDocumentStore

# 1. Document store (same config as indexing)
document_store = ChromaDocumentStore(
    persist_path="./chroma_db",
    collection_name="haystack_docs"
)

# 2. Query embedder
query_embedder = OpenAITextEmbedder(
    model="text-embedding-3-small",
    api_key=os.getenv("OPENAI_API_KEY")
)

# 3. Retriever
retriever = ChromaEmbeddingRetriever(
    document_store=document_store,
    top_k=3
)

# 4. Pipeline
retrieval_pipeline = Pipeline()
retrieval_pipeline.add_component("query_embedder", query_embedder)
retrieval_pipeline.add_component("retriever", retriever)
retrieval_pipeline.connect("query_embedder.embedding", "retriever.query_embedding")

# 5. Run a query
query = "What is RAG and how does it relate to Haystack?"
result = retrieval_pipeline.run({"query_embedder": {"text": query}})

print(f"Query: {query}\n")
for i, doc in enumerate(result["retriever"]["documents"], 1):
    print(f"  {i}. Score: {doc.score:.4f} | Content: {doc.content}")

Run it:

python retrieval_pipeline.py

Expected output (scores will vary slightly):

Query: What is RAG and how does it relate to Haystack?

  1. Score: 0.8234 | Content: RAG stands for Retrieval-Augmented Generation.
  2. Score: 0.7912 | Content: Haystack is an open-source LLM framework for building RAG pipelines.
  3. Score: 0.7105 | Content: Embeddings convert text into high-dimensional vectors for semantic search.

The retriever returns Document objects with score, content, meta, and id. You can feed these directly into a PromptBuilder + Generator for a full RAG loop.

Add metadata filtering

Chroma supports metadata filtering at query time. Haystack’s ChromaEmbeddingRetriever accepts a filters parameter using Chroma’s filter syntax.

# filtering.py
import os
from haystack import Pipeline, Document
from haystack.components.embedders import OpenAITextEmbedder
from haystack_integrations.components.retrievers.chroma import ChromaEmbeddingRetriever
from haystack_integrations.document_stores.chroma import ChromaDocumentStore

document_store = ChromaDocumentStore(
    persist_path="./chroma_db",
    collection_name="haystack_docs"
)

# Re-index with metadata
from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack.components.writers import DocumentWriter

embedder = OpenAIDocumentEmbedder(model="text-embedding-3-small", api_key=os.getenv("OPENAI_API_KEY"))
writer = DocumentWriter(document_store=document_store)

indexing = Pipeline()
indexing.add_component("embedder", embedder)
indexing.add_component("writer", writer)
indexing.connect("embedder.documents", "writer.documents")

docs_with_meta = [
    Document(content="Haystack 2.0 released with new pipeline syntax.", meta={"version": "2.0", "topic": "release"}),
    Document(content="Chroma 0.5 adds HNSW index persistence.", meta={"version": "0.5", "topic": "release"}),
    Document(content="RAG patterns: retrieve-then-read, generate-then-retrieve.", meta={"version": "N/A", "topic": "pattern"}),
]
indexing.run({"embedder": {"documents": docs_with_meta}})

# Now retrieve with a filter: only documents where topic == "release"
query_embedder = OpenAITextEmbedder(model="text-embedding-3-small", api_key=os.getenv("OPENAI_API_KEY"))
retriever = ChromaEmbeddingRetriever(document_store=document_store, top_k=5)

retrieval = Pipeline()
retrieval.add_component("query_embedder", query_embedder)
retrieval.add_component("retriever", retriever)
retrieval.connect("query_embedder.embedding", "retriever.query_embedding")

# Chroma filter syntax: {"field": {"$eq": "value"}}
filters = {"topic": {"$eq": "release"}}

result = retrieval.run({
    "query_embedder": {"text": "What new releases are there?"},
    "retriever": {"filters": filters}
})

print("Filtered results (topic=release):")
for doc in result["retriever"]["documents"]:
    print(f"  - {doc.content} | meta: {doc.meta}")

Run it:

python filtering.py

Expected output:

Filtered results (topic=release):
  - Haystack 2.0 released with new pipeline syntax. | meta: {'version': '2.0', 'topic': 'release'}
  - Chroma 0.5 adds HNSW index persistence. | meta: {'version': '0.5', 'topic': 'release'}

The third document (topic: “pattern”) is excluded. Chroma’s filter language supports $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, and logical operators $and, $or, $not. Pass complex filters as nested dictionaries.

Persistence and production notes

• Concurrency: Chroma’s local persistent mode is single-process. For multi-process or multi-machine access, run Chroma as a server (chroma run --path ./chroma_db --host 0.0.0.0 --port 8000) and connect with chromadb.HttpClient. Haystack’s ChromaDocumentStore accepts host and port parameters for this mode.
• Embedding consistency: Always use the same embedding model for indexing and query. Mixing models produces meaningless similarity scores.
• Collection management: ChromaDocumentStore creates the collection if it doesn’t exist. To reset, delete the ./chroma_db directory or call client.delete_collection("haystack_docs") before re-initializing.
• Batch indexing: For large corpora, batch documents in chunks of 100–500 to avoid memory pressure and API rate limits. Haystack’s DocumentWriter handles batches automatically when you pass a list of documents.

Wiring into a full RAG pipeline (sketch)

# rag_pipeline.py
import os
from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator
from haystack.components.embedders import OpenAITextEmbedder
from haystack_integrations.components.retrievers.chroma import ChromaEmbeddingRetriever
from haystack_integrations.document_stores.chroma import ChromaDocumentStore

document_store = ChromaDocumentStore(persist_path="./chroma_db", collection_name="haystack_docs")

prompt_template = """
Answer the question using only the provided context.

Context:
{% for doc in documents %}
  {{ doc.content }}
{% endfor %}

Question: {{ question }}
Answer:
"""

rag = Pipeline()
rag.add_component("query_embedder", OpenAITextEmbedder(model="text-embedding-3-small", api_key=os.getenv("OPENAI_API_KEY")))
rag.add_component("retriever", ChromaEmbeddingRetriever(document_store=document_store, top_k=4))
rag.add_component("prompt_builder", PromptBuilder(template=prompt_template))
rag.add_component("generator", OpenAIGenerator(model="gpt-4o-mini", api_key=os.getenv("OPENAI_API_KEY")))

rag.connect("query_embedder.embedding", "retriever.query_embedding")
rag.connect("retriever.documents", "prompt_builder.documents")
rag.connect("prompt_builder.prompt", "generator.prompt")

result = rag.run({
    "query_embedder": {"text": "How does Haystack relate to RAG?"},
    "prompt_builder": {"question": "How does Haystack relate to RAG?"}
})

print(result["generator"]["replies"][0])

This produces a grounded answer citing the indexed documents. The pattern scales: swap the generator, adjust the prompt, add a Ranker or AnswerBuilder as needed.

Common pitfalls

Next steps

• Replace the local Chroma with a hosted instance (Chroma Cloud, or self-hosted via Docker) by changing ChromaDocumentStore to use host/port instead of persist_path.
• Add a DocumentSplitter before embedding to chunk long texts.
• Experiment with hybrid retrieval: combine ChromaEmbeddingRetriever with a BM25Retriever via Haystack’s JoinDocuments component.
• Monitor embedding costs — text-embedding-3-small is inexpensive, but large corpora add up. Consider local embedders (e.g., SentenceTransformersDocumentEmbedder) for high-volume workloads.

You now have a working Chroma + Haystack stack: persistent vector storage, metadata filtering, and a retrieval pipeline ready for RAG. The same patterns apply whether you’re prototyping locally or deploying to production.