Haystack RAG pipeline tutorial: from PDF to answer

If you’ve ever tried to build a RAG system from scratch, you know the gap between “it works on my machine” and “it handles real PDFs at scale.” Haystack closes that gap with a component-based architecture that lets you swap parsers, embedders, retrievers, and generators without rewriting your pipeline. This tutorial walks through a complete haystack rag pipeline pdf to answer setup — from raw PDF to cited answer — using only open-source components you can run locally.

Prerequisites

You’ll need Python 3.10+ and a virtual environment. Install the core dependencies:

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install haystack-ai==2.7.0 \
    sentence-transformers==3.0.1 \
    pypdf==4.2.0 \
    rank-bm25==0.2.2 \
    tqdm==4.66.5

For the generator, we’ll use a local model via Ollama. Install Ollama and pull a small instruct model:

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b-instruct-q4_K_M

If you prefer an API-backed generator, you can swap the Ollama component for any OpenAI-compatible endpoint — including gateways that route across 240+ models with automatic fallback when a provider degrades.

Project structure

haystack-rag-pdf/
├── data/
│   └── sample.pdf          # your test document
├── pipeline.py             # main pipeline definition
├── query.py                # interactive query script
└── requirements.txt

Download a sample PDF for testing — any multi-page document with headings and tables works. The Haystack docs include a sample if you need one.

Step 1: PDF parsing and preprocessing

Haystack’s PyPDFToDocument converter extracts text, but raw PDF text is noisy. We’ll chain a converter, a cleaner, and a splitter to produce clean chunks with metadata preserved.

# pipeline.py
from pathlib import Path
from haystack import Pipeline
from haystack.components.converters import PyPDFToDocument
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import SentenceTransformersDocumentEmbedder

document_store = InMemoryDocumentStore()

pdf_converter = PyPDFToDocument()
cleaner = DocumentCleaner(
    remove_empty_lines=True,
    remove_extra_whitespaces=True,
    remove_repeated_substrings=False,
)
splitter = DocumentSplitter(
    split_by="sentence",
    split_length=5,
    split_overlap=2,
    split_respect_sentence_boundary=True,
)
embedder = SentenceTransformersDocumentEmbedder(
    model="sentence-transformers/all-MiniLM-L6-v2",
    batch_size=32,
)
writer = DocumentWriter(document_store=document_store)

indexing_pipeline = Pipeline()
indexing_pipeline.add_component("converter", pdf_converter)
indexing_pipeline.add_component("cleaner", cleaner)
indexing_pipeline.add_component("splitter", splitter)
indexing_pipeline.add_component("embedder", embedder)
indexing_pipeline.add_component("writer", writer)

indexing_pipeline.connect("converter", "cleaner")
indexing_pipeline.connect("cleaner", "splitter")
indexing_pipeline.connect("splitter", "embedder")
indexing_pipeline.connect("embedder", "writer")

Run it against your PDF:

# run_indexing.py
from pipeline import indexing_pipeline

result = indexing_pipeline.run({
    "converter": {"sources": [Path("data/sample.pdf")]}
})
print(f"Indexed {result['writer']['documents_written']} documents")

Expected output:

Indexed 47 documents

Each document in the store now carries content, embedding, and metadata like page_number and source_id.

Step 2: Hybrid retrieval — dense + sparse

Pure vector search misses exact-match keywords (error codes, product names). Pure BM25 misses semantic matches. Haystack’s HybridRetriever combines both with reciprocal rank fusion.

# add to pipeline.py
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.joiners import DocumentJoiner
from haystack.components.rankers import TransformersSimilarityRanker

embedding_retriever = InMemoryEmbeddingRetriever(
    document_store=document_store,
    top_k=20,
)
bm25_retriever = InMemoryBM25Retriever(
    document_store=document_store,
    top_k=20,
)
joiner = DocumentJoiner(join_mode="reciprocal_rank_fusion", top_k=10)
ranker = TransformersSimilarityRanker(
    model="cross-encoder/ms-marco-MiniLM-L-6-v2",
    top_k=5,
)

The ranker re-ranks the fused results using a cross-encoder — this is where quality jumps noticeably.

Step 3: Prompt construction with citations

We need a prompt that forces the model to cite sources. Haystack’s PromptBuilder handles Jinja2 templates cleanly.

# add to pipeline.py
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OllamaGenerator

prompt_template = """
You are a precise technical assistant. Answer the question using ONLY the provided documents.
Cite sources inline using [doc_N] where N is the document index (starting from 1).
If the answer cannot be found in the documents, say "I don't know."

Documents:
{% for doc in documents %}
[doc_{{ loop.index }}] (page {{ doc.meta.page_number }}): {{ doc.content }}
{% endfor %}

Question: {{ question }}

Answer:
"""

prompt_builder = PromptBuilder(template=prompt_template)
generator = OllamaGenerator(
    model="llama3.2:3b-instruct-q4_K_M",
    url="http://localhost:11434",
    generation_kwargs={
        "temperature": 0.1,
        "top_p": 0.9,
        "num_predict": 512,
    },
)

The low temperature keeps answers grounded. The num_predict cap prevents runaway generations.

Step 4: Assemble the query pipeline

# add to pipeline.py
from haystack.components.embedders import SentenceTransformersTextEmbedder

text_embedder = SentenceTransformersTextEmbedder(
    model="sentence-transformers/all-MiniLM-L6-v2",
)

query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", text_embedder)
query_pipeline.add_component("embedding_retriever", embedding_retriever)
query_pipeline.add_component("bm25_retriever", bm25_retriever)
query_pipeline.add_component("joiner", joiner)
query_pipeline.add_component("ranker", ranker)
query_pipeline.add_component("prompt_builder", prompt_builder)
query_pipeline.add_component("generator", generator)

query_pipeline.connect("text_embedder.embedding", "embedding_retriever.query_embedding")
query_pipeline.connect("bm25_retriever", "joiner")
query_pipeline.connect("embedding_retriever", "joiner")
query_pipeline.connect("joiner", "ranker")
query_pipeline.connect("ranker", "prompt_builder.documents")
query_pipeline.connect("prompt_builder", "generator")

Step 5: Interactive querying

# query.py
from pipeline import query_pipeline

def ask(question: str) -> str:
    result = query_pipeline.run({
        "text_embedder": {"text": question},
        "bm25_retriever": {"query": question},
        "prompt_builder": {"question": question},
        "ranker": {"query": question},
    })
    return result["generator"]["replies"][0]

if __name__ == "__main__":
    print("Haystack RAG ready. Type 'exit' to quit.\n")
    while True:
        q = input("❯ ").strip()
        if q.lower() in {"exit", "quit"}:
            break
        print(ask(q))
        print()

Run it:

python query.py

Example session:

❯ What is the maximum throughput reported for the API gateway?
The API gateway achieves a maximum throughput of 12,000 requests per second [doc_3] (page 4) under sustained load with 99th percentile latency under 50ms [doc_5] (page 5).

❯ Which authentication methods are supported?
Supported authentication methods include OAuth 2.0, API keys, and mTLS [doc_1] (page 2). JWT validation is handled at the edge [doc_2] (page 3).

❯ What is the pricing for the enterprise tier?
I don't know.

The “I don’t know” response confirms the model isn’t hallucinating — it only answers from retrieved context.

Step 6: Persisting the document store

In-memory storage vanishes on restart. Swap to a persistent backend in one line:

# pipeline.py - replace InMemoryDocumentStore
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.document_stores.chroma import ChromaDocumentStore

document_store = ChromaDocumentStore(
    persist_path="./chroma_db",
    collection_name="rag_docs",
    duplicate_policy=DuplicatePolicy.SKIP,
)

Add haystack-integrations[chroma]==2.7.0 to requirements. The rest of the pipeline stays identical — Haystack’s document store interface is consistent across backends.

Step 7: Evaluating retrieval quality

Before shipping, measure retrieval precision. Haystack’s evaluation module makes this straightforward:

# eval.py
from haystack import Pipeline
from haystack.components.evaluators import DocumentRecallEvaluator
from haystack.components.evaluators import DocumentMRREvaluator

eval_pipeline = Pipeline()
eval_pipeline.add_component("recall", DocumentRecallEvaluator())
eval_pipeline.add_component("mrr", DocumentMRREvaluator())

ground_truth = [
    {"question": "max throughput", "expected_doc_ids": ["doc_3"]},
    {"question": "auth methods", "expected_doc_ids": ["doc_1", "doc_2"]},
]

# Run retriever for each question, collect retrieved doc_ids, then evaluate

Aim for recall@5 > 0.8 and MRR > 0.6 on your domain-specific eval set before promoting to production.

Common failure modes and fixes

Symptom	Likely cause	Fix
“I don’t know” on known content	Chunk size too small, context fragmented	Increase `split_length` to 8–10, reduce overlap
Hallucinated citations	Prompt template allows free-form citation format	Enforce `[doc_N]` pattern strictly, lower temperature
Slow retrieval on >10k docs	In-memory store, no HNSW index	Migrate to Chroma/Weaviate/Qdrant with HNSW
Cross-encoder OOM	Batch size too large for GPU	Set `batch_size=16` in `TransformersSimilarityRanker`

Scaling considerations

This pipeline runs entirely locally. For production workloads:

Document store: Move to a managed vector database (Qdrant Cloud, Pinecone, Weaviate Cloud) with horizontal scaling
Embedder: Batch-embed offline; store embeddings in the document store to avoid re-embedding on every deploy
Generator: Route through an inference gateway that handles model fallback, caching, and per-token metering — especially useful when you need to swap between open-weight and proprietary models without code changes
Observability: Add Haystack’s PipelineTracer to emit spans to OpenTelemetry; correlate retrieval latency with generation quality

What’s next

You now have a working haystack rag pipeline pdf to answer system that parses PDFs, retrieves hybrid, re-ranks with a cross-encoder, and generates cited answers. From here:

Add a query rewriter component to expand ambiguous questions before retrieval
Implement metadata filtering (by date, author, document type) at the retriever level
Build a feedback loop: log user thumbs-up/down, mine false negatives for eval set expansion
Containerize with a multi-stage Dockerfile — keep the model weights in a separate layer for faster rebuilds

The component graph you built is portable. Swap the generator for GPT-4o, the document store for Elasticsearch, the embedder for Cohere — the pipeline definition barely changes. That’s the point of Haystack.