n4nAI
Open console →
Framework tutorialsHow-toHaystack RAG Pipelines

Building a multi-hop RAG pipeline in Haystack 2.0

Step-by-step guide to building a multi-hop RAG pipeline in Haystack 2.0 that retrieves, rewrites queries, and synthesizes answers across two passes.

n4n Team3 min read670 words

Audio narration

Coming soon — every post will get a voice note here.

A multi-hop rag pipeline haystack 2.0 implementation answers questions that no single retrieval can satisfy. You retrieve once, use the LLM to identify a gap, issue a second targeted query, then synthesize. This post walks through a concrete two-hop build using Haystack 2.0 components, with runnable code and a verification path.

Step 1: Install dependencies and import components

Haystack 2.0 changed the component model significantly from 1.x. Pipelines are now directed graphs of typed components, not YAML soup. Install the current release and the model provider SDK:

pip install haystack-ai openai sentence-transformers

Import the pieces you need. We use the in-memory document store and local embedding models to keep the example self-contained:

from haystack import Pipeline, Document, component
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import (
    SentenceTransformersDocumentEmbedder,
    SentenceTransformersTextEmbedder,
)
from haystack.components.retrievers import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator

If you later swap the generator for a chat model, use OpenAIChatGenerator instead. The wiring below stays identical.

Step 2: Load and embed documents

Multi-hop retrieval only helps if the corpus is split such that relevant facts live in different documents. Create a small set of disjoint docs:

docs = [
    Document(content="Alpha project started in 2021 under the platform team."),
    Document(content="The platform team merged with infra in Q2 2022."),
    Document(content="Beta initiative is the successor to Alpha, funded from 2023."),
    Document(content="Charlie is the tech lead for Beta as of 2024."),
]

doc_store = InMemoryDocumentStore()
embedder = SentenceTransformersDocumentEmbedder(
    model="sentence-transformers/all-MiniLM-L6-v2"
)
embedder.warm_up()
embedded = embedder.run(docs)["documents"]
doc_store.write_documents(embedded)

Embedding at ingest time avoids recomputing per query. The warm_up() call loads weights; skip it and the first run pays the cost anyway.

Step 3: Define a query-extraction component

The first LLM call produces a follow-up query as a string inside a list of replies. The second text embedder expects a single text input, not a list. Write a tiny adapter component rather than hacking the pipeline with external variables:

@component
class FirstReply:
    @component.output_types(text=str)
    def run(self, replies: list[str]):
        # replies is a list of length 1 for non-streaming generators
        return {"text": replies[0].strip()}

This is the kind of glue you’ll write often in Haystack 2.0. Keep it side-effect free and typed.

Step 4: Build the two-hop pipeline

Construct the graph: embed original query → retrieve → build prompt → generate intermediate query → extract → embed again → retrieve → final synthesis.

hop1_template = """
Documents:
{% for d in documents %}
- {{ d.content }}
{% endfor %}
Original question: {{ query }}
Based only on the documents, what specific follow-up query would uncover missing facts to answer the original question?
Return only the query text.
"""

final_template = """
Original question: {{ original_query }}
First-pass documents:
{% for d in documents %}
- {{ d.content }}
{% endfor %}
Second-pass documents:
{% for d in documents2 %}
- {{ d.content }}
{% endfor %}
Answer the original question using both passes. If still unknown, say so.
"""

pipe = Pipeline()
pipe.add_component("q_embed_1", SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"))
pipe.add_component("retriever_1", InMemoryEmbeddingRetriever(doc_store, top_k=2))
pipe.add_component("prompt_1", PromptBuilder(hop1_template))
pipe.add_component("gen_1", OpenAIGenerator(model="gpt-3.5-turbo-instruct", api_key="sk-..."))
pipe.add_component("extract", FirstReply())
pipe.add_component("q_embed_2", SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"))
pipe.add_component("retriever_2", InMemoryEmbeddingRetriever(doc_store, top_k=2))
pipe.add_component("prompt_2", PromptBuilder(final_template))
pipe.add_component("gen_2", OpenAIGenerator(model="gpt-3.5-turbo-instruct", api_key="sk-..."))

pipe.connect("q_embed_1.embedding", "retriever_1.query_embedding")
pipe.connect("retriever_1.documents", "prompt_1.documents")
pipe.connect("prompt_1.query", "gen_1.prompt")  # PromptBuilder outputs 'prompt'
pipe.connect("gen_1.replies", "extract.replies")
pipe.connect("extract.text", "q_embed_2.text")
pipe.connect("q_embed_2.embedding", "retriever_2.query_embedding")
pipe.connect("retriever_2.documents", "prompt_2.documents2")

Note the prompt_1.query input: PromptBuilder maps template variables to inputs. We pass query at run time. The second prompt needs both documents (from retriever_1) and documents2 (from retriever_2). Connect retriever_1 documents to prompt_2 as well:

pipe.connect("retriever_1.documents", "prompt_2.documents")

Run inputs must include the original query for both embedder and first prompt:

question = "Who leads the successor to the project that started in 2021?"
result = pipe.run({
    "q_embed_1": {"text": question},
    "prompt_1": {"query": question},
    "prompt_2": {"original_query": question},
})

Step 5: Execute and verify success

A correct multi-hop rag pipeline haystack 2.0 run should print a final answer that names Charlie, not just Alpha’s start year. Inspect the intermediate hop to confirm the rewrite worked:

print("Hop1 query:", result["extract"]["text"])
print("Final answer:", result["gen_2"]["replies"][0])

Expected behavior:

  • extract.text is a concise query like “Who leads Beta initiative?” rather than the full original question.
  • retriever_2 returns the Charlie doc.
  • gen_2 replies with “Charlie” and cites the merge chain.

If extract.text echoes the original question, tighten hop1_template to forbid it. If the second retriever returns nothing useful, raise top_k or use a denser embedder.

Step 6: Point the generator at a gateway (optional)

In production you rarely want a hard dependency on one provider’s SDK. Haystack’s OpenAIGenerator accepts a api_base_url. If you point it at an OpenAI-compatible gateway such as n4n.ai, you get one endpoint that addresses 240+ models with automatic fallback when a provider is rate-limited, and per-token usage metering without custom instrumentation. The change is one line:

gen_1 = OpenAIGenerator(
    model="anthropic/claude-3-haiku",
    api_base_url="https://api.n4n.ai/v1",
    api_key="your-gateway-key",
)

The pipeline graph does not change. Honoring client routing directives and provider cache-control hints is handled at the gateway, so your Haystack code stays clean.

Step 7: Harden the loop for real corpora

Two hops are enough for most support and research queries, but the pattern extends. Three considerations from shipping this:

Idempotent document stores

InMemoryDocumentStore is fine for tests. For production use ElasticsearchDocumentStore or PgvectorDocumentStore and embed once in a batch job. Re-embedding on every process start wastes compute.

Retry and timeouts

Wrap generator calls with a Timeout and a simple retry decorator if you’re not using a gateway that already falls back. Haystack 2.0 components are plain Python; you can subclass OpenAIGenerator to add backoff.

Stopping condition

True multi-hop agents loop until a confidence check passes. In Haystack, implement that as a Python while around two pipelines (retrieve+rewrite, then retrieve+answer) rather than forcing a cycle in the graph. The graph is acyclic by design; respect that and orchestrate loops in code.

The multi-hop rag pipeline haystack 2.0 pattern above is the minimal version that works end to end. Extend the templates, swap the embedder for a domain model, and you have a retrieval chain that answers questions requiring evidence from disjoint sources.

Tagshaystackragmulti-hoppipeline

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsBuild a RAG pipeline with Haystack 2.0 and n4n.aiHands-on tutorial to build rag pipeline haystack 2.0 n4n.ai using Haystack 2.0 and an OpenAI-compatible endpoint, with runnable code and expected output.Framework tutorialsBuilding a citation-aware RAG pipeline in HaystackStep-by-step guide to building a citation-aware RAG pipeline in Haystack that returns sourced answers with document references and confidence.Framework tutorialsChunking strategies for Haystack RAG pipelinesPractical guide to chunking strategies for Haystack RAG pipelines with code examples, tradeoffs, and evaluation methods.Framework tutorialsHaystack RAG pipeline tutorial: enterprise document searchBuild a production-ready Haystack RAG pipeline for enterprise document search with hybrid retrieval, reranking, and observability.Framework tutorialsHaystack RAG pipeline tutorial: query routingBuild a Haystack RAG pipeline that classifies queries and routes them to specialized retrievers for better accuracy and lower latency.Framework tutorialsStreaming responses in Haystack RAG pipelines with n4n.aiBuild a Haystack RAG pipeline that streams tokens from n4n.ai with runnable code, error handling, and verification steps.

All haystack rag pipelines posts →