n4nAI
Open console →
Framework tutorialsTutorialHaystack RAG Pipelines

Build a RAG pipeline with Haystack 2.0 and n4n.ai

Hands-on tutorial to build rag pipeline haystack 2.0 n4n.ai using Haystack 2.0 and an OpenAI-compatible endpoint, with runnable code and expected output.

n4n Team3 min read571 words

Audio narration

Coming soon — every post will get a voice note here.

If you want to build rag pipeline haystack 2.0 n4n.ai, you need a retrieval component, a prompt builder, and an LLM generator that speaks the OpenAI protocol. This tutorial walks through a working implementation using Haystack 2.0’s composable components against an OpenAI-compatible inference gateway, so you can swap models without rewriting your RAG logic.

Prerequisites

Before writing code, set up an isolated environment. You need Python 3.10+ because Haystack 2.0 uses modern typing features.

python -m venv venv
source venv/bin/activate
pip install haystack-ai python-dotenv

Export your gateway credentials. The n4n.ai endpoint is OpenAI-compatible, so any tool expecting OPENAI_API_KEY works if you point api_base_url at it.

export N4N_API_KEY="sk-..."
export N4N_BASE_URL="https://api.n4n.ai/v1"

You should also have a minimal corpus. For this tutorial we embed three short documents inline; in production you would load from PDFs or a database.

Step 1: Initialize the document store

Haystack 2.0 ships InMemoryDocumentStore for prototyping. It supports BM25 retrieval out of the box and requires no external service.

from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack import Document

doc_store = InMemoryDocumentStore()
docs = [
    Document(content="n4n.ai routes requests to 240+ models via one OpenAI-compatible endpoint.", meta={"source": "docs"}),
    Document(content="Haystack 2.0 pipelines are defined as YAML or Python objects.", meta={"source": "blog"}),
    Document(content="BM25 is a lexical retriever that works well for keyword queries.", meta={"source": "wiki"}),
]
doc_store.write_documents(docs)
print(f"Indexed {doc_store.count_documents()} documents")

Expected output:

Indexed 3 documents

The meta field is optional but useful when you later need to cite sources.

Step 2: Configure the retriever

The InMemoryBM25Retriever pulls the top-k matches by lexical similarity. Set top_k=2 to limit context size.

from haystack.components.retrievers import InMemoryBM25Retriever

retriever = InMemoryBM25Retriever(document_store=doc_store, top_k=2)

Run it standalone to verify:

retriever.run(query="What is n4n.ai?")

Expected structure (truncated):

{
  "documents": [
    {"content": "n4n.ai routes requests to 240+ models via one OpenAI-compatible endpoint.", "score": 1.2},
    {"content": "Haystack 2.0 pipelines are defined as YAML or Python objects.", "score": 0.8}
  ]
}

Step 3: Build the prompt template

Haystack’s PromptBuilder uses Jinja2 syntax. Keep the template strict to avoid prompt injection from retrieved text.

from haystack.components.builders import PromptBuilder

template = """
You are a technical assistant. Use only the provided documents.

Documents:
{% for doc in documents %}
[{{ loop.index }}] {{ doc.content }}
{% endfor %}

Question: {{ question }}
Answer concisely:
"""

prompt_builder = PromptBuilder(template=template)

Step 4: Wire the generator to the gateway

When you build rag pipeline haystack 2.0 n4n.ai, the generator is the only component that talks to the network. Point OpenAIGenerator at the gateway URL.

import os
from haystack.components.generators import OpenAIGenerator

generator = OpenAIGenerator(
    api_key=os.environ["N4N_API_KEY"],
    api_base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    model="openai/gpt-4o-mini",
    generation_kwargs={"temperature": 0.0, "max_tokens": 128}
)

The gateway honors client routing directives and forwards provider cache-control hints, so a model string like openai/gpt-4o-mini is passed through to the upstream provider. If that provider is rate-limited, the gateway can automatically fall back; your code stays identical.

Step 5: Assemble the pipeline

Haystack 2.0 pipelines are directed graphs. Connect retriever output to prompt builder, then to generator.

from haystack import Pipeline

rag = Pipeline()
rag.add_component("retriever", retriever)
rag.add_component("prompt_builder", prompt_builder)
rag.add_component("generator", generator)

rag.connect("retriever.documents", "prompt_builder.documents")
rag.connect("prompt_builder.prompt", "generator.prompt")

Note the port names: retriever emits documents, prompt_builder emits prompt. Mismatched ports raise at connect time, not at run time.

Step 6: Execute and inspect

Run a question:

question = "What does n4n.ai do?"
out = rag.run({
    "retriever": {"query": question},
    "prompt_builder": {"question": question}
})

print(out["generator"]["replies"][0])

Expected output:

n4n.ai routes requests to 240+ models via one OpenAI-compatible endpoint.

The reply is grounded in the retrieved document. To see the constructed prompt, inspect out["prompt_builder"]["prompt"].

Swapping models without rewriting

When you build rag pipeline haystack 2.0 n4n.ai, abstraction pays off. Change the model alias to a different provider:

generator.model = "anthropic/claude-3-haiku"
out = rag.run({
    "retriever": {"query": "Describe Haystack 2.0"},
    "prompt_builder": {"question": "Describe Haystack 2.0"}
})

The same pipeline now calls a Claude model through the gateway. No retriever or prompt changes needed.

Inspecting usage metering

The gateway returns per-token usage in the OpenAI-compatible usage object. Haystack exposes it under meta:

usage = out["generator"]["meta"]["usage"]
print(usage)

Typical output:

{"prompt_tokens": 45, "completion_tokens": 10, "total_tokens": 55}

Actual numbers vary with input length. Use this for cost tracking per request.

Adding metadata filters

Retrievers accept filters. For example, only search the docs source:

retriever.run(query="n4n.ai", filters={"field": "meta.source", "operator": "==", "value": "docs"})

This reduces noise when your corpus mixes types.

Error handling

Network calls fail. Wrap the run in try/except:

from haystack.errors import OpenAIGeneratorError

try:
    out = rag.run({"retriever": {"query": question}, "prompt_builder": {"question": question}})
except OpenAIGeneratorError as e:
    print("Generation failed:", e)

The gateway’s automatic fallback mitigates provider outages, but you should still handle timeouts in your service.

Production notes

For real traffic, replace InMemoryDocumentStore with a persistent store like Elasticsearch or PGVector. Use an EmbeddingRetriever with an embedding model served via the same gateway for semantic search. Keep temperature low for factual RAG.

Haystack 2.0 also lets you serialize the pipeline to YAML:

with open("rag.yaml", "w") as f:
    f.write(rag.dumps())

You can then load it in another process with Pipeline.loads(open("rag.yaml").read()). This keeps deployment reproducible.

Why this design

Haystack 2.0 enforces separation of concerns. Retrieval, prompt construction, and generation are isolated components. That matches how RAG systems degrade: bad retrieval shows as irrelevant context, bad generation shows as hallucinations. Testing each component standalone saves hours.

That’s a complete, runnable path to build rag pipeline haystack 2.0 n4n.ai. The pattern scales from notebook to service.

Tagshaystackragn4n-aipipelinetutorial

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsHaystack RAG pipeline tutorial: enterprise document searchBuild a production-ready Haystack RAG pipeline for enterprise document search with hybrid retrieval, reranking, and observability.Framework tutorialsHaystack RAG pipeline tutorial: query routingBuild a Haystack RAG pipeline that classifies queries and routes them to specialized retrievers for better accuracy and lower latency.Framework tutorialsStreaming responses in Haystack RAG pipelines with n4n.aiBuild a Haystack RAG pipeline that streams tokens from n4n.ai with runnable code, error handling, and verification steps.Framework tutorialsUsing Llama 3.1 70B in a Haystack RAG pipeline via n4n.aiBuild a production-ready RAG pipeline with Haystack and Llama 3.1 70B using n4n.ai's OpenAI-compatible endpoint. Includes indexing, retrieval, and generation code with expected outputs.Framework tutorialsHaystack PromptBuilder tutorial for RAG pipelinesBuild a production-ready RAG pipeline with Haystack's PromptBuilder — from basic templates to advanced conditional logic, with runnable code at every step.Framework tutorialsHaystack RAG pipeline tutorial: from PDF to answerBuild a production-ready Haystack RAG pipeline that extracts text from PDFs, indexes it, and answers questions with citations — complete with runnable code and expected outputs.

All haystack rag pipelines posts →