Haystack setup tutorial: connecting n4n.ai generators

Wiring Haystack to a non-OpenAI inference gateway is straightforward if you treat it as an OpenAI-compatible backend. This walkthrough covers a complete haystack n4n.ai generator setup that points Haystack’s OpenAIChatGenerator at the n4n.ai endpoint, so you can route to 240+ models without rewriting your pipeline code.

Step 1: Install Haystack and dependencies

Haystack 2.x changed the component API significantly from 1.x. Use a clean virtual environment to avoid version conflicts with older farm-haystack installs.

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install haystack-ai>=2.5.0 python-dotenv

Confirm the import path before writing any pipeline code:

from haystack.components.generators.openai import OpenAIChatGenerator
print("ok")

If that raises ModuleNotFoundError, you installed the legacy package. haystack-ai is the correct PyPI name for 2.x.

Step 2: Set up credentials and environment

The gateway expects a bearer token. Put it in an environment variable rather than hardcoding it in source.

export N4N_API_KEY="sk-your-key-here"

For local development, a .env file loaded via python-dotenv works:

from dotenv import load_dotenv
load_dotenv()
import os
assert os.environ.get("N4N_API_KEY"), "N4N_API_KEY missing"

Never log the key. Haystack’s generator will read it from api_key parameter; we pass os.environ["N4N_API_KEY"] directly.

Step 3: Configure the OpenAIChatGenerator for the haystack n4n.ai generator setup

The core of the haystack n4n.ai generator setup is the OpenAIChatGenerator configuration. Point api_base_url at the gateway’s OpenAI-compatible base, and use the model string format the gateway expects (typically provider/model-name).

from haystack.components.generators.openai import OpenAIChatGenerator
import os

generator = OpenAIChatGenerator(
    api_key=os.environ["N4N_API_KEY"],
    api_base_url="https://api.n4n.ai/v1",
    model="anthropic/claude-3-5-sonnet",
    generation_kwargs={
        "temperature": 0.2,
        "max_tokens": 512,
    },
)

A few concrete notes from shipping this:

• Model strings are routing directives. The gateway forwards them to the underlying provider. If you request openai/gpt-4o-mini you get that model; if you request meta-llama/llama-3-70b, the gateway routes accordingly. Invalid strings return a 400 with a model list reference.
• api_base_url must end without a trailing slash. Haystack appends /chat/completions. A stray slash doubles the path and yields 404s.
• Timeouts. Default HTTP timeout in Haystack is 30s. Set timeout in generation_kwargs if your workloads run long prompts through slower providers.

The gateway honors client routing directives and forwards provider cache-control hints, so if you pass extra_body={"cache": True} it propagates where supported.

Step 4: Build a minimal pipeline

A generator alone is useful, but Haystack shines when you compose it with a PromptBuilder. Below is a runnable script that asks a model to summarize a technical paragraph.

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators.openai import OpenAIChatGenerator
import os

prompt_template = """
Summarize the following text in one sentence, preserving technical accuracy:
{{ text }}
"""

builder = PromptBuilder(template=prompt_template)
generator = OpenAIChatGenerator(
    api_key=os.environ["N4N_API_KEY"],
    api_base_url="https://api.n4n.ai/v1",
    model="anthropic/claude-3-5-sonnet",
)

pipe = Pipeline()
pipe.add_component("builder", builder)
pipe.add_component("generator", generator)
pipe.connect("builder", "generator")

result = pipe.run(
    data={"builder": {"text": "Haystack is a composable LLM framework. It lets engineers wire generators, retrievers, and evaluators into typed pipelines without custom orchestration code."}}
)
print(result["generator"]["replies"][0])

This is the minimal end-to-end path. In production you would load the template from a file and inject api_base_url from config, but the wiring is identical.

Step 5: Run and verify success

Execute the script:

python summarize.py

Expected output

You should see a single string reply similar to:

Haystack is a composable LLM framework that enables engineers to build typed pipelines connecting generators, retrievers, and evaluators without custom orchestration.

Verification checklist

• The process exits 0.
• result["generator"]["replies"] is a non-empty list of strings.
• result["generator"]["meta"] contains usage with prompt_tokens and completion_tokens greater than zero. The gateway performs per-token usage metering, so those counts reflect what you will be billed.

meta = result["generator"]["meta"]
assert meta["usage"]["prompt_tokens"] > 0
assert meta["usage"]["completion_tokens"] > 0

Common failures and debugging

• 401 Unauthorized: Key missing or malformed. Check echo $N4N_API_KEY.
• 404 Not Found: Usually bad api_base_url (trailing slash) or wrong model string.
• TimeoutError: Increase timeout in generation_kwargs or reduce max_tokens.
• JSON decode error: The gateway returned an error HTML page because the model string included an unsupported character. Stick to provider/model lowercase with hyphens.

Step 6: Streaming and advanced routing

Haystack’s OpenAIChatGenerator supports streaming via streaming_callback. This is useful for chat UIs where time-to-first-token matters.

def on_token(token: str):
    print(token, end="", flush=True)

generator = OpenAIChatGenerator(
    api_key=os.environ["N4N_API_KEY"],
    api_base_url="https://api.n4n.ai/v1",
    model="openai/gpt-4o-mini",
    streaming_callback=on_token,
)

Because the underlying gateway performs automatic fallback when a provider is rate-limited or degraded, a streaming request that hits a transient provider error may seamlessly continue on a secondary provider if you have enabled that behavior in your routing policy. Your callback code does not need to change; the reply tokens simply resume.

If you need to pin a specific provider version or forbid fallback, pass routing hints in extra_body. Haystack forwards unknown kwargs to the request payload:

generator = OpenAIChatGenerator(
    api_key=os.environ["N4N_API_KEY"],
    api_base_url="https://api.n4n.ai/v1",
    model="anthropic/claude-3-5-sonnet",
    extra_body={"routing": {"fallback": False}},
)

Step 7: Production considerations

A haystack n4n.ai generator setup that works locally needs hardening before serving traffic.

Retries. Haystack does not retry by default. Wrap the pipeline call in a tenacity retry if you see intermittent 5xx from the gateway during provider degradation:

from tenacity import retry, stop_after_attempt(3), wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=1, max=10))
def run_pipe(text):
    return pipe.run(data={"builder": {"text": text}})

Concurrency. The generator uses a synchronous requests client. For high throughput, run the pipeline in a thread pool or switch to the async OpenAIChatGenerator (AsyncOpenAIChatGenerator) inside an asyncio event loop.

Model aliasing. Define a small mapping in your config so code references summary_model rather than a raw provider/model string. This lets you shift models without touching pipeline definitions.

Usage accounting. Capture meta["usage"] on every run and ship it to your metrics stack. Per-token metering means a sudden prompt-size increase shows up immediately as cost variance.

Cache control. When you send repeated system prompts, set extra_body={"cache": True} to leverage provider prompt caching where the gateway forwards the hint. This cuts latency and token cost on long system prefixes.

The haystack n4n.ai generator setup is now complete: you have a typed pipeline, verified output, streaming option, and production guards. From here, drop in a InMemoryDocumentStore and a SentenceTransformersDocumentEmbedder to extend the same generator into a RAG pipeline without changing the connection logic.