Haystack’s OpenAI components talk to any endpoint that mimics the OpenAI REST contract. This guide shows how to connect haystack n4n.ai openai-compatible api in Haystack 2.x, then run a retrieval-augmented generation (RAG) pipeline against 240+ models without vendor lock-in. You’ll point the stock OpenAIChatGenerator and OpenAITextEmbedder at a single base URL and get automatic provider fallback for free.
Step 1: Install Haystack and prepare the environment
Use Python 3.10 or newer. Haystack 2.x dropped the old farm-haystack namespace and ships a cleaner component API. Create a clean virtual environment and install the framework:
python -m venv .venv
source .venv/bin/activate
pip install haystack-aiThe core package includes OpenAIChatGenerator, OpenAITextEmbedder, OpenAIDocumentEmbedder, and InMemoryDocumentStore. You do not need the haystack-experimental or provider-specific integrations because the gateway speaks the OpenAI protocol natively.
Verify the install:
python -c "import haystack; print(haystack.__version__)"Set your gateway credential as an environment variable so it never lands in source control:
export N4N_API_KEY="sk-..." # your n4n.ai gateway keyIf you run inside a container or CI, inject the secret via your orchestrator’s secret store. The key is a Bearer token sent in the Authorization header, exactly as OpenAI expects.
Step 2: Point Haystack at the gateway base URL
To connect haystack n4n.ai openai-compatible api, you only need to override api_base_url on the Haystack OpenAI classes. The gateway exposes one OpenAI-compatible endpoint that fronts 240+ models, so you do not manage per-provider keys or base URLs.
from haystack.utils import Secret
from haystack.components.generators import OpenAIChatGenerator
from haystack.components.embedders import OpenAITextEmbedder
API_KEY = Secret.from_env_var("N4N_API_KEY")
BASE_URL = "https://api.n4n.ai/v1"
chat_generator = OpenAIChatGenerator(
api_key=API_KEY,
api_base_url=BASE_URL,
model="openai/gpt-4o-mini", # any model ID from the gateway catalog
generation_kwargs={"temperature": 0.1, "max_tokens": 512},
)
text_embedder = OpenAITextEmbedder(
api_key=API_KEY,
api_base_url=BASE_URL,
model="openai/text-embedding-3-small",
)Model strings follow the gateway’s catalog format: {provider}/{model-name}. If you omit the provider prefix, the gateway applies its default routing directive. Because the endpoint is OpenAI-compatible, Haystack’s client signs requests identically to how it would for OpenAI’s API—no custom transport code.
For local experimentation you can bypass the env var:
API_KEY = Secret.from_token("sk-...") # not recommended for productionBut keep from_env_var in anything you ship.
Step 3: Index documents for retrieval
A RAG pipeline needs a document store and embeddings. We’ll use the in-memory store for simplicity; swap in Elasticsearch or PGVector for production scale.
from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack.components.writers import DocumentWriter
doc_store = InMemoryDocumentStore()
doc_embedder = OpenAIDocumentEmbedder(
api_key=API_KEY,
api_base_url=BASE_URL,
model="openai/text-embedding-3-small",
)
docs = [
Document(content="n4n.ai routes LLM traffic across providers with automatic fallback."),
Document(content="Haystack is a Python framework for building RAG and agent pipelines."),
Document(content="OpenAI-compatible APIs share the /v1/chat/completions request shape."),
]
# Embed and write in one pass
embedded = doc_embedder.run(docs)["documents"]
DocumentWriter(doc_store).run(embedded)The embedder calls the same base URL you configured for the generator. You pay per token via the gateway’s metering, so embedding cost is visible alongside generation cost in one bill. If your source corpus is large, batch documents and wrap the loop with a simple rate-limit backoff—the gateway returns standard 429s.
Step 4: Assemble the RAG pipeline
Wire a retriever, a prompt builder, and the chat generator. Haystack’s Pipeline object declares connections by output/input variable names.
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder
from haystack import Pipeline
retriever = InMemoryEmbeddingRetriever(doc_store, top_k=2)
prompt_builder = PromptBuilder(
template="""
Answer the question using only the context.
Context:
{% for doc in documents %}
{{ doc.content }}
{% endfor %}
Question: {{ question }}
Answer:
"""
)
rag = Pipeline()
rag.add_component("text_embedder", text_embedder)
rag.add_component("retriever", retriever)
rag.add_component("prompt_builder", prompt_builder)
rag.add_component("generator", chat_generator)
rag.connect("text_embedder.embedding", "retriever.query_embedding")
rag.connect("retriever.documents", "prompt_builder.documents")
rag.connect("prompt_builder.prompt", "generator.prompt")Run it with a question:
question = "How does Haystack connect to an OpenAI-compatible gateway?"
result = rag.run({
"text_embedder": {"text": question},
"prompt_builder": {"question": question},
})
print(result["generator"]["replies"][0])The pipeline embeds the query, retrieves the two closest chunks, builds a strict prompt, and streams the answer from the gateway. Because you used the gateway to connect haystack n4n.ai openai-compatible api, the same model string can be changed to anthropic/claude-3.5-sonnet or meta-llama/llama-3.1-70b-instruct without code changes beyond the constructor.
Step 5: Verify success and observe metering
Success means three things: the pipeline returns a coherent answer, the retrieved documents are relevant, and the gateway logs show token usage.
Run the script. You should see a reply that mentions Haystack and the gateway. If you get a 401, check N4N_API_KEY. A 404 on the model means the catalog ID is wrong—list available models from the gateway’s /v1/models endpoint.
To confirm token metering, inspect the gateway’s usage dashboard or call its usage API with the same key. Each chat completion returns usage in the response body; Haystack exposes it via result["generator"]["meta"] if you enable debug on the generator:
chat_generator = OpenAIChatGenerator(
api_key=API_KEY,
api_base_url=BASE_URL,
model="openai/gpt-4o-mini",
debug=True,
)The n4n.ai gateway performs automatic fallback when a provider is rate-limited or degraded, so a transient provider error should not surface as a pipeline exception—the same request silently routes to a healthy provider. That resilience is the main reason to put a gateway in front of Haystack rather than calling providers directly.
Add a quick assertion to lock behavior in tests:
assert result["generator"]["replies"], "Generator returned empty reply"
assert len(result["retriever"]["documents"]) > 0, "Retriever found nothing"Step 6: Swap models without refactoring
Change the model argument on both embedder and generator to move across providers. For embeddings, dimension changes require rebuilding the document store. For chat, you can A/B test by instantiating two generators and selecting at runtime:
generator_a = OpenAIChatGenerator(api_key=API_KEY, api_base_url=BASE_URL, model="openai/gpt-4o")
generator_b = OpenAIChatGenerator(api_key=API_KEY, api_base_url=BASE_URL, model="anthropic/claude-3.5-sonnet")
chosen = generator_b if use_claude else generator_aNo HTTP client changes, no new SDK. That is the payoff of treating the gateway as a drop-in OpenAI substitute.
Production notes
Timeouts and retries. Haystack’s OpenAI client inherits httpx defaults. Set timeout on the generator if your gateway tail latency is high. Wrap pipeline runs in a retry decorator for idempotent queries.
Cache control. The gateway forwards provider cache-control hints. If you repeatedly send the same long system prompt, annotate it according to the provider’s cache schema; the gateway passes the hint through, reducing cost on supported models.
Streaming. For chat UIs, pass streaming_callback to OpenAIChatGenerator. The gateway streams SSE chunks identical to OpenAI’s format, so your existing frontend works unchanged.
Embedding dimension mismatch. If you switch embedding models, recreate the document store or add a dimension check. InMemoryDocumentStore does not auto-resize vectors.
When you connect haystack n4n.ai openai-compatible api, you collapse multi-provider LLM infrastructure into one base URL, one key, and one set of Haystack components. That’s the entire integration.