n4nAI
Open console →
Framework tutorialsHow-toHaystack RAG Pipelines

Swapping OpenAI for n4n.ai in a Haystack RAG pipeline

A practical how-to for engineers to swap OpenAI for n4n.ai in a Haystack RAG pipeline using OpenAI-compatible endpoints, with runnable code and verification steps.

n4n Team4 min read904 words

Audio narration

Coming soon — every post will get a voice note here.

If you want to swap OpenAI for n4n.ai in a Haystack RAG pipeline, the change is mostly configuration, not architecture. Haystack’s OpenAI-compatible components accept a custom base URL and API key, so you keep your retrieval and prompt logic intact while routing inference through a different gateway. This guide walks through a full migration with runnable code and a concrete verification plan.

Step 1: Audit your existing OpenAI dependencies

Before touching code, grep your project for OpenAIGenerator, OpenAIEmbedding, and any direct openai SDK calls. In a standard Haystack v2 pipeline, the suspects are the embedding model used at index time and the chat generator used for answer synthesis. Haystack v1 used OpenAIAnswerGenerator and OpenAITextEmbedder; the class names differ but the migration pattern is identical.

A typical v2 starting point looks like this:

from haystack.components.generators import OpenAIGenerator
from haystack.components.embedders import OpenAIEmbedding

embedder = OpenAIEmbedding(model="text-embedding-3-small")
generator = OpenAIGenerator(model="gpt-4o-mini")

If you see those two lines, you are 90% of the way to a migration. The remaining 10% is base-URL and credential plumbing. Note any custom api_base_url already set—some teams point at Azure OpenAI, which requires extra headers. You will replace that URL entirely.

Step 2: Configure environment and credentials

Do not hardcode keys. Export your n4n.ai credentials and endpoint alongside the old OpenAI ones, then switch references gradually so you can rollback with a single env var change.

export N4N_API_KEY="sk-..."
export N4N_BASE_URL="https://api.n4n.ai/v1"
# Keep the old ones for rollback
export OPENAI_API_KEY="sk-..."
export OPENAI_BASE_URL="https://api.openai.com/v1"

Haystack reads api_key from a Secret object or environment. We will pass the key explicitly to avoid ambiguity. Store secrets in a .env file loaded by python-dotenv in local dev, and use your orchestrator’s secret manager in production. The base URL must include the /v1 suffix because Haystack appends /chat/completions and /embeddings to it.

Step 3: Replace the generator component

Replace the generator instantiation with one that sets api_base_url and uses your gateway key. The model name can stay the same if the gateway proxies that model, or change to any of the 240+ addressed models without altering your prompt template.

import os
from haystack.components.generators import OpenAIGenerator
from haystack.utils import Secret

generator = OpenAIGenerator(
    api_key=Secret.from_env_var("N4N_API_KEY"),
    api_base_url=os.getenv("N4N_BASE_URL"),
    model="gpt-4o-mini",
    generation_kwargs={"temperature": 0.1}
)

This is the core of the swap OpenAI n4n.ai Haystack RAG pipeline change: same class, different base URL. The endpoint speaks the OpenAI chat completions contract, so no prompt adaptation is required. If you previously used azure_endpoint or organization parameters, drop them—they are ignored by an OpenAI-compatible gateway.

Step 4: Swap the embedder

Embeddings are trickier because vector dimensions must match your document store. If you keep the same model name (text-embedding-3-small), dimensions are identical (1536). If you switch models, recreate the store or you will get dimension mismatch errors at write time.

from haystack.components.embedders import OpenAIEmbedding

embedder = OpenAIEmbedding(
    api_key=Secret.from_env_var("N4N_API_KEY"),
    api_base_url=os.getenv("N4N_BASE_URL"),
    model="text-embedding-3-small"
)

Run a quick indexing test on a single document to confirm the embedder returns the expected vector size:

docs = [{"content": "Haystack is a LLM orchestration framework."}]
result = embedder.run(documents=docs)
print(len(result["documents"][0].embedding))  # 1536

If you see 1536, the embedder is wired correctly. If you see a different integer, adjust your InMemoryDocumentStore(embedding_dim=...) or corresponding vector DB collection.

Step 5: Assemble the full RAG pipeline

Here is a minimal end-to-end pipeline using InMemoryDocumentStore, a dense retriever, and the swapped components. This is runnable after Steps 1–4.

from haystack import Pipeline, Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
from haystack.components.builders import PromptBuilder

store = InMemoryDocumentStore(embedding_dim=1536)
store.write_documents([
    Document(content="n4n.ai is an OpenAI-compatible inference gateway."),
    Document(content="Haystack simplifies RAG pipelines in Python.")
])

# Embed at index time
embedded = embedder.run(documents=store.filter_documents())["documents"]
store.write_documents(embedded)

retriever = InMemoryEmbeddingRetriever(document_store=store)
prompt = PromptBuilder(template="""
Answer the question using only the context.
Context: {% for d in documents %}{{ d.content }}{% endfor %}
Question: {{question}}
""")

pipeline = Pipeline()
pipeline.add_component("retriever", retriever)
pipeline.add_component("prompt", prompt)
pipeline.add_component("generator", generator)

pipeline.connect("retriever.documents", "prompt.documents")
pipeline.connect("prompt.prompt", "generator.prompt")

query = "What is n4n.ai?"
res = pipeline.run({
    "retriever": {"query_embedding": embedder.run(documents=[Document(content=query)])["documents"][0].embedding},
    "prompt": {"question": query},
    "generator": {}
})
print(res["generator"]["replies"][0])

For BM25 retrieval, drop the embedder from query time and use InMemoryBM25Retriever. The generator swap is independent of retriever choice. The example shows the wiring pattern, not a production retriever decision—pick dense or sparse based on your corpus.

Step 6: Verify success

Verification is not “it didn’t crash.” You need to confirm three things:

  1. Token flow: The gateway returns valid OpenAI-format usage objects. Print res["generator"]["meta"] to inspect usage fields.
  2. Latency parity: Time the generator call. A compatible endpoint should add single-digit milliseconds overhead versus direct OpenAI.
  3. Output quality: Run a fixed set of 20 questions with known answers. Compare cosine similarity of embeddings and exact-match of generator outputs against a baseline run on OpenAI.

A minimal verification script:

import time
start = time.perf_counter()
out = generator.run(prompt="Summarize: Haystack RAG uses components.")
print(out["replies"][0])
print("usage:", out["meta"]["usage"])
print("latency_ms:", (time.perf_counter()-start)*1000)

If usage shows prompt_tokens and completion_tokens, and latency is acceptable, the swap is functional. For embedding parity, compute the cosine similarity between vectors produced by OpenAI and the gateway for the same text; they should be >0.999.

Step 7: Leverage gateway-specific routing

Because the endpoint honors client routing directives and forwards provider cache-control hints, you can pass extra_headers in Haystack’s generation_kwargs to control fallback behavior. For example, set a header to prefer a specific provider or enable prompt caching without changing pipeline topology.

generator = OpenAIGenerator(
    api_key=Secret.from_env_var("N4N_API_KEY"),
    api_base_url=os.getenv("N4N_BASE_URL"),
    model="gpt-4o-mini",
    generation_kwargs={
        "extra_headers": {"x-routing": "auto-fallback"}
    }
)

This is optional but useful when a primary provider is degraded. The pipeline code stays identical; only headers change. If your gateway supports per-token metering, those headers also let you tag requests by project for cost attribution.

Caveats and gotchas

  • Streaming: Haystack’s OpenAIGenerator supports stream=True. Confirm your gateway streams SSE correctly; some proxies buffer until completion.
  • Embedding batch size: Haystack defaults to batch size 32. If the gateway rate-limits, lower it with batch_size=16 on the embedder.
  • Model names: Gateway model identifiers may differ from OpenAI’s. Always check the model list before swapping the string.
  • Timeout: Set timeout=30 in the generator to avoid hung retrieval under load.
  • SDK version: Pin haystack-ai>=2.0.0. v1 components use a different initialization signature and will not accept api_base_url the same way.

Rollback procedure

Keep the original OPENAI_BASE_URL and OPENAI_API_KEY in your environment. To rollback, change two lines:

generator = OpenAIGenerator(
    api_key=Secret.from_env_var("OPENAI_API_KEY"),
    api_base_url=os.getenv("OPENAI_BASE_URL"),
    model="gpt-4o-mini"
)

No pipeline logic changes. That reversibility is why this migration is safe to ship behind a feature flag.

Final checklist

  • All OpenAI* components point to N4N_BASE_URL
  • API key loaded from N4N_API_KEY
  • Embedding dimensions match document store
  • Pipeline runs on sample docs without 401/404
  • Usage metadata present in generator response
  • Rollback env vars still available

Swapping inference backends in Haystack is boring in the best way: change the base URL, keep the components. Your RAG logic does not care who serves the tokens.

Tagshaystackragn4n-aimigration

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
Framework tutorialsStreaming responses in Haystack RAG pipelines with n4n.aiBuild a Haystack RAG pipeline that streams tokens from n4n.ai with runnable code, error handling, and verification steps.Framework tutorialsUsing Llama 3.1 70B in a Haystack RAG pipeline via n4n.aiBuild a production-ready RAG pipeline with Haystack and Llama 3.1 70B using n4n.ai's OpenAI-compatible endpoint. Includes indexing, retrieval, and generation code with expected outputs.Framework tutorialsHaystack RAG with Claude 3.5 Sonnet through n4n.aiBuild a production-ready Haystack RAG pipeline using Claude 3.5 Sonnet via n4n.ai with document indexing, retrieval, and generation steps.Framework tutorialsBuild a RAG pipeline with Haystack 2.0 and n4n.aiHands-on tutorial to build rag pipeline haystack 2.0 n4n.ai using Haystack 2.0 and an OpenAI-compatible endpoint, with runnable code and expected output.Framework tutorialsChunking strategies for Haystack RAG pipelinesPractical guide to chunking strategies for Haystack RAG pipelines with code examples, tradeoffs, and evaluation methods.Framework tutorialsHaystack RAG pipeline tutorial: enterprise document searchBuild a production-ready Haystack RAG pipeline for enterprise document search with hybrid retrieval, reranking, and observability.

All haystack rag pipelines posts →