Haystack n4n.ai setup: choosing embedding models

You’re building a RAG system with Haystack and want to route embedding calls through n4n.ai instead of calling providers directly. This guide walks through the haystack n4n.ai embedding model setup from dependency installation to a verified end-to-end pipeline. We’ll cover model selection trade-offs, connection configuration, and a minimal working example you can run today.

Step 1: install the required packages

Haystack’s OpenAI-compatible integrations work with n4n.ai because the gateway exposes an OpenAI-compatible /v1/embeddings endpoint. You need the core Haystack package plus the OpenAI document embedder and text embedder components.

pip install haystack-ai "haystack-ai[openai]" python-dotenv

If you’re on an older Haystack version (1.x), the package name is farm-haystack and the extras syntax differs. This guide assumes Haystack 2.x.

Verify the install:

python -c "from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder; print('OK')"

Step 2: choose an embedding model

n4n.ai routes to 240+ models across providers. For embeddings, your choice depends on three factors: dimension size (affects vector store schema and query latency), multilingual support, and cost per million tokens.

For most English-first workloads, text-embedding-3-small hits the sweet spot. If you need strong multilingual retrieval without re-indexing, bge-m3 or nomic-embed-text-v1.5 are solid picks. Note that changing models later means re-embedding your entire corpus — choose deliberately.

Step 3: configure the n4n.ai connection

n4n.ai uses an OpenAI-compatible base URL and your gateway API key. Store credentials in environment variables — never hardcode them.

Create a .env file:

N4N_API_KEY=sk-ng-xxxxxxxxxxxxxxxx
N4N_BASE_URL=https://api.n4n.ai/v1
EMBEDDING_MODEL=text-embedding-3-small

Load these in your application:

import os
from dotenv import load_dotenv

load_dotenv()

N4N_API_KEY = os.getenv("N4N_API_KEY")
N4N_BASE_URL = os.getenv("N4N_BASE_URL")
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "text-embedding-3-small")

if not N4N_API_KEY:
    raise RuntimeError("N4N_API_KEY not set")

The gateway honors client routing directives — you can pass model at request time to override the default, and it forwards provider cache-control hints so repeated embeddings of identical text can hit cached responses.

Step 4: wire the document embedder for indexing

Haystack’s OpenAIDocumentEmbedder writes embeddings into each Document’s embedding field. Point it at the n4n.ai base URL and pass your API key.

from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack import Document

doc_embedder = OpenAIDocumentEmbedder(
    api_key=N4N_API_KEY,
    api_base_url=N4N_BASE_URL,
    model=EMBEDDING_MODEL,
    dimensions=1536,  # match your model; 3-small = 1536, 3-large = 3072
    batch_size=32,
    progress_bar=True,
)

# Example documents
docs = [
    Document(content="Haystack 2.x uses a component-based pipeline architecture."),
    Document(content="n4n.ai provides an OpenAI-compatible embeddings endpoint."),
    Document(content="Embedding dimensions must match your vector store schema."),
]

# Run embedding
result = doc_embedder.run(documents=docs)
embedded_docs = result["documents"]

for d in embedded_docs:
    print(f"Doc id: {d.id}, embedding dim: {len(d.embedding)}")

Verify success: each document should now have a non-empty embedding list of length matching your model’s dimensions (1536 for text-embedding-3-small). The progress_bar=True flag shows throughput — expect roughly 100-300 docs/sec depending on network latency.

Step 5: wire the text embedder for queries

At query time, use OpenAITextEmbedder to embed the user’s question into the same vector space.

from haystack.components.embedders import OpenAITextEmbedder

text_embedder = OpenAITextEmbedder(
    api_key=N4N_API_KEY,
    api_base_url=N4N_BASE_URL,
    model=EMBEDDING_MODEL,
    dimensions=1536,
)

query = "How does Haystack connect to n4n.ai for embeddings?"
result = text_embedder.run(text=query)
query_embedding = result["embedding"]

print(f"Query embedding dim: {len(query_embedding)}")
print(f"First 5 values: {query_embedding[:5]}")

Verify success: the returned embedding length matches your document embeddings. If dimensions differ, you’ve mismatched the dimensions parameter or the model — fix before indexing.

Step 6: build a minimal indexing pipeline

Haystack pipelines chain components. Here’s a minimal indexing pipeline that embeds and writes to an in-memory document store (swap for Weaviate, Qdrant, Pinecone, etc. in production).

from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.writers import DocumentWriter

document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")

indexing_pipeline = Pipeline()
indexing_pipeline.add_component("embedder", doc_embedder)
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
indexing_pipeline.connect("embedder.documents", "writer.documents")

# Run on a larger sample
sample_docs = [
    Document(content=f"Document {i}: " + " ".join(["token"] * 100))
    for i in range(200)
]

indexing_pipeline.run({"embedder": {"documents": sample_docs}})

print(f"Indexed {document_store.count_documents()} documents")

Verify success: document_store.count_documents() returns 200. Spot-check a stored document:

stored = document_store.filter_documents()[0]
print(f"Has embedding: {stored.embedding is not None}")
print(f"Embedding length: {len(stored.embedding)}")

Step 7: build a minimal query pipeline

The query pipeline embeds the question, retrieves top-k documents, and (optionally) passes them to a generator. Here’s the retrieval-only version:

from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever

retriever = InMemoryEmbeddingRetriever(document_store=document_store, top_k=5)

query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", text_embedder)
query_pipeline.add_component("retriever", retriever)
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")

question = "What embedding model should I use with Haystack?"
result = query_pipeline.run({"text_embedder": {"text": question}})

for doc in result["retriever"]["documents"]:
    print(f"Score: {doc.score:.4f} | Content: {doc.content[:80]}...")

Verify success: you get back ranked documents with cosine similarity scores between 0 and 1. Scores above 0.7 typically indicate strong semantic match for this model family.

Step 8: handle rate limits and retries

n4n.ai automatically falls back across providers when one is rate-limited or degraded, but transient errors can still surface. Configure retries at the component level:

from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack.utils import Secret

doc_embedder_resilient = OpenAIDocumentEmbedder(
    api_key=Secret.from_env_var("N4N_API_KEY"),
    api_base_url=N4N_BASE_URL,
    model=EMBEDDING_MODEL,
    dimensions=1536,
    batch_size=32,
    max_retries=3,
    timeout=60.0,
)

The Secret.from_env_var pattern keeps keys out of serialized pipelines. Set max_retries and timeout based on your SLA — embedding calls are idempotent, so retries are safe.

Step 9: monitor usage and costs

n4n.ai meters per-token usage. If you need observability in your application, wrap the embedder to log token counts:

import logging
from functools import wraps

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def log_embedding_usage(component):
    original_run = component.run
    
    @wraps(original_run)
    def wrapper(*args, **kwargs):
        result = original_run(*args, **kwargs)
        # Haystack embedders return meta with usage when available
        meta = result.get("meta", {})
        if meta:
            logger.info(f"Embedding usage: {meta}")
        return result
    
    component.run = wrapper
    return component

doc_embedder = log_embedding_usage(doc_embedder)
text_embedder = log_embedding_usage(text_embedder)

The gateway returns provider-standard usage fields (prompt_tokens, total_tokens) in the response metadata. Aggregate these to track spend per pipeline run.

Step 10: swap models without re-indexing? you can’t

A common misconception: you can change the embedding model at query time only. This breaks retrieval because the query vector lives in a different space than the indexed document vectors. The dimensions may differ, and even with matching dimensions, the semantic geometry shifts.

If you must change models:

1. Update EMBEDDING_MODEL and dimensions in both embedders
2. Re-run the full indexing pipeline on your corpus
3. Verify query pipeline returns sensible results

Plan for this migration. It’s a full re-index, not a config toggle.

Common pitfalls

Mismatched dimensions: Setting dimensions=1536 for text-embedding-3-large (3072 dims) truncates vectors silently in some providers. Always match the model’s native dimension or omit the parameter to use the default.

Wrong base URL: The n4n.ai embeddings endpoint is https://api.n4n.ai/v1 — include the /v1 suffix. The OpenAI client libraries expect it.

Batch size too large: Haystack’s default batch_size=32 works well. Pushing to 128+ can trigger gateway timeouts on large documents. Tune based on your average document length.

Forgetting to embed queries: The retriever expects a vector. Passing raw text to InMemoryEmbeddingRetriever fails with a cryptic error. Always run text_embedder first in the pipeline.

Next steps

• Swap InMemoryDocumentStore for a production vector store (Qdrant, Weaviate, Pinecone) — the embedder components stay identical
• Add a generator component (OpenAIGenerator pointed at n4n.ai) for full RAG
• Implement hybrid search with bge-m3 sparse vectors if your workload benefits from keyword matching
• Set up CI to run the verification steps above on every deploy

The haystack n4n.ai embedding model setup is intentionally boring: standard OpenAI-compatible components, environment-driven config, and predictable failure modes. That’s the point — you get provider diversity and fallback without rewriting your pipeline logic.