LlamaIndex base URL and auth setup for n4n.ai

LlamaIndex assumes an OpenAI-compatible endpoint by default, but pointing it at a gateway like n4n.ai requires explicit base URL configuration and a few authentication adjustments. This guide walks through the complete setup — from the minimal client instantiation to production-grade patterns for retries, timeouts, and observability — so you can swap the backend without rewriting your index or query logic.

Minimal working configuration

The fastest way to verify connectivity is to instantiate OpenAI (or OpenAILike) with the gateway’s base URL and your n4n.ai API key. The key difference from a standard OpenAI call is the base_url parameter and ensuring the client sends the Authorization header in the format the gateway expects.

from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

llm = OpenAI(
    model="meta-llama/llama-3.1-70b-instruct",
    api_key="n4n-your-api-key-here",
    base_url="https://api.n4n.ai/v1",
    temperature=0.2,
    max_tokens=4096,
)

embed_model = OpenAIEmbedding(
    model="text-embedding-3-large",
    api_key="n4n-your-api-key-here",
    base_url="https://api.n4n.ai/v1",
)

# Quick sanity check
resp = llm.complete("Return the string 'ok' and nothing else.")
print(resp.text.strip())  # should print: ok

If that prints ok, the plumbing works. The rest of this guide hardens that minimal snippet for real workloads.

Why base URL and auth need explicit handling

LlamaIndex’s OpenAI and OpenAIEmbedding classes inherit from openai.OpenAI under the hood. When you omit base_url, the SDK defaults to https://api.openai.com/v1. Most gateways — including n4n.ai — expose an OpenAI-compatible surface at a different host, so you must override the base URL on both the LLM and embedding clients.

Authentication is the second gotcha. OpenAI expects Bearer sk-.... n4n.ai accepts the same header format but issues keys prefixed with n4n-. The SDK does not validate the prefix; it simply forwards the header. Treat the key as an opaque string and store it in your secret manager, not in source control.

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="meta-llama/llama-3.1-70b-instruct",
    api_key=os.environ["N4N_API_KEY"],  # injected at deploy time
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
)

Centralizing configuration with Settings

LlamaIndex’s global Settings object lets you define the LLM and embedding model once and have every index, query engine, and agent inherit them. This is the recommended pattern for applications larger than a single script.

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
import os

Settings.llm = OpenAI(
    model=os.environ.get("N4N_CHAT_MODEL", "meta-llama/llama-3.1-70b-instruct"),
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    temperature=0.1,
    max_tokens=8192,
    timeout=60.0,
    max_retries=3,
)

Settings.embed_model = OpenAIEmbedding(
    model=os.environ.get("N4N_EMBED_MODEL", "text-embedding-3-large"),
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    embed_batch_size=100,
    timeout=60.0,
    max_retries=3,
)

# Now any downstream component picks up the gateway automatically
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)  # uses Settings.embed_model
query_engine = index.as_query_engine()              # uses Settings.llm
response = query_engine.query("Summarize the key findings.")
print(response)

Handling provider fallback and routing hints

One reason to use a gateway is automatic fallback when a downstream provider is rate-limited or degraded. n4n.ai honors client-supplied routing directives via extra headers. LlamaIndex lets you inject those through the default_headers parameter on the client.

from llama_index.llms.openai import OpenAI
import os

llm = OpenAI(
    model="meta-llama/llama-3.1-70b-instruct",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    default_headers={
        "x-n4n-prefer": "latency",        # or "cost", "quality"
        "x-n4n-fallback": "true",         # enable automatic fallback
        "x-n4n-max-provider-latency": "5000",  # ms before fallback triggers
    },
    timeout=30.0,
    max_retries=2,
)

The gateway also returns provider-level cache-control hints in response headers (x-n4n-provider, x-n4n-cached, x-n4n-latency-ms). If you need those for observability, access the raw response via the raw attribute on the completion object.

resp = llm.complete("What is the capital of France?")
print(resp.raw.headers.get("x-n4n-provider"))      # e.g., "together", "fireworks"
print(resp.raw.headers.get("x-n4n-cached"))        # "true" or "false"
print(resp.raw.headers.get("x-n4n-latency-ms"))    # e.g., "342"

Streaming responses

Streaming works the same way as with OpenAI — call stream_complete or astream_complete. The gateway forwards chunks as they arrive from the selected provider. Set a reasonable timeout on the client to avoid hanging on stalled streams.

from llama_index.llms.openai import OpenAI
import os

llm = OpenAI(
    model="meta-llama/llama-3.1-70b-instruct",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    streaming=True,
    timeout=120.0,
)

for chunk in llm.stream_complete("Write a 200-word essay on distributed tracing."):
    print(chunk.delta, end="", flush=True)
print()

Async streaming is identical with astream_complete and async for.

Embedding batch sizing and throughput

OpenAIEmbedding defaults to embed_batch_size=10. For large document ingestion, increase this to reduce round trips — but respect the gateway’s per-request token limit. A batch size of 50–100 is usually safe for text-embedding-3-large at 8k context.

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-large",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    embed_batch_size=100,
    timeout=120.0,
    max_retries=3,
)

If you hit 413 Payload Too Large, drop the batch size or switch to a model with a larger context window.

Common pitfalls

Forgetting base_url on the embedding client

The LLM client works but embeddings still hit api.openai.com. Always set base_url on both Settings.llm and Settings.embed_model.

Hardcoding model names that don’t exist on the gateway

n4n.ai exposes 240+ models under their provider-native IDs (e.g., meta-llama/llama-3.1-70b-instruct, mistralai/mixtral-8x7b-instruct). OpenAI model names like gpt-4o will 404 unless the gateway has an explicit alias. Use the provider-native ID or check the gateway’s model catalog.

Ignoring timeout and retry configuration

Default SDK timeouts are often 600 seconds. For production, set explicit timeouts (30–120s) and retries (2–3) on both clients. The gateway’s own fallback logic adds latency; your client timeout should exceed the gateway’s fallback threshold.

# Bad: relies on defaults
llm = OpenAI(model="...", api_key=key, base_url=url)

# Good: explicit boundaries
llm = OpenAI(
    model="...",
    api_key=key,
    base_url=url,
    timeout=60.0,
    max_retries=3,
)

Leaking API keys in logs

LlamaIndex’s default logging can dump request/response bodies. Disable debug logging in production or sanitize the Authorization header.

import logging

# Reduce verbosity
logging.getLogger("llama_index").setLevel(logging.WARNING)
logging.getLogger("openai").setLevel(logging.WARNING)
logging.getLogger("httpx").setLevel(logging.WARNING)

Production checklist

Before deploying a workload that routes through the gateway, verify each item:

Advanced: per-request model override

Sometimes you need a different model for a specific query (e.g., a smaller model for classification, a larger one for synthesis). Override at call time without mutating global settings.

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

# Global default
Settings.llm = OpenAI(
    model="meta-llama/llama-3.1-70b-instruct",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
)

# One-off with a different model
classifier = OpenAI(
    model="meta-llama/llama-3.1-8b-instruct",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    temperature=0.0,
    max_tokens=10,
)

label = classifier.complete("Classify as SPAM or HAM: 'Buy now!!!'")
print(label.text.strip())  # SPAM

Async patterns for high-throughput services

If you’re building an API layer, use the async clients throughout. LlamaIndex’s async methods (aquery, aretrieve, astream_chat) propagate asyncio correctly when the underlying LLM and embedding clients are async-compatible.

from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import Settings, VectorStoreIndex
import os

Settings.llm = OpenAI(
    model="meta-llama/llama-3.1-70b-instruct",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    timeout=60.0,
    max_retries=3,
)

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-large",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ.get("N4N_BASE_URL", "https://api.n4n.ai/v1"),
    embed_batch_size=100,
    timeout=60.0,
    max_retries=3,
)

# In your FastAPI/Starlette handler
async def handle_query(question: str) -> str:
    index = VectorStoreIndex.from_documents(await load_docs_async())
    query_engine = index.as_query_engine(streaming=True)
    response = await query_engine.aquery(question)
    return response.response

The gateway’s fallback and routing work identically for sync and async paths.

Debugging connectivity issues

When something fails, start with a raw HTTP request to isolate whether the problem is LlamaIndex, the gateway, or the network.

curl -X POST https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer n4n-your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/llama-3.1-70b-instruct", "messages": [{"role": "user", "content": "ok"}], "max_tokens": 5}'

If that returns a completion, the gateway and key are fine — check your LlamaIndex client configuration. If it returns 401/403, rotate the key. If it times out, check egress firewall rules and DNS.

For LlamaIndex-specific issues, enable debug logging temporarily:

import logging
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("httpx").setLevel(logging.DEBUG)

Look for the base_url in the request log lines — it should show https://api.n4n.ai/v1/chat/completions, not the OpenAI hostname.

Migration from direct provider SDKs

If you’re moving from calling Together, Fireworks, or Anyscale directly, the changes are minimal:

1. Replace the provider’s base_url with the gateway’s.
2. Replace the provider-specific API key with your n4n.ai key.
3. Update model IDs to the gateway’s unified catalog format (provider/model).
4. Remove any provider-specific retry/fallback logic — the gateway handles it.
5. Add routing headers if you want to express preferences.

The rest of your LlamaIndex code (indices, retrievers, query engines, agents) remains untouched.

Summary

Pointing LlamaIndex at n4n.ai comes down to three things: set base_url on both the LLM and embedding clients, pass the gateway API key via api_key, and configure timeouts, retries, and routing headers for production reliability. Centralize this in Settings so every component inherits the gateway automatically. Test with a raw curl first, then validate streaming, embeddings, and fallback behavior under load. Once the plumbing is verified, your application code stays pure LlamaIndex — the gateway becomes an implementation detail.