Switch LlamaIndex from OpenAI to n4n.ai

This llamaindex switch openai to n4n.ai tutorial shows how to repoint an existing LlamaIndex project from OpenAI’s hosted API to an OpenAI-compatible inference endpoint. The migration is mechanical: LlamaIndex’s OpenAI wrapper accepts a custom base_url and api_key, so your indexing and query logic stays untouched.

Step 1: Audit Your OpenAI Coupling

Before changing anything, find every place your code constructs an OpenAI client or relies on OpenAI-only behavior. LlamaIndex splits LLM and embedding concerns, so the two most common touchpoints are llama_index.llms.openai.OpenAI and llama_index.embeddings.openai.OpenAIEmbedding.

grep -rn "llms.openai\|embeddings.openai" .
grep -rn "Settings.llm\|Settings.embed_model" .

If you use OpenAIAgent, GPTVectorStoreIndex (legacy), or fine-tuned model names, note them. The goal is to isolate constructor calls, not rewrite RAG pipelines.

Step 2: Pin LlamaIndex and Provider Packages

LlamaIndex moved LLM and embedding classes into optional provider packages. Install or upgrade them explicitly to avoid version drift.

pip install -U llama-index-core llama-index-llms-openai llama-index-embeddings-openai

Check that your llama-index-core is at least 0.10.0 so Settings is the global config surface. Older 0.9.x code using ServiceContext still works but requires wrapping in ServiceContext.from_defaults(llm=..., embed_model=...).

Step 3: Externalize Endpoint Configuration

Do not hardcode credentials or URLs. Read them from environment so you can flip providers without code changes. For example, n4n.ai exposes an OpenAI-compatible base URL at https://api.n4n.ai/v1; set OPENAI_BASE_URL to that value (or your gateway’s equivalent).

export OPENAI_API_KEY="sk-your-gateway-key"
export OPENAI_BASE_URL="https://api.n4n.ai/v1"

If you run locally, put these in .env and load with python-dotenv. The key here is whatever the gateway issues; it is not your OpenAI key unless the gateway proxies OpenAI directly.

Step 4: Replace the LLM Constructor

Instantiate OpenAI with the same model name your gateway supports. Model aliases differ across backends, so verify the exact string from the gateway’s model list.

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-4o-mini",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_BASE_URL"],
    temperature=0.1,
    max_tokens=1024,
)

The OpenAI class sends requests to {base_url}/chat/completions. If your gateway uses a different path suffix, this wrapper will not work without a subclass—but OpenAI-compatible means the path matches.

Streaming Note

Set is_chat_model=True (default) and use llm.stream_complete() if you need token streaming. The base_url propagates to streaming sessions automatically.

Step 5: Swap the Embedding Model

Embeddings are the second hard dependency. Use OpenAIEmbedding with the same base_url. Dimension mismatches break vector stores, so keep the dimension consistent with your existing index or rebuild it.

from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_BASE_URL"],
)

If your prior index was built with OpenAI’s text-embedding-ada-002 (1536 dims) and you switch to a 512-dim model, queries will fail at cosine similarity. Either keep the same dimension or call index.refresh() after re-embedding.

Step 6: Wire Into Global Settings

LlamaIndex reads from Settings when you do not pass llm/embed_model explicitly. Set them once at startup.

from llama_index.core import Settings

Settings.llm = llm
Settings.embed_model = embed_model

If you use a VectorStoreIndex.from_documents(...) call without passing models, it now uses the gateway. For multi-tenant apps, prefer passing llm and embed_model per index to avoid global state races.

Step 7: Run a Smoke Test Query

Create a tiny corpus and execute a query. This validates auth, routing, and response parsing.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("sample_data").load_data()
index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the first document in one sentence.")

print(str(response))
print("Tokens:", response.metadata.get("token_usage"))

Verify Success

A successful migration shows:

1. No ConnectionError or 401 from the gateway.
2. str(response) is non-empty and coherent.
3. token_usage is populated (confirms the gateway returns OpenAI-style usage JSON).

If you see 404, the model name is not registered at the gateway. If you see 422, the gateway rejected a parameter LlamaIndex sent—inspect the request body and adjust the constructor kwargs.

Step 8: Handle Cache-Control and Fallback

OpenAI-compatible gateways often forward cache-control hints or perform automatic provider fallback. LlamaIndex does not need code changes for this; the base_url receives standard headers. If you want to force a specific provider behind the gateway, pass extra_headers={"x-router-directive": "openai"} to the OpenAI constructor—assuming the gateway honors client routing directives.

llm = OpenAI(
    model="gpt-4o-mini",
    api_key=os.environ["OPENAI_API_KEY"],
    base_url=os.environ["OPENAI_BASE_URL"],
    extra_headers={"x-router-directive": "openai"},
)

Do not assume embeddings are cached the same way across providers. Re-check latency on the first embed call after a cold start.

Step 9: Update Tests and CI

Your unit tests likely mocked openai.ChatCompletion. Now they should mock the base_url host. Use respx or pytest-httpx to intercept {OPENAI_BASE_URL}/chat/completions and return a fixture response shaped like OpenAI’s.

import respx
import httpx

@respx.mock
def test_query():
    respx.post("https://api.n4n.ai/v1/chat/completions").mock(
        return_value=httpx.Response(200, json={
            "choices": [{"message": {"content": "ok"}}],
            "usage": {"total_tokens": 5}
        })
    )
    # call your query engine

This catches regression if someone reverts the base_url change.

Step 10: Roll Back Safely

Keep the old OpenAI path behind a flag during transition.

import os
from llama_index.llms.openai import OpenAI

if os.environ.get("USE_GATEWAY") == "1":
    llm = OpenAI(model="gpt-4o-mini", api_key=os.environ["GW_KEY"], base_url=os.environ["OPENAI_BASE_URL"])
else:
    llm = OpenAI(model="gpt-4o-mini", api_key=os.environ["OPENAI_KEY"])

Run a shadow evaluation: send 100 queries to both, compare answer similarity. Only then flip USE_GATEWAY=1 in production.

Common Pitfalls

• Model name drift: gpt-3.5-turbo on OpenAI may map to a different snapshot on a gateway. Pin the full version string if the gateway exposes it.
• Embedding dim mismatch: Rebuild indexes when changing embedding models.
• Timeout defaults: LlamaIndex’s OpenAI uses a 60s timeout. Gateways with slower cold starts need timeout=120.
• API key scoping: Gateway keys often have per-token metering; a leaked key costs money, not just data.

Following these steps gives you a LlamaIndex app that talks to an OpenAI-compatible endpoint with zero changes to retrieval or orchestration code. The only durable edits are constructor arguments and environment variables.