Getting started with LlamaIndex and n4n.ai

This llamaindex n4n.ai setup tutorial shows how to connect LlamaIndex to the n4n.ai OpenAI-compatible inference gateway so you can swap between 240+ models without changing client code. We’ll build a small retrieval pipeline from scratch, using n4n.ai for both completions and embeddings.

Prerequisites

Before you start, confirm you have the following:

Python 3.10 or newer
pip and venv available on your PATH
An n4n.ai API key (sign up, then export it as N4N_API_KEY)
Basic familiarity with Python virtual environments and environment variables

You do not need a local vector database or GPU. The index will live in memory.

Step 1: Install dependencies

Create an isolated environment and install the LlamaIndex core plus the OpenAI-compatible LLM and embedding wrappers.

python -m venv .venv
source .venv/bin/activate
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv

Verify the installs resolved correctly:

pip show llama-index | grep Version
# Expected: Version: 0.10.x (or newer)

LlamaIndex splits providers into separate packages, so the two *-openai extras are required even though we are not calling OpenAI directly—they speak the OpenAI request shape that n4n.ai mirrors.

Step 2: Configure credentials

Keep the key out of source control. Create a .env file in your project root:

# .env
N4N_API_KEY=sk-your-actual-key-here

Load it in Python with python-dotenv. If you prefer exporting in the shell, that works too—just skip the loader.

Step 3: Point LlamaIndex at n4n.ai

The key move in this llamaindex n4n.ai setup tutorial is setting api_base on both the LLM and the embedding model. n4n.ai exposes a single OpenAI-compatible endpoint that addresses 240+ models, so you only maintain one URL.

import os
from dotenv import load_dotenv
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import Settings

load_dotenv()

N4N_BASE = "https://api.n4n.ai/v1"

llm = OpenAI(
    model="openai/gpt-4o-mini",
    api_key=os.environ["N4N_API_KEY"],
    api_base=N4N_BASE,
    temperature=0.1,
)

embed_model = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.environ["N4N_API_KEY"],
    api_base=N4N_BASE,
)

Settings.llm = llm
Settings.embed_model = embed_model

Because n4n.ai forwards provider cache-control hints and provides automatic fallback when a provider is degraded, the same endpoint handles both calls without extra retry logic in your code. The model string uses a provider/slug convention; switching to anthropic/claude-3-haiku later requires no structural change.

Step 4: Ingest documents and build an index

LlamaIndex treats everything as Document objects. For a quick check, use three inline strings:

from llama_index.core import Document, VectorStoreIndex

docs = [
    Document(text="n4n.ai is an inference gateway routing to many LLM providers."),
    Document(text="LlamaIndex abstracts retrieval and indexing for RAG applications."),
    Document(text="OpenAI-compatible endpoints accept the same request shape as OpenAI."),
]

index = VectorStoreIndex.from_documents(docs)

If the cell returns without raising, the embeddings were computed through n4n.ai and the in-memory vector store is populated. For larger corpora, swap from_documents for a SimpleDirectoryReader call—the rest of the pipeline stays identical.

Step 5: Query the index

Turn the index into a query engine and ask a question that requires retrieving the first document:

query_engine = index.as_query_engine()
response = query_engine.query("What does n4n.ai do?")
print(response.response)

Expected output:

n4n.ai is an inference gateway that routes requests to many LLM providers.

The response object also carries source nodes. Inspect them to confirm retrieval worked:

for node in response.source_nodes:
    print(node.score, node.text[:60])

You should see a non-zero score for the gateway sentence and lower or empty scores for the others.

Step 6: Switch models without refactoring

The value of routing through a gateway shows up when you change models. Edit the Settings.llm assignment:

llm = OpenAI(
    model="anthropic/claude-3-sonnet",
    api_key=os.environ["N4N_API_KEY"],
    api_base=N4N_BASE,
)
Settings.llm = llm

Re-run the query. The embedding model can stay on OpenAI-compatible text embeddings; n4n.ai honors client routing directives per call, so mixed provider stacks are legal in one process.

Step 7: Inspect per-token usage

n4n.ai returns per-token usage metering on the underlying OpenAI-style response. With LlamaIndex, the raw payload is accessible via the LLM wrapper directly:

completion = llm.complete("Return the word 'pong' as JSON")
print(completion.raw.usage)

Typical shape:

{
  "prompt_tokens": 12,
  "completion_tokens": 8,
  "total_tokens": 20
}

Log these fields to your own metrics pipeline if you need cost attribution across tenants.

Troubleshooting

401 Unauthorized
Check that N4N_API_KEY is loaded. Print os.environ.get("N4N_API_KEY") before client construction.

Model not found (404)
Model slugs are case-sensitive and prefixed by provider. Verify the exact string in the n4n.ai model list.

Embedding dimension mismatch
If you later change embed_model to a different dimension (e.g., text-embedding-3-large), rebuild the index—old vectors will not align.

Slow first call
Cold starts on some providers behind the gateway can take a few seconds. Subsequent calls hit warm paths. Set timeout on the OpenAI constructor if you need stricter bounds.

Final script

Here is the complete llamaindex n4n.ai setup tutorial code in one file:

import os
from dotenv import load_dotenv
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core import Settings, Document, VectorStoreIndex

load_dotenv()
N4N_BASE = "https://api.n4n.ai/v1"

Settings.llm = OpenAI(
    model="openai/gpt-4o-mini",
    api_key=os.environ["N4N_API_KEY"],
    api_base=N4N_BASE,
)
Settings.embed_model = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.environ["N4N_API_KEY"],
    api_base=N4N_BASE,
)

docs = [Document(text="n4n.ai is an inference gateway routing to many LLM providers.")]
index = VectorStoreIndex.from_documents(docs)
print(index.as_query_engine().query("What does n4n.ai do?").response)

Run it with python main.py after activating the venv. You now have a working RAG loop against a unified inference gateway, and you can rotate models by editing one string.