Configure LlamaIndex to use n4n.ai as the LLM

This llamaindex configure n4n.ai llm tutorial shows how to route LlamaIndex requests through n4n.ai’s OpenAI-compatible gateway. You’ll point the LlamaIndex OpenAI LLM wrapper at a single endpoint that fronts 240+ models with automatic fallback, then run a real query to confirm it works.

Step 1: Install the LlamaIndex OpenAI integration

LlamaIndex split its providers into separate packages in the v0.10+ line. You need the core library and the OpenAI LLM wrapper (which speaks the OpenAI chat protocol that the gateway emulates).

pip install llama-index llama-index-llms-openai python-dotenv

If you already have a LlamaIndex project, pin the versions to avoid surprises:

pip install "llama-index>=0.11.0" "llama-index-llms-openai>=0.2.0"

The python-dotenv dependency is optional but keeps credentials out of source control.

Step 2: Export the endpoint and API key

The gateway exposes one OpenAI-compatible base URL. Store the key and URL in environment variables so they aren’t hardcoded.

export N4N_API_KEY="sk-your-key-here"
export N4N_BASE_URL="https://api.n4n.ai/v1"

A quick sanity check before writing Python: hit the endpoint with curl to confirm the key is valid and the model slug works.

curl $N4N_BASE_URL/chat/completions \
  -H "Authorization: Bearer $N4N_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"ping"}]}'

You should get a standard OpenAI-style JSON response with choices[0].message.content.

Step 3: Instantiate the LLM wrapper

LlamaIndex’s OpenAI class accepts base_url and api_key. The model field is passed through verbatim, so use the provider-qualified slug the gateway expects (e.g., openai/gpt-4o, anthropic/claude-3-5-sonnet-20240620).

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="openai/gpt-4o",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ["N4N_BASE_URL"],
    temperature=0.1,
    max_tokens=512,
    timeout=30,
)

Opinion: always set an explicit timeout. The gateway handles provider degradation with fallback, but your code should fail fast if the network stalls.

Step 4: Register the LLM globally (or per-call)

For most applications, set the LLM on Settings so higher-level abstractions (query engines, agents, routers) pick it up automatically.

from llama_index.core import Settings

Settings.llm = llm

If you prefer isolation, pass llm= to the specific component instead:

query_engine = index.as_query_engine(llm=llm)

Avoid mixing global and per-call settings in the same module; it makes debugging routing issues painful.

Step 5: Run a bare completion to verify wiring

Before building an index, confirm the wrapper returns text and usage metadata.

resp = llm.complete("Write a Python function to reverse a string.")
print(resp.text)
print("usage:", resp.usage)

Expected output: a code snippet and a dict like {'prompt_tokens': 12, 'completion_tokens': 45, 'total_tokens': 57}. The gateway performs per-token metering, so those numbers reflect what you’ll be billed.

If you get a 401, check the env var name. A 404 on the model usually means the slug is wrong—the gateway does not silently alias gpt-4o to openai/gpt-4o.

Step 6: Build a minimal RAG pipeline

A completion test proves the LLM works; a retrieval pipeline proves LlamaIndex integration is correct. Use an in-memory vector store with a dummy document.

from llama_index.core import VectorStoreIndex, Document

docs = [
    Document(text="The gateway fronts 240+ models behind one OpenAI-compatible endpoint."),
    Document(text="Fallback is automatic when a provider is rate-limited or degraded."),
]

index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine()

out = query_engine.query("What happens when a provider is degraded?")
print(out.response)

This exercises the embedding step (default text-embedding-3-small from OpenAI directly unless you override Settings.embed_model) and the LLM call. If the answer mentions automatic fallback, the LLM received the retrieved context.

Note: embedding models are a separate concern. This tutorial focuses on the LLM; point Settings.embed_model at your own embedding provider if you don’t want to call OpenAI directly.

Step 7: Switch models and use routing directives

The value of the gateway is model diversity without code changes. Swap the model argument:

llm_claude = OpenAI(
    model="anthropic/claude-3-5-sonnet-20240620",
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ["N4N_BASE_URL"],
)
print(llm_claude.complete("Summarize the term 'inference gateway' in one sentence.").text)

n4n.ai honors client routing directives and forwards provider cache-control hints. To pass cache hints, construct an openai.OpenAI client with default_headers and hand it to the wrapper:

from openai import OpenAI as OpenAIClient

raw_client = OpenAIClient(
    api_key=os.environ["N4N_API_KEY"],
    base_url=os.environ["N4N_BASE_URL"],
    default_headers={"x-provider-cache-control": "max-age=3600"},  # consult gateway docs for exact header
)

llm_cached = OpenAI(model="openai/gpt-4o", client=raw_client)

This pattern is useful when you generate long system prompts that rarely change—cache hits cut latency and cost.

Step 8: Verify success in production-like conditions

A single synchronous call isn’t enough. Add a tiny retry wrapper and log the usage field so you can confirm metering end-to-end.

import time

def guarded_query(llm, prompt, retries=3):
    for i in range(retries):
        try:
            r = llm.complete(prompt)
            assert r.text.strip()
            return r
        except Exception as e:
            if i == retries - 1:
                raise
            time.sleep(2 ** i)

resp = guarded_query(llm, "Explain automatic fallback in LLM gateways.")
print(resp.usage)

If you run this against the gateway and watch your metering dashboard, you’ll see token counts appearing per request. That’s the definitive proof the integration is live.

Common pitfalls

• Model slug mismatch: The gateway is strict. gpt-4o will 404; use openai/gpt-4o.
• Streaming: LlamaIndex supports stream_complete. The gateway streams compatible chunks, but test your parser—some providers send different finish reasons.
• Temperature bounds: Some models behind the gateway reject temperature=0 (e.g., certain reasoning models). Keep it in [0.0, 1.0] and catch ValueError.
• Settings leakage: In notebooks, Settings.llm persists across cells. Reassign explicitly when switching models.

Final check

You now have a working llamaindex configure n4n.ai llm tutorial setup: one OpenAI wrapper, one base URL, any model. Run the Step 6 script, see a coherent answer, and check the usage dict. That’s the whole integration.