LlamaIndex Settings.llm configuration with n4n.ai

If you’re building with LlamaIndex and want to route LLM calls through a gateway that handles provider fallback, usage metering, and model routing without changing your application code, this llamaindex settings.llm n4n.ai tutorial walks through the exact configuration. The Settings.llm global is the single place where LlamaIndex resolves which model to use for indexing, querying, and agent loops. Pointing it at an OpenAI-compatible endpoint lets you swap providers, enable fallbacks, and centralize observability without touching every VectorStoreIndex or QueryEngine call.

Step 1: Install the required packages

You need LlamaIndex core, the OpenAI integration (which provides the OpenAI class that works with any OpenAI-compatible endpoint), and the HTTP client it depends on.

pip install llama-index-core llama-index-llms-openai httpx

If you’re using a virtual environment, activate it first. Verify the imports work:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
print("Imports OK")

Step 2: Get your n4n.ai API key and base URL

Log into the n4n.ai dashboard and create an API key. The gateway exposes a single OpenAI-compatible base URL: https://api.n4n.ai/v1. This endpoint fronts 240+ models and honors routing directives you pass in headers.

Store the key in your environment — never hardcode it.

export N4N_API_KEY="sk-..."
export N4N_BASE_URL="https://api.n4n.ai/v1"

Step 3: Configure Settings.llm with the gateway endpoint

Create the OpenAI instance pointed at the n4n.ai base URL, then assign it to Settings.llm. This makes it the default for every LlamaIndex operation that needs an LLM.

import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

api_key = os.getenv("N4N_API_KEY")
base_url = os.getenv("N4N_BASE_URL", "https://api.n4n.ai/v1")

if not api_key:
    raise RuntimeError("N4N_API_KEY not set in environment")

Settings.llm = OpenAI(
    model="gpt-4o-mini",           # any model the gateway serves
    api_key=api_key,
    api_base=base_url,
    temperature=0.1,
    max_tokens=1024,
    timeout=60.0,
    # Optional: pass routing hints via extra headers
    default_headers={
        "X-n4n-Route": "cost-optimized",  # or "latency-optimized", "provider:anthropic"
    },
)

What each parameter does:

• model: The logical model name. The gateway resolves this to an actual provider model based on your routing rules.
• api_base: Points the OpenAI client at the gateway instead of api.openai.com.
• default_headers: Optional routing directives. The gateway reads X-n4n-Route to select a provider or strategy. You can also pass X-n4n-Model to pin a specific provider model (e.g., anthropic/claude-3-5-sonnet-20241022).

Step 4: Optionally configure the embedding model

If your pipeline uses embeddings (RAG, semantic search), point Settings.embed_model at the same gateway. The gateway serves embedding models through the same endpoint.

from llama_index.embeddings.openai import OpenAIEmbedding

Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_key=api_key,
    api_base=base_url,
    dimensions=1536,  # optional, for models that support truncation
)

Now both LLM and embedding calls route through the gateway with the same fallback and metering behavior.

Step 5: Verify the configuration with a minimal test

Run a quick completion to confirm the wiring works end-to-end.

from llama_index.core import Settings

response = Settings.llm.complete("Say 'pong' if you receive this.")
print(response.text.strip())

Expected output: pong (or whatever the model returns). If you see an authentication error, check N4N_API_KEY. If you see a connection error, verify N4N_BASE_URL and network egress.

Step 6: Use the configured LLM in a real pipeline

With Settings.llm set, every LlamaIndex component that needs an LLM uses it automatically — no need to pass llm= everywhere.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core import Settings

# Load documents
documents = SimpleDirectoryReader("./data").load_data()

# Build index — uses Settings.embed_model for embeddings
index = VectorStoreIndex.from_documents(documents)

# Query — uses Settings.llm for synthesis
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What are the key risks mentioned in the document?")
print(response)

The same Settings.llm instance handles:

• Query synthesis in QueryEngine
• Agent reasoning loops in ReActAgent or OpenAIAgent
• Sub-question decomposition in SubQuestionQueryEngine
• Any custom component that calls Settings.llm.complete() or Settings.llm.achat()

Step 7: Override per-request when needed

Global settings are convenient, but sometimes a specific query needs a different model (e.g., a cheaper model for classification, a stronger one for synthesis). Pass llm= explicitly to override just that call.

from llama_index.llms.openai import OpenAI
from llama_index.core import Settings

# One-off stronger model for final synthesis
strong_llm = OpenAI(
    model="gpt-4o",
    api_key=api_key,
    api_base=base_url,
    temperature=0.0,
)

query_engine = index.as_query_engine(
    similarity_top_k=3,
    llm=strong_llm,  # overrides Settings.llm only for this engine
)

This keeps your default cost-optimized while allowing surgical upgrades.

Step 8: Enable async for throughput

If you’re running a server or batch pipeline, use the async methods. The OpenAI client supports acomplete, achat, and astream_complete natively.

import asyncio
from llama_index.core import Settings

async def batch_complete(prompts: list[str]) -> list[str]:
    tasks = [Settings.llm.acomplete(p) for p in prompts]
    responses = await asyncio.gather(*tasks)
    return [r.text for r in responses]

prompts = [
    "Summarize in one sentence: " + doc.text[:500]
    for doc in documents[:10]
]
summaries = asyncio.run(batch_complete(prompts))
for s in summaries:
    print(s[:120], "...")

Async avoids blocking the event loop and lets the gateway’s connection pooling work efficiently.

Step 9: Observe usage and routing in the dashboard

The gateway meters per-token usage per request and surfaces it in the n4n.ai dashboard. You’ll see:

• Input/output tokens per model and provider
• Latency percentiles (p50, p95, p99)
• Fallback events when a provider returns 429 or 5xx
• Routing directive effectiveness

No additional instrumentation code is required — the gateway captures this from the OpenAI-compatible request/response flow.

Step 10: Handle streaming for responsive UIs

For chat interfaces, stream tokens as they arrive. LlamaIndex’s stream_complete and astream_complete work unchanged.

def stream_response(query: str):
    query_engine = index.as_query_engine(streaming=True, similarity_top_k=3)
    streaming_response = query_engine.query(query)
    for token in streaming_response.response_gen:
        print(token, end="", flush=True)
    print()

stream_response("Explain the architecture in simple terms")

The gateway forwards provider cache-control hints (e.g., x-cache-status: hit) through the response headers, which the OpenAI client exposes via response.raw.headers if you need them for debugging.

Common pitfalls

Wrong base URL path: The gateway expects /v1 in the base URL. https://api.n4n.ai without /v1 will 404 on /chat/completions.

Model name mismatch: Use a model name the gateway recognizes. Run curl -H "Authorization: Bearer $N4N_API_KEY" https://api.n4n.ai/v1/models to list available models.

Missing async in async contexts: Calling complete() inside an async function blocks the loop. Use acomplete() instead.

Header casing: The gateway reads X-n4n-Route case-insensitively, but some proxies normalize headers. Use the exact casing shown.

Verification checklist

• Settings.llm.complete("test") returns a response without error
• Settings.embed_model.get_text_embedding("test") returns a 1536-dim vector (or your model’s dimension)
• A full RAG query returns a coherent answer grounded in your documents
• Dashboard shows the request with correct token counts and model name
• Fallback works: temporarily block the primary provider in dashboard rules, re-run, verify response still succeeds

Next steps

• Set up routing rules in the dashboard to direct traffic by model, cost, or latency
• Add X-n4n-Request-ID headers from your application for end-to-end tracing
• Configure per-project API keys if you need separate billing or rate limits
• Explore the gateway’s model aliasing to map logical names (e.g., my-chat-model) to specific provider models without code changes

The Settings.llm pattern keeps your application code clean while the gateway handles the operational complexity of multi-provider LLM inference.