Routing LlamaIndex agents through n4n.ai's model gateway

Llamaindex model routing through a gateway decouples your agent logic from provider SDKs and rate limits. By targeting a single OpenAI-compatible endpoint, you keep LlamaIndex’s agent abstractions intact while moving model selection, fallback, and metering to the infrastructure layer. This guide builds a working ReAct-style agent that calls a math tool and shows exactly where to inject routing directives.

Step 1: Install the stack

Use a clean virtualenv. The only hard dependencies are the LlamaIndex core, the OpenAI LLM bridge, and the official OpenAI client (pulled in transitively).

pip install llama-index-core llama-index-llms-openai llama-index-agents-openai

If you are on an older LlamaIndex version (<0.10), the agent class may live in llama_index.agents instead of llama_index.agents.openai. The code below uses the split packages that match the current index structure. Keep your installs pinned in production; the LLM bridge changes function-calling schemas occasionally.

Step 2: Configure the gateway as the LLM backend

LlamaIndex’s OpenAI class is just a thin wrapper over the OpenAI SDK. Point its api_base at the gateway and supply a key. n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded, so a single base URL replaces dozens of vendor-specific clients.

from llama_index.llms.openai import OpenAI

llm = OpenAI(
    api_base="https://gateway.n4n.ai/v1",
    api_key="sk-gateway-xxxx",
    model="openai/gpt-4o-mini",
    temperature=0.1,
    timeout=30,
    max_retries=2,
)

The model string follows the gateway’s routing convention: provider/-model-name. You can swap it at runtime without reconstructing the agent (see Step 7). The timeout and max_retries are passed through to the underlying HTTP client; the gateway handles provider-side retries separately. Do not set max_retries high—let the gateway’s fallback switch providers instead of hammering a dead one.

Step 3: Define a tool the agent can call

LlamaIndex tools are plain Python functions with type hints. The framework generates the JSON schema for the LLM from the signature and docstring.

from llama_index.core.tools import FunctionTool

def multiply(a: float, b: float) -> float:
    """Multiply two numbers and return the product."""
    return a * b

tool = FunctionTool.from_defaults(fn=multiply)

Keep docstrings precise. The LLM uses them to decide when to call the tool, so vague descriptions degrade routing accuracy. If your tool raises exceptions, catch them internally and return a string error—LlamaIndex does not automatically serialize Python tracebacks for the model.

Step 4: Construct the agent

We use OpenAIAgent because it natively speaks the function-calling protocol that the gateway forwards unchanged. Verbose mode prints the thought trace, which is the fastest way to confirm llamaindex model routing is working.

from llama_index.agents.openai import OpenAIAgent

agent = OpenAIAgent.from_tools(
    tools=[tool],
    llm=llm,
    verbose=True,
    system_prompt="You are a concise math assistant. Use tools for calculations.",
)

If you prefer the newer workflow API, the same llm object drops into AgentWorkflow without modification. The gateway is agnostic to whether the traffic comes from an agent loop or a one-shot completion. Set system_prompt explicitly; defaults are fine for testing but too generic for production routing where token waste matters.

Step 5: Run a query

Execute a simple prompt that forces tool use.

response = agent.chat("What is 17.5 times 4?")
print(response.response)

Expected output: the agent calls multiply(17.5, 4), gets 70.0, and returns a natural-language answer. The verbose log shows the function_call payload sent to the gateway and the tool message returned. If you see the model answering directly without a tool call, tighten the system prompt or lower temperature.

Step 6: Verify success and metering

Success is not just a correct answer. Confirm three things:

1. Request reached the gateway. Check the x-request-id response header (accessible via the underlying client if you wrap the call). A valid UUID means the gateway accepted and routed the request.
2. Model routing applied. The verbose trace prints the model field. It should match what you set (openai/gpt-4o-mini), proving llamaindex model routing forwarded your directive.
3. Usage metered per token. The gateway returns standard OpenAI usage objects. In LlamaIndex, access them after the call:

raw = llm._client.chat.completions.last_response
if raw:
    print(raw.headers.get("x-ratelimit-remaining"))
    print(raw.usage)

If x-ratelimit-remaining is present, the gateway is enforcing and reporting limits. The usage object confirms prompt and completion tokens are counted, which is what per-token billing relies on. For automated tests, assert that raw.usage.completion_tokens > 0 to catch silent failures where the gateway returns an empty completion.

Step 7: Advanced llamaindex model routing patterns

Per-request model override

You do not need a new agent to change models. Mutate the model attribute on the LLM before the next chat:

agent.llm.model = "anthropic/claude-3-haiku"
response = agent.chat("Now compute 99 * 3 without tools.")

The gateway honors the new routing directive and forwards it to the specified provider. This is useful for cheap pre-processing (small model) versus reasoning (large model) in the same agent loop. Because LlamaIndex reads llm.model at call time, the swap is immediate and thread-local if you use separate agent instances.

Provider cache-control hints

Some providers support prompt caching. The gateway forwards cache-control hints if you pass them as extra headers. In LlamaIndex, inject them via additional_kwargs:

llm = OpenAI(
    api_base="https://gateway.n4n.ai/v1",
    api_key="sk-gateway-xxxx",
    model="openai/gpt-4o",
    additional_kwargs={"extra_headers": {"cache-control": "max-age=300"}},
)

This tells the upstream provider to cache the system prompt for five minutes, cutting latency and cost on repeated agent boots. Not every provider honors the header; the gateway passes it through untouched, so test with the specific model before relying on it.

Automatic fallback in practice

If the primary provider returns 429 or 503, the gateway shifts the request to a configured secondary without LlamaIndex noticing. Your agent code stays identical. To test, temporarily set an invalid model that the gateway knows maps to a degraded provider, then watch the verbose log: the response still arrives, but the x-routed-provider header (if exposed) differs from the requested one. Build your agent tests against the gateway’s fallback behavior rather than mocking provider errors locally.

Step 8: Streaming and async notes

For production agents, use achat and streaming to avoid blocking the event loop:

async def run():
    handler = await agent.achat("Calculate 2.5 * 8.")
    print(handler.response)

import asyncio
asyncio.run(run())

The gateway streams SSE chunks compatible with the OpenAI format, so LlamaIndex’s stream_chat works unchanged. Do not implement your own retry logic for provider errors; the gateway’s fallback already covers that layer. If you need to cancel mid-stream, close the underlying HTTPX client—LlamaIndex propagates the cancellation.

Step 9: Common pitfalls

• Model string typos. The gateway rejects unknown provider/model pairs with a 400. LlamaIndex surfaces this as an APIError; log the model field in your exception handler.
• Tool schema drift. Changing a function signature without rebuilding the FunctionTool leaves the old schema cached in the agent. Recreate the tool object whenever the signature changes.
• Mixed sync/async. Calling agent.chat inside an async loop blocks the loop. Use achat consistently.
• Header leakage. extra_headers set on the LLM apply to every request. Don’t put debug headers in production code.

Why this beats direct provider wiring

Hard-coding openai or anthropic clients inside tools scatters credentials and rate-limit handling across your codebase. Centralizing via llamaindex model routing keeps the agent focused on orchestration. You gain one place to rotate keys, enforce budget caps, and A/B test models. When a new model drops, you change a string, not a dependency.

The setup above is production-minimal: nine steps from install to verified metering. Extend it with more tools, multi-agent workflows, or a custom llm subclass that picks models based on input length. The gateway absorbs the provider complexity so your LlamaIndex code stays boring—in the best way.