LlamaIndex with OpenAI-compatible gateway endpoints

Most LlamaIndex applications assume the OpenAI client. Pointing that same code at a LlamaIndex OpenAI-compatible gateway takes a single base-URL change and unlocks hundreds of models behind one interface. This walkthrough shows the exact steps to stand up a query pipeline against such a gateway, including streaming, provider hints, and how to verify the call landed where you expected.

Step 1: Install the LlamaIndex OpenAI integration

LlamaIndex split its providers into optional packages. You need the core plus the OpenAI LLM adapter; the adapter talks to any endpoint that mimics the OpenAI chat completions shape.

pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai

If you are on an older monolithic install (llama-index<0.10), the OpenAI and OpenAIEmbedding classes are already bundled, but the code below still applies.

Step 2: Configure the gateway base URL and key

The gateway issues an API key and exposes a /v1 endpoint that accepts the same request schema as OpenAI. Set them as environment variables or pass directly. Using env vars keeps secrets out of source control.

export LLM_GATEWAY_URL="https://gateway.example.com/v1"
export LLM_GATEWAY_KEY="sk-your-gateway-key"

In Python, read them and construct the LLM object. The api_base parameter is the only required deviation from vanilla OpenAI usage.

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    api_base=os.environ["LLM_GATEWAY_URL"],
    api_key=os.environ["LLM_GATEWAY_KEY"],
    model="gpt-4o-mini",
    temperature=0.1,
)

If your gateway prefixes model names with a provider (e.g. openai/gpt-4o-mini), use that exact string. The gateway routes the call downstream; LlamaIndex is oblivious.

Step 3: Instantiate the LLM with routing hints

A LlamaIndex OpenAI-compatible gateway typically proxies 200+ models. Pick one that fits latency and cost. Some gateways let you send routing directives via headers—for example, preferring a specific provider region or allowing fallback. n4n.ai honors client routing directives and forwards provider cache-control hints, which you set on the underlying OpenAI client.

from openai import OpenAI as RawOpenAI
from llama_index.llms.openai import OpenAI as LlamaOpenAI

raw_client = RawOpenAI(
    base_url=os.environ["LLM_GATEWAY_URL"],
    api_key=os.environ["LLM_GATEWAY_KEY"],
    default_headers={"X-Route": "auto-fallback", "X-Cache-Control": "max-age=600"},
)

llm = LlamaOpenAI(client=raw_client, model="anthropic/claude-3.5-sonnet")

This pattern gives you full control over transport-level headers while keeping LlamaIndex’s high-level abstractions.

Step 4: Point embeddings at the same gateway

Vector indices need embeddings. LlamaIndex’s default OpenAIEmbedding also respects api_base. If you skip this, the index will call OpenAI directly while your queries go through the gateway—split billing and possible model mismatch.

from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    api_base=os.environ["LLM_GATEWAY_URL"],
    api_key=os.environ["LLM_GATEWAY_KEY"],
    model="text-embedding-3-small",
)

Some gateways expose non-OpenAI embedding models under prefixed ids like cohere/embed-english. Pass those strings exactly. The embedding call now flows through the same gateway, so token metering and routing policies apply uniformly.

Step 5: Build an index and run a synchronous query

Create a few in-memory documents and a vector index. For a real corpus you would swap SimpleDirectoryReader in, but the LLM and embedding wiring is identical.

from llama_index.core import VectorStoreIndex, Document

docs = [
    Document(text="n4n.ai routes to 240+ models behind one endpoint."),
    Document(text="LlamaIndex abstracts query engines over vector stores."),
]

index = VectorStoreIndex.from_documents(docs, embed_model=embed_model)
query_engine = index.as_query_engine(llm=llm)

response = query_engine.query("Which gateway exposes many models?")
print(str(response))

The query engine calls llm.chat under the hood. Because we pointed api_base at the gateway, the completion request never touches OpenAI directly.

Step 6: Enable streaming

For chat-style apps, stream tokens to avoid blocking the UI. LlamaIndex exposes stream_chat on the LLM and stream_query on the engine.

stream = query_engine.stream_query("Summarize the documents in one sentence.")
for delta in stream.response_gen:
    print(delta, end="", flush=True)

If you call the LLM directly:

for chunk in llm.stream_chat([
    {"role": "user", "content": "What model are you?"}
]):
    print(chunk.delta, end="", flush=True)

Streaming works unchanged because the gateway speaks the OpenAI SSE protocol.

Step 7: Inspect usage and verify the route

Verification matters: you need proof the request hit the gateway and which upstream model served it. Most gateways return OpenAI-compatible usage objects. LlamaIndex attaches them to response.metadata for query engines, or to the final chunk for streams.

# After a non-streaming query
print(response.metadata.get("usage"))
# Example: {'prompt_tokens': 12, 'completion_tokens': 8, 'total_tokens': 20}

For a stream, accumulate the last chunk:

final = None
for chunk in llm.stream_chat([{"role": "user", "content": "hi"}]):
    final = chunk
print(final.additional_kwargs.get("usage"))

Cross-check the gateway’s per-token usage metering dashboard (if provided) against these numbers. A LlamaIndex OpenAI-compatible gateway should show the same token counts and the resolved model ID. If the model string in the dashboard differs from your request (e.g. claude-3.5-sonnet vs anthropic/claude-3.5-sonnet), the gateway rewritten the route—expected behavior.

Step 8: Handle fallback and degradation

Providers rate-limit. A gateway that implements automatic fallback saves you from writing retry loops. With n4n.ai, when a provider is rate-limited or degraded, the request reroutes without client changes. To test, temporarily pass a model that you know is throttled and watch the latency; the response should still arrive, and the usage metadata will reflect the actually-served provider.

You can also force a behavior with routing headers as shown in Step 3. If your gateway ignores unknown headers, it should at least forward standard Cache-Control to the upstream—useful when the upstream supports prompt caching.

How gateways differ (and why it matters)

Not every OpenAI-compatible endpoint is equal. Some are thin proxies that only rewrite the model field. Others, like n4n.ai, add automatic fallback, per-token metering, and honor client routing directives. When you build a LlamaIndex OpenAI-compatible gateway integration, decide which guarantees you need:

• Model coverage: Does the gateway expose the long-tail models your eval needs?
• Reliability: Will it retry across providers, or 500 on first failure?
• Observability: Can you reconcile token usage from the response with a billing view?

LlamaIndex doesn’t care about these differences—it just sends chat completions. Your operational posture does.

Final verification checklist

1. api_base points to the gateway /v1 URL for both LLM and embeddings.
2. A query returns text and usage metadata.
3. Streaming yields incremental deltas without errors.
4. Gateway logs show the expected model and token count.
5. Routing headers (if sent) appear in gateway request traces.

Follow those steps and your LlamaIndex app talks to any compliant gateway with zero changes to index or query logic.