Configuring LlamaIndex to use an OpenAI-compatible LLM gateway

The llamaindex openai-compatible gateway config is straightforward if you treat the gateway as a drop-in replacement for the OpenAI client. Most teams hit friction only when they assume LlamaIndex’s defaults match their gateway’s auth scheme, base URL, or model naming. This guide walks through a working setup end to end, with runnable Python and the exact points where gateway behavior diverges from OpenAI’s reference API.

Step 1: Validate the gateway with a raw request

Before touching LlamaIndex, confirm the gateway speaks the OpenAI chat protocol. This isolates auth and model-name problems from framework noise.

curl https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "Say pong."}],
    "max_tokens": 16
  }'

A 200 with choices[0].message.content containing “pong” means the base URL, key, and model alias are correct. A 401 means the key is wrong. A 404 means the model string is not registered at the gateway. A proxy error (often 502 or a custom 4xx) indicates the gateway could not reach the upstream—irrelevant to LlamaIndex config but good to know early.

Step 2: Install the LlamaIndex stack

Use a clean virtual environment. In LlamaIndex v0.10+, the OpenAI LLM and embedding classes live in separate packages from the core.

pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai

If you need a local vector store for the example, the in-memory default is sufficient to verify the wiring. The code below uses that default so the focus stays on the gateway.

Step 3: Export gateway credentials

Never hard-code keys. LlamaIndex’s OpenAI classes read OPENAI_API_KEY and OPENAI_API_BASE by default, but explicit passing is clearer when a gateway is involved.

export GATEWAY_API_KEY="sk-your-gateway-key"
export GATEWAY_BASE_URL="https://gateway.example.com/v1"

If you are using a service like n4n.ai, that single OpenAI-compatible endpoint fronts 240+ models and applies automatic fallback when an upstream provider is rate-limited, so the base URL and key are the only network-level details you need.

For local development, a .env file with python-dotenv works equally well:

from dotenv import load_dotenv
load_dotenv()

Step 4: Configure the LLM object

Construct the LLM with the gateway’s base URL and key. Pass the model name exactly as the gateway expects it—gateways often expose vendor models under aliased names like anthropic/claude-3-sonnet or openai/gpt-4o-mini.

from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="openai/gpt-4o-mini",
    api_base="https://gateway.example.com/v1",
    api_key="sk-your-gateway-key",
    temperature=0.1,
    max_tokens=1024,
)

If your gateway honors client routing directives or provider cache-control hints, build the underlying OpenAI client yourself and inject it. This avoids losing headers that LlamaIndex’s convenience wrapper might strip:

from openai import OpenAI as OpenAIClient
from llama_index.llms.openai import OpenAI

oai_client = OpenAIClient(
    base_url="https://gateway.example.com/v1",
    api_key="sk-your-gateway-key",
    default_headers={"Cache-Control": "max-age=300"},
)

llm = OpenAI(client=oai_client, model="openai/gpt-4o-mini")

The llamaindex openai-compatible gateway config at this layer is just a base-URL swap plus a model-string change.

Step 5: Configure the embedding model

Embeddings are a separate endpoint on most gateways. Use OpenAIEmbedding with the same base URL. Pick an embedding model the gateway actually serves; many gateways proxy text-embedding-3-small.

from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_base="https://gateway.example.com/v1",
    api_key="sk-your-gateway-key",
)

Set both on the global Settings so any downstream LlamaIndex module picks them up:

from llama_index.core import Settings

Settings.llm = llm
Settings.embed_model = embed_model

Note the dimension: text-embedding-3-small outputs 1536 dimensions, text-embedding-3-large outputs 3072. Mismatched dimensions against an existing index will throw at query time.

Step 6: Build a minimal index and query engine

With the gateway-backed models registered, the rest of LlamaIndex is unchanged. Create a vector index from a few documents and run a query.

from llama_index.core import VectorStoreIndex, Document

docs = [
    Document(text="LlamaIndex abstracts data connectors and retrieval."),
    Document(text="An OpenAI-compatible gateway routes to multiple providers."),
    Document(text="Token metering is reported per request by the gateway."),
]

index = VectorStoreIndex.from_documents(docs)
query_engine = index.as_query_engine(similarity_top_k=2)

response = query_engine.query("What does LlamaIndex abstract?")
print(str(response))

If you want streaming to verify token delivery through the gateway, use as_query_engine(streaming=True) and iterate:

query_engine = index.as_query_engine(streaming=True)
streaming_response = query_engine.query("Explain gateway fallback.")
for token in streaming_response.response_gen:
    print(token, end="")

Step 7: Verify success and meter usage

A correct llamaindex openai-compatible gateway config produces three observable results:

1. Auth passes: No 401 from the gateway. If you see one, check that api_base does not include /chat/completions—LlamaIndex appends the path.
2. Model resolves: The gateway returns 200 with logprobs or usage. A 404 means the model string is wrong for that gateway’s namespace.
3. Usage meters: The response object carries response.metadata with token counts. Print it:

print(response.metadata)
# {'token_count': 128, 'prompt_tokens': 54, 'completion_tokens': 74}

If your gateway provides per-token usage metering, those numbers should match its dashboard within a small rounding window. For async workloads, aquery returns the same metadata shape.

Troubleshooting checklist

Base URL trailing slash

LlamaIndex concatenates /chat/completions to api_base. If you set api_base="https://gateway.example.com/v1/", you get a double slash, which some gateways reject. Use no trailing slash.

Model name mapping

Gateways that aggregate providers rarely accept bare gpt-4. They need a prefix or a mapped alias. Check the gateway’s model list; if it exposes openai/gpt-4o-mini, use that exact string in the OpenAI(model=...) call.

Timeouts on large contexts