The fastest way to broaden model coverage without rewriting your LLM call sites is to change one configuration value. The openai python sdk switch base_url pattern lets you keep the familiar chat.completions.create interface while routing to a gateway that fronts 240+ models from dozens of providers. Below is the exact sequence to repoint the client, send cross-provider requests, and confirm the traffic lands where you expect.
Step 1: Install or upgrade the OpenAI Python SDK
The modern SDK (v1.0+) uses a client object and supports arbitrary base_url values. If you are on an older version, upgrade first.
pip install -U openai
python -c "import openai; print(openai.__version__)"Anything >= 1.40.0 is safe for the patterns here. The legacy 0.x openai.ChatCompletion module will not accept a custom base URL the same way, so don’t skip the upgrade.
Step 2: Configure the client with the new base_url
Instantiate OpenAI with base_url pointing at the gateway’s OpenAI-compatible root. The root must include the /v1 suffix if the gateway follows the standard path layout. Keep credentials in the environment, not in source.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.example.com/v1", # replace with your gateway
api_key=os.environ["LLM_GATEWAY_KEY"],
)The api_key is what the gateway uses to authenticate you; it is not forwarded to upstream providers. If your gateway supports unauthenticated local dev, you can pass api_key="dummy" but never ship that.
Step 3: List available models
Before sending a request, confirm the gateway responds and inspect the model namespace. A single models.list() call returns the full catalog.
models = client.models.list()
print(f"Model count: {len(models.data)}")
for m in models.data[:10]:
print(m.id)Model IDs are typically namespaced as provider/family or provider/model-name. Examples: anthropic/claude-3.5-sonnet, meta-llama/llama-3-70b-instruct, openai/gpt-4o-mini. The openai python sdk switch base_url approach means this list reflects the gateway’s unified catalog, not just OpenAI’s native models.
Step 4: Send your first cross-provider request
Pick a model ID from the list and call it exactly as you would with OpenAI’s API.
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[
{"role": "system", "content": "You are terse."},
{"role": "user", "content": "Say 'pong'."},
],
temperature=0.2,
max_tokens=16,
)
print(resp.choices[0].message.content)
print(resp.usage)Streaming works identically:
stream = client.chat.completions.create(
model="meta-llama/llama-3-70b-instruct",
messages=[{"role": "user", "content": "Count to 3."}],
stream=True,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")Step 5: Pass provider-specific hints and routing directives
Some gateways accept HTTP headers to control routing or caching. The OpenAI SDK exposes extra_headers on every request. For example, to prefer cheaper routes and forward a cache hint to the upstream provider:
resp = client.chat.completions.create(
model="openai/gpt-4o",
messages=[{"role": "user", "content": "Explain base_url."}],
extra_headers={
"X-Routing-Directive": "prefer-lowest-cost",
"Cache-Control": "max-age=3600",
},
)Gateways such as n4n.ai honor client routing directives and forward provider cache-control hints, so the same header block propagates end to end without extra plumbing. If your gateway ignores unknown headers, they are harmless.
Step 6: Handle fallback and degradation
A multi-provider gateway typically provides automatic fallback when a provider is rate-limited or degraded. Your code should still catch exceptions, because a total outage or invalid model ID will raise.
from openai import APIError, RateLimitError
try:
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": "Hi"}],
)
except RateLimitError as e:
print("Gateway rate limited:", e)
except APIError as e:
print("API error:", e.status_code, e.message)Because the gateway already shifts traffic on provider errors, you usually don’t need custom retry loops for upstream 429s. Add a short backoff only for gateway-level 429s.
Step 7: Verify success and inspect metering
Verification is more than a non-empty string. Check the model actually used, the usage block, and the response headers if your gateway returns them.
def verify_call(client):
resp = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role": "user", "content": "Return the word OK."}],
)
assert resp.choices[0].message.content.strip().upper() == "OK", "unexpected content"
assert resp.usage.total_tokens > 0, "no token metering"
assert resp.model.startswith("openai/"), f"unexpected model {resp.model}"
print("Verification passed:", resp.usage)
verify_call(client)Per-token usage metering appears in resp.usage. If your gateway bills per token, this object is the source of truth for reconciliation. Log resp.usage.prompt_tokens and completion_tokens to your metrics pipeline.
You can also verify at the network layer with curl:
curl -s $BASE_URL/v1/models \
-H "Authorization: Bearer $LLM_GATEWAY_KEY" | head -c 200A JSON list confirms the base_url is correct and reachable.
Step 8: Refactor existing code safely
If you have dozens of call sites, centralize client creation. Don’t scatter base_url across modules.
# llm_client.py
from functools import lru_cache
from openai import OpenAI
import os
@lru_cache(maxsize=1)
def get_client() -> OpenAI:
return OpenAI(
base_url=os.environ["LLM_BASE_URL"],
api_key=os.environ["LLM_GATEWAY_KEY"],
)Then replace from openai import OpenAI at call sites with from llm_client import get_client and client = get_client(). This makes the openai python sdk switch base_url change a one-line environment configuration rather than a code edit.
Common pitfalls
Trailing slashes. base_url="https://gateway.example.com/v1/" often doubles the slash and 404s on /v1//chat/completions. Omit the trailing slash.
Model name format. Native OpenAI SDK calls use gpt-4o. Behind a gateway you must use the gateway’s ID, usually openai/gpt-4o. A bare gpt-4o may resolve or may error depending on the gateway’s default namespace.
Key scoping. The gateway key is not the OpenAI key. If you accidentally leave api_key=os.environ["OPENAI_API_KEY"] you’ll get auth errors at the gateway.
Timeout defaults. The SDK default timeout is 600s. For interactive apps, lower it:
client = OpenAI(base_url=..., api_key=..., timeout=30.0)Streaming cleanup. Always consume or close streams. An unread stream can hold a connection open.
What you get now
After these steps, your Python service talks to 240+ models through the same chat.completions surface you already know. The openai python sdk switch base_url maneuver is reversible: point it back at https://api.openai.com/v1 and your original behavior returns. That isolation makes it cheap to experiment with model routing, fallbacks, and cost metering without forking your codebase.