Building a python client multiple llm models that spans GPT-5, Claude Opus 4.8, and Gemini 3 used to mean three SDKs, three auth schemes, and three sets of response types. A single OpenAI-compatible gateway collapses that into one HTTP interface: you point the standard openai package at a different base URL and switch the model string. This tutorial builds that client from scratch and runs it against real model identifiers.
Prerequisites
- Python 3.10 or newer
openaiPython package (v1.40+)- An API key from an OpenAI-compatible inference gateway. Set it as
LLM_API_KEYin your environment. - Familiarity with
asynciois helpful but not required; we start synchronous.
python -m venv .venv
source .venv/bin/activate
pip install openai python-dotenv
export LLM_API_KEY="sk-..."If you prefer a .env file, load it with python-dotenv before importing the client.
Point one client at the gateway
The openai library accepts a base_url argument. Services such as n4n.ai provide a single OpenAI-compatible endpoint that fronts 240+ models and handles fallback when a provider is degraded, which is the simplest way to run a python client multiple llm models without vendor lock. The code below configures the client once.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.n4n.ai/v1", # replace with your gateway's URL
api_key=os.environ["LLM_API_KEY"],
timeout=30.0,
)Everything downstream uses this single client object. No conditional imports, no vendor-specific exception handling.
Verify model availability
Before sending traffic, confirm the gateway exposes the models you intend to use. The models.list() call returns identifiers exactly as they should be passed to chat.completions.create.
response = client.models.list()
ids = [m.id for m in response.data]
print(f"Total models: {len(ids)}")
for model_id in ["gpt-5", "claude-opus-4-8", "gemini-3"]:
print(model_id, "available" if model_id in ids else "missing")Expected output (truncated):
Total models: 242
gpt-5 available
claude-opus-4-8 available
gemini-3 availableIf any identifier is missing, check the gateway’s model catalog; naming follows each provider’s published slug.
First call: GPT-5
The request shape is identical to OpenAI’s own API. Only the model field changes.
resp = client.chat.completions.create(
model="gpt-5",
messages=[
{"role": "system", "content": "You are a concise senior engineer."},
{"role": "user", "content": "What is head-of-line blocking in HTTP/2?"},
],
temperature=0.2,
max_tokens=200,
)
print(resp.choices[0].message.content)
print("Usage:", resp.usage.model_dump())Expected output:
HTTP/2 multiplexes streams over one TCP connection. Head-of-line blocking occurs when a single stalled stream (e.g., a slow packet) delays all other streams behind it at the TCP layer, undermining concurrency.
Usage: {'prompt_tokens': 24, 'completion_tokens': 41, 'total_tokens': 65}The usage object is populated per token by the gateway, so you get per-token metering without extra calls.
Same client, different model: Claude Opus 4.8
Switch the model string. No other code changes.
resp = client.chat.completions.create(
model="claude-opus-4-8",
messages=[{"role": "user", "content": "Summarize that in one line."}],
max_tokens=80,
)
print(resp.choices[0].message.content)Expected output:
HTTP/2 head-of-line blocking is a TCP-level delay where one stalled stream holds up all others sharing the connection.This python client multiple llm models approach means you can A/B test providers by changing one variable, not rewriting modules.
Gemini 3 with streaming
Gemini models often benefit from streaming to surface latency. The openai client supports stream=True uniformly.
stream = client.chat.completions.create(
model="gemini-3",
messages=[{"role": "user", "content": "List three caching strategies for LLM gateways."}],
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
if chunk.usage:
print("Stream usage:", chunk.usage.model_dump())Expected output (streamed, then usage):
1. Prompt prefix caching at the provider.
2. Gateway-level response caching keyed by hashed request.
3. Semantic cache using embeddings for near-duplicate hits.
Stream usage: {'prompt_tokens': 15, 'completion_tokens': 32, 'total_tokens': 47}Building a small router
Hardcoding model strings is fine for a script, but a real service wants a thin wrapper. Below is a minimal MultiLLM that picks a model from a config and falls back on exception.
from openai import OpenAI, APIError
class MultiLLM:
def __init__(self, client: OpenAI, default="gpt-5"):
self.client = client
self.default = default
def complete(self, prompt: str, model: str | None = None, **kw):
model = model or self.default
try:
resp = self.client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
**kw,
)
return resp.choices[0].message.content
except APIError as e:
# gateway may already have tried automatic fallback;
# if not, escalate or switch model here
raise RuntimeError(f"Model {model} failed: {e}") from e
llm = MultiLLM(client)
print(llm.complete("Ping", model="claude-opus-4-8"))If your gateway supports client routing directives, you can pass them via extra_headers. For example, to force a specific provider region:
resp = client.chat.completions.create(
model="gemini-3",
messages=[{"role": "user", "content": "Hi"}],
extra_headers={"x-routing": "provider:google;region:us-central"},
)Gateways that honor such hints forward them upstream and respect provider cache-control headers, which keeps prefix-cache hits intact across calls.
Batch across all three models
A common task: run the same prompt through every model and compare. The python client multiple llm models pattern makes this a loop.
prompt = "Define eventual consistency."
for model in ["gpt-5", "claude-opus-4-8", "gemini-3"]:
try:
out = llm.complete(prompt, model=model, max_tokens=60)
print(f"[{model}] {out}\n")
except RuntimeError as e:
print(f"[{model}] error: {e}\n")Expected output (abridged):
[gpt-5] Eventual consistency guarantees that, absent new updates, all replicas converge to the same value over time.
[claude-opus-4-8] It is a consistency model where replicated data becomes consistent asynchronously after a write.
[gemini-3] Eventual consistency: a storage system property where reads may lag but all copies sync eventually.Handling rate limits and degradation
Even with a gateway that performs automatic fallback when a provider is rate-limited, your client should back off on 429 or 5xx. The openai library raises RateLimitError and APIConnectionError. Catch them, sleep, retry with a different model if needed.
import time
from openai import RateLimitError, APIConnectionError
def safe_complete(llm, prompt, models, max_retries=2):
for attempt in range(max_retries):
for m in models:
try:
return llm.complete(prompt, model=m, max_tokens=50)
except RateLimitError:
time.sleep(2 ** attempt)
except APIConnectionError:
time.sleep(1)
return NoneBecause the gateway already collapses 240+ models behind one endpoint, this retry logic stays simple—no per-vendor SDK quirks.
Meter and log usage
Per-token usage metering is returned on every response. Aggregate it in your service:
from collections import defaultdict
totals = defaultdict(int)
for model in ["gpt-5", "claude-opus-4-8", "gemini-3"]:
r = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Count this."}],
max_tokens=10,
)
totals[model] += r.usage.total_tokens
print(dict(totals))Expected output:
{'gpt-5': 12, 'claude-opus-4-8': 11, 'gemini-3': 10}Feed this to your billing or observability pipeline. The gateway forwards exact provider counts, so you are not estimating.
Wrapping up the pattern
You now have a single OpenAI client instance that talks to GPT-5, Claude Opus 4.8, and Gemini 3 with no conditional code paths. The python client multiple llm models strategy reduces dependency surface, simplifies testing, and lets you shift traffic between providers by changing a string. For production, add structured logging, a circuit breaker, and a config-driven model map—but the core integration is exactly the code above.
If you need to support a new model next quarter, you add one slug to your routing table. No new package, no new auth, no new response parser.