One endpoint for GPT-5, Claude, Gemini, and Llama

An openai-compatible endpoint multiple models approach lets you swap GPT-5, Claude, Gemini, and Llama without rewriting your integration layer. Instead of maintaining four SDKs and four auth flows, you target one base URL and address each model by a namespaced ID. This guide walks a concrete path from a single client instance to production-grade routing with fallback and metering.

Step 1: Point your OpenAI client at one base URL

The fastest way to unify access is to reuse the OpenAI Python client and override base_url. Every major LLM vendor now speaks enough of the OpenAI chat schema that a gateway can translate the rest. You keep your existing call sites; only the constructor changes.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.your-gateway.example/v1",
    api_key="sk-your-gateway-key",
)

The core promise of an openai-compatible endpoint multiple models is that the request shape stays identical across vendors. You send messages, model, max_tokens, and optional tools. The gateway maps those to Anthropic, Google, or Meta wire formats.

Pitfall: do not hardcode api.openai.com in middleware or proxies. Use an environment variable for base_url so you can switch gateways without code changes. If you later adopt a gateway that provides automatic fallback, no application code needs to move.

Step 2: Address models by vendor-prefixed IDs

Once the client is pointed correctly, you select the backend purely by the model string. A consistent naming convention avoids ambiguity when two vendors ship similarly named models.

def complete(prompt: str, model: str):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=512,
    )
    return resp.choices[0].message.content

# Swap with one argument
complete("Explain Raft.", "openai/gpt-5")
complete("Explain Raft.", "anthropic/claude-3-5-sonnet")
complete("Explain Raft.", "google/gemini-1.5-pro")
complete("Explain Raft.", "meta/llama-3.1-70b")

This is where the openai-compatible endpoint multiple models pattern pays off—no branch on vendor, no separate anthropic.Client import. Your test suite can iterate over a list of model IDs and assert the same behavior.

Tradeoff: model IDs are not standardized across gateways. Document your namespace internally; treat the ID as a contract, not a string literal scattered in code. Avoid baking provider-specific suffixes like -latest into business logic.

Namespacing rule of thumb

Use vendor/model-family as the minimal safe format. If you run multiple versions of Llama from different hosts, extend to vendor/host/model. Centralize the list:

MODELS = {
    "flagship": "openai/gpt-5",
    "balanced": "anthropic/claude-3-5-sonnet",
    "long_ctx": "google/gemini-1.5-pro",
    "local": "meta/llama-3.1-70b",
}

Step 3: Normalize what providers quietly diverge on

Behind the compatible surface, providers differ in context limits, token counting, and tool schemas. A thin adapter layer prevents silent truncation or schema errors.

MODEL_LIMITS = {
    "openai/gpt-5": 128_000,
    "anthropic/claude-3-5-sonnet": 200_000,
    "google/gemini-1.5-pro": 1_000_000,
    "meta/llama-3.1-70b": 8_000,
}

def safe_complete(prompt: str, model: str, max_tokens: int = 512):
    limit = MODEL_LIMITS.get(model, 8_000)
    # crude estimate: 4 chars ~ 1 token
    if len(prompt) // 4 + max_tokens > limit:
        raise ValueError(f"{model} context overflow risk")
    return complete(prompt, model)

Claude expects system prompts inside a top-level system field in its native API, but the OpenAI translation usually lifts a system message correctly. Verify with a quick smoke test; don’t assume.

Tool calling is the sharpest edge

OpenAI uses tools with function objects; Anthropic requires input_schema instead of parameters. Gemini flattens function declarations differently. If you pass tools, test on each model. If a model doesn’t support them, the gateway may return an error or silently ignore—log the response headers.

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {"type": "object", "properties": {"zip": {"type": "string"}}}
    }
}]

# Works on GPT-5 and Claude via translation; verify on Llama
resp = client.chat.completions.create(
    model="openai/gpt-5",
    messages=[{"role": "user", "content": "Weather in 94107?"}],
    tools=tools,
)

Common mistake: trusting the same strict JSON mode across vendors. GPT-5 supports response_format with strict schema; Llama hosted instances often do not. Gate the feature by model capability flag.

Step 4: Implement fallback without masking errors

Providers degrade. Rate limits, regional outages, or cold starts will hit you. A gateway that offers automatic fallback when a provider is rate-limited or degraded removes most of the operational burden. For example, n4n.ai routes to a healthy backend if your primary model returns 429, so your code stays linear. You can also implement client-side fallback:

FALLBACK_CHAIN = [
    "openai/gpt-5",
    "anthropic/claude-3-5-sonnet",
    "google/gemini-1.5-pro",
]

def complete_with_fallback(prompt: str):
    last_err = None
    for model in FALLBACK_CHAIN:
        try:
            return complete(prompt, model)
        except Exception as e:
            last_err = e
            continue
    raise last_err

Pitfall: fallback hides quality regressions. A prompt tuned for GPT-5 may produce worse outputs on Llama. Log which model actually served the request (resp.model) and sample outputs for offline evaluation.

Tradeoff: cross-vendor fallback adds tail latency. If you only need resilience, prefer a gateway-level fallback over chaining three client calls in series. Client-side chains also double-count failed tokens if errors occur after partial generation.

Step 5: Meter usage and enforce budgets

Per-token usage metering is non-negotiable when many models share one code path. The OpenAI response object includes usage regardless of backend.

resp = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "Summarize this RFC."}],
)
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Pipe these to your metrics system tagged by resp.model. A single endpoint makes cost attribution trivial: one meter, many backends. Set hard caps per model in your gateway config to avoid a misconfigured batch job draining the budget on GPT-5.

Common mistake: assuming prompt_tokens is computed identically across vendors. Google’s tokenization differs from OpenAI’s; treat cross-model token counts as approximate for budgeting, exact only within a single model. If your gateway exposes cached_tokens, subtract those from billed prompt tokens when reconciling invoices.

Step 6: Forward cache-control hints to cut cost

Providers support prompt caching, but the hint fields differ. An OpenAI-compatible gateway that honors client routing directives and forwards provider cache-control hints lets you send one schema. For OpenAI-style caching, pass extra_body:

client.chat.completions.create(
    model="openai/gpt-5",
    messages=[{"role": "system", "content": LONG_SYSTEM_PROMPT}],
    extra_body={"cache_control": {"type": "ephemeral"}},
)

On Anthropic, the same gateway translates that to cache_control blocks. Without this passthrough, you pay full prompt tokens on every call for static prefixes.

The openai-compatible endpoint multiple models design means you write the cache hint once and the gateway maps it to each provider’s native field. Tradeoff: cached tokens often have a minimum lifetime and may expire under load. Don’t cache highly dynamic content; measure cache hit rate via usage cached_tokens if exposed.

Common pitfalls when swapping models in production

• System message placement: Some gateways map a system message to a different field; always test with a model from each vendor.
• Max output caps: GPT-5 may allow 4k completion tokens; Llama hosted instances may cap at 2k. Set max_tokens per model.
• Streaming differences: Streaming SSE chunks may include different finish_reason semantics. Parse defensively.
• Tool response format: If you use function calls, validate the arguments schema on the client. A model may return JSON that violates your spec.
• Latency variance: Gemini and Claude have different time-to-first-token; your timeout budgets must accommodate the slowest fallback.
• Version drift: gpt-5 may point to a patched checkpoint next week. Pin to dated aliases in production and promote deliberately.

Final checklist

1. One OpenAI client, base_url from env.
2. Model IDs namespaced and centralized in a config dict.
3. Context and token limits enforced before send.
4. Fallback chain with logging of actual served model.
5. Usage metering tagged by model, budgets enforced at gateway.
6. Cache-control hints passed via extra_body for static prefixes.

An openai-compatible endpoint multiple models strategy collapses integration complexity into a single client and a string. The work shifts from SDK wrangling to model governance—exactly where it should be.