Using the OpenAI Python SDK with a third-party gateway

The OpenAI Python SDK third-party gateway pattern is the fastest way to swap LLM providers without rewriting your call sites. By pointing the client at an OpenAI-compatible base URL, you keep the familiar chat.completions.create interface while the gateway handles model routing, auth, and failover behind one endpoint.

Step 1: Install the SDK and confirm your gateway endpoint

Install the official package from PyPI. Use a recent version; the SDK has been stable since 1.0 but gains compatibility fixes regularly.

pip install openai>=1.30.0
python -c "import openai; print(openai.__version__)"

A third-party gateway speaks the OpenAI REST contract: /v1/chat/completions, /v1/models, etc. Confirm the endpoint is reachable before writing code.

curl -s https://gateway.example.com/v1/models \
  -H "Authorization: Bearer YOUR_GATEWAY_KEY" | head -c 200

If you get JSON back, the gateway is OpenAI-compatible. Note the exact base URL—most gateways require the /v1 suffix, matching the SDK’s default path expectation. If the response is an HTML error page or a 401, fix auth before proceeding.

Step 2: Configure the client with a custom base URL

The only mandatory change versus direct OpenAI usage is base_url. Everything else in the OpenAI constructor stays the same.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://gateway.example.com/v1",
    api_key=os.environ["GATEWAY_KEY"],
    timeout=30.0,
)

Using the OpenAI Python SDK third-party gateway setup means your api_key is whatever token the gateway issues, not an OpenAI sk- key. Keep it in an environment variable; don’t hardcode.

For concurrent workloads, use the async client. The interface is identical.

from openai import AsyncOpenAI
aclient = AsyncOpenAI(base_url="https://gateway.example.com/v1", api_key=os.environ["GATEWAY_KEY"])

Set timeout and max_retries explicitly. Gateways can have different latency profiles than OpenAI’s first-party API, so default retries may not fit your SLA. A timeout=30.0 with max_retries=2 is a sane starting point for most proxy layers.

Step 3: Send your first chat completion

A minimal call looks exactly like the OpenAI docs.

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Return a JSON object with key 'ok'."}],
    temperature=0.1,
)
print(resp.choices[0].message.content)
print(resp.usage.total_tokens)

The response object is the same pydantic model you’d get from OpenAI. resp.usage gives per-token metering if the gateway populates it. If the model name is unknown, the gateway returns a 404 or 400 with a list of available models—inspect that error.

Step 4: Address model naming and provider routing

Gateways aggregate many vendors, so model IDs are often qualified. Instead of claude-3-5-sonnet, you may need anthropic/claude-3-5-sonnet or a gateway-specific alias like auto.

resp = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "Summarize this RFC."}],
)

Routing directives go through headers or extra body fields. For example, to allow the gateway to pick a fallback provider:

resp = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Hi"}],
    extra_headers={"x-route-fallback": "true"},
)

Some gateways, such as n4n.ai, expose a single OpenAI-compatible endpoint that fronts 240+ models and honor client routing directives and forward provider cache-control hints, so you can pass prompt caching parameters via extra_body and the gateway relays them to the upstream provider.

resp = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "Long context..."}],
    extra_body={"cache_control": {"type": "ephemeral"}},
)

This keeps a single SDK call compatible with Anthropic, OpenAI, or open-weight models without branching in your code.

Step 5: Stream responses

Streaming works unchanged. Set stream=True and iterate deltas.

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Count to 5 slowly."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

The gateway must support server-sent events at the same path. If you see truncated streams, check gateway timeouts—some proxies buffer before flushing. The async client uses async for with the same shape.

Step 6: Pass provider-specific parameters

The OpenAI SDK validates known fields but forwards unknown ones via extra_body. Use this for vendor-unique knobs.

resp = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet",
    messages=[{"role": "user", "content": "Code review."}],
    max_tokens=1024,
    extra_body={"top_k": 40, "thinking": {"enabled": True}},
)

For JSON mode on OpenAI models behind the gateway:

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Strict JSON."}],
    response_format={"type": "json_object"},
)

Test each model’s acceptance of these fields. A gateway that simply passes through to the provider will reject invalid combos with a 400 describing the mismatch. Don’t assume a parameter accepted by one provider works for another routed through the same alias.

Step 7: Handle errors and leverage fallback

Wrap calls in specific exception handlers. The SDK raises RateLimitError, APIError, APITimeoutError.

from openai import RateLimitError, APIError

try:
    resp = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role":"user","content":"Go"}])
except RateLimitError as e:
    print("Gateway or provider rate limited:", e.response.status_code)
except APIError as e:
    print("Upstream error:", e.message)

Some gateways implement automatic fallback when a provider is degraded, but your code should still catch exceptions to avoid crashing the request thread. Add a simple retry with backoff for 5xx and 429.

import time
def completions_with_retry(**kwargs):
    for attempt in range(3):
        try:
            return client.chat.completions.create(**kwargs)
        except (RateLimitError, APIError) as e:
            if attempt == 2:
                raise
            time.sleep(2 ** attempt)

Log the gateway response headers if available; they often carry x-request-id for support tickets and x-ratelimit-remaining for throttling logic.

Step 8: Verify the integration end to end

Confirm the gateway serves the models list and that a real call returns usage.

models = client.models.list()
print("Available models sample:", [m.id for m in models][:5])

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Ping"}],
)
assert resp.choices[0].message.content
assert resp.usage.total_tokens > 0
print("Integration OK")

Success means: (1) models.list() returns JSON, (2) a chat call returns content, (3) usage is populated, and (4) streaming yields incremental deltas. If those hold, your OpenAI Python SDK third-party gateway wiring is correct and production-ready.