From vendor lock-in to gateway: migrating OpenAI SDK code

Most Python codebases that call OpenAI look identical: a client constructed with an api_key, a model string, and a handful of parameter tweaks. To avoid OpenAI vendor lock-in gateway pain, you can keep that code almost intact and instead change where the client sends requests, then offload provider selection to a compatible gateway. This guide gives an ordered path from a hard-coded OpenAI dependency to a routing-agnostic call site.

1. Audit your current OpenAI SDK usage

Before changing a line, find every direct touchpoint. In a Python service this is usually openai.OpenAI() or AsyncOpenAI().

from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "hi"}],
    temperature=0.2,
)

Run a quick static scan:

grep -rn "openai" --include="*.py" . | grep -E "OpenAI\(|chat.completions"

Record three things: model identifiers, timeout/retry configuration, and any extra_headers or extra_body usage. You cannot migrate what you have not enumerated. Pay special attention to helper functions that wrap the client—those are the only places you should later edit. If you run the SDK inside Lambda or background workers, grep those repos too. The goal is to avoid OpenAI vendor lock-in gateway lock by making the client dumb and the gateway smart.

2. Swap the base URL and credentials

The OpenAI SDK is an HTTP client that speaks a specific JSON contract. It accepts base_url. Point it at your gateway’s OpenAI-compatible endpoint and use a gateway-issued key.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GATEWAY_KEY"],
    base_url="https://gateway.example.com/v1",
)

That is the entire cutover for a trivial call. If your gateway exposes one OpenAI-compatible endpoint that addresses 240+ models, you do not need to import provider-specific packages. The create method shape stays identical.

A common mistake is leaving OPENAI_API_KEY in the environment. The SDK will not complain if you pass api_key explicitly, but a stray env var can cause confusion in tests. Unset it in your deployment manifest.

3. Normalize model identifiers

Gateways route by model name, but they rarely accept bare OpenAI strings when multiple vendors are behind the same endpoint. Introduce a single mapping function at the edge of your code.

MODEL_MAP = {
    "gpt-4o": "openai/gpt-4o",
    "gpt-4o-mini": "openai/gpt-4o-mini",
    "claude-3-5-sonnet": "anthropic/claude-3-5-sonnet",
}

def route(model: str) -> str:
    return MODEL_MAP.get(model, model)

Use it immediately before the call:

resp = client.chat.completions.create(
    model=route("gpt-4o"),
    messages=messages,
)

Do not inline qualified names across modules. If a model is deprecated upstream, you change one dictionary, not fifty call sites. This is also where you can implement weighted routing (“use anthropic for 10% of traffic”) without touching business logic.

4. Disable client-side retries

The OpenAI SDK ships with exponential backoff on 429/5xx. That logic conflicts with gateway-level fallback. If the gateway already reshuffles to a healthy provider, your client should fail fast and surface the error.

client = OpenAI(
    base_url="https://gateway.example.com/v1",
    api_key=os.environ["GATEWAY_KEY"],
    max_retries=0,
    timeout=20,
)

A gateway like n4n.ai provides automatic fallback when a provider is rate-limited or degraded, so an application retry loop only doubles latency during incidents. Set max_retries=0 and let the gateway return a single aggregated response.

5. Forward cache-control and provider hints

OpenAI’s extra_headers pass through to the gateway if it is transparent. Prompt caching saves money; do not drop it during migration.

resp = client.chat.completions.create(
    model=route("gpt-4o"),
    messages=system_plus_user,
    extra_headers={"cache-control": "max-age=600"},
)

Gateways that honor client routing directives and forward provider cache-control hints will propagate max-age to the upstream vendor. Verify in the gateway logs that the hint arrived; some gateways strip unknown headers by default.

6. Shift usage metering to the gateway

Your code probably logs response.usage.prompt_tokens. After migration, the gateway’s per-token usage metering is the source of truth because it accounts for fallback models and internal routing overhead. Keep logging the response object, but bill from the gateway’s usage field.

usage = resp.usage
# gateway may add `resp.headers["x-gateway-tokens"]`

Do not build your own token counter. Provider tokenization differs; the gateway normalizes it.

7. Handle streaming and async

Streaming uses the same method with stream=True. The SSE payload format is unchanged.

stream = client.chat.completions.create(
    model=route("gpt-4o"),
    messages=messages,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

For async, swap OpenAI for AsyncOpenAI and await the call. No gateway-specific changes are needed beyond the base URL. Test that your streaming parser tolerates an extra done event some gateways emit.

8. Run shadow traffic before cutover

Stand up a dual-write harness. Send the same request to the old client and the gateway client, compare outputs, and log divergence.

def shadow(req: dict):
    legacy = legacy_client.chat.completions.create(**req)
    gateway = client.chat.completions.create(**req)
    if legacy.choices[0].message.content != gateway.choices[0].message.content:
        log.warning("drift", req, legacy, gateway)

Run this for a week on low-risk endpoints. Watch p95 latency; a gateway hop adds 5–15 ms typically, not a dealbreaker. Only then flip the default client in production. The whole point is to avoid OpenAI vendor lock-in gateway risk while proving parity.

9. Mock the gateway in unit tests

Use respx or httpx mock to assert the correct base URL and model qualifier.

import respx

@respx.mock
def test_route():
    respx.post("https://gateway.example.com/v1/chat/completions").mock(
        return_value=respx.Response(200, json={"choices": []})
    )
    client.chat.completions.create(model=route("gpt-4o"), messages=[])
    assert respx.calls.last.request.headers["authorization"].startswith("Bearer")

This catches regressions when someone “fixes” the client and hardcodes a model.

Common pitfalls and tradeoffs

Hidden defaults. The OpenAI SDK defaults temperature to 1.0; your gateway may default to 0.0. Pin every parameter you care about.

Non-chat APIs. The assistants API, fine-tuning, and embeddings have different shapes. A gateway focused on chat completions will not mirror them. Isolate those calls behind an if USE_GATEWAY flag and keep the OpenAI client for them.

Key scoping. Gateway keys often restrict routes or models. A key valid for openai/* will 401 on anthropic/*. Read the gateway’s auth error, not the SDK’s generic message.

Streaming termination. Some gateways send a final usage chunk; others do not. If you compute cost from the stream, reconcile with the response headers.

The tradeoff is clear: you avoid OpenAI vendor lock-in gateway dependency by adding a routing layer, but you accept a network hop and a thinner feature surface. For 90% of chat workloads, that is the right call.

Rollback plan

Keep the legacy client instantiated but dormant. A single environment variable flips the route() function to return bare names and the client constructor to use the old base URL. Test that flip in staging monthly. Vendor lock-in is a process problem, not a code problem—make exit cheap.