Migrating from the OpenAI SDK to a unified LLM gateway

When you migrate OpenAI SDK to a unified gateway, the first surprise is how little application code changes. The OpenAI Python and TypeScript clients speak an HTTP dialect that most gateways implement verbatim, so the work is configuration, not refactoring. This guide walks through a complete cutover, from auditing your calls to verifying fallback behavior with real models like GPT-5, Claude Opus 4.8, Gemini 3, and Llama 4.

Step 1: Audit your current OpenAI SDK usage

Find every place you construct a client and every hardcoded model string. Most codebases have one or two client factories and a sprawl of model="gpt-4o" calls. Run a quick static scan:

rg "OpenAI\(" -l
rg "chat\.completions\.create" -l
rg "model=\"[a-z0-9-]+\"" -o

Capture which endpoints you use beyond chat: embeddings, audio transcription, and image generation often are not supported by every gateway. A unified gateway typically covers chat and completions first.

from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this PR"}],
    temperature=0.2,
    response_format={"type": "json_object"},
)

Note the parameters you rely on: temperature, response_format, tools, stream, seed. The unified gateway must support these unchanged, or you need an adapter.

Step 2: Swap the base URL and API key

Set a single environment variable for the gateway endpoint and key. The OpenAI SDK reads base_url from the constructor; nothing else in the call site changes. Keep the old key available behind a flag during canary testing.

import os
from openai import OpenAI

USE_GATEWAY = os.environ.get("USE_LLM_GATEWAY") == "1"

client = OpenAI(
    base_url=os.environ["LLM_GATEWAY_URL"] if USE_GATEWAY else None,
    api_key=os.environ["LLM_GATEWAY_KEY"] if USE_GATEWAY else os.environ["OPENAI_API_KEY"],
)

For TypeScript:

import OpenAI from "openai";

const useGateway = process.env.USE_LLM_GATEWAY === "1";
const client = new OpenAI({
  baseURL: useGateway ? process.env.LLM_GATEWAY_URL : undefined,
  apiKey: useGateway ? process.env.LLM_GATEWAY_KEY : process.env.OPENAI_API_KEY,
});

Export the vars in your shell or CI:

export LLM_GATEWAY_URL="https://api.example.com/v1"
export LLM_GATEWAY_KEY="sk-gw-..."
export USE_LLM_GATEWAY=1

If your gateway is OpenAI-compatible, the chat.completions.create method works without edits. This is the core of how you migrate OpenAI SDK to a unified gateway with minimal risk.

Step 3: Remap model identifiers

Unified gateways namespace models by provider. Instead of gpt-4o, you pass openai/gpt-5 or a gateway-specific alias. Centralize the mapping so you can add Gemini 3 or Llama 4 without touching call sites.

Legacy string	Gateway model ID
`gpt-4o`	`openai/gpt-5`
`claude-3-opus`	`anthropic/claude-opus-4-8`
`gemini-pro`	`google/gemini-3`
`llama-3-70b`	`meta/llama-4`

MODEL_MAP = {
    "gpt-4o": "openai/gpt-5",
    "claude-3-opus": "anthropic/claude-opus-4-8",
    "gemini-pro": "google/gemini-3",
    "llama-3-70b": "meta/llama-4",
}

def mapped(model: str) -> str:
    return MODEL_MAP.get(model, model)

Call sites become:

resp = client.chat.completions.create(
    model=mapped("gpt-4o"),
    messages=[{"role": "user", "content": "Ping"}],
)

Route by task, not by habit. GPT-5 leads on reasoning, Claude Opus 4.8 on long context, Gemini 3 on multimodal, Llama 4 on self-hosted cost.

Step 4: Handle streaming and response shapes

Streaming works the same way—iterate the async generator. Verify that chunk.choices[0].delta carries the expected fields. Some gateways add provider metadata in response_headers.

stream = client.chat.completions.create(
    model="anthropic/claude-opus-4-8",
    messages=[{"role": "user", "content": "Stream a haiku"}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

If you use response_format={"type": "json_object"}, test it against each provider. Claude and Gemini accept it via the gateway’s translation layer, but edge cases exist with strict schemas. Also check finish_reason mapping: stop, length, and tool_calls should surface identically.

For async apps:

async for chunk in await client.chat.completions.create(
    model="google/gemini-3",
    messages=[{"role": "user", "content": "Stream"}],
    stream=True,
):
    ...

Step 5: Implement fallback and routing directives

Providers fail. A unified gateway should automatically reroute when a backend is rate-limited. With n4n.ai, a single OpenAI-compatible endpoint addresses 240+ models and triggers automatic fallback when a provider is degraded, so you don’t write retry loops.

You can also force routing with headers if you need deterministic provider selection:

resp = client.chat.completions.create(
    model="openai/gpt-5",
    messages=[{"role": "user", "content": "Critical task"}],
    extra_headers={"X-Route-Preference": "openai,anthropic"},
)

The gateway honors client routing directives and forwards provider cache-control hints. If you use prompt caching on Claude, set extra_headers={"cache_control": "ephemeral"} and the gateway passes it through. Disable fallback explicitly when debugging:

extra_headers={"X-Allow-Fallback": "false"}

Step 6: Meter usage and enforce budgets

Per-token metering is non-negotiable when you span four providers. The gateway returns usage in the standard OpenAI shape; log it centrally.

usage = resp.usage
print(f"prompt:{usage.prompt_tokens} completion:{usage.completion_tokens}")

Wrap the client in a thin middleware to emit metrics:

def metered_create(client, **kwargs):
    resp = client.chat.completions.create(**kwargs)
    statsd.incr("llm.tokens", resp.usage.total_tokens, tags=[f"model:{kwargs['model']}"])
    return resp

Set a monthly token budget via the gateway API. Your app should handle 429 with Retry-After gracefully—the SDK does this by default, but tune max_retries=3 and timeout=30.

Step 7: Verify the migration

Success means identical output shape, working fallback, and correct metering. Write a smoke test that hits each provider model and asserts the response structure.

import pytest

@pytest.mark.parametrize("model", [
    "openai/gpt-5",
    "anthropic/claude-opus-4-8",
    "google/gemini-3",
    "meta/llama-4",
])
def test_gateway_model(client, model):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": "Return the word OK"}],
        temperature=0,
    )
    assert resp.choices[0].message.content
    assert resp.usage.total_tokens > 0

To verify fallback, temporarily block one provider via gateway config (or use a bad model suffix) and confirm the request succeeds with a different backend. Check the X-Provider response header if your gateway emits one. Run the suite with USE_LLM_GATEWAY=1.

Step 8: Roll out gradually

Flip the USE_LLM_GATEWAY flag for 5% of traffic via your feature flag system. Compare latency percentiles and error rates against the direct OpenAI baseline for a day. Then ramp to 100%.

if flag_enabled("llm_gateway", user_id):
    client = gateway_client
else:
    client = openai_client

Keep the legacy client instantiation until the gateway proves stable across all four model families.

Common pitfalls

Timeout tuning. Default SDK timeout is 10 minutes; gateways often have stricter upstream limits. Set timeout=30 on the client.

Cache headers lost. If you rely on OpenAI’s system_fingerprint for caching, note that unified gateways map cache hints per provider. Forward cache_control explicitly.

Tool calling differences. Gemini 3 supports parallel tool calls differently than GPT-5. Test your agent loops against each model before flipping traffic.

Logging PII. Gateway logs may capture prompts for debugging. Redact before sending if you migrate openai sdk unified gateway calls in regulated environments.

SDK version drift. OpenAI SDK v1.x changed the client constructor. Pin openai>=1.0 and update imports from openai.ChatCompletion to client.chat.completions.

Final verification checklist

All client instantiations use the gateway base URL behind a flag
Model map covers every legacy string including GPT-5, Claude Opus 4.8, Gemini 3, Llama 4
Streaming and JSON mode tested per provider
Fallback verified by injecting a provider outage
Usage tokens logged and budget enforced
Integration test passes for all four model families
Canary at 5% shows no regression in p95 latency

Cutting over takes an afternoon if your calls are centralized. The payoff is one contract, many models, and no 2 a.m. provider outage pages.