Migrating a raw OpenAI SDK app to support multiple providers

If your application talks directly to the OpenAI Python SDK, the client initialization and model name are probably baked into your business logic. To migrate openai sdk app to multi-provider routing, you need to extract those couplings and point the same API surface at a gateway that speaks the OpenAI protocol. This article walks through a step-by-step refactor that keeps your code changes minimal while unlocking access to dozens of models.

Step 1: Inventory your OpenAI SDK calls

Before changing anything, find every place you construct a client or call a completion endpoint. A typical raw app looks like this:

from openai import OpenAI

client = OpenAI(api_key="sk-...")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Summarize this ticket"}],
    temperature=0.2,
)
print(response.choices[0].message.content)

Grepping for OpenAI(, chat.completions.create, and hardcoded model= strings gives you the full blast radius. Most small services have one or two call sites. Larger ones may have the client instantiated per request inside helper functions—those are the ones that cause pain later.

The goal of this audit is to confirm that you only use the OpenAI-compatible subset: chat.completions.create, embeddings.create, and standard message shapes. If you rely on assistants, fine_tuning, or files, those endpoints are not portable to generic gateways and need separate handling.

Step 2: Extract a client factory

Replace direct OpenAI() instantiation with a function that reads configuration from the environment. This isolates the base URL and key, which is the only thing that changes when you migrate openai sdk app to multi-provider access.

import os
from openai import OpenAI

def get_llm_client() -> OpenAI:
    return OpenAI(
        api_key=os.environ["LLM_API_KEY"],
        base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
    )

Now every call site becomes get_llm_client().chat.completions.create(...). You haven’t changed behavior yet, but you’ve created a single seam to inject a different backend.

Step 3: Route through an OpenAI-compatible gateway

The fastest way to support multiple vendors without rewriting request logic is to target a gateway that implements the OpenAI REST contract. For example, n4n.ai provides one OpenAI-compatible endpoint covering 240+ models, with automatic fallback when a provider is rate-limited and per-token usage metering. You switch by setting LLM_BASE_URL and a gateway-specific key:

export LLM_BASE_URL="https://api.n4n.ai/v1"
export LLM_API_KEY="your-gateway-key"

Your existing get_llm_client() now sends traffic to the gateway. Model strings can stay as the provider-native IDs (e.g., gpt-4o, claude-3.5-sonnet) or use prefixed routing hints if the gateway supports them. No code change beyond the environment is required to fan out to multiple upstreams.

If you prefer self-hosting, LiteLLM or similar proxies work the same way—the SDK never knows it isn’t talking to OpenAI.

Step 4: Parameterize model selection

Hardcoding model="gpt-4o" defeats the purpose. Move model choices into configuration keyed by task:

{
  "summarization": "anthropic/claude-3.5-sonnet",
  "classification": "openai/gpt-4o-mini",
  "embeddings": "text-embedding-3-small"
}

Load it and pass the value through:

import json

with open("models.json") as f:
    MODEL_MAP = json.load(f)

def summarize(text: str) -> str:
    client = get_llm_client()
    resp = client.chat.completions.create(
        model=MODEL_MAP["summarization"],
        messages=[{"role": "user", "content": f"Summarize: {text}"}],
        temperature=0.2,
    )
    return resp.choices[0].message.content

This step is where the migrate openai sdk app to multi-provider effort pays off: swapping a model is a config edit, not a deploy.

Step 5: Normalize request and response differences

Providers diverge on details even inside the OpenAI shape. Three concrete gotchas:

1. JSON mode: response_format={"type": "json_object"} is honored by OpenAI and some Anthropic routes, but not all open-weight models. Detect support via config and omit otherwise.
2. Token limits: Older SDK defaults to max_tokens; some gateways forward max_completion_tokens for newer models. Pass max_tokens explicitly to avoid 400s.
3. Content parts: Vision models need content as a list of {"type": "image_url"} blocks. If you build messages dynamically, write a helper that only adds image parts when the selected model supports vision (track this in your config).

def build_messages(prompt: str, image_url: str | None = None):
    content = [{"type": "text", "text": prompt}]
    if image_url:
        content.append({"type": "image_url", "image_url": {"url": image_url}})
    return [{"role": "user", "content": content if image_url else prompt}]

Keep these normalizations at the edge of your call sites. Don’t leak provider specifics into domain logic.

Step 6: Add fallback or trust the gateway

If your gateway already does automatic fallback, you can skip manual retry. If you’re running a bare proxy or direct multi-client setup, catch errors and retry with a secondary model:

from openai import APIError

def chat_with_fallback(model: str, fallback: str, messages: list):
    client = get_llm_client()
    try:
        return client.chat.completions.create(model=model, messages=messages)
    except APIError as e:
        if e.status_code in (429, 503):
            return client.chat.completions.create(model=fallback, messages=messages)
        raise

When you migrate openai sdk app to multi-provider using a gateway with built-in degradation handling, this block becomes dead code—delete it after confirming the gateway’s behavior in load tests.

Step 7: Verify the migration

Verification is concrete: run a script that exercises each task model and asserts the response shape.

def test_routing():
    for task, model in MODEL_MAP.items():
        if task == "embeddings":
            continue
        resp = get_llm_client().chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": "ping"}],
        )
        assert resp.choices[0].message.content
        assert resp.usage.total_tokens > 0
        print(f"{task}:{model} ok, {resp.usage.total_tokens} tokens")

Run it with the gateway URL set. You should see each task return content and a non-zero token count. If you used n4n.ai or similar, the usage metering in the response confirms the gateway forwarded the call correctly. For embeddings, assert len(resp.data[0].embedding) > 0.

Beyond unit checks, watch logs for model fields that don’t match your config—some SDKs silently rewrite IDs. Finally, load test with pytest-xdist or a simple loop to confirm fallback triggers when you artificially cap rate limits.

Step 8: Clean up and document routing directives

Once stable, document which model backs which task and the cache-control hints you send. The OpenAI SDK forwards extra_headers, so you can pass provider cache directives through the gateway:

client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=messages,
    extra_headers={"x-cache-control": "ephemeral"},
)

Gateways that honor client routing directives will pass this to the upstream. Record these in your model config so the next engineer doesn’t guess.

The migrate openai sdk app to multi-provider path is mostly about removing hardcoded assumptions. After these steps, your app treats LLMs as swappable utilities, and a vendor outage becomes a config flip rather than a outage of your own service.