Migrating from direct OpenAI integration to a gateway

To migrate from direct OpenAI integration to gateway, you need to change a handful of call sites and add resilience logic without rewriting business logic. This guide walks through the concrete steps we used to shift a production Python service off api.openai.com and onto an OpenAI-compatible inference gateway, keeping the same SDK and request shapes.

Step 1: Audit your existing OpenAI call sites

Start by finding every place your code talks to OpenAI. In a Python codebase using the official SDK, that means locating OpenAI() client instantiations and the model strings passed to chat.completions.create.

grep -rn "openai" --include="*.py" . | grep -i "OpenAI("
grep -rn "model=" --include="*.py" . | grep -E "gpt-|text-embedding"

A typical direct integration looks like this:

from openai import OpenAI

client = OpenAI(api_key="sk-...")  # defaults to https://api.openai.com/v1

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this ticket"}],
    temperature=0.2,
)

Note three things you will need to preserve: the model identifier, the request parameters (temperature, max_tokens, etc.), and any assumptions about response shape. Gateways speak the same OpenAI REST contract, so the response object is unchanged. The only mandatory swap is where the client sends bytes.

Step 2: Swap the base URL and credentials

The fastest path is to keep the OpenAI SDK and point it at a gateway endpoint. Set base_url and use a gateway-issued key. A gateway such as n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models, so the same client works for many providers without vendor-specific branches.

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["LLM_GATEWAY_URL"],   # e.g. https://api.n4n.ai/v1
    api_key=os.environ["GATEWAY_API_KEY"],
)

Do not hardcode the URL. Read it from environment or config so you can run tests against a mock. If you previously set openai.api_key globally, delete that pattern; explicit client objects are easier to reason about during a migration.

Verify the client connects by sending a trivial request in a REPL:

r = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "ping"}],
)
print(r.choices[0].message.content)

If you get a completion, the transport swap worked.

Step 3: Externalize model routing

Direct integrations tend to scatter model names like "gpt-4o" across the code. A gateway earns its keep when you treat model names as routing hints rather than hard contracts. Define a small config layer:

{
  "default_chat_model": "gpt-4o-mini",
  "fallback_chat_model": "gpt-4o",
  "embedding_model": "text-embedding-3-small"
}

Load it and pass the resolved string to the client. Gateways that honor client routing directives let you send a qualified name (e.g. openai/gpt-4o-mini) or rely on the gateway’s own mapping. Either way, your application code should reference a logical key, not a provider SKU.

If you use prompt caching, forward provider cache-control hints through the gateway. With the OpenAI SDK you can pass extra headers:

resp = client.chat.completions.create(
    model=cfg["default_chat_model"],
    messages=[...],
    extra_headers={"x-cache-control": "ephemeral"},
)

A correct gateway forwards those hints to the upstream provider instead of stripping them.

Step 4: Handle authentication and per-token metering

Gateways issue their own API keys, and they meter usage per token. You should capture the usage block on each response for your own cost attribution:

resp = client.chat.completions.create(
    model=cfg["default_chat_model"],
    messages=payload,
)
usage = resp.usage
# usage.prompt_tokens, usage.completion_tokens, usage.total_tokens
logger.info("llm_call", extra={"model": resp.model, "usage": usage.model_dump()})

Do not assume the model field in the response equals the string you sent. Gateways performing automatic fallback when a provider is rate-limited or degraded will return the actual served model. Log it. If your billing system expects a fixed model tag, reconcile against the gateway’s per-token usage metering instead of the request parameter.

Step 5: Add minimal client-side resilience

A gateway handles upstream degradation, but your client still needs to cope with gateway-level 429s or network blips. You do not need a heavy framework; a tight retry loop with backoff suffices.

import time
from openai import APIError, RateLimitError

def complete_with_retry(client, **kwargs):
    for attempt in range(3):
        try:
            return client.chat.completions.create(**kwargs)
        except RateLimitError:
            time.sleep(2 ** attempt)
        except APIError as e:
            if attempt == 2:
                raise
            time.sleep(2 ** attempt)
    raise RuntimeError("exhausted retries")

Keep the retry count low. If the gateway is already doing provider fallback, repeated client retries mostly add latency. Treat 5xx as retryable, 4xx (except 429) as fatal.

Step 6: Update tests and staging verification

Replace recorded cassettes or mocked api.openai.com hosts with your gateway URL. In pytest, point the client at a local stub:

import pytest
from openai import OpenAI

@pytest.fixture
def gateway_client(monkeypatch):
    monkeypatch.setenv("LLM_GATEWAY_URL", "http://localhost:8080/v1")
    monkeypatch.setenv("GATEWAY_API_KEY", "test")
    return OpenAI()

Write a smoke test that asserts the request left for the gateway, not OpenAI:

def test_base_url_used(gateway_client):
    assert "localhost:8080" in gateway_client.base_url

Run your full suite against staging with the real gateway in shadow mode: duplicate traffic, compare responses semantically, and confirm token counts are within expected bounds.

Step 7: Cutover and monitor

Flip the environment variable in production. Keep the old OpenAI key disabled but accessible in case you need to revert. Watch three signals for the first 24 hours:

• Error rate on the gateway path vs the historical OpenAI direct path.
• usage.total_tokens per request type—gateways may route to a different model than you requested.
• p95 latency; fallback can add a hop.

If you see sustained 429s from the gateway itself, you have a quota issue, not a code bug.

Verifying success

Success means your application code no longer references api.openai.com, all model names resolve through config, and a single base_url change routes traffic. Concrete checks:

1. grep -rn "api.openai.com" . returns nothing.
2. A canary request with extra_headers={"x-test": "true"} appears in gateway logs with correct routing.
3. resp.usage.total_tokens > 0 on a real call, and the logged model matches the served model after fallback.
4. Your test suite passes with the gateway stub and with the live gateway in staging.

The migration is complete when you can swap providers by editing a JSON file rather than touching Python. That is the real payoff of choosing to migrate from direct OpenAI integration to gateway architecture: vendor lock-in becomes a configuration concern, not a code rewrite.