Swapping your base_url to point at a gateway feels like a one-line change, but the openai sdk migration gotchas surface weeks later when a provider goes down or a model rejects a parameter you thought was universal. The OpenAI SDK encodes assumptions about a single backend; a unified gateway breaks several of them in ways that are invisible until production traffic hits edge cases.
The compatibility illusion
OpenAI-compatible means the request path /v1/chat/completions accepts similar JSON and returns a similar shape. That compatibility is real for the happy path. It collapses the moment you depend on behavior the SDK implicitly guarantees rather than the wire protocol.
The SDK validates requests against OpenAI’s documented schema. A gateway that fronts 240+ models cannot enforce the same constraints, because not every underlying provider shares OpenAI’s parameter ranges, response fields, or error vocabulary. Your client code likely trusts those constraints.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.example/v1",
api_key="sk-gateway-123"
)
# This worked against api.openai.com; on a gateway it may route to a model that ignores `logprobs`
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": "Summarize this"}],
logprobs=True
)The call may succeed and silently drop the unsupported field, or return a 400 with a different error body. Both break naive expectations.
Error shapes and status codes
The OpenAI SDK raises typed exceptions: RateLimitError, APITimeoutError, BadRequestError. Those types map to specific status codes and error.type strings from OpenAI. A gateway sits between you and the provider, so it may transform, wrap, or synthesize errors.
A provider-side 429 becomes a 429 at the gateway, but the error.message and error.type reflect the upstream provider, not OpenAI. If your retry logic keys off err.type == "rate_limit_exceeded", you may miss err.type == "throttled" from a different backend.
from openai import APIStatusError
try:
client.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":"hi"}])
except APIStatusError as e:
body = e.response.json()
# Gateway may return: {"error": {"type": "upstream_rate_limit", "provider": "openai"}}
print(e.status_code, body["error"].get("type"))Another silent break: model fallback. If the gateway substitutes gpt-4o-mini when gpt-4o is saturated, the response model field will not match your request. Tests that assert resp.model == "gpt-4o" fail intermittently.
Streaming and SSE nuances
Streaming is where most teams get burned. The OpenAI SDK streams Server-Sent Events with choices[0].delta chunks. The gateway must normalize streams from providers with different chunking cadences, finish-reason semantics, and usage reporting.
OpenAI supports stream_options={"include_usage": True} to emit a final chunk with token counts. Many non-OpenAI providers do not support that flag. The gateway may omit usage entirely in the stream, or synthesize it after the fact. Your token logging silently records zeros.
stream = client.chat.completions.create(
model="mistral/mixtral-8x7b",
messages=[{"role": "user", "content": "Explain raft"}],
stream=True,
stream_options={"include_usage": True}
)
for chunk in stream:
if chunk.usage:
print(chunk.usage.prompt_tokens) # None for providers that don't support usage in streamFinish reasons also diverge. OpenAI sends stop, length, content_filter, tool_calls. A gateway might surface error or max_tokens from an upstream that uses different terminology. If your completion pipeline only checks for stop, truncated generations slip through as successes.
Token counting and usage metering
OpenAI’s usage.prompt_tokens is computed with OpenAI’s tokenizer. A gateway routing to Claude or Llama uses the provider’s tokenizer, which counts differently for the same text. Your cost dashboards built on OpenAI token math will be wrong.
Per-token metering at the gateway level is the source of truth for billing, not the SDK response. A gateway such as n4n.ai provides per-token usage metering and honors client routing directives, so you can tag requests for internal chargeback instead of trusting the response object.
Prompt caching is another leak. OpenAI exposes prompt_tokens_details.cached_tokens in usage. Anthropic uses cache_control breakpoints in the message body. The gateway forwards provider cache-control hints, but the OpenAI SDK has no native field for it. You pass it via extra_body:
client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Long context here..."}
],
extra_body={"cache_control": {"type": "ephemeral", "prefix": True}}
)If you migrate without auditing where cached token counts come from, you will misattribute savings.
Provider-specific parameters
The SDK rejects parameters it does not recognize unless placed in extra_body. That is actually helpful, but it means your gateway calls accumulate extra_body plumbing that varies per model. Temperature bounds differ: OpenAI caps at 2; some open-weight models accept higher. response_format with strict JSON schema is OpenAI-only; a gateway may pass it to a model that ignores it and returns free text.
# Passes SDK validation, but behavior depends entirely on routed provider
client.chat.completions.create(
model="meta/llama-3.1-70b",
messages=[{"role": "user", "content": "Output JSON"}],
response_format={"type": "json_object"},
extra_body={"repeat_penalty": 1.1}
)Your integration tests must assert on actual output shape, not on the SDK call succeeding.
Retries, timeouts, and fallback
The OpenAI SDK retries on 429 and 5xx with exponential backoff by default (max_retries=2). A gateway often performs its own automatic fallback when a provider is rate-limited or degraded. If you keep SDK retries enabled, you stack two recovery layers: the SDK retries a gateway that already retried a different provider, multiplying latency and possibly double-charging tokens.
Decide who owns resilience. If the gateway handles fallback, disable SDK retries:
client = OpenAI(
base_url="https://gateway.example/v1",
api_key="sk-gateway-123",
max_retries=0,
timeout=30.0
)If you need fine-grained control over which models are allowable fallbacks, use routing headers instead of catching and re-calling:
client = OpenAI(
base_url="https://gateway.example/v1",
api_key="sk-gateway-123",
default_headers={"X-Route-Allow": "gpt-4o,gpt-4o-mini"}
)Client code changes
The mechanical change is small:
# Before
client = OpenAI(api_key="sk-openai-123")
# After
client = OpenAI(
base_url="https://gateway.example/v1",
api_key="sk-gateway-123"
)But you should also centralize model names. Gateway model identifiers are namespaced (provider/model), not bare (gpt-4o). Hardcoded model strings across your codebase become a migration tax. Put them in a config layer.
Tradeoffs
A unified gateway buys you provider diversity, failover, and a single billing surface. You lose provider-native SDK conveniences: OpenAI’s client.beta.assistants, Realtime API, or fine-tune endpoints are not gateway concerns. If your system is single-model and single-provider, the migration adds complexity with no payoff.
If you are already juggling two or more providers via conditional code, the gateway removes more complexity than it introduces. The openai sdk migration gotchas are concentrated in the first two weeks: error typing, stream usage, and token accounting. After that, the heterogeneity is explicit rather than hidden in if provider == "openai" branches.
Takeaway
Treat the gateway as a distinct backend, not a transparent proxy with a different DNS name. Before cutover, write tests that assert on gateway response shapes for errors, streaming finish reasons, and usage fields. Disable redundant retries so the gateway’s fallback is the only recovery path. Audit every response_format and logprobs usage, and move model names into configuration. The openai sdk migration gotchas are real but bounded; teams that confront them explicitly ship a more resilient inference layer than the one they left behind.