The decision to switch from direct provider APIs to a unified gateway usually starts with a pile of half-maintained SDK wrappers and a spreadsheet of rate limits. You can cut that surface area by routing every model call through a single OpenAI-compatible endpoint, but the migration needs a plan or you’ll trade one mess for another.
Audit what you actually call
Before changing any code, list every provider endpoint your services hit. Most teams discover they use three providers but only five models total, plus a handful of abandoned experiments that nobody deleted. Pull request logs from your API gateway, or grep your repos for client instantiations:
# Old pattern: one client per provider, scattered across services
openai_client = OpenAI(api_key=os.environ["OPENAI_KEY"])
anthropic_client = Anthropic(api_key=os.environ["ANTHROPIC_KEY"])
# plus raw requests to a self-hosted vLLM at http://gpu-1:8000Record the exact model strings, the parameters passed (temperature, max_tokens, stop, top_p), and whether you use streaming or function calling. This inventory is the contract your unified gateway must satisfy. If you skip this step, you will learn about a missing claude-2.1 call when a customer hits it post-migration.
Pay attention to non-chat endpoints too. Embeddings, audio transcription, and image generation often have different base paths. A gateway that only normalizes chat completions forces you to keep direct clients for the rest. Note those exceptions; they stay on direct paths until the gateway supports them.
Choose a gateway that speaks OpenAI
The fastest way to switch from direct provider APIs to a unified gateway is to pick one that is OpenAI-compatible. Your existing openai Python or TypeScript client can target a different base_url with no other changes for basic chat completions. That single property eliminates most response-parsing rewrites.
from openai import OpenAI
client = OpenAI(
base_url="https://api.gateway.example/v1", # your gateway
api_key=os.environ["GATEWAY_KEY"],
)
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "ping"}],
)If the gateway fronts 240+ models through that single shape, you avoid rewriting response parsing. Some gateways, including n4n.ai, expose exactly this: one OpenAI-compatible endpoint addressing hundreds of models, forwarding provider cache-control hints without extra plumbing. You still need to verify behavior, but the client code stays identical.
Avoid gateways that require a custom SDK unless they also offer an OpenAI shim. A custom SDK locks you in deeper than the providers you are leaving.
Map model names and provider params
Provider model IDs are not standardized. Anthropic uses claude-3-5-sonnet-20241022, OpenAI uses gpt-4o, and a self-hosted model might be local/mistral-7b. Your gateway will have its own naming scheme, often prefixed by provider. Document the translation table in one place:
{
"openai/gpt-4o": "gpt-4o",
"anthropic/claude-3-5-sonnet": "claude-3-5-sonnet-20241022",
"local/mistral": "mistral-7b-instruct"
}Read this from config; never hardcode strings across handlers. When you switch from direct provider APIs to a unified gateway, a central map prevents drift.
Provider-specific parameters also diverge. Anthropic accepts system as a top-level field; OpenAI expects it inside messages. A good gateway normalizes this, but verify. If you send extra_body={"system": "..."} to an OpenAI client pointed at a gateway, ensure the gateway translates it rather than passing invalid JSON to the upstream. Test with a system prompt that changes behavior visibly.
Handle streaming and function calling
Streaming works over SSE in both OpenAI and most gateways, but chunk shapes can differ for tool calls. Test your parser against the gateway’s raw stream before flipping traffic.
stream = client.chat.completions.create(
model="gpt-4o",
messages=[...],
tools=[{"type": "function", "function": {"name": "get_weather", "parameters": {...}}}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta
if delta.tool_calls:
# accumulate name and arguments across chunksIf you previously used Anthropic’s stream_events API, you’ll need to adapt to the OpenAI chunk format. Budget time for this; it’s where most subtle bugs hide. In TypeScript, the openai package’s chat.completions.stream helper works against a compatible gateway, but confirm that tool_calls deltas are emitted incrementally and not buffered.
Function calling also differs in argument serialization. Some providers stream partial JSON strings; others wait. Your agent loop must tolerate both. Write a test that asserts a tool call completes with valid JSON after the stream ends.
Implement routing and fallback
A key reason to switch from direct provider APIs to a unified gateway is centralized fallback. Instead of catching RateLimitError in your code and retrying another provider, the gateway can do it. But blindly enabling fallback hides cost spikes.
Set explicit routing directives when you have a preference, and let the gateway fall back only on 429/5xx:
resp = client.chat.completions.create(
model="anthropic/claude-3-5-sonnet",
messages=[...],
extra_headers={
"x-routing-preference": "provider:anthropic",
"x-fallback": "allowed",
},
)Gateways that honor client routing directives let you keep control. n4n.ai automatically falls back when a provider is degraded but respects your preference header first. Use that during partial migrations so you don’t surprise your finance team with a sudden shift to a premium model.
Define fallback policy per route. A background summarization job can fall back to a cheaper model; a user-facing agent should not. Encode that in headers or gateway config, not in scattered try/except blocks.
Migrate behind a shim
Don’t rewrite all call sites at once. Insert a thin wrapper that swaps the client based on an env flag:
def get_client():
if os.environ.get("USE_GATEWAY") == "1":
return OpenAI(base_url=GATEWAY_URL, api_key=GATEWAY_KEY)
return OpenAI(api_key=OPENAI_KEY) # legacy direct pathRun both paths in production for a subset of traffic. Compare responses and latencies. Only when the gateway path is stable for a week do you remove the legacy branch. This incremental switch from direct provider APIs to a unified gateway contains blast radius.
Use a header or tag on requests so your observability stack can split metrics by path. You want to see gateway p95 versus direct p95 on the same endpoint.
Watch the tradeoffs
Latency and TLS overhead
Adding a proxy hop adds 10–30ms typically, but cross-region calls can add more. Measure p95 before and after. If your gateway is in a different continent than your compute, negotiate a private link or accept the cost.
Cache-control and provider hints
Providers like OpenAI and Anthropic support prompt caching via specific headers or body fields. A unified gateway should forward those hints; otherwise you lose cache hits and pay full price. Verify the gateway passes cache_control in the body or x-cache headers.
client.chat.completions.create(
model="claude-3-5-sonnet",
messages=[
{"role": "user", "content": "long context..."},
{"role": "user", "content": "question",
"extra_body": {"cache_control": {"type": "ephemeral"}}}
],
)If the gateway strips unknown fields, your cache rate drops. Test with a repeated prefix and check token usage in the response. A gateway that honors provider cache-control hints keeps your bill predictable.
Per-token metering
You need per-token usage metering to attribute cost by team or feature. A gateway that returns accurate usage objects and aggregates them saves you building your own meter. Confirm the gateway’s usage matches provider billing, especially for cached tokens which some providers discount. Discrepancies of even 5% matter at scale.
Reconcile gateway metered tokens against provider invoices monthly. The switch from direct provider APIs to a unified gateway consolidates billing, but it also creates a new middle layer that can miscount.
Common pitfalls
Silent model drift. A gateway might map gpt-4 to a newer snapshot without telling you. Pin versions in your translation table and alert on unknown model IDs.
Missing function-call streaming. Some gateways buffer tool calls and emit them only at the end. Your agent loop breaks if it expects incremental deltas. Write a streaming test that fails on buffering.
Auth leakage. Never log the gateway key. When you centralize, one leaked key compromises all providers. Rotate and scope it to specific routes.
Over-reliance on fallback. Automatic fallback can mask a broken primary provider for weeks while you burn 5x cost on a premium model. Set alerts on fallback rates and page on sustained diversion.
Ignoring rate limits on the gateway itself. The gateway has its own limits separate from providers. Treat its 429s as first-class. Backoff with jitter, same as you did with providers.
Assuming embeddings are covered. Many gateways focus on chat. If you use embeddings, confirm the gateway proxies them or keep that client direct.
Final checklist
- Inventory models, params, and non-chat endpoints.
- Stand up gateway with OpenAI-compatible endpoint.
- Build model-name map and shim with env toggle.
- Validate streaming, tools, and cache headers on shadow traffic.
- Enable fallback with routing preferences per route criticality.
- Flip production gradually; monitor p95, error rate, and token cost.
- Delete direct client code and reconcile first invoice.
The switch from direct provider APIs to a unified gateway pays off when you treat the gateway as infrastructure, not a black box. Audit, map, test, and keep control of routing.