Teams rarely plan to run five LLM provider SDKs. They accumulate them: OpenAI for chat, Anthropic for long context, Cohere for rerank, a local vLLM for cheap classification, maybe Gemini for multimodal. The decision of one gateway vs five SDKs is where you choose to absorb complexity—scattered across your services or behind a single normalized interface. This guide gives an ordered path to consolidate without surrendering provider-specific leverage.
1. Audit what the SDKs actually do
Start by grepping your repos for import statements and HTTP clients. Most teams discover 80% of traffic uses three capabilities: chat completion, embeddings, and one custom endpoint like rerank or fine-tune. The remaining 20% touches provider-unique features: streaming tool calls, prompt caching, structured output, or batch inference.
# Before: three SDKs, three shapes, three error models
import openai, anthropic, cohere
openai.chat.completions.create(model="gpt-4o", messages=[])
anthropic.messages.create(model="claude-3-5-sonnet", max_tokens=1024, messages=[])
cohere.rerank(query="", documents=[])Map each call site to a capability, not a library. If you only use chat.completions and embeddings, consolidation is trivial. If you depend on Anthropic’s system block or Cohere’s return_documents, you need a gateway that forwards those fields or a wrapper that branches internally.
Pitfall: ignoring offline and batch endpoints. They often have separate auth, slower rate limits, and a gateway may not proxy them. Log every requests.post to provider domains before you plan the cutover.
2. Define the unified interface you need
Do not adopt a gateway’s entire surface area. Define a minimal internal client that matches your audit:
class LLMClient:
def complete(self, model: str, messages: list, **opts) -> Completion: ...
def embed(self, model: str, inputs: list) -> list: ...
def rerank(self, model: str, query: str, docs: list) -> list: ...Keep a typed extra: dict parameter on each method. This preserves access to cache-control, seed, or provider-specific stop sequences without polluting your core API.
def complete(self, model, messages, extra=None):
return self._backend.complete(model, messages, **(extra or {}))Trade-off: a too-thin interface forces escape hatches in every service; a too-fat one recreates the SDK mess. Target the 80% and document the escape hatch explicitly in your internal README.
3. Choose normalization: gateway or thin wrapper
A gateway gives one base URL, one auth token, one error model. A thin wrapper keeps five SDKs but fronts them with your LLMClient.
Gateways shine when you want automatic fallback. For example, a service like n4n.ai exposes one OpenAI-compatible endpoint covering 240+ models and will reroute when a provider is rate-limited or degraded, while still honoring client routing directives and provider cache-control hints. That removes retry scaffolding from your code and centralizes per-token usage metering.
A wrapper shines when you need debug fidelity: stepping into the exact SDK version, or using beta endpoints not yet proxied. It also avoids a new single point of failure outside your infrastructure.
Decision rule: if your team is small and model variety is high, start with a gateway. If you run regulated workloads with strict egress or need unreleased features, wrap the SDKs yourself with pinned versions.
4. Migrate provider-by-provider
Do not flip a flag and switch all traffic. Pick the lowest-risk path: embeddings. They are stateless and easy to diff against the old output.
# Old
vec = openai.embeddings.create(model="text-embedding-3", input=txt).data[0].embedding
# New (gateway, OpenAI-compatible)
vec = client.embeddings.create(model="openai/text-embedding-3", input=txt).data[0].embedding
assert cosine(vec, old_vec) > 0.999Then move chat completions for internal tools, not user-facing latency-sensitive ones. Keep the old SDK import behind a feature flag for at least two weeks.
if flag.enabled("use_gateway_chat"):
resp = client.chat.completions.create(model="anthropic/claude-3-5-sonnet", messages=msgs)
else:
resp = anthropic.messages.create(model="claude-3-5-sonnet", max_tokens=1024, messages=msgs)Common pitfall: assuming token counts match. Providers count whitespace and unicode differently; verify your cost metering against per-token usage reports before retiring the old path. A 2% drift on a million-token bill is real money.
5. Handle streaming, retries, and fallbacks
Streaming is where SDKs diverge most. OpenAI streams SSE with choices.delta. Anthropic streams content_block_delta. A gateway normalizes these to one format; a wrapper must translate each provider’s event shape.
# Gateway stream looks identical to OpenAI SDK
stream = client.chat.completions.create(model="mistral/large", stream=True, messages=msgs)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")If you use a wrapper, implement a single StreamAdapter per provider and never let application code see raw chunks. Otherwise you will duplicate parsing logic in ten places.
Retries: gateways often provide automatic fallback, but you still need idempotency keys for billing safety. Set max_retries=0 on the client and handle exponential backoff yourself if you cache responses. Fallback is not free—when a primary provider is down, the gateway calls a secondary that may be slower or pricier.
6. Preserve provider-specific leverage
Consolidation fails when you lose prompt caching or structured output. Forward cache hints explicitly through the unified client:
client.chat.completions.create(
model="anthropic/claude-3-5-sonnet",
messages=[{"role":"user","content":"..."}],
extra_headers={"x-cache-control": "ephemeral"} # forwarded by compliant gateway
)If your gateway does not forward such headers, keep that one call on the native SDK. Measure the cost: a missed cache can 10x latency and inflate the bill on long system prompts.
Similarly, OpenAI’s response_format for JSON mode and Anthropic’s thinking blocks must survive the abstraction. Add integration tests that assert the field appears in the upstream request log.
7. Watch the hidden costs
A gateway adds a proxy hop. Expect 10–30ms p99 overhead in the same region; cross-region calls can add 100ms. For same-region inference it is negligible; for edge services it stings.
Billing aggregation can hide which team burns tokens. Use per-token usage metering to tag by service or environment. If the gateway does not support labels, ship its access logs to your own warehouse and build the pivot table.
Single point of failure: if the gateway goes down, all providers are dark. Run a synthetic health check that, on failure, falls back to a direct SDK for one critical model. This hybrid keeps you alive during gateway incidents.
8. Decision checklist
Use one gateway vs five SDKs when:
- You support more than three providers in production.
- Your error handling and retry logic are duplicated across services.
- You want centralized fallback and metering without writing it.
Keep native SDKs when:
- You rely on unreleased provider features or private beta endpoints.
- Compliance forbids third-party proxy of request payloads.
- Your traffic is single-provider and stable enough that abstraction is pure overhead.
Consolidation is not all-or-nothing. A hybrid—gateway for long-tail models, native SDK for flagship latency-critical path—is often the right end state.
Common pitfalls summary
- Treating the gateway as a drop-in without testing token counts and latency.
- Flattening provider params into a lowest-common-denominator interface.
- Forgetting to forward cache-control, seed, or stop headers.
- Ignoring the proxy’s own rate limits and outage blast radius.
- Not keeping a kill switch to native SDKs for critical paths.
Pick the seam that matches your org, migrate incrementally, and keep the escape hatch. The goal is less code, not less control.