Replacing five provider SDKs with one gateway API

Most teams accumulate a stack of provider SDKs—OpenAI, Anthropic, Cohere, Google, Mistral—each with its own auth, pagination, and error shapes. The fastest way to cut that surface area is to replace provider SDKs with one gateway API that speaks the OpenAI protocol and routes to all of them behind a single base URL.

1. Audit what your SDK calls actually do

Before deleting anything, map every LLM call in your codebase. In practice, 90% of integrations are one of three things: chat completions, embeddings, or token counting. The rest is retry logic, timeout tuning, and ad-hoc type wrappers.

Pull a representative sample from each provider:

# Old: openai
openai.ChatCompletion.create(model="gpt-4o", messages=[...])

# Old: anthropic
anthropic_client.messages.create(model="claude-3-5-sonnet", max_tokens=1024, messages=[...])

# Old: google
genai_client.generate_content(model="gemini-1.5-pro", contents=[...])

The request schemas differ, but the semantic payload—messages, max tokens, temperature—is identical. That overlap is what makes consolidation possible.

Pitfall: Custom streaming parsers. If you wrote per-provider SSE handlers, those are the costliest lines to retire. Flag them explicitly.

2. Choose an OpenAI-compatible gateway

The OpenAI request/response format has become the lingua franca for LLM inference. A gateway that exposes an OpenAI-compatible REST surface lets you keep your existing OpenAI client and point it elsewhere.

A gateway like n4n.ai exposes a single OpenAI-compatible endpoint covering 240+ models and performs automatic fallback when a provider is rate-limited or degraded. You trade direct provider coupling for a thin routing layer. The tradeoff is real: the gateway is now a dependency, but you eliminate five others and gain cross-provider redundancy.

Verify the gateway supports:

• /v1/chat/completions with standard fields
• /v1/embeddings
• Streaming via SSE
• Model namespacing (e.g., anthropic/claude-3.5-sonnet)

3. Swap the client initialization

You do not need five clients. One openai Python package (or the TypeScript equivalent) with a different base_url is enough.

from openai import OpenAI

# Before: five clients
# from openai import OpenAI as OA
# import anthropic, cohere, google.generativeai as genai

# After: one client
client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # gateway endpoint
    api_key="your-gateway-key",
)

resp = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": "Summarize this PR"}],
    temperature=0.2,
)

In Node:

import OpenAI from "openai";
const client = new OpenAI({
  baseURL: "https://api.n4n.ai/v1",
  apiKey: process.env.GATEWAY_KEY,
});

Delete the provider-specific imports. Your CI will immediately show what code actually depended on them.

4. Normalize model identifiers

Provider SDKs use bare model names that collide (gpt-4o vs claude-3-5-sonnet). Gateways qualify them with a provider prefix. Centralize this in a config map so you can change routing without touching call sites.

{
  "chat_primary": "anthropic/claude-3.5-sonnet",
  "chat_cheap": "openai/gpt-4o-mini",
  "embed": "openai/text-embedding-3-small"
}

If the gateway honors client routing directives, you can pin a provider or request cache hints in the same call:

resp = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[...],
    extra_headers={"x-routing": "prefer:anthropic; cache-control: max-age=3600"},
)

Tradeoff: Namespacing couples your code to the gateway’s naming scheme. Keep the map behind a small helper so a provider rename is a one-line change.

5. Unify streaming and tool calls

Streaming is where provider differences bite. Anthropic streams content_block_delta; OpenAI streams choices[0].delta. The gateway abstracts this into the OpenAI SSE shape.

stream = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[...],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Function calling / tool use follows the same rule: define tools in OpenAI’s tools array and let the gateway translate to the target provider. Test at least one tool call per provider you use—translation bugs show up in argument parsing.

Pitfall: Some providers don’t support parallel tool calls or strict JSON mode. The gateway may silently downgrade. Validate the finish_reason and tool call structure in tests.

6. Configure fallback and rate-limit handling

Automatic fallback at the gateway reduces 429s, but your client still needs to handle a gateway-level failure. Wrap calls in a retry with backoff:

import time
def complete_with_retry(**kwargs):
    for attempt in range(3):
        try:
            return client.chat.completions.create(**kwargs)
        except Exception as e:
            if "rate_limit" in str(e).lower():
                time.sleep(2 ** attempt)
                continue
            raise
    raise RuntimeError("Gateway exhausted retries")

If you need explicit control, pass a fallback list via routing header (if supported) or implement a secondary model in code:

try:
    return complete_with_retry(model="anthropic/claude-3.5-sonnet", ...)
except RuntimeError:
    return complete_with_retry(model="openai/gpt-4o", ...)

Tradeoff: Client-side fallback duplicates gateway logic. Prefer gateway-native fallback; keep client retries only for network errors.

7. Consolidate usage tracking

Per-provider metering means five dashboards. A gateway with per-token usage metering returns standardized usage objects:

resp = client.chat.completions.create(model="openai/gpt-4o", messages=[...])
print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Pipe these to your own analytics or pull from the gateway’s metered billing endpoint. Either way, you get one cost view instead of five CSV exports.

Pitfall: Token counts differ slightly across providers for the same text. Don’t assert exact equality in tests; assert ranges.

8. Delete the old SDK dependencies

Once tests pass against the gateway, remove the packages:

pip uninstall anthropic cohere google-generativeai

Update lockfiles, strip provider env vars from deploy configs, and add a lint rule forbidding imports of those packages. Keep one integration test per model family to catch gateway regressions early.

Common pitfalls when you replace provider SDKs with one gateway API

• Timeout defaults: Provider SDKs often set generous timeouts. The OpenAI client defaults to 10 minutes, but gateways may proxy slower providers. Set timeout=30 explicitly.
• Logging sensitive prompts: Centralizing calls means one log stream sees everything. Scrub before logging at the gateway boundary.
• Version drift: The OpenAI client library evolves. Pin it; gateways lag slightly on new fields like response_format variants.
• Cold-start latency: Routing through a gateway adds one network hop. Measure p99 before and after; usually <30ms but non-zero.

The ordered path is straightforward: audit, pick a compatible gateway, swap clients, map models, unify streams, handle fallback, consolidate metering, delete SDKs. Doing this cuts your integration code by roughly half and turns five vendor negotiations into one.