Consolidating multi-provider integrations: a migration guide

Most teams accumulate provider-specific SDKs as they experiment with different LLMs, ending up with brittle glue code and duplicated auth logic. This consolidating multi-provider integrations guide lays out an ordered migration path from scattered client libraries to a single inference gateway, without rewriting your application logic.

Audit your current integration surface

Start by mapping every direct call to an LLM provider. Grep your repos for openai, anthropic, cohere, google.generativeai, or whatever SDKs you installed. Record the model strings, parameter shapes, retry wrappers, and timeout settings.

A simple inventory file helps:

services:
  - name: summarizer
    provider: anthropic
    sdk: "@anthropic-ai/sdk"
    models: [claude-3-5-sonnet]
    call_sites: [src/summarize.ts:42]
  - name: router
    provider: openai
    sdk: "openai"
    models: [gpt-4o-mini]
    call_sites: [src/route.py:88]
  - name: embedder
    provider: cohere
    sdk: "cohere"
    models: [embed-english-v3]
    call_sites: [src/vector.py:21]

The pitfall here is shadow calls inside serverless functions, background workers, or notebook experiments. If a cron job hits a provider directly, your gateway migration will silently miss it. Pull provider spend from billing APIs for the last 30 days and reconcile against this list. Any line item with no matching call site is a leak.

Define a unified request shape

The OpenAI chat completions schema has become the lingua franca. Even if you currently use Anthropic’s system/user blocks or Cohere’s chat endpoint, map them to messages with role. Keep your internal domain objects unchanged; write a thin adapter that translates intent to the unified shape.

# Before: anthropic SDK
# client.messages.create(model="claude-3-5-sonnet", system="You are terse.",
#                        messages=[{"role":"user","content":"Hi"}])

# After: OpenAI-compatible
payload = {
    "model": "anthropic/claude-3-5-sonnet",
    "messages": [
        {"role": "system", "content": "You are terse."},
        {"role": "user", "content": "Hi"}
    ]
}

Embeddings need the same treatment. Map Cohere’s texts array to OpenAI’s input list:

# Cohere: cohere.embed(texts=["a","b"], model="embed-english-v3")
openai_payload = {"model": "cohere/embed-english-v3", "input": ["a", "b"]}

Do not let provider field names leak into business logic. If a product manager asks for “temperature 0.2”, that should be a config value, not a scattered keyword argument.

Stand up an OpenAI-compatible gateway

Point all clients at one base URL. This removes per-provider base URLs, API key management, and model name spelling differences. An OpenAI-compatible endpoint that addresses 240+ models lets you keep the same model string convention while the gateway resolves the backend.

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.example.com/v1",
    api_key=os.environ["GATEWAY_KEY"]
)

resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Status?"}]
)

Standardize the auth header. Most gateways accept Authorization: Bearer, same as OpenAI. Rotate keys at the gateway level, not in every microservice. Tradeoff: you lose provider-native SDK conveniences like typed tool-calling helpers. You can re-implement those on the client side or use the gateway’s extension fields.

Migrate read paths first

Move non-mutating workloads—summarization, embeddings, classification—before any agent that triggers side effects. This limits blast radius. Use feature flags to route a percentage of traffic:

if flags.use_gateway("summarizer"):
    client = gateway_client
else:
    client = legacy_anthropic_client

Run both paths in parallel for a week. Diff the outputs for a fixed input set. If your eval harness shows <1% semantic drift, flip the flag permanently. For embeddings, compare cosine similarity between legacy and gateway outputs; a drop below 0.999 means a mapping bug.

Handle provider-specific features via routing directives

Some models need extra controls: seed, cache hints, or safety thresholds. A good gateway honors client routing directives and forwards provider cache-control hints. Pass these as extensions, not as magic query params.

{
  "model": "anthropic/claude-3-5-sonnet",
  "messages": [{"role": "user", "content": "Long doc..."}],
  "extensions": {
    "provider_route": {"prefer": ["anthropic"], "fallback": ["openai"]},
    "cache_control": {"type": "ephemeral", "ttl": 300}
  }
}

Do not embed provider names as hardcoded strings across your codebase. Centralize them in a routing config so you can shift traffic when one provider degrades. If you use Anthropic prompt caching, confirm the gateway translates your cache_control block into the provider’s native field; otherwise you pay full token cost.

Implement fallback and degradation testing

Automatic fallback when a provider is rate-limited or degraded is a gateway feature you should still verify. n4n.ai provides this, but your code must handle the latency tail and possible model swap gracefully. Write a chaos test that disables a provider key at the gateway level:

curl -X POST https://gateway.example.com/admin/simulate \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -d '{"provider":"anthropic","status":503}'

Then assert your app still returns a completion (possibly from openai/gpt-4o) within your SLA. If your retry loop doubles the timeout, you’ll see it here. Set a retry budget: max two attempts, then return cached or degraded response. Never let a fallback cascade into a 30-second hang for a user-facing request.

Meter and attribute usage

Per-token usage metering is non-negotiable for cost control. Read the x-usage response header or the usage field in the JSON body.

curl -i https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "x-tenant-id: team-search" \
  -d '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"hi"}]}'

Look for:

x-usage: {"prompt_tokens":5,"completion_tokens":3,"total_tokens":8}

Attribute these to internal teams via a x-tenant-id header you send on every call. Billing becomes a join on gateway logs, not a scrape of three provider dashboards. In Python, capture it directly:

resp = client.chat.completions.create(model="openai/gpt-4o", messages=[...])
print(resp.usage.model_dump())  # {'prompt_tokens':5,'completion_tokens':3,'total_tokens':8}

Common pitfalls and tradeoffs

Streaming differences. Provider SSE formats vary. Gateways normalize them, but you may see altered finish_reason semantics. Test your streaming parser against the gateway, not the native SDK. A missing content delta can break JSON accumulators.

Latency tax. Adding a hop costs 10–30ms. For synchronous user-facing calls, measure p99 before and after. If it breaks your budget, keep latency-critical paths on a direct connection or use a regional gateway edge.

Abstraction leak. When you need a brand-new provider feature (e.g., Anthropic’s prompt caching before gateway support), you’ll wait for the gateway to expose it. Keep a thin escape hatch in your client factory that can bypass the gateway for flagged experiments.

Version drift. The OpenAI schema evolves. Pin your client library and review gateway changelogs. A missing strict param can silently drop compliance checks. Write a schema snapshot test that fails when the gateway response shape changes.

Cost opacity. Unified billing hides which provider consumed tokens. Without tenant tags and per-model metering, you’ll lose the ability to negotiate enterprise discounts.

Cutover checklist

1. Inventory all call sites and reconcile with provider bills.
2. Write adapter to unified OpenAI-compatible shape for chat and embeddings.
3. Deploy gateway client behind a feature flag.
4. Migrate read paths, diff evals and embedding similarities.
5. Add routing directives for cache/seed/safety as extensions.
6. Chaos-test fallback with provider simulation endpoints.
7. Enable per-token metering and tenant tags on every request.
8. Remove legacy SDK dependencies from the build.

Following this consolidating multi-provider integrations guide reduces your dependency surface from N providers to one authenticated endpoint, while keeping the escape hatches you actually need. The migration is mechanical, not architectural—if you treat the gateway as a dumb pipe with smart routing, you ship in weeks, not quarters.