From provider sprawl to one API: a consolidation guide

Most teams accumulate LLM provider integrations the way they accumulate AWS accounts: one per project, one per experiment, until you’re juggling a dozen SDKs, auth schemes, and billing dashboards. Collapsing that provider sprawl to one API is less about deleting code and more about extracting a stable interface from a mess of inconsistent shapes.

The payoff is concrete: one retry policy, one token accounting pipeline, one place to inject fallbacks when a vendor throws 429s at 2 a.m.

1. Inventory what you actually call

Before writing any abstraction, list every LLM request your services make. Don’t trust memory; grep your repos for SDK imports.

grep -r "openai\|anthropic\|google.generativeai\|cohere" --include="*.py" --include="*.ts" .

For each call site, record:

• Model name(s) requested
• Input shape (chat vs completion vs embeddings)
• Timeout and retry assumptions
• Whether responses are streamed
• Any provider-specific params (temperature, top_p, safety_settings)

You will find the bulk of traffic hits a handful of models. The long tail is where sprawl hides.

2. Define a provider-agnostic request shape

Pick one lingua franca. OpenAI’s chat completions schema has become the de facto standard; every major vendor now mirrors it or offers a translation shim. Define your internal type around that.

from pydantic import BaseModel
from typing import Literal, Optional

class ChatMessage(BaseModel):
    role: Literal["system", "user", "assistant"]
    content: str

class UnifiedChatRequest(BaseModel):
    model: str
    messages: list[ChatMessage]
    temperature: float = 0.7
    max_tokens: Optional[int] = None
    stream: bool = False
    # provider-specific passthrough
    extra: dict = {}

Keep extra for things like anthropic_version or safety_settings. Resist the urge to flatten every vendor knob into top-level fields; you’ll regret it when the next model adds a new param.

3. Implement a thin translation layer

Your gateway should accept UnifiedChatRequest and emit the correct provider call. A dispatch table beats inheritance here.

def route_request(req: UnifiedChatRequest):
    if req.model.startswith("gpt-"):
        return call_openai(req)
    if req.model.startswith("claude-"):
        return call_anthropic(req)
    if req.model.startswith("gemini-"):
        return call_gemini(req)
    raise ValueError(f"no route for {req.model}")

Each call_* function maps fields and sends the request. For Anthropic, you convert messages to their prompt format; for Gemini, you wrap in contents. Keep these mappers pure and unit-tested.

A common mistake: writing the translation inside your business logic. That leaks provider details back into call sites and defeats the consolidation.

4. Route with explicit fallbacks

Provider degradation is not theoretical. When you consolidate provider sprawl to one API, you gain the ability to fail over without touching product code.

Define a priority list per capability:

{
  "chat": ["claude-3-5-sonnet", "gpt-4o", "mixtral-8x22b"],
  "cheap_classification": ["gpt-4o-mini", "claude-3-haiku"]
}

Your router tries the first; on 429 or 5xx within a timeout, moves down the list. If you use n4n.ai, it honors client routing directives and forwards provider cache-control hints, so you keep cache wins across backends while getting automatic fallback when a provider is rate-limited or degraded.

Implement the fallback loop with a hard cap to avoid cascading latency:

def with_fallback(models: list[str], req: UnifiedChatRequest):
    last_err = None
    for m in models:
        try:
            return route_request(req.model_replace(m))
        except (ProviderTimeout, RateLimitError) as e:
            last_err = e
            continue
    raise last_err

Tradeoff: fallback changes which model answers, so non-deterministic outputs may shift. Log the actually-used model on every response.

5. Meter and observe per token

You cannot manage what you don’t measure. After consolidation, emit a structured log line for every call with: requested model, actual model, input tokens, output tokens, latency, cost (if you have a price table).

logger.info("llm.call", extra={
    "req_model": req.model,
    "actual_model": actual,
    "in_tokens": usage.prompt_tokens,
    "out_tokens": usage.completion_tokens,
    "ms": elapsed,
})

Per-token usage metering lets you spot a silent fallback to an expensive model that’s blowing the budget. If you build your own gateway, persist these to a columnar store; if you use a managed gateway, ship the metrics to your existing pipeline.

6. Migrate call sites incrementally

Do not rewrite everything in a flag day. Wrap the old SDK calls behind a compatibility shim that speaks your unified request, then swap implementations behind it.

// old: await openai.chat.completions.create({...})
// new:
await gateway.chat({
  model: "gpt-4o",
  messages,
  temperature: 0.3,
});

Start with internal tools, then low-risk user flows. Keep the old client available but deprecated; delete after 30 days of no references in logs.

Common pitfalls and tradeoffs

Cache semantics differ. Provider prompt caches have different minimum token thresholds and TTLs. Forwarding cache-control hints is necessary but not sufficient; measure cache hit rate per provider.

Streaming shapes diverge. SSE formats vary in field names and finish-reason signaling. Normalize to a single chunk schema early or your frontend will special-case providers.

Rate limit headers are liars. Some vendors return 429 with Retry-After; others return 200 with an error object inside. Your translation layer must treat both as rate limits.

Model capability gaps. A unified max_tokens param means different things across contexts. Set sane caps per model in config, not in code.

Auth rotation. Centralizing API keys simplifies compliance but creates a single point of failure. Use a secrets manager with per-environment isolation and rotate quarterly.

When to stop consolidating

Not every integration belongs behind one API. Proprietary fine-tuned endpoints with custom schemas, or hardware-bound inference, may cost more to abstract than they save. The goal of moving from provider sprawl to one API is operational leverage, not theological purity. If a call site would need more than ten lines of extra passthrough, leave it native and monitor it separately.

A minimal reference topology

A reasonable self-hosted layout:

service -> unified-gateway (auth, routing, metering)
                     |
   -----------------------------------------
   |               |               |
openai-shim   anthropic-shim   gemini-shim
   |               |               |
provider API   provider API   provider API

The gateway is a stateless reverse proxy with a config file of model routes. n4n.ai provides this as a single OpenAI-compatible endpoint covering 240+ models, but the architecture above works without it if you want to own the code.

Final checklist

• Inventory complete, long tail documented
• Unified request schema defined and typed
• Mappers covered by tests for each provider
• Fallback lists set with latency caps
• Token metering in production logs
• Call sites migrated behind shim
• Cache hit rates monitored per provider

Collapsing provider sprawl to one API is a few days of focused work for most teams, and it pays back the first time a provider goes down and your users don’t notice.