Migrating from a single provider to multi-provider routing

Most teams start with one LLM provider because the SDK is simple and the docs are clear. The move to a single to multi-provider LLM migration becomes necessary when you hit rate limits, need a cheaper model for certain tasks, or want resilience against outages. Treat this as a refactor of your I/O boundary, not a rewrite of your product logic.

1. Decouple your calling code from provider SDKs

Stop importing openai or anthropic directly in business logic. Define an internal CompletionClient interface, or simply point the OpenAI client at a proxy. The OpenAI-compatible request shape is the de facto standard; most gateways and open-weights servers accept it. This isolation means your product code never learns which provider served the token.

from openai import OpenAI

# Instead of hardcoding api.openai.com
client = OpenAI(
    base_url="https://your-gateway.example/v1",
    api_key="sk-your-key",
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this log"}],
    temperature=0.2,
)

If you use typed languages, define a ModelRouter trait with one method: complete(req: ChatRequest): ChatResponse. The implementation can later hold the routing table. This one change lets you swap the base_url without touching call sites, which is the foundation of a single to multi-provider LLM migration.

2. Introduce a routing layer

You need a component that decides which provider and model handles a request. Reasons: cost, latency, capability, and fallback. A simple policy map keyed by task type works for most teams and avoids premature complexity.

{
  "routes": {
    "summarize": {"model": "mixtral-8x7b", "provider": "groq"},
    "codegen": {"model": "claude-3-5-sonnet", "provider": "anthropic"},
    "fallback": {"model": "gpt-4o", "provider": "openai"}
  }
}

The router should accept a route_key from the caller, not infer it from prompt content. Inference logic belongs in the caller’s domain layer. If you want to avoid building the fallback and model translation yourself, an OpenAI-compatible gateway like n4n.ai gives you one endpoint addressing 240+ models with automatic fallback when a provider is rate-limited, which removes most of the boilerplate in a single to multi-provider LLM migration.

3. Normalize request and response shapes

Providers diverge on tool calling, response formats, and system message handling. Normalize at the edge:

• Map system prompt to the correct field per model.
• Strip unsupported parameters (e.g., logprobs on some endpoints).
• Convert response tool_calls to a uniform internal struct.

def normalize_request(messages, model_profile):
    # model_profile tells us where system goes, what params are allowed
    if not model_profile["supports_system_separate"]:
        sys = next((m for m in messages if m["role"] == "system"), None)
        if sys:
            messages = [m for m in messages if m["role"] != "system"]
            messages[0]["content"] = sys["content"] + "\n" + messages[0]["content"]
    return messages

Test this with recorded fixtures from each provider. Differences in tokenization mean the same text yields different token counts; bill on what the provider returns, not your estimate. Also note that some models require max_tokens while others default sensibly—set it explicitly to avoid surprises.

4. Handle provider-specific quirks

Context windows are the obvious one, but there are subtle ones:

• Some models reject temperature=0 (use 0.01).
• Streaming chunk shapes vary; accumulate before parsing.
• Certain models don’t support response_format: json_object.
• Tool call argument serialization differs (JSON vs. stringified JSON).

Build a validation step that runs before the call:

def validate(model_profile, kwargs):
    if kwargs.get("response_format") == {"type": "json_object"} and not model_profile["json_mode"]:
        raise ValueError(f"{model_profile['model']} lacks JSON mode")
    if kwargs.get("temperature", 0) == 0 and model_profile["rejects_zero_temp"]:
        kwargs["temperature"] = 0.01
    return kwargs

Tradeoff: you lose access to cutting-edge features that only one provider exposes. Decide deliberately; don’t let the abstraction leak. If a feature is core to your product, route those calls to the capable provider explicitly rather than forcing uniformity.

5. Implement fallback and degradation

Fallback is not just try/except on 429. Include semantic checks: if the response is empty or truncated, switch model.

def complete_with_fallback(prompt, primary, backup):
    try:
        r = call(primary, prompt)
        if not r.choices[0].message.content.strip():
            raise ValueError("empty")
        return r
    except (RateLimitError, ValueError, TimeoutError):
        return call(backup, prompt)

Set a max fallback depth to avoid cascading latency. For user-facing calls, cap at one backup; for async jobs, allow two. Also consider degrading gracefully: if all providers fail, return a cached answer or a structured error the UI can render.

6. Meter and observe

You cannot manage what you don’t measure. Capture per-token usage from the response usage field and tag it with route name, model, and provider.

log = {
    "route": "summarize",
    "model": resp.model,
    "prompt_tokens": resp.usage.prompt_tokens,
    "completion_tokens": resp.usage.completion_tokens,
    "latency_ms": elapsed,
}

Per-token usage metering lets you attribute cost accurately across providers and spot when a cheap model silently falls back to an expensive one. Feed these logs to your existing dashboards; alert on fallback rate exceeding 5% for any route.

7. Roll out gradually

Don’t flip the switch. Run shadow mode: send production traffic to the new router but discard the result. Compare output quality with heuristics or a small human panel.

Then ramp: 5% → 25% → 100%. Keep the old single-provider path as a kill switch for the first week.

curl -X POST https://your-gateway.example/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"router:summarize","messages":[{"role":"user","content":"hi"}]}'

If your router honors client routing directives (n4n.ai does), you can force a specific provider via header during the ramp to validate a suspect path without global impact.

Common pitfalls and tradeoffs

Latency. Multi-provider means more network hops and possible cold starts. Mitigate with connection pooling and regional endpoints.

Behavioral drift. Same prompt, different model, different output. Lock evaluation sets so you notice regressions.

Cost opacity. Fallback can quietly multiply spend. Meter every call as in step 6.

Over-abstraction. Don’t build a universal DSL. Keep the interface close to chat completions; specialize only where needed.

Cache invalidation. Provider prefix caches save money but break if you change system prompt ordering. Forward cache-control hints where supported.

A single to multi-provider LLM migration is mostly disciplined plumbing. Keep the calling code dumb, the router smart, and the metrics loud.