Failover routing for open-weight models across providers

Building resilient inference for open-weight models means you cannot depend on a single provider staying healthy. Failover routing open-weight models across providers is the practice of mapping equivalent model weights to multiple endpoints and shifting traffic when one degrades. This guide walks through a concrete implementation path, from equivalence classes to retry logic, with the tradeoffs you will hit in production.

1. Define model equivalence classes

Not every endpoint labeled Llama-3-70B-Instruct is interchangeable. Quantization (fp8 vs bf16), context window, and system prompt handling differ. Group providers by weight lineage, quantization, and max context.

EQUIVALENCE_CLASSES = {
    "llama3-70b": [
        {"provider": "together", "model": "meta-llama/Llama-3-70b-chat-hf",
         "base_url": "https://api.together.xyz/v1"},
        {"provider": "fireworks", "model": "accounts/fireworks/models/llama-v3p3-70b-instruct",
         "base_url": "https://api.fireworks.ai/inference/v1"},
        {"provider": "groq", "model": "llama-3.3-70b-versatile",
         "base_url": "https://api.groq.com/openai/v1"},
    ],
}

Treat a class as your unit of routing. If you need strict reproducibility, narrow the class to a single provider and skip failover. For most chat use cases, cross-provider equivalence at the same weight family is good enough.

2. Normalize request and response shapes

OpenAI-compatible endpoints reduce friction, but gaps remain. Tool calling, logprobs, seed, and response_format support vary. Write a per-provider sanitizer so your calling code stays clean.

def sanitize_params(provider: str, payload: dict) -> dict:
    if provider == "groq":
        payload.pop("logprobs", None)  # unsupported on some Groq models
    if provider == "fireworks":
        payload.pop("seed", None)
    # conservative cap based on class context limit
    payload["max_tokens"] = min(payload.get("max_tokens", 1024), 4096)
    return payload

Response parsing should tolerate extra fields. Never assume finish_reason values are identical—some providers emit stop, others length or tool_calls.

3. Health checks and provider scoring

Naive round-robin hides degradation. Run active probes: a cheap completion every 10–30 seconds per provider. Track rolling error rate and p95 latency.

class ProviderHealth:
    def __init__(self, providers):
        self.score = {p: 1.0 for p in providers}
    def report_failure(self, p):
        self.score[p] *= 0.5
    def report_success(self, p):
        self.score[p] = min(1.0, self.score[p] + 0.05)
    def ranked(self, candidates):
        return sorted(candidates, key=lambda c: self.score[c["provider"]], reverse=True)

A score that decays on failure and recovers slowly prevents flapping. Weight latency into scoring only if you have tight SLOs.

4. Implement the failover loop

The core of failover routing open-weight models is an ordered retry across ranked candidates. Distinguish retryable errors (429, 500, 503, timeouts, connection resets) from fatal ones (400, 401, content filter). Respect Retry-After on 429.

from openai import OpenAI, RateLimitError, APIStatusError, APIConnectionError
import time

def complete_with_failover(class_id, messages, api_keys, health, **params):
    candidates = health.ranked(EQUIVALENCE_CLASSES[class_id])
    last_err = None
    for attempt, cfg in enumerate(candidates):
        client = OpenAI(base_url=cfg["base_url"], api_key=api_keys[cfg["provider"]])
        try:
            resp = client.chat.completions.create(
                model=cfg["model"], messages=messages,
                **sanitize_params(cfg["provider"], params)
            )
            health.report_success(cfg["provider"])
            return resp
        except RateLimitError as e:
            health.report_failure(cfg["provider"])
            last_err = e
            backoff = 0.5 * (2 ** attempt)
            time.sleep(backoff)
        except (APIConnectionError, APIStatusError) as e:
            if isinstance(e, APIStatusError) and e.status_code < 500:
                raise  # non-retryable
            health.report_failure(cfg["provider"])
            last_err = e
    raise last_err

Keep the loop tight. If you have three candidates, max two shifts. More than that signals a systemic outage.

5. Streaming and partial failures

Streaming breaks naive failover. Once you send the first token to the client, you cannot rewind. Two patterns work:

• Buffer-then-stream: Hold the first chunk for ~50 ms; if the provider errors before that, fail over silently. Adds latency but saves clients from broken streams.
• Gateway-mediated reconnect: Some gateways can swap providers mid-stream. If you delegate to a gateway such as n4n.ai, you get an OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded, but you still need to handle client-visible interruptions.

Token counting mismatch

Providers tokenize differently. A max_tokens of 4096 on one may exceed another’s limit. Always set a class-wide conservative cap and log used tokens from the response.

6. Observability and routing directives

Log the winning provider on every request. Emit metrics: per-provider success rate, latency, and failover count. When debugging, you need to know if traffic pinned to a degraded provider.

If you build your own, add a header like x-prefer-provider to force a specific candidate during incidents. Gateways often honor similar client routing directives and forward provider cache-control hints, letting you reuse a warm context on the same provider when possible.

7. Common pitfalls and tradeoffs

Equivalence drift. fp8 serving can produce subtly different outputs on reasoning tasks. If your eval fails intermittently, check which provider served it.

Tool call schema differences. One provider may return arguments as a string, another as an object. Normalize before passing to your function executor.

Cold-start latency. Failover often lands on a provider with zero warm capacity. Expect 2–5x latency on the first request after a shift.

Cost variability. Per-token price differs across providers. Automatic failover can quietly move traffic to a 2x-cost endpoint. Set a cost ceiling in your scorer.

Stateful retries. If you retry after a 500 but the provider actually processed the request, you may duplicate a side-effecting tool call. Make tools idempotent.

8. Build vs. buy

Building failover routing open-weight models yourself gives full control over scoring and sanitization. You own the health probes and can tune for your specific model classes. The cost is ongoing maintenance: provider APIs change, new models appear weekly, and token metering becomes your responsibility.

A gateway offloads health checks, per-token usage metering, and fallback logic. The tradeoff is less visibility into the exact scoring function. For most teams shipping a product, starting with a gateway and extracting the router only for latency-critical paths is the pragmatic order.

9. Actionable checklist

1. List every open-weight model you use and map it to at least two provider endpoints.
2. Write a sanitizer for each provider’s unsupported parameters.
3. Stand up a health scorer with exponential decay on failure.
4. Implement the retry loop with backoff and Retry-After handling.
5. Decide streaming strategy: buffer or accept mid-stream errors.
6. Emit per-provider metrics from day one.
7. Set a cost ceiling so failover does not silently 3x your bill.

Failover routing open-weight models is not a one-time feature; it is a operational discipline. The code above is a starting point, not a finished system. The moment a provider changes its max_context or drops logprobs, your sanitizer and equivalence classes need a pull request.