Automatic retries vs automatic failover explained

Automatic retries vs automatic failover describe two distinct resilience mechanisms for LLM API calls. Retries re-attempt the same request against the same endpoint after a transient error, while failover routes the request to a different provider or model when the primary becomes unavailable or degraded. Confusing the two leads to brittle inference stacks and surprising bill spikes.

What automatic retries actually do

A retry wraps a single outbound call and replays it when the response signals a retryable condition. The target stays identical: same base URL, same model name, same auth token.

Mechanics

Typical retry logic watches for HTTP 429, 500, 502, 503, 504, or socket timeouts. It applies backoff—usually exponential with jitter—to avoid synchronized reconnect storms. The OpenAI Python client defaults to max_retries=2 against its own endpoint; most SDKs behave similarly.

from tenacity import retry, wait_exponential_jitter, stop_after_attempt

@retry(wait=wait_exponential_jitter(max=10), stop=stop_after_attempt(4))
def call_llm(messages):
    # single provider, same model, same credentials
    return client.chat.completions.create(model="gpt-4o", messages=messages)

The decorator above attempts four times against the same endpoint. If the provider is hard-down, all four fail with the same root cause.

When retries help

Retries absorb transient blips: a load balancer restart, a brief network partition, a sporadic 500 from a provider’s inference worker. They are cheap because they require no extra configuration or alternate credentials. For temperature=0 completions they are mostly safe—the output is deterministic enough that a replay yields the same tokens.

When retries hurt

Retries amplify load during incidents. If every client retries with the same fixed interval, you create a retry storm that extends the outage. They also waste tokens when the first attempt actually succeeded but the response was lost in flight—LLM calls are not always idempotent if the model invokes tools. A retry of a request that already triggered a send_email function will send a second email.

What automatic failover actually does

Failover changes the destination. When the primary provider returns a fatal or sustained degraded signal, the system selects a secondary provider or model and sends a fresh request.

Mechanics

A failover controller needs a health view and a routing table. It may key off HTTP status, latency thresholds, or explicit provider outage webhooks. Passive health uses response outcomes; active health pings synthetic requests.

{
  "route": {
    "primary": "openai/gpt-4o",
    "fallback": ["anthropic/claude-3-5-sonnet", "meta/llama-3-70b"]
  },
  "trigger": {"status": [429, 500, 503], "latency_ms": 2000}
}

The gateway evaluates the trigger, then reissues the request to the next entry. Crucially, the prompt and conversation state must be portable across models—not a given when tokenization or tool schemas differ.

State portability

OpenAI-style tools arrays do not map 1:1 to Anthropic’s tools format or Google’s function declarations. A failover layer must translate or reject the call. Context windows also differ: a 32k conversation cannot silently jump to an 8k model.

Routing logic in practice

n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited or degraded. The client sends a single request; the gateway honors routing directives and forwards provider cache-control hints. That removes the need to hand-roll multi-provider logic for many workloads.

Why the distinction matters for LLM workloads

LLM inference differs from classic RPC. Generation is stateful in the sense that context drives output, and many calls invoke tools or write to databases.

Cost and latency

Retries cost the same per-token price as the original call, but failover can shift you to a more expensive model. If your primary is a cheap quantized model and failover jumps to a frontier model, a single user action can 10x the spend. Gateways with per-token usage metering expose the exact cost delta when failover promotes a request to a pricier model, letting you set tier boundaries.

Idempotency is rare

A retry of a temperature=0 completion is mostly safe. A retry of a streaming call that already emitted partial tokens is not—the user may see duplicated text. Failover after partial stream is worse: you cannot resume the stream from another provider. Your UX must handle “switching models mid-answer” or buffer before rendering.

Provider-specific errors

A 400 with “max tokens exceeded” is not retryable. Failover won’t fix it either. Both mechanisms must respect error semantics, or you burn calls blindly. Map status codes explicitly: retry only on 408, 429, 500–504; failover on sustained 429/503 or connection-refused; fatal on 400–403.

Observability requirements

You cannot operate these mechanisms blind. Emit distinct counters: retry_attempts_total, failover_events_total, final_provider. A dashboard that mixes them hides whether you are burning tokens on replays or silently upgrading to costly models. Trace IDs should carry the original model request so support can reconstruct the path.

Concrete example

Suppose you run a support bot. The primary is openai/gpt-4o-mini. You configure client retries and gateway failover.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible, 240+ models
    api_key="sk-...",
    max_retries=2,  # client-side retries on same endpoint
)

# Gateway config (illustrative header)
# X-Route-Fallback: anthropic/claude-3-5-haiku, google/gemini-flash
resp = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Refund policy?"}],
    extra_headers={"X-Cache-Control": "max-age=300"},
)

If the provider returns 429 twice, the client retries within the endpoint. If the provider is fully degraded, the gateway fails over to the specified fallback model, preserving the cache hint. Your application code never changes.

Without gateway failover, you’d write:

primary = "openai/gpt-4o-mini"
fallbacks = ["anthropic/claude-3-5-haiku", "google/gemini-flash"]
for model in [primary, *fallbacks]:
    try:
        return client.chat.completions.create(model=model, messages=messages)
    except ProviderError as e:
        if e.status in (429, 500, 502, 503):
            continue
        raise

That manual loop mixes retries and failover and breaks on partial streaming. It also loses cache context across providers.

A curl equivalent for header-based routing:

curl https://api.n4n.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Route-Fallback: anthropic/claude-3-5-haiku" \
  -H "X-Cache-Control: max-age=300" \
  -d '{"model":"openai/gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

Common misconceptions

“Retries are enough for reliability”

They are not. A regional provider outage lasts longer than any backoff budget. Retries just delay the failure and inflate tail latency.

“Failover is just retries with extra steps”

False. Retries reuse the same brittle dependency. Failover introduces a new dependency that may have different behavior, latency, or compliance posture. You must test output equivalence and tool translation.

“Failover always preserves context”

JSON messages port, but system prompt nuances, token limits, and tool definitions may not. A model with 8k context cannot silently take a 32k conversation. A failover controller must reject or truncate.

“Retries are safe for any request”

If the request triggers a side effect—say, an LLM calling a send_email tool—a retry can double-send. Make tools idempotent or disable retries for mutating steps.

Implementation checklist

• Classify errors: map status codes and exception types to retryable, failoverable, or fatal.
• Set retry budgets: max 2–3 attempts with jitter; never retry 4xx except 408/429.
• Define failover tiers: same model family, same price class, then escalation.
• Preserve cache directives: forward cache-control so fallback hits cached prefixes.
• Log routing decisions: emit which mechanism fired and the final destination.
• Test partial failure: kill the stream mid-token and verify UX.
• Alert on failover rate: a rising rate signals primary instability, not success.

Bottom line

Automatic retries vs automatic failover solve different failure modes. Retries handle flaky networks; failover handles dead providers. Ship both, but treat failover as a separate code path with its own guardrails around cost, context portability, and observability.