Incident response checklist for multi-provider AI outages

When a single LLM provider goes down, your customer-facing features go dark unless you’ve already wired up alternatives. A solid multi-provider AI outage checklist turns a 3am page into a routine failover. This is the checklist we use for production systems routing across multiple inference backends, derived from incidents where a provider’s 503 took out a checkout copilot.

1. Map provider dependencies and failure modes

Start with a blunt inventory. List every model endpoint your service calls, the critical user paths that depend on it, and the acceptable degradation (e.g., “search summarization can fall back to cached results; chat cannot”). Most teams discover they have three silent dependencies on a provider they thought was optional.

Use a static manifest checked into repo, but back it with runtime tracing. Export an OpenTelemetry span per provider call so you can reconstruct the dependency graph under load.

{
  "services": {
    "chat": {
      "primary": "gpt-4o",
      "fallback": ["claude-3-5-sonnet", "mixtral-8x22b"],
      "degrade_to": "static_help_text"
    },
    "doc_summarizer": {
      "primary": "claude-3-opus",
      "fallback": ["gpt-4-turbo"],
      "degrade_to": "queue_for_later"
    }
  }
}

Without this map, on-call engineers guess. The multi-provider AI outage checklist starts with knowing what breaks when provider X returns 503, and what business impact that carries.

2. Implement health-aware automatic fallback

Client code must catch provider-specific errors and retry against the next candidate. Don’t block on a dead endpoint; use timeouts shorter than user patience and treat 429/500/503 as failover signals. Semantic failures (content filter, malformed output) should also trigger fallback if your SLA allows.

def complete_with_fallback(messages, models):
    for model in models:
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                timeout=8.0
            )
        except (RateLimitError, APIConnectionError, InternalServerError) as e:
            log.warning("model %s failed: %s", model, e)
            continue
    raise AllProvidersDown()

If you sit behind a gateway such as n4n.ai, which provides one OpenAI-compatible endpoint with automatic fallback when a provider is rate-limited or degraded, you still need this loop for partial degradations where the gateway returns a synthesized error or a non-transient refusal. The multi-provider AI outage checklist demands you verify fallback paths in staging, not prod.

3. Cache prompts and honor cache-control hints

LLM calls are expensive and slow; many outages are self-inflicted via thundering herds. Use semantic caching for repeated prompts and forward provider cache-control headers to avoid stampeding a recovered provider. Anthropic and OpenAI both support prefix caching—use it.

curl -X POST https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Cache-Control: max-age=3600" \
  -d '{ "model": "gpt-4o", "messages": [{"role":"user","content":"status?"}] }'

A good multi-provider AI outage checklist includes a cache hit-rate SLO. If hit rate drops below 40% during an incident, you’re likely retrying blindly. Cache keys should incorporate model family, not just raw string, to allow cross-provider reuse where safe.

4. Instrument per-token usage and latency percentiles

You can’t debug what you don’t measure. Emit per-token usage tagged by provider, model, and route. When a provider degrades, token throughput drops before errors spike—p50 token latency doubling is your early warning.

metrics.histogram(
    "llm.token.throughput",
    value=tokens / elapsed_sec,
    tags={"provider": "openai", "model": "gpt-4o"}
)

Per-token metering also exposes cost anomalies when fallback routes shift traffic to a pricier model. The multi-provider AI outage checklist requires dashboards that show cross-provider token share in real time, with alerts on sudden share flips.

5. Deploy circuit breakers per provider

A circuit breaker stops sending traffic to a provider after N consecutive failures, then half-opens after a cooldown. This prevents retry storms from prolonging an outage and gives the provider room to recover.

class CircuitBreaker {
  private failures = 0;
  private open = false;
  private nextAttempt = 0;
  async call(fn: () => Promise<any>) {
    if (this.open && Date.now() < this.nextAttempt) throw new Error("circuit open");
    try {
      const res = await fn();
      this.failures = 0;
      this.open = false;
      return res;
    } catch (e) {
      if (++this.failures > 5) {
        this.open = true;
        this.nextAttempt = Date.now() + 30_000;
      }
      throw e;
    }
  }
}

Wire breakers into your fallback chain. The multi-provider AI outage checklist is incomplete without explicit breaker state in your incident timeline and a log line when a breaker trips.

6. Build a manual override and feature kill-switch

Automated fallback isn’t infallible. Ship a runtime flag that degrades AI features to non-LLM paths (cached answers, “try again later” UI) without a deploy. The flag must be readable by every edge node within seconds.

{
  "feature_flags": {
    "ai_chat_enabled": true,
    "force_provider": null,
    "degrade_mode": "none"
  }
}

During a multi-provider AI outage, flipping degrade_mode to readonly can save your error budget. The checklist must list who can flip the flag, the command to verify propagation, and a default re-enable timer to avoid stuck degradation.

7. Run fault-injection game days

You only trust fallback if you’ve broken it on purpose. Use proxy failures or env vars to simulate 429s from your primary, then watch the system shift. Do this in prod with a canary cohort.

# toxiproxy example
toxiproxy-cli toxic add -n latency -t latency \
  -a latency=2000 -a jitter=500 my_llm_proxy

A mature multi-provider AI outage checklist includes quarterly drills with a recorded postmortem. If fallback works but p99 latency triples, that’s still an incident. Track mean time to detect (MTTD) across providers as a drill metric.

8. Write provider-specific postmortems

After the fire, document which provider failed, what triggered it, and whether your fallback behaved. Use a table to compare expected vs actual. The multi-provider AI outage checklist closes with this artifact so the next outage is cheaper.

Include the exact error codes, the breaker timings, and the cache hit rate during the window. The multi-provider AI outage checklist is only as good as the lessons filed afterward.

Synthesis

Operating LLMs in production means assuming every provider will fail quarterly. The eight items above convert panic into procedure:

Keep the manifest updated, test the breakers, and watch token metrics—the rest is practice.