How to detect LLM provider outages before your users do

Your users don’t care that Anthropic’s API returned 503s for nine minutes; they care that your chat feature stalled. Detecting LLM provider outages early means treating model endpoints as unstable dependencies and monitoring them with the same rigor you give PostgreSQL or Stripe. This guide walks through a concrete pipeline to catch degradations before support tickets spike.

Step 1: Instrument Every LLM Call With Structured Telemetry

You cannot detect an outage you cannot see. Wrap your provider calls so that every request emits a structured log with provider, model, HTTP status, latency, and token count. Do not rely on aggregate dashboard charts alone—you need per-request events to slice by provider and model after the fact.

import time, json, logging

logger = logging.getLogger("llm_client")

def call_llm(provider: str, model: str, payload: dict) -> dict:
    start = time.monotonic()
    try:
        # raw_post is your internal HTTP helper to the provider endpoint
        resp = raw_post(provider, model, payload)
        latency = time.monotonic() - start
        logger.info(json.dumps({
            "event": "llm_call",
            "provider": provider,
            "model": model,
            "status": resp.status_code,
            "latency_ms": round(latency * 1000),
            "tokens": resp.json().get("usage", {}).get("total_tokens", 0)
        }))
        return resp.json()
    except Exception as e:
        latency = time.monotonic() - start
        logger.error(json.dumps({
            "event": "llm_error",
            "provider": provider,
            "model": model,
            "error": type(e).__name__,
            "latency_ms": round(latency * 1000)
        }))
        raise

Ship these logs to a system that supports low-latency queries (Elasticsearch, Loki, or ClickHouse). Tag the provider field as a high-cardinality dimension only if your backend handles it; otherwise hash the model name.

What to capture beyond status codes

A 200 with a truncated stream is still an outage. If you use streaming, record time-to-first-token and whether the stream closed before the finish_reason arrived. Detecting LLM provider outages early requires watching those signals, not just status >= 500.

Step 2: Run Provider-Specific Canaries Every 30 Seconds

Synthetic traffic catches outages that your real traffic might miss due to caching or low volume. Stand up a separate process that calls each provider with a minimal payload: max_tokens: 1, a trivial prompt, and a hard 5-second timeout.

import requests, time, os

PROVIDERS = {
    "openai": (
        "https://api.openai.com/v1/chat/completions",
        {"Authorization": f"Bearer {os.environ['OPENAI_KEY']}"},
        {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 1}
    ),
    "anthropic": (
        "https://api.anthropic.com/v1/messages",
        {"x-api-key": os.environ["ANTHROPIC_KEY"], "anthropic-version": "2023-06-01"},
        {"model": "claude-3-haiku-20240307", "max_tokens": 1, "messages": [{"role": "user", "content": "hi"}]}
    ),
}

def ping(name):
    url, headers, body = PROVIDERS[name]
    try:
        r = requests.post(url, headers=headers, json=body, timeout=5)
        return r.status_code == 200
    except Exception:
        return False

while True:
    for name in PROVIDERS:
        ok = ping(name)
        # emit metric llm_canary_up{provider=name} 1 or 0
    time.sleep(30)

Run this outside your primary deployment region if you can—provider degradations are often regional. The canary should never hit your production model routing; it tests the raw provider edge.

Step 3: Aggregate Signals and Set Adaptive Thresholds

Static thresholds (“alert if latency > 2s”) break the moment you add a slower model. Use a rolling baseline per provider-model pair and alert on deviation.

from collections import deque

class LatencyTracker:
    def __init__(self, window=20):
        self.window = deque(maxlen=window)
    def add(self, ms):
        self.window.append(ms)
    def is_degraded(self, current_ms):
        if len(self.window) < self.window.maxlen:
            return False
        avg = sum(self.window) / len(self.window)
        return current_ms > avg * 3  # 3x baseline = incident

Feed both canary results and production telemetry into the same tracker. A provider is effectively down if either the canary fails twice consecutively or production error rate exceeds 5% over two minutes.

Prometheus example

If you export metrics, a rule like this catches early drift:

alert: LLMProviderDegraded
expr: |
  rate(llm_error_total[2m]) / rate(llm_call_total[2m]) > 0.05
  and on(provider) llm_canary_up == 0
for: 2m

Detecting LLM provider outages early depends on correlating the synthetic signal with real errors—false positives drop when both agree.

Step 4: Alert On-Call and Trigger Fallbacks

Page a human, but also let automation reduce blast radius. If you front your traffic with a gateway such as n4n.ai, its automatic fallback when a provider is rate-limited or degraded will mask transient errors from users. Still, detecting LLM provider outages early via canaries lets you know which provider is unhealthy and tune routing directives or disable a bad model before the gateway’s fallback budget exhausts.

Wire the alert to a runbook:

{
  "alert": "llm_provider_outage",
  "condition": "llm_canary_up == 0 for 1m",
  "routes": ["pagerduty:oncall-llm"],
  "annotations": {
    "runbook": "https://wiki.internal/llm-outage",
    "fallback_override": "disable_model: claude-3-opus-20240229"
  }
}

Do not silently swallow the error in the gateway without notification. The worst case is a fallback chain that burns per-token budget across three providers while no engineer knows.

Streaming-specific action

For streaming endpoints, if time-to-first-token exceeds your baseline by 5x, cancel the request server-side and retry on a secondary provider. Expose a x-llm-fallback-attempt header so clients can log it.

Step 5: Close the Loop With Postmortems

After every provider incident, write a short postmortem focused on detection lag: the time between canary failure and user impact (should be negative—canary first) and the time between canary failure and page (should be under 60s).

Template:

Provider:
Canary fail time:
First user error time:
Detection gap:
Action taken:
Follow-up: tune threshold / add region / disable model

Over three incidents you will calibrate the adaptive multiplier. Some providers have daily latency wobble; others fail hard. Detecting LLM provider outages early becomes cheaper as your baselines mature.

How to Verify Success

Success is not “we have dashboards.” Success is measured in incident reviews: your canary alert fires and a human acknowledges it before your user-facing error budget burns. A concrete test—once a month, block one provider’s egress in staging via firewall rule and confirm the canary pages within one minute and the fallback path serves traffic. If the page arrives after a user complaint, your thresholds are still wrong.