How to monitor LLM API uptime across multiple providers

To monitor LLM API uptime multiple providers without trusting vendor status pages, you need synthetic probes that exercise the real inference path. A status page shows broad outages hours late; your users feel degraded latency or 429s immediately. This guide gives you a runnable Python harness that checks each provider on a schedule, emits metrics, and triggers alerts when a backend goes dark.

Step 1: Enumerate providers and load secrets

Start by listing the providers you depend on and reading credentials from the environment. Keep the roster in a single config so you can add new models without rewriting probe logic. For a competitor-comparisons exercise, include the major commercial APIs and at least one aggregator so you can see relative reliability side by side.

import os

PROVIDERS = {
    "openai": {
        "base_url": "https://api.openai.com/v1",
        "api_key": os.environ["OPENAI_API_KEY"],
        "model": "gpt-4o-mini",
    },
    "anthropic": {
        "base_url": "https://api.anthropic.com/v1",
        "api_key": os.environ["ANTHROPIC_API_KEY"],
        "model": "claude-3-haiku-20240307",
    },
    "mistral": {
        "base_url": "https://api.mistral.ai/v1",
        "api_key": os.environ["MISTRAL_API_KEY"],
        "model": "mistral-small-latest",
    },
    "groq": {
        "base_url": "https://api.groq.com/openai/v1",
        "api_key": os.environ["GROQ_API_KEY"],
        "model": "llama3-8b-8192",
    },
}

If you route through a unified gateway, the same pattern works with one base URL. For example, n4n.ai exposes a single OpenAI-compatible endpoint that addresses 240+ models and honors client routing directives, so you can target a specific backend by setting a header and still monitor LLM API uptime multiple providers from one probe loop.

Step 2: Send a minimal completion request

A health check should cost as little as possible. Request a single token with a trivial prompt. Use the OpenAI Python client pointed at each base URL; for providers that speak the OpenAI protocol, this just works. For Anthropic’s native API you would swap to their SDK, but the probe shape is identical: one shot, minimal tokens, catch exceptions.

from openai import OpenAI
import time

def probe_provider(name, cfg):
    client = OpenAI(base_url=cfg["base_url"], api_key=cfg["api_key"])
    start = time.monotonic()
    try:
        resp = client.chat.completions.create(
            model=cfg["model"],
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=1,
            timeout=10,
        )
        latency_ms = (time.monotonic() - start) * 1000
        ok = bool(resp.choices[0].message.content)
        return {"provider": name, "ok": ok, "latency_ms": latency_ms, "error": None}
    except Exception as e:
        latency_ms = (time.monotonic() - start) * 1000
        return {"provider": name, "ok": False, "latency_ms": latency_ms, "error": str(e)}

When debugging a flaky provider, a raw curl against the endpoint confirms whether the failure is in your wrapper or the network:

curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  https://api.openai.com/v1/models

A 200 with a sub-second total tells you the control plane is up even if the inference path is slow.

Step 3: Run probes concurrently

Sequential checks waste minutes when you have ten providers. Run them in a thread pool or with asyncio. Below is a thread pool version that finishes in the time of the slowest timeout.

from concurrent.futures import ThreadPoolExecutor

def run_all_probes(providers):
    results = []
    with ThreadPoolExecutor(max_workers=len(providers)) as ex:
        futures = [ex.submit(probe_provider, n, c) for n, c in providers.items()]
        for f in futures:
            results.append(f.result())
    return results

if __name__ == "__main__":
    snap = run_all_probes(PROVIDERS)
    for r in snap:
        print(r)

If you prefer async, the same logic maps to asyncio.gather with an httpx.AsyncClient. The point is to avoid serial blocking calls that turn a 10-second check into 100 seconds.

Step 4: Persist and emit metrics

Raw prints do not page anyone. Convert each result to a Prometheus metric or a JSON line that your log pipeline can parse. For historical comparison across providers, write each snapshot to SQLite.

import sqlite3, json, time

def persist(results, db_path="uptime.db"):
    conn = sqlite3.connect(db_path)
    conn.execute(
        "CREATE TABLE IF NOT EXISTS probes "
        "(ts REAL, provider TEXT, ok INTEGER, latency_ms REAL, error TEXT)"
    )
    for r in results:
        conn.execute(
            "INSERT INTO probes VALUES (?,?,?,?,?)",
            (time.time(), r["provider"], int(r["ok"]), r["latency_ms"], r["error"]),
        )
    conn.commit()
    conn.close()

def to_prometheus(results):
    lines = []
    for r in results:
        status = 1 if r["ok"] else 0
        lines.append(f'llm_up{{provider="{r["provider"]}"}} {status}')
        lines.append(f'llm_latency_ms{{provider="{r["provider"]}"}} {r["latency_ms"]:.1f}')
    return "\n".join(lines)

Ship the Prometheus lines to a Pushgateway, or stdout if you use Vector/Fluent Bit. The key is that llm_up is a gauge per provider, not a single global flag.

Step 5: Track state and alert on sustained failure

A single failed probe could be a transient network blip. Maintain a failure counter per provider and only alert after N consecutive misses or when latency exceeds a hard ceiling.

from collections import defaultdict

fail_streak = defaultdict(int)
LATENCY_CAP_MS = 5000
ALERT_AFTER = 3

def evaluate(results):
    alerts = []
    for r in results:
        if not r["ok"] or r["latency_ms"] > LATENCY_CAP_MS:
            fail_streak[r["provider"]] += 1
        else:
            fail_streak[r["provider"]] = 0
        if fail_streak[r["provider"]] >= ALERT_AFTER:
            alerts.append(f'{r["provider"]} down for {fail_streak[r["provider"]]} checks')
    return alerts

def send_slack(alerts, webhook):
    if not alerts:
        return
    payload = {"text": "LLM provider alerts:\n" + "\n".join(alerts)}
    # use requests.post(webhook, json=payload) in real code

Wire alerts into Slack, PagerDuty, or a webhook. Keep the threshold tight for providers you call synchronously in user requests; loosen it for batch-only backends.

Step 6: Schedule the probe

Run the harness every 60 seconds. On Kubernetes, wrap it in a CronJob. On a VPS, use systemd timers.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: llm-uptime-probe
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: probe
            image: your-registry/llm-probe:latest
            envFrom:
            - secretRef:
                name: provider-keys
          restartPolicy: Never

If you prefer serverless, a Cloud Scheduler → Cloud Run job achieves the same with zero always-on cost. The schedule is what turns point-in-time checks into a reliability time series you can use to monitor LLM API uptime multiple providers over weeks.

Step 7: Verify success

Verification is concrete:

Run python probe.py locally with valid keys. You should see one line per provider with ok: True and a latency under a few hundred milliseconds.
Corrupt one key (export OPENAI_API_KEY=invalid). Re-run. That provider’s ok flips to False, error shows AuthenticationError, and after three runs the alert list includes it.
Check your metrics backend: llm_up{provider="openai"} should be 0 for the bad key and 1 for others. Graph it over an hour to confirm the schedule fires.
Query SQLite: SELECT provider, COUNT(*) FROM probes WHERE ok=0 GROUP BY provider; should show only the poisoned entry.
Restore the key and watch fail_streak reset and the alert clear.

If all five hold, you have a working system to monitor LLM API uptime multiple providers continuously.

Caveats that will bite you

Provider “uptime” is not binary. A 200 response with a 30-second latency is worse than a fast 503 for interactive apps. Track p95 latency separately from success rate, and graph both.

Some providers rate-limit health checks if you hammer them. Keep max_tokens=1 and the interval at 60s or above. If you must check more often, use a gateway that aggregates and caches model availability—n4n.ai forwards provider cache-control hints and falls back automatically when a backend is degraded, which reduces the need for per-provider probing at high frequency.

Model names change. Pin a cheap, stable model per provider for the probe, and update the roster when the vendor deprecates it. A probe that fails because the model ID vanished is a false outage that trains your team to ignore alerts.

Finally, regional endpoints behave differently. A provider may be green in us-east-1 and red in eu-west-1. Encode region in the provider key (openai-us, openai-eu) so your metrics reflect reality.

What you get

You now own a low-cost, vendor-independent uptime signal. When a vendor status page says “all systems operational” but your probes show llm_up{provider="openai"}=0, you trust your data. That is the entire point: to monitor LLM API uptime multiple providers you need first-party evidence, not press releases. Build the harness once, point it at every backend you use, and let the graphs tell you which competitor actually delivers.