How to write a postmortem for an LLM API outage

A solid LLM API outage postmortem goes beyond uptime graphs; it must capture model-specific failures, fallback behavior, and token cost impact. Traditional microservice postmortems miss the nuances of provider rate limits, non-deterministic responses, and cached prompt mismatches that uniquely plague LLM integrations. A good LLM API outage postmortem also accounts for token budget exhaustion as a first-class failure mode, not just HTTP errors.

Step 1: Define incident scope and timeline

Open the incident in your tracker and pin down the exact start and end times using server logs, not human memory. An LLM API outage often begins with elevated error rates on a specific model before cascading to others because clients retry with different models.

Write a machine-readable summary first. This forces precision and prevents the “we think it was about 20 minutes” drift that ruins later analysis.

{
  "incident_id": "INC-2024-042",
  "title": "GPT-4o timeouts via primary provider",
  "started_at": "2024-03-12T14:02:00Z",
  "resolved_at": "2024-03-12T15:18:00Z",
  "severity": "SEV2",
  "affected_models": ["gpt-4o", "claude-3-opus"],
  "error_signature": "429/500 from provider A, fallback disabled"
}

Severity should reflect business impact, not just technical scope. If gpt-4o powers your paid summarization feature and claude-3-opus is a free tier fallback, a 30-minute gpt-4o outage is SEV2 even if overall request volume dropped only 5%.

If you cannot align the timestamps from your gateway logs with provider status pages, the postmortem is not ready. Go back and export raw logs.

Step 2: Collect provider-level telemetry

Pull raw request/response logs for the window. Most OpenAI-compatible gateways emit structured logs with model, status, latency_ms, and usage fields. Parse them with a small script rather than eyeballing the UI.

import json
from collections import defaultdict

errors = defaultdict(int)
latencies = defaultdict(list)
with open("llm_logs.jsonl") as f:
    for line in f:
        rec = json.loads(line)
        model = rec["model"]
        if rec.get("status", 200) >= 400:
            errors[(model, rec["status"])] += 1
        else:
            latencies[model].append(rec["latency_ms"])

for k, v in sorted(errors.items(), key=lambda x: -x[1]):
    print("ERR", k, v)

for m, vals in latencies.items():
    vals.sort()
    p95 = vals[int(len(vals)*0.95)]
    print("LAT", m, "p95_ms=", p95)

Look for patterns: a single model returning 529 (provider overloaded) while others stay green indicates a provider-specific degradation, not a global config bug. Status codes matter—429 means rate limit, 500 means provider internal error, 408 means timeout on a streaming connection. Each implies a different remediation.

Step 3: Reproduce the failure with minimal code

A postmortem without a reproduction is a rumor. Write a 15-line client that hits the same endpoint with the same parameters observed in the logs.

from openai import OpenAI

client = OpenAI(base_url="https://api.your-gateway.com/v1", api_key="test")

try:
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Summarize: <long doc>"}],
        temperature=0.2,
        max_tokens=1024,
    )
    print(resp.choices[0].message.content)
except Exception as e:
    print("REPRO:", type(e).__name__, str(e)[:200])

Run this against the provider that failed, using the same region and API version. If the error does not recur in isolation, suspect request shaping (e.g., batch size, concurrency) or cache-control headers.

For timeout-style outages, reproduce under concurrency:

import threading
def hit():
    try:
        client.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":"hi"}])
    except Exception as e:
        print("thread err", e)

threads = [threading.Thread(target=hit) for _ in range(20)]
[t.start() for t in threads]; [t.join() for t in threads]

Verify reproduction

Success means the script throws the same exception class and status code as the production logs within three attempts, or the concurrency test triggers the timeout at the same threshold.

Step 4: Evaluate fallback and routing logic

Document whether your routing layer actually diverted traffic. If your stack routed through n4n.ai, note whether its automatic fallback to a healthy provider engaged when the primary was rate-limited, and whether client routing directives were honored. A postmortem that ignores a working mitigation teaches the wrong lesson.

Inspect your routing config:

{
  "route": {
    "primary": "provider-a/gpt-4o",
    "fallback": ["provider-b/gpt-4o", "provider-c/claude-3-opus"],
    "honor_cache_control": true
  }
}

If fallback was disabled by a flag flip, say so. If it fired but increased p95 latency 3x, quantify that. Also check whether provider cache-control hints were forwarded—a missed cache-read hint can silently multiply token cost during an outage as retries bypass cache.

Step 5: Quantify token consumption and cost drift

LLM outages rarely stop all traffic; they shift it. Compute the token delta between the incident window and a baseline week. Use per-token metering exports if available.

import json
from datetime import datetime

baseline_tokens = 1_200_000  # from previous Tuesday same hour
incident_tokens = 0
with open("usage.jsonl") as f:
    for line in f:
        rec = json.loads(line)
        ts = datetime.fromisoformat(rec["ts"].replace("Z", "+00:00"))
        if datetime(2024,3,12,14,0) <= ts <= datetime(2024,3,12,15,30):
            incident_tokens += rec["usage"].get("total_tokens", 0)

drift_pct = (incident_tokens - baseline_tokens) / baseline_tokens * 100
print(f"Token drift: {drift_pct:.1f}%")

A 20% drop in completion tokens with a spike in prompt tokens suggests users retried with longer contexts after failures. That is a product behavior worth noting. If your gateway provides per-token metering, reconcile the incident cost against the expected run-rate to show finance the blast radius.

Step 6: Write the blame-free narrative

Engineers read postmortems to learn, not to assign fault. Use this skeleton:

Summary

One paragraph: what broke, who felt it, how long. Example: “Between 14:02 and 15:18 UTC, requests to gpt-4o via provider A failed with 529 errors. Paid summarization latency exceeded 30s for 8% of users. Fallback was misconfigured and did not engage.”

Impact

Number of failed requests, affected models, dollar cost from retry storms.

Root cause

The technical trigger (e.g., provider A rotated a cert without notice; our client pinned an old CA bundle). Avoid “engineer X forgot”—write “the CA bundle update runbook was not executed because the alert referenced a deprecated playbook.”

Mitigation

What stopped the bleeding (fallback enable, traffic shift, circuit breaker).

Detection gap

Why alerts fired 12 minutes late. Fix this with a concrete metric like “no synthetic probe for provider-A/gpt-4o status.”

A good LLM API outage postmortem separates summary from root cause without editorializing.

Step 7: Assign remediation with owners

Vague actions like “improve monitoring” fail. Write tickets with acceptance criteria.

Link each ticket in the postmortem. If a ticket already exists, reference it. Unowned actions are the most common reason a second outage happens the same way.

Step 8: Verify the postmortem is complete

Before closing, run this checklist:

• Timeline matches logs within 60 seconds
• Reproduction script attached and runs
• Token drift calculated
• Fallback behavior described (worked or not)
• Every remediation has an owner and date
• No individual named as culprit

How to verify success

A postmortem is successful when an on-call engineer who was asleep during the event can reproduce the LLM API outage postmortem findings using your script, understand the root cause, and confirm the remediation tickets are in the current sprint. If they have a question, the doc failed.

Writing an LLM API outage postmortem is a discipline: capture the weird parts of model infra, prove them with code, and ship fixes that survive contact with the next provider hiccup.