Postmortem template for AI and LLM incidents

Most teams reuse a generic ops postmortem for their first LLM outage and immediately hit gaps: token budgets, model drift, and provider degradation aren’t in the standard template. This AI incident postmortem template gives you an ordered path to capture what actually broke when your inference stack misbehaves, from misrouted requests to silent context truncation.

1. Write the Summary Before the Analysis

Start with a 3-sentence summary. Non-negotiable. It forces the author to state impact and cause before digging into logs.

• What broke: e.g., “GPT-4 class requests to /v1/chat/completions returned 429 from provider X for 22 minutes.”
• Who felt it: “Internal summarization pipeline and 14% of chat users.”
• Why: “Provider X rotated a quota without notice; our retry logic amplified the load.”

Keep this section under 100 words. If you can’t write it, you don’t understand the incident yet. The AI incident postmortem template below assumes this summary sits at the top of every report.

2. Quantify Impact in Tokens, Not Just Requests

Traditional postmortems count errors per minute. LLM systems need token-level impact because cost and latency scale with generation size.

Capture:

• Total prompt tokens affected
• Total completion tokens lost or regenerated
• Dollar-equivalent waste (use your metering, not guesswork)

A minimal structured log line looks like this:

{
  "event": "llm_request",
  "model": "openai/gpt-4o",
  "status": 503,
  "usage": { "prompt_tokens": 1200, "completion_tokens": 0 },
  "ts": "2024-06-15T14:02:11Z"
}

Aggregate with standard tools:

jq 'select(.event=="llm_request" and .status>=500) | .usage.prompt_tokens' logs.jsonl | jq -s 'add'

If you lack per-request token logging, that’s a finding for the action items.

3. Build the Timeline with Model Identity

A timeline that says “API slow” is useless. You need which model, which provider, and which version. Model IDs drift: a provider can silently shift a snapshot under the same friendly name.

Example timeline entry

14:02 UTC - Request to model "anthropic/claude-3-opus-20240229" via gateway returned 503.
14:03 UTC - Retry with "openai/gpt-4-turbo-2024-04-09" succeeded but with 2.1x p95 latency.
14:05 UTC - Circuit breaker tripped, queued 1.2k requests.

Store the model string exactly as sent. “Claude” is not a model ID; “anthropic/claude-3-opus-20240229” is. Include the region if your client passes it.

4. Root Cause: Model, Provider, or Glue?

Classify the failure into one of three buckets:

1. Model behavior: hallucinations, refusals, format drift, regression in fine-tune.
2. Provider infrastructure: 5xx, timeouts, rate limits, regional outage.
3. Glue code: bad prompt assembly, wrong temperature, missing fallback, incorrect parsing.

Distinguishing model failures from pipeline failures

Model failures often show as valid HTTP 200 with garbage payload. Provider failures show as transport errors. Glue failures show as correct calls to the wrong target.

Code: isolating glue bugs

A common glue bug is silently swapping the model based on a misread config:

# Bug: env var overrides intended model in staging
model = os.getenv("LLM_MODEL", "openai/gpt-4o")
if config.env == "staging":
    model = "mock/local"  # forgot to guard this
response = client.chat.completions.create(model=model, messages=msgs)

Your postmortem should include the minimal repro. Paste the offending function, not the whole service. If the model itself emitted malformed JSON, include the raw completion and the schema you expected.

5. Detection and Alerting Gaps

If your alert fired 40 minutes after users complained on Twitter, write that down. LLM incidents often evade standard HTTP checks because the endpoint returns 200 with garbage.

Add a check for:

• Response schema validation (does it parse as JSON when expected?)
• Token throughput drop vs baseline
• Embedding mismatch (if used for retrieval)

try:
    data = json.loads(response.choices[0].message.content)
except json.JSONDecodeError:
    metrics.incr("llm.schema_violation")

If you didn’t have that metric, add it to the template’s follow-up. Also note whether your latency alert used wall-clock or token-normalized latency; the latter catches slow generation that status checks miss.

6. Mitigation and Fallback Behavior

Document what stopped the bleeding. If you have automatic fallback, state whether it engaged and what it masked.

If you front your traffic with a gateway such as n4n.ai, you get one OpenAI-compatible endpoint for 240+ models, automatic fallback when a provider is rate-limited or degraded, per-token usage metering, and it honors client routing directives and forwards provider cache-control hints—useful raw material, but your AI incident postmortem template must still record the originally requested model and the ultimately served one. The cache-control forwarding matters: a missed cache hint can force expensive regeneration that looks like a model slowdown.

Code: logging the served model

Even with fallback, log the effective route:

resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=msgs,
    extra_headers={"x-router-prefer": "provider-a"}
)
# Gateway may return a different model if fallback triggered
served = resp.model  # e.g. "azure/gpt-4o" 
logger.info("llm_served", orig="openai/gpt-4o", served=served, 
            usage=resp.usage.model_dump())

Without that log line, you’ll guess during the review.

7. Lessons and Action Items

Each item needs an owner and a date. Vague “improve monitoring” is not actionable. The AI incident postmortem template above forces ownership; use it.

Example:

• Add schema validation alert for /v1/extract (owner: @j, due 2024-07-01)
• Cap retry count at 2 to prevent thundering herd (owner: @m)
• Document model ID mapping in runbook (owner: @s)

Link each item to the root cause category so you can track whether glue bugs dominate.

8. The Template Skeleton

Copy this into your incident repo:

## AI Incident Postmortem: <date> <service>
### Summary
- What: 
- Impact: 
- Cause: 

### Impact (tokens)
- Prompt tokens affected: 
- Completion tokens wasted: 

### Timeline
- HH:MM UTC: 

### Root Cause Category
- [ ] Model [ ] Provider [ ] Glue
- Details: 

### Detection Gap
- 

### Mitigation
- 

### Action Items
- [ ] 

Common Pitfalls

• Blaming the model for glue errors. A prompt injection from your own concatenation is not “Claude being dumb.”
• Ignoring token cost. A 10% error rate on a 100k-token batch is a real financial event.
• Skipping the repro. If you can’t reproduce the bad output in a notebook, you haven’t finished.
• Mixing model versions. Treating “gpt-4” as immutable across May and June snapshots hides regressions.

Tradeoffs in Postmortem Depth

Writing a full AI incident postmortem template for a 30-second blip on a dev sandbox is overhead. Use severity tiers:

• Sev1 (prod user impact): full template.
• Sev2 (degraded latency): sections 1–4 only.
• Sev3 (internal test failure): timeline + root cause.

Don’t force the heavy process on every hiccup; you’ll get filler docs nobody reads.

Rehearse Before the Real Thing

Run a game day where you kill a provider and use this template live. The first real incident isn’t the time to discover your log schema lacks model IDs or that your fallback silently downgrades to a weaker model that breaks your parser.