When your production LLM calls start returning 529s or empty completions, you don’t want to improvise. A practical AI outage incident response runbook turns panic into procedure: detect the degradation, route around it, alert the right people, and capture the postmortem. This tutorial builds that runbook as executable Python you can run from a cron job or incident bot.
Prerequisites
- Python 3.10+ with
pip openaiSDK v1.0+ (pip install openai)requestsfor health probes (pip install requests)- An OpenAI-compatible endpoint and API key exported as
LLM_BASE_URLandLLM_API_KEY - A Slack incoming webhook URL in
SLACK_WEBHOOK(optional but used below) - Basic familiarity with Python exception handling
Verify your environment before continuing:
python -c "import openai, requests; print('deps ok')"
echo $LLM_BASE_URLExpected output if configured:
deps ok
https://your-gateway/v1Step 1: Health probe and failure classification
An AI outage incident response runbook starts with objective signal. We probe the models endpoint and a trivial completion to separate “provider down” from “model throttled”.
import os
import time
import requests
from openai import OpenAI, RateLimitError, APIConnectionError, APITimeoutError
BASE_URL = os.environ["LLM_BASE_URL"]
API_KEY = os.environ["LLM_API_KEY"]
client = OpenAI(base_url=BASE_URL, api_key=API_KEY)
def probe_health() -> dict:
status = {"models_endpoint": False, "completion": False, "latency_ms": None}
try:
r = requests.get(f"{BASE_URL}/models", headers={"Authorization": f"Bearer {API_KEY}"}, timeout=5)
status["models_endpoint"] = r.status_code == 200
except requests.RequestException:
pass
start = time.time()
try:
client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "ping"}],
max_tokens=1,
)
status["completion"] = True
status["latency_ms"] = int((time.time() - start) * 1000)
except (RateLimitError, APIConnectionError, APITimeoutError):
pass
return status
print(probe_health())Run it. A healthy gateway returns:
{'models_endpoint': True, 'completion': True, 'latency_ms': 412}If completion is False but models_endpoint is True, you have a model-level outage or rate limit. That distinction drives the next step.
Step 2: Encode runbook states
We model the incident as a small state machine. This makes the AI outage incident response runbook auditable and testable.
from enum import Enum
class Severity(Enum):
NONE = 0
DEGRADED = 1
OUTAGE = 2
class Runbook:
def __init__(self):
self.severity = Severity.NONE
self.actions_taken = []
def evaluate(self, health: dict) -> Severity:
if not health["models_endpoint"]:
self.severity = Severity.OUTAGE
elif not health["completion"]:
self.severity = Severity.DEGRADED
else:
self.severity = Severity.NONE
return self.severity
rb = Runbook()
print(rb.evaluate(probe_health()))Expected output during normal operation: <Severity.NONE: 0>.
Step 3: Implement fallback and routing
Detection alone does nothing. The runbook must act. We wrap the completion call with fallback models and a retry budget.
If you front your calls with n4n.ai, its automatic fallback when a provider is rate-limited or degraded reduces the need for manual model switching, but your AI outage incident response runbook should still track which fallback occurred and alert on repeated degradations.
FALLBACK_MODELS = ["gpt-4o-mini", "claude-3-haiku", "mixtral-8x7b"]
def complete_with_fallback(prompt: str, primary: str = "gpt-3.5-turbo") -> tuple[str, str]:
models_to_try = [primary] + [m for m in FALLBACK_MODELS if m != primary]
last_err = None
for model in models_to_try:
try:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=50,
)
return resp.choices[0].message.content, model
except (RateLimitError, APIConnectionError, APITimeoutError) as e:
last_err = e
continue
raise RuntimeError(f"All models failed: {last_err}")
text, used = complete_with_fallback("What is 2+2?")
print(f"Used {used}: {text}")Sample success output:
Used gpt-4o-mini: 2 + 2 = 4When the primary is throttled, the runbook logs the switch. That log is the core evidence for the postmortem.
Step 4: Alerting and checkpoint
An incident without notification is just a silent failure. We post a Slack message when severity is DEGRADED or OUTAGE.
import json
import requests as http
def alert(webhook: str, severity: Severity, health: dict, used_model: str = "none"):
payload = {
"text": f":rotating_light: LLM {severity.name} detected",
"blocks": [
{"type": "section", "text": {"type": "mrkdwn", "text": f"*Severity:* {severity.name}\n*Health:* {json.dumps(health)}\n*Fallback model:* {used_model}"}}
]
}
http.post(webhook, json=payload, timeout=5)
webhook = os.environ.get("SLACK_WEBHOOK")
if webhook and rb.severity != Severity.NONE:
alert(webhook, rb.severity, probe_health(), used_model="gpt-4o-mini")If Slack is configured, your channel receives a formatted message. If not, the script skips silently.
Step 5: Generate postmortem stub
After the incident clears, you need a written record. The AI outage incident response runbook should emit a Markdown template pre-filled with timestamps and observed models.
from datetime import datetime
def write_postmortem(path: str, health: dict, used_model: str):
now = datetime.utcnow().isoformat()
content = f"""# Postmortem {now}
## Impact
LLM calls degraded or failing.
## Detection
Health probe: {health}
## Mitigation
Fallback model used: {used_model}
## Action Items
- [ ] Review provider status page
- [ ] Tune retry budget
- [ ] Add synthetic monitoring
"""
with open(path, "w") as f:
f.write(content)
return path
write_postmortem("/tmp/postmortem.md", probe_health(), "gpt-4o-mini")
print("postmortem written")Check the file:
head -5 /tmp/postmortem.md# Postmortem 2025-03-04T12:34:56.789012
## Impact
LLM calls degraded or failing.Running the runbook end-to-end
Wire the pieces into a single entrypoint. This is the executable form of your AI outage incident response runbook.
def main():
health = probe_health()
severity = rb.evaluate(health)
used = "none"
if severity == Severity.OUTAGE:
try:
_, used = complete_with_fallback("healthcheck")
except RuntimeError:
pass
if severity != Severity.NONE:
if webhook := os.environ.get("SLACK_WEBHOOK"):
alert(webhook, severity, health, used)
write_postmortem(f"/tmp/pm-{int(time.time())}.md", health, used)
print(f"severity={severity.name} used={used}")
if __name__ == "__main__":
main()Execute it under normal conditions:
python runbook.pyseverity=NONE used=noneThen simulate an outage by pointing LLM_BASE_URL at a black hole:
LLM_BASE_URL=http://127.0.0.1:9 python runbook.pyseverity=OUTAGE used=noneIf Slack is set, the team gets paged; a postmortem stub lands in /tmp.
Maintaining the runbook
The runbook is only useful if it evolves. Review the generated postmortems weekly. If the same fallback model appears repeatedly, promote it to primary in FALLBACK_MODELS. Add new providers to the probe list as you adopt them.
Keep the code in your incident repo, not a wiki. A wiki rots; a linted Python file gets code-reviewed.
Your AI outage incident response runbook is now a deployable artifact. When the next 529 hits, the procedure is already running.