Setting up on-call alerts for LLM latency spikes

Latency at the tail is where LLM-powered products break. This tutorial walks through building on-call alerts for LLM latency spikes using a lightweight Python probe, Prometheus, and Alertmanager so you page a human before users notice.

Prerequisites

• Python 3.10+ with requests and prometheus_client (pip install requests prometheus_client).
• Docker and Docker Compose to run Prometheus and Alertmanager locally.
• An OpenAI-compatible /v1/chat/completions endpoint. Export LLM_ENDPOINT, LLM_KEY, and optionally LLM_MODEL.
• curl and jq for scraping metrics during verification.

If you route through a gateway such as n4n.ai, which offers an OpenAI-compatible endpoint across 240+ models with automatic fallback when a provider is degraded, client-side measurement is still essential—fallback may add seconds you didn’t account for.

Why time-to-first-token

Users perceive LLM responsiveness as the gap between sending a prompt and seeing the first characters. Total completion time matters less for interactive UX. Your on-call alerts for LLM latency spikes should therefore trigger on time-to-first-token (TTFT), not just on HTTP errors or total request time.

Step 1: Write the latency probe

The probe opens a streaming request, parses Server-Sent Events, and records the clock delta at the first data: chunk.

import time
import requests
import os

URL = os.environ["LLM_ENDPOINT"]
KEY = os.environ["LLM_KEY"]
MODEL = os.environ.get("LLM_MODEL", "gpt-3.5-turbo")

def probe():
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": "ping"}],
        "stream": True,
        "max_tokens": 5,
    }
    headers = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
    start = time.perf_counter()
    ttft = None
    with requests.post(URL, json=payload, headers=headers, stream=True) as r:
        for line in r.iter_lines():
            if line and line.startswith(b"data:"):
                chunk = line[5:].strip()
                if chunk and chunk != b"[DONE]":
                    if ttft is None:
                        ttft = time.perf_counter() - start
                    break
    total = time.perf_counter() - start
    return ttft, total

if __name__ == "__main__":
    ttft, total = probe()
    print(f"TTFT={ttft:.3f}s total={total:.3f}s")

Run it against a healthy endpoint:

LLM_ENDPOINT=https://api.example.com/v1/chat/completions \
LLM_KEY=sk-... python probe.py

Expected output:

TTFT=0.437s total=0.452s

Step 2: Export metrics to Prometheus

A long-running exporter reuses probe() every 15 seconds and exposes gauges.

from prometheus_client import start_http_server, Gauge
import time, threading
from probe import probe

TTFT = Gauge("llm_request_ttft_seconds", "Time to first token")
TOTAL = Gauge("llm_request_total_seconds", "Total request time")

def loop():
    while True:
        try:
            ttft, total = probe()
            TTFT.set(ttft)
            TOTAL.set(total)
        except Exception:
            TTFT.set(float("nan"))
        time.sleep(15)

if __name__ == "__main__":
    start_http_server(8000)
    threading.Thread(target=loop, daemon=True).start()
    while True:
        time.sleep(3600)

Verify the metrics endpoint:

curl -s localhost:8000/metrics | grep llm_request

Output:

llm_request_ttft_seconds 0.441
llm_request_total_seconds 0.459

Step 3: Prometheus scrape and rules

prometheus.yml scrapes the exporter and loads alert rules:

global:
  scrape_interval: 15s
rule_files:
  - /etc/prometheus/alerts.yml
scrape_configs:
  - job_name: llm_probe
    static_configs:
      - targets: ['host.docker.internal:8000']

alerts.yml defines the spike condition:

groups:
  - name: llm_latency
    rules:
      - alert: LLMHighTTFT
        expr: llm_request_ttft_seconds > 2
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "LLM TTFT > 2s for 2m"

The for: 2m prevents a single slow request from paging on-call.

Step 4: Alertmanager routing

alertmanager.yml sends severity: page to PagerDuty and everything else to Slack:

route:
  receiver: slack
  group_by: ['alertname']
  routes:
    - match:
        severity: page
      receiver: pagerduty
receivers:
  - name: slack
    slack_configs:
      - api_url: ${SLACK_URL}
        channel: '#llm-alerts'
  - name: pagerduty
    pagerduty_configs:
      - service_key: ${PD_KEY}

Alertmanager expands env vars at startup when passed via --config.file and the process environment.

Step 5: Run the stack

docker-compose.yml:

services:
  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerts.yml:/etc/prometheus/alerts.yml
    ports:
      - "9090:9090"
  alertmanager:
    image: prom/alertmanager:v0.27.0
    environment:
      SLACK_URL: $SLACK_URL
      PD_KEY: $PD_KEY
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

docker compose up -d

Check targets at http://localhost:9090/targets — the llm_probe job should be green.

Step 6: Fault-inject to test the alerts

Stand up a slow mock endpoint to validate the full path:

from flask import Response
import time

app = Flask(__name__)

@app.route('/v1/chat/completions', methods=['POST'])
def slow():
    time.sleep(3)  # force TTFT > 2s
    def gen():
        yield "data: {\"choices\":[{\"delta\":{\"content\":\"hi\"}}]}\n\n"
        yield "data: [DONE]\n\n"
    return Response(gen(), mimetype='text/event-stream')

Point the exporter at it (LLM_ENDPOINT=http://localhost:5000/v1/chat/completions) and restart. Within two minutes, Prometheus evaluates the rule and fires LLMHighTTFT. Alertmanager shows the alert at http://localhost:9093/#/alerts and routes to PagerDuty.

This confirms your on-call alerts for LLM latency spikes work end-to-end.

Tuning and operational notes

Set thresholds from observed p95 TTFT, not guesswork. If your p95 is 800ms, a 2s page threshold is reasonable; for batch workloads, 10s may be the right line.

Use a second alert for slower burn: llm_request_ttft_seconds > 1 with for: 10m and severity: warn to Slack. This gives early signal before the page fires.

When you ship a postmortem for an AI outage, include the TTFT graph from Prometheus and the exact alert rule that should have fired. If the alert existed but didn’t page, the routing tree is the bug.

The probe measures client-observed latency. If you use a gateway that honors provider cache-control hints or routes to different backends, label the metric with the resolved model via response headers to catch per-provider regressions.

Cleaning up

Stop the stack with docker compose down. The exporter process can be killed or left running as a systemd unit in production. For production, run the exporter alongside a node-exporter and scrape both from a central Prometheus.

Effective on-call alerts for LLM latency require measuring what users feel, exporting it cleanly, and routing pages without flapping. The code above is production-shaped; harden the probe’s timeout and add auth before deploying.