Setting up alerts for AI agent failures in production

Most teams ship an AI agent to production, watch it answer a few prompts correctly, and move on. Then a model provider starts returning 429s, a tool schema changes, or the agent loops silently until it hits a token limit. Setting up alerts for AI agent failures turns those invisible incidents into actionable pages before users notice.

Prerequisites

• Python 3.11+ with prometheus_client, opentelemetry-api, and opentelemetry-sdk installed.
• A running agent loop you can modify. We’ll use a minimal ReAct-style stub.
• Prometheus 2.45+ and Alertmanager 0.25+ (Docker Compose is fine).
• A Slack incoming webhook URL for notifications.
• Basic familiarity with PromQL and YAML config.

Failure modes you must alert on

The foundation of alerts for AI agent failures is classifying what actually breaks. In practice, four classes cover 90% of incidents:

• llm_error: the model API returned an error or timed out.
• tool_error: a called function raised an exception or returned malformed data.
• validation_error: the agent’s output failed schema validation.
• max_iterations: the loop hit its step cap without producing a final answer.

A fifth signal, llm_fallback, is not an error but a leading indicator of provider stress. We’ll track all five.

Instrument the agent with failure counters

Use a Prometheus Counter with type and agent_id labels. This gives you per-failure breakdowns without spawning separate metrics.

from prometheus_client import Counter, start_http_server

AGENT_FAILURES = Counter(
    "agent_failures_total",
    "Count of AI agent failures by type",
    ["type", "agent_id"]
)

def record_failure(failure_type: str, agent_id: str = "default"):
    AGENT_FAILURES.labels(type=failure_type, agent_id=agent_id).inc()

Wrap your agent step so every exception path is captured:

class LLMProviderError(Exception): pass
class ToolExecutionError(Exception): pass
class ValidationError(Exception): pass
class MaxIterationsError(Exception): pass

def run_agent_step(state):
    try:
        return execute_step(state)  # your LLM + tool logic
    except LLMProviderError:
        record_failure("llm_error")
        raise
    except ToolExecutionError:
        record_failure("tool_error")
        raise
    except ValidationError:
        record_failure("validation_error")
        raise
    except MaxIterationsError:
        record_failure("max_iterations")
        raise

Expose metrics on port 8000:

if __name__ == "__main__":
    start_http_server(8000)
    run_agent_loop()

Expected output from curl localhost:8000/metrics | grep agent_failures:

# HELP agent_failures_total Count of AI agent failures by type
# TYPE agent_failures_total counter
agent_failures_total{type="llm_error",agent_id="default"} 0.0
agent_failures_total{type="tool_error",agent_id="default"} 0.0
agent_failures_total{type="validation_error",agent_id="default"} 0.0
agent_failures_total{type="max_iterations",agent_id="default"} 0.0

Track LLM provider degradation and fallback

If you route LLM calls through a gateway like n4n.ai, you get automatic fallback across 240+ models when a provider is rate-limited or degraded. That prevents a hard outage, but fallback frequency is a stress signal you should meter. The gateway honors client routing directives, so we can tag a preference and detect when it wasn’t honored.

def call_llm(messages, primary_model="gpt-4o"):
    resp = gateway.chat.completions.create(
        model=primary_model,
        messages=messages,
        extra_headers={"x-n4n-routing": "prefer:primary"}
    )
    if resp.headers.get("x-n4n-fallback-used") == "true":
        record_failure("llm_fallback")
    return resp

Now llm_fallback increments on degraded upstreams. This feeds your alerts for AI agent failures with a leading indicator before error rates spike.

Simulate a failure to validate the pipeline

Before wiring Prometheus, confirm the counter increments. Run a one-off script:

from agent_metrics import record_failure

record_failure("llm_error")
record_failure("llm_error")
record_failure("tool_error")

After running it, curl localhost:8000/metrics | grep llm_error shows:

agent_failures_total{type="llm_error",agent_id="default"} 2.0
agent_failures_total{type="tool_error",agent_id="default"} 1.0

You now have a verifiable signal.

Export metrics to Prometheus

Create prometheus.yml:

scrape_configs:
  - job_name: 'agent'
    static_configs:
      - targets: ['host.docker.internal:8000']

Launch it:

docker run -p 9090:9090 \
  -v ./prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

In the Prometheus UI (http://localhost:9090), run rate(agent_failures_total[5m]). You should see a vector with your labels. If targets show DOWN, check the host.docker.internal mapping or use the container IP.

Define alert rules

Create alerts.yml:

groups:
  - name: agent_alerts
    rules:
      - alert: AgentFailureRateHigh
        expr: sum(rate(agent_failures_total[10m])) > 0.1
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "AI agent failure rate above 0.1/s"

      - alert: AgentStalled
        expr: increase(agent_failures_total{type="max_iterations"}[30m]) > 0
        for: 0m
        labels:
          severity: warn
        annotations:
          summary: "Agent hitting max iterations"

      - alert: LLMFallbackSpiking
        expr: sum(rate(agent_failures_total{type="llm_fallback"}[15m])) > 0.05
        for: 10m
        labels:
          severity: warn
        annotations:
          summary: "LLM fallback rate elevated"

Reload Prometheus with curl -X POST http://localhost:9090/-/reload (started with --web.enable-lifecycle). In the Alerts tab, AgentFailureRateHigh will show PENDING after the rate crosses threshold, then FIRING after for: 5m. These rules power alerts for AI agent failures across any environment where the metric is scraped.

Route to Slack via Alertmanager

alertmanager.yml:

route:
  receiver: 'slack'
  group_by: ['alertname']
  group_wait: 30s

receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#agent-alerts'
        text: "{{ .CommonAnnotations.summary }}: {{ .CommonAnnotations.description }}"

Run it:

docker run -p 9093:9093 \
  -v ./alertmanager.yml:/etc/alertmanager/alertmanager.yml \
  prom/alertmanager

Tell Prometheus where Alertmanager is via command-line flag --alertmanager.url=http://localhost:9093. When the failure rate holds, Slack receives:

AgentFailureRateHigh: AI agent failure rate above 0.1/s
Aggregate failure rate 0.14 per second over 10m.

Correlate with traces

Metrics page you; traces tell you why. Add OpenTelemetry spans around each step:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
tracer = trace.get_tracer("agent")

def run_agent_step(state):
    with tracer.start_as_current_span("agent_step") as span:
        try:
            return execute_step(state)
        except Exception as e:
            span.record_exception(e)
            span.set_attribute("error", True)
            raise

Export to Jaeger or Tempo. When an alert fires, pull the trace ID from your logs and see exactly which tool call or LLM request timed out.

Keep alerts actionable

• Label metrics with agent_id and env so staging doesn’t page you at 3 a.m.
• Set for: longer than your normal retry bursts; a single transient 429 shouldn’t fire a page.
• Add a deadman switch: alert: AgentDown if up{job="agent"} == 0 so silent metric loss is itself alerted.
• Review llm_fallback weekly. A steady fallback rate means you should negotiate higher primary-model quota.

Building these alerts for AI agent failures takes an afternoon and replaces customer-reported outages with engineer-owned signals.