AI agents vs runbooks for SRE incident triage

The debate over AI agents vs runbooks incident triage starts when your pager stops being a rare event and becomes a stream. Runbooks give you deterministic, reviewable procedures; agents give you adaptive reasoning that can correlate signals across systems without a human in the loop. Both have a place, but the trade-offs are sharper than most posts admit.

Capabilities

Runbooks: deterministic and auditable

A runbook encodes a known fix. It assumes the failure mode is understood and the response is mechanical. That is perfect for certificate renewals, failover triggers, or restarting a stuck worker.

incident: worker_stuck
trigger:
  metric: "queue_depth > 1000 for 5m"
steps:
  - action: "kubectl rollout restart deploy/worker"
    verify: "queue_depth < 100 for 2m"
  - notify: "#db-oncall"

The strength is predictability. If the runbook is wrong, you debug the runbook, not the execution environment. It will not invent a new command.

AI agents: adaptive correlation

An agent wraps an LLM with tools: log search, metric queries, shell, ticket updates. It forms a hypothesis from incomplete signals and chooses among actions that no one pre-scripted.

def triage_agent(alert):
    ctx = gather_context(alert)  # prometheus + logs + deploy diff
    resp = llm.chat(
        system="You are an SRE agent. Use tools to remediate.",
        user=ctx,
        tools=[rollback, scale, page_human]
    )
    if resp.tool_call == "rollback":
        execute_rollback(resp.target, dry_run=True)
    elif resp.tool_call == "scale":
        scale_deployment(resp.target, resp.replicas)

The agent shines when the incident is novel—a weird dependency loop between two services that no one documented. It can read the recent deploy diff and propose a revert. When the agent calls an LLM gateway, using a single OpenAI-compatible endpoint that fronts 240+ models with automatic fallback keeps the agent resilient if a provider is rate-limited mid-incident.

Cost model

Runbooks cost almost nothing at runtime: a few compute seconds in a CI runner or a lambda. The real cost is authoring and maintenance—engineer hours to keep steps current as the system evolves. A stale runbook is worse than none, because it creates false confidence.

AI agents invert that. The marginal run costs tokens: input context (often large logs) and output reasoning. At scale, a busy incident channel can burn thousands of input tokens per minute. You pay for capability, not just execution. Per-token metering lets you attribute cost to specific incidents, which is useful for chargebacks to product teams.

Latency and throughput

A runbook step executes in milliseconds to seconds. Throughput is bounded by your automation platform, not by reasoning. You can fan out hundreds of runbooks concurrently with no sweat.

An agent adds LLM latency: first token often 200–2000ms depending on model and context size, plus tool round-trips. For a single incident that is fine. For 50 concurrent alerts, you need queueing and model throughput headroom. Agents are not a drop-in for high-frequency automated remediation unless you constrain scope to low-cardinality events.

Ergonomics

Runbooks are plain text or YAML. Any engineer can read, diff, and review them in PRs. On-call can scan a runbook in 10 seconds at 3am. Testing a runbook is straightforward: simulate the trigger in a staging cluster.

Agents require prompt engineering, tool schemas, and eval harnesses. The behavior is non-deterministic; you need tracing to understand why it acted. That is a heavier operational burden, but it pays off when the long tail of incidents is large. A good pattern is to record every agent session and auto-generate a proposed runbook from successful resolutions.

Ecosystem and integration

Runbooks plug into existing ops tools: PagerDuty, Grafana, Terraform, kubectl. They are native to CI/CD and service meshes.

Agents need an LLM provider, tool wrappers, and a sandbox. They benefit from a model gateway that honors client routing directives and forwards provider cache-control hints, so repeated context (like a service topology doc) hits cache and cuts cost. The surrounding ecosystem is younger but moving fast, with open standards for tool calling stabilizing.

Limits

Runbooks break on unknown-unknowns. If the symptom is not matched, they sit idle or, worse, execute a wrong fixed step because a condition drifted.

Agents hallucinate. They can invent a log line or misread a metric. Guardrails—policy checks, dry-run mode, human approval for destructive actions—are mandatory. Also, agents depend on the quality of their tools; a half-baked metric API yields confident nonsense. They also struggle with strict ordering constraints unless you encode them in the prompt and verify in code.

Head-to-head summary

Which to choose

Use runbooks when

• The failure modes are known and repeated (disk full, cert expiry, replica lag).
• You need auditability and instant execution.
• Regulatory or risk constraints forbid autonomous action.
• Your team is small and cannot staff agent eval pipelines.

Use AI agents when

• Incidents are heterogeneous and low-frequency but high-impact.
• You have rich telemetry and want correlation across services.
• Your team can invest in eval, tracing, and guardrails.
• You already operate an LLM gateway with fallback and caching.

Hybrid (recommended)

Encode the top 20% of incidents as runbooks. Route the long tail to an agent that can propose a runbook patch after resolution. This keeps deterministic speed where it matters and captures institutional knowledge continuously. The agent becomes a runbook author, not a black box.