Building an AI agent to auto-remediate Kubernetes pod failures

Building an AI agent Kubernetes auto-remediation loop starts with a simple premise: pods fail, and most failures repeat known patterns. This tutorial shows how to wire the Kubernetes API to an LLM so the system can diagnose a crashed pod and apply a safe fix without paging a human.

Prerequisites

• A running Kubernetes cluster (kind or minikube) with kubectl pointing at it.
• Python 3.11+ and pip.
• An API key for an OpenAI-compatible endpoint. We’ll route through n4n.ai, an OpenAI-compatible gateway that provides automatic fallback if a provider is degraded and per-token metering.
• The kubernetes and openai Python packages:

pip install kubernetes openai

• A test deployment that crashes on purpose:

kubectl create deployment crashy --image=busybox -- sleep 5

This pod exits every 5 seconds, landing in CrashLoopBackOff.

Architecture

The agent has four stages:

1. Detect – watch pod events for terminal or waiting states.
2. Collect – pull logs, events, and pod spec.
3. Reason – send context to an LLM, get back a structured remediation plan.
4. Act – apply only allow-listed actions via the K8s API.

The LLM is not given cluster admin. It proposes; the code disposes with guardrails.

Step 1: Detect failing pods

Load kube config and stream pod changes. We filter for CrashLoopBackOff or Error reasons.

from kubernetes import client, config, watch

config.load_kube_config()
v1 = client.CoreV1Api()
w = watch.Watch()

def is_failing(pod):
    for cs in (pod.status.container_statuses or []):
        if cs.state.waiting and cs.state.waiting.reason in ("CrashLoopBackOff", "Error"):
            return True
    return False

for ev in w.stream(v1.list_pod_for_all_namespaces, timeout_seconds=30):
    pod = ev["object"]
    if is_failing(pod):
        print(f"FAILING: {pod.metadata.namespace}/{pod.metadata.name}")

Run this against the crashy deployment. Expected output:

FAILING: default/crashy-5d8b9c7f8-abcde

Step 2: Collect diagnostic context

Before calling the model, gather the last 50 log lines and recent events. Truncate to keep token count sane.

def collect_context(namespace, name):
    pod = v1.read_namespaced_pod(name, namespace)
    try:
        logs = v1.read_namespaced_pod_log(name, namespace, tail_lines=50)
    except client.exceptions.ApiException:
        logs = ""
    evs = v1.list_namespaced_event(
        namespace,
        field_selector=f"involvedObject.name={name}",
    )
    return {
        "pod": pod.to_dict(),
        "logs": logs,
        "events": [e.to_dict() for e in evs.items[-5:]],
    }

For the busybox crash, logs will be empty (it never writes stdout) and events show BackOff.

Step 3: Reason with the LLM

We use the OpenAI Python SDK pointed at the compatible gateway. The system prompt forces JSON output with an action enum.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")

SYSTEM = """You are an SRE agent. Given Kubernetes diagnostic context, return strict JSON:
{"action": "restart"|"scale"|"noop", "params": {}, "reason": "short string"}
Only restart for transient crashes. Scale for load. noop if unsafe."""

def propose_remediation(ctx):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": json.dumps(ctx)[:6000]},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Call it inside the watch loop:

ctx = collect_context(pod.metadata.namespace, pod.metadata.name)
plan = propose_remediation(ctx)
print("PLAN:", plan)

Expected plan for the crashy pod:

{"action": "restart", "params": {}, "reason": "Transient exit code 0 with sleep loop; restart to clear backoff."}

The AI agent Kubernetes auto-remediation design relies on this tight loop: detect, context, propose, act.

Step 4: Apply remediation with guardrails

Never let the model execute arbitrary Kubernetes operations. Map its action to a fixed code path.

def apply_plan(namespace, name, plan):
    action = plan.get("action")
    if action == "restart":
        v1.delete_namespaced_pod(name, namespace)
        return f"deleted pod {name} for restart"
    if action == "scale":
        # require explicit deployment + replicas in params
        dep = plan["params"].get("deployment")
        rep = plan["params"].get("replicas")
        if not dep or not isinstance(rep, int):
            return "rejected scale: missing params"
        apps = client.AppsV1Api()
        apps.patch_namespaced_deployment_scale(
            dep, namespace, {"spec": {"replicas": rep}}
        )
        return f"scaled {dep} to {rep}"
    return "noop: " + plan.get("reason", "no reason")

Wire it up:

if is_failing(pod):
    ctx = collect_context(pod.metadata.namespace, pod.metadata.name)
    plan = propose_remediation(ctx)
    result = apply_plan(pod.metadata.namespace, pod.metadata.name, plan)
    print(f"APPLIED: {result}")

After deletion, Kubernetes recreates the pod. The crash repeats, but the agent proves the loop works. In production you’d add a cooldown and max-retry count.

Step 5: Run the full agent

Save as agent.py and run:

python agent.py

Sample output:

FAILING: default/crashy-5d8b9c7f8-abcde
PLAN: {'action': 'restart', 'params': {}, 'reason': 'Transient exit code 0 with sleep loop; restart to clear backoff.'}
APPLIED: deleted pod crashy-5d8b9c7f8-abcde for restart

The pod returns, crashes again, and the loop triggers once more. That’s expected for a deliberately broken workload.

Beyond simple restarts

The AI agent Kubernetes auto-remediation can handle OOMKilled by proposing a scale with higher memory limits, but you should extend apply_plan to patch the deployment spec rather than just replica count. Example patch:

apps = client.AppsV1Api()
dep = apps.read_namespaced_deployment(dep_name, namespace)
dep.spec.template.spec.containers[0].resources.limits["memory"] = "256Mi"
apps.patch_namespaced_deployment(dep_name, namespace, dep)

Add that as a new action type after validating the model’s params contain a positive integer.

Safety notes

• Run the agent with a ServiceAccount scoped to pods and deployments in a single namespace.
• Keep an allow-list of actions; never exec into a pod based on model output.
• Log every plan and applied action to an audit sink. The per-token metering from the gateway helps track cost per remediation.
• Add a jitter and exponential backoff so a persistent bug doesn’t trigger a deletion storm.

Where to take it next

Replace the simple watch with a controller using informers for lower API pressure. Cache pod context to avoid re-sending the same logs. If you run multi-cloud, the gateway’s automatic fallback keeps the reasoning step alive when one provider is throttled.

The core pattern—detect, contextualize, propose with constraints, act with guardrails—transfers to most SRE toil. Start with crash loops, then expand to liveness probe failures and pending pods.