Deploying AI agents on Kubernetes: a practical guide

Deploying AI agents on Kubernetes is not the same as shipping a stateless web service. An agent typically runs a loop that calls a language model, invokes tools, and mutates external state, which means you need to plan for partial failure, secret rotation, and noisy neighbor model APIs. This guide lays out an ordered path we use for getting agents into production without drowning in YAML.

1. Define the agent runtime contract

Before writing manifests, define what “running” means for your agent. Is it a long-lived process that polls a queue? A batch job that processes one task and exits? An interactive session backed by a websocket? Most teams skip this and regret it when liveness probes start killing agents mid-tool-call.

A reactive agent that consumes from a queue should ack only after the step succeeds. A proactive poller should sleep between iterations. The key property is clean shutdown: the process must exit after finishing the current unit of work when it receives SIGTERM.

import os, json, signal

stop = False

def handle_term(signum, frame):
    global stop
    stop = True

signal.signal(signal.SIGTERM, handle_term)

def run_step(msg):
    # call LLM, run tool, persist
    ...

while not stop:
    msg = queue.pop()
    if msg:
        run_step(msg)

Idempotency is non-negotiable. Kubernetes will reschedule pods; duplicate deliveries happen.

2. Containerize with a minimal, reproducible image

Use a multi-stage build. Install only what the agent needs; do not ship a Jupyter or ML notebook image to production. Pin the base tag and a dependency lockfile. Run as a non-root user to limit blast radius.

FROM python:3.12-slim AS build
WORKDIR /app
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock

FROM python:3.12-slim
WORKDIR /app
COPY --from=build /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY agent.py .
ENV PYTHONUNBUFFERED=1
USER 1000:1000
CMD ["python", "agent.py"]

Tradeoff: slim images lack shells and debuggers. If you need curl for probes, add it explicitly in the final stage. Avoid :latest; unreproducible images turn incidents into archaeology.

3. Choose the right Kubernetes primitive

For deploying AI agents on Kubernetes, the default Deployment is wrong for anything that is not safely horizontally scalable. If your agent holds in-memory conversation state, run it as a StatefulSet or back it with an external store like Redis or Postgres. For task-oriented agents, a Job or CronJob is cheaper and simpler.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: report-agent
spec:
  schedule: "0 6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: agent
            image: registry.example.com/report-agent:1.4.0
            envFrom:
            - secretRef:
                name: agent-secrets

Pitfall: running an always-on Deployment for an agent that fires once per day burns cluster resources and creates a false sense of “availability.” If you need scheduling, use a controller built for it.

4. Set resource requests from observed behavior

Agents are usually CPU-bound on orchestration logic and I/O-bound on model calls. They rarely need GPUs unless you self-host models. Set requests conservatively and limits slightly above to avoid OOM kills during context serialization.

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "512Mi"

If you self-host a 7B model, budget memory for weights plus KV cache; a limit of 4Gi may be required. Kubernetes schedules on requests, so lying about them leads to noisy neighbors and unpredictable latency.

5. Manage secrets and config separately

Never embed API keys in the image. Use Secret objects or an external secrets operator (e.g., Vault, AWS Secrets Manager). Mount them as env vars for legacy SDKs or as files for modern ones.

kubectl create secret generic agent-secrets \
  --from-literal=OPENAI_API_KEY=sk-... \
  --from-literal=TOOL_TOKEN=abc123

Config that changes per environment (poll interval, model name, feature flags) belongs in a ConfigMap. Keep prompts in a versioned store or a hashed ConfigMap key, not hardcoded in the binary. Rotate secrets by updating the Secret and rolling the pod; do not rebuild images for credential changes.

6. Route model traffic through a stable gateway

Most agents call multiple model providers. Writing fallback logic in the agent couples infrastructure to code. An OpenAI-compatible endpoint that fronts many models simplifies client code. For example, n4n.ai exposes one endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded, and it honors client routing directives and forwards provider cache-control hints. That lets you switch models via env var instead of code deploys.

import os, openai

client = openai.OpenAI(
    base_url=os.environ["OPENAI_BASE_URL"],
    api_key=os.environ["OPENAI_API_KEY"],
)
resp = client.chat.completions.create(
    model=os.environ.get("AGENT_MODEL", "gpt-4o-mini"),
    messages=[{"role": "user", "content": "Summarize tickets"}],
)

Per-token usage metering at the gateway gives you cost attribution per agent without building your own middleware. Pitfall: hardcoding a single provider URL makes incident response slower. Abstract the base URL from day one.

7. Scale on the right signal

CPU autoscaling is meaningless when your agent is blocked on a 30-second LLM call. Use queue depth or custom metrics. KEDA scales a Deployment from zero based on Redis list length:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: agent-scaler
spec:
  scaleTargetRef:
    name: ticket-agent
  triggers:
  - type: redis
    metadata:
      address: redis:6379
      listName: agent:queue
      listLength: "5"

Set a maxReplicaCount that matches your model API quota. Tradeoff: scaling to zero introduces cold starts. For latency-sensitive agents, keep min replicas at one and scale on concurrency instead.

8. Instrument everything, but watch the logs

Agents fail silently when a tool returns malformed JSON or a model stalls. Emit structured logs with task IDs and step names. Add OpenTelemetry traces around LLM calls to see which model is slow or which tool throws.

import logging, json
log = logging.getLogger("agent")
log.info(json.dumps({"event": "step_start", "task": task_id, "model": model}))

Export metrics: prompt tokens, completion tokens, step latency, error rate. Pitfall: logging full prompts may leak PII into cluster log aggregation. Redact or hash sensitive fields before emit, and set retention limits.

9. Handle termination gracefully

Kubernetes sends SIGTERM, then kills after terminationGracePeriodSeconds (default 30). Your agent must finish the current loop iteration and ack the queue message only after success. Extend the grace period for long steps.

import signal, os

def handle_term(signum, frame):
    os.environ["AGENT_STOP"] = "1"

signal.signal(signal.SIGTERM, handle_term)

For stateful leaders, use a Lease object to avoid split-brain during rolling updates. Acquire the lease on start; release on shutdown. If you use StatefulSet, the pod ordinal helps but does not replace distributed locking.

10. CI/CD and pre-deployment testing

Build the image in CI, run unit tests for tool calls with mocked LLM responses, and scan for vulnerabilities. Push a tagged image and let GitOps (Argo CD, Flux) sync the manifest. Do not apply manifests from a laptop.

A smoke test job in the cluster validates that the agent can pull a secret, reach the model gateway, and process one fake task. This catches misconfigured ServiceAccount or egress policies before pager alerts fire.

11. Common pitfalls and tradeoffs

In-memory state. Deploying AI agents on Kubernetes with conversation state in process memory means every crash loses context. Push state to an external store; treat pods as cattle.

Over-provisioning. Always-on replicas for sporadic agents waste money. Prefer Jobs or scale-to-zero.

Prompt versioning. A prompt change is a code change. Store prompts in a ConfigMap with a hash suffix and roll gradually.

Model drift. Providers change behavior. Pin model versions where possible, and monitor output quality with offline eval jobs.

Network egress. Model calls leave the cluster. Ensure egress policies allow it, but restrict to known endpoints to prevent data exfiltration via a compromised agent.

Resource fights. Agents that spawn subprocesses (e.g., headless browsers for scraping) need higher CPU limits; isolate them on node pools.

12. A minimal manifest to start

Combine the above into a single deployment for a queue-driven agent:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ticket-agent
spec:
  replicas: 1
  selector:
    matchLabels: {app: ticket-agent}
  template:
    metadata:
      labels: {app: ticket-agent}
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: agent
        image: registry.example.com/ticket-agent:1.0.0
        envFrom:
        - secretRef: {name: agent-secrets}
        - configMapRef: {name: agent-config}
        resources:
          requests: {cpu: "250m", memory: "256Mi"}
          limits: {cpu: "1", memory: "512Mi"}

Iterate from here. Deploying AI agents on Kubernetes becomes manageable once you separate runtime contract, infrastructure, and model access into distinct layers, and you resist the urge to treat the cluster like a VM with extra steps.