Autoscaling GPU pools for high-traffic AI agents

Running autoscaling GPU pools for AI agents at high traffic forces you to balance cold-start latency against compute cost. Most teams underestimate the metric plumbing required before Kubernetes can make sane scaling decisions, and they over-provision GPU nodes “just in case.”

1. Profile agent inference before scaling

You cannot autoscale what you haven’t measured. Capture per-request token counts, batch sizes, and time-to-first-token for each agent workflow. A typical agent loop issues many short completions plus occasional long generations; your pool must handle both.

Use a structured metrics emitter:

from prometheus_client import Histogram

TOKEN_HIST = Histogram("agent_tokens", "Tokens per request", ["model"])
LATENCY = Histogram("agent_ttft", "Time to first token", ["model"])

def serve(model, prompt):
    with LATENCY.labels(model).time():
        tokens = generate(prompt)
        TOKEN_HIST.labels(model).observe(len(tokens))

Profile under realistic concurrency. A single A100 may saturate at 4 concurrent 7B batches but only 1 concurrent 70B stream. That ratio drives your replica count.

2. Pick instance types and bin-packing strategy

GPU instances are expensive and slow to attach. Prefer a few large instances (e.g., 8x A100) over many small ones if your scheduler can bin-pack pods with MIG or time-slicing. Kubernetes Device Plugins expose fractional GPUs; use them.

# Pod spec excerpt
resources:
  limits:
    nvidia.com/gpu: 1
    # or fractional with MIG: nvidia.com/mig-1g.5gb: 1

Tradeoff: fractional GPUs raise scheduling complexity and can starve neighbors under noisy workloads. For high-traffic AI agents, isolate heavy models on dedicated nodes using taints.

kubectl taint nodes gpu-heavy node-role=gpu-heavy:NoSchedule

3. Export the right autoscaling signals

The default CPU/memory HPA is useless for GPUs. Export a custom metric: queue depth per model, or tokens-in-flight. Push to Prometheus and expose via the Prometheus Adapter.

# prometheus-adapter config (partial)
rules:
- seriesQuery: 'agent_queue_depth{model!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
  name:
    as: "gpu_queue_depth"
  metricsQuery: 'avg(agent_queue_depth{model!=""}) by (namespace)'

Pitfall: scaling on GPU utilization alone lags because utilization spikes only after requests land. Queue depth predicts load before the GPU heats up.

4. Tune the Horizontal Pod Autoscaler

Set target queue depth, not utilization. Use behavior flags to slow scale-down and speed scale-up.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: agent-pool
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: agent-deployment
  minReplicas: 2
  maxReplicas: 32
  metrics:
  - type: External
    external:
      metric:
        name: gpu_queue_depth
      target:
        type: AverageValue
        averageValue: "4"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

This expands aggressively when queue depth exceeds 4, but waits five minutes before removing capacity. Cold GPU pods take 20–40s to pull images and load weights; premature scale-down causes flapping.

5. Keep a warm buffer for spikes

Autoscaling GPU pools for AI agents must absorb bursty agent traffic. Maintain a warm minimal replica set (minReplicas >= expected baseline) and use preemptible/spot nodes for the overflow tier.

# Node pool label for spot
kubectl label node spot-gpu-1 workload=spot

Route non-latency-critical batch agents to spot. If spot reclaims, your HPA pulls on-demand capacity. Tradeoff: spot interruptions cause request failures if you don’t cap concurrent spot usage below 30% of total.

6. Offload long-tail models to a gateway

Not every agent call needs your GPUs. Routing rare model requests externally keeps your autoscaling GPU pools for AI agents focused on the dominant 80% of traffic. For models you don’t self-host, an OpenAI-compatible endpoint with broad model coverage works.

n4n.ai provides one endpoint addressing 240+ models with automatic fallback when a provider is rate-limited, which avoids building your own multi-provider retry logic. Your internal pool handles the hot paths; the gateway covers the tail.

{
  "routing": {
    "self_hosted": ["llama-3-70b", "mixtral-8x22b"],
    "gateway": ["gpt-4o", "claude-3-opus", "rare-embedding-model"]
  }
}

Honor cache-control hints from the gateway to reduce repeat cost.

7. Validate with synthetic load and chaos

Write a loader that replays agent traces. Don’t use uniform random traffic; agent patterns are bursty with think-then-act gaps.

import asyncio, random

async def agent_session(client):
    while True:
        await client.complete(prompt=random_prompt())
        await asyncio.sleep(random.expovariate(0.1))  # think gap

# launch 500 sessions

Then kill a node mid-run. Confirm HPA replaces capacity within SLO. Pitfall: many teams test scaling only at startup, never during steady state, missing slow metric scrapes.

Common pitfalls and tradeoffs

Metric scrape interval too long: Default 30s scrape hides sub-minute spikes. Drop to 10s for the queue metric.
Ignoring model memory footprint: Loading two 70B models on one 80GB GPU fails silently. Use init containers to preload and fail fast.
Over-reliance on serverless GPU: Cold starts of 60s+ break interactive agents. Use warm pools.
No request prioritization: A background summarization job can starve a user-facing agent. Use priority classes.

Final checklist

Profile token and concurrency per model.
Export queue-depth custom metric.
HPA on queue depth, aggressive up / conservative down.
Warm baseline + spot overflow.
Offload tail to gateway.
Load test with realistic traces and node failure.

Autoscaling GPU pools for AI agents is an operational discipline, not a YAML toggle. Get the metrics right and the scaling follows.