Blue-green deployment AI agents lets you ship new agent logic without dropping a single in-flight request. The pattern duplicates your serving stack and swaps traffic only after the shadow environment proves healthy, which matters when your agents hold long-lived conversations with LLM backends and cannot afford a cold restart.
Step 1: Provision isolated blue and green stacks
Start with two identical infrastructure environments differentiated by a tag, not by code. Use the same container orchestrator (Kubernetes, Nomad, or plain Docker Compose) and parameterize the image tag.
# docker-compose.blue-green.yml
services:
blue:
image: registry.internal/agent:1.0.2
environment:
- AGENT_ENV=blue
ports: ["8080:8000"]
green:
image: registry.internal/agent:1.1.0-rc1
environment:
- AGENT_ENV=green
ports: ["8081:8000"]Keep the agent process stateless. Any conversation state belongs in Redis or a database, not in process memory, or you cannot shift traffic mid-session. If you must keep warm connections to an LLM gateway, use a single shared connection pool outside the per-request path.
Example session store:
import redis, json
r = redis.Redis(host="session-db", port=6379, decode_responses=True)
def save_ctx(session_id, messages):
r.set(f"agent:ctx:{session_id}", json.dumps(messages), ex=3600)Network isolation should be identical: both stacks in the same subnet, same security groups, same outbound access to your model endpoint. Differences in egress rules are a classic source of “works in green, fails in prod” bugs.
Step 2: Deploy the new agent to green without touching blue
Build and push the new image, then start only the green service. Blue continues serving production traffic.
docker compose -f docker-compose.blue-green.yml up -d greenYour agent code should read its model endpoint from an environment variable. A minimal FastAPI agent looks like this:
# agent.py
from fastapi import FastAPI
from openai import OpenAI
import os
app = FastAPI()
client = OpenAI(
base_url=os.environ["LLM_BASE_URL"],
api_key=os.environ["LLM_API_KEY"],
default_headers={"x-routing-directive": os.environ.get("MODEL_ROUTE", "")}
)
@app.post("/chat")
def chat(prompt: str):
resp = client.chat.completions.create(
model=os.environ["MODEL_NAME"],
messages=[{"role": "user", "content": prompt}]
)
return {"text": resp.choices[0].message.content}Setting x-routing-directive lets a gateway pin the upstream provider. n4n.ai honors client routing directives, so both blue and green hit the same model backend even while you cut over, avoiding behavioral drift from provider fallback. If your gateway forwards provider cache-control hints, prompt caching stays effective across the swap because the cache key is stable.
Step 3: Smoke-test green in isolation
Never shift live traffic to untested code. Hit the green port directly with a representative payload.
curl -s localhost:8081/chat -H 'content-type: application/json' \
-d '{"prompt":"Summarize: blue-green deployment AI agents reduce risk"}'Assert the response shape and latency. Add a scripted check that fails on non-200 or missing fields:
# smoke.py
import requests, sys
r = requests.post("http://localhost:8081/chat",
json={"prompt":"test"}, timeout=10)
assert r.status_code == 200, r.status_code
assert "text" in r.json(), "no text field"
print("green OK")Run it in CI before promoting green. Beyond happy-path, test the exact prompt templates your production agent uses, including any function-calling schemas. A changed schema is the most common breaking change in agent deploys.
# schema_check.py
payload = {"prompt": "Book a flight", "tools": [{"type": "function", "name": "book"}]}
r = requests.post("http://localhost:8081/chat", json=payload)
assert "tool_calls" in r.json(), "schema regression"Step 4: Shift traffic with weighted routing
Put both stacks behind a reverse proxy that supports weight adjustment. Envoy or NGINX Plus do this; plain NGINX requires a reload. Below is a minimal NGINX upstream config you can swap without downtime by editing weights and sending reload.
upstream agent_pool {
server 127.0.0.1:8080 weight=100; # blue
server 127.0.0.1:8081 weight=0; # green
}
server {
listen 80;
location / {
proxy_pass http://agent_pool;
}
}To begin cutover, set green to 10 and blue to 90, then reload:
sed -i 's/8080 weight=100/8080 weight=90/; s/8081 weight=0/8081 weight=10/' /etc/nginx/conf.d/agent.conf
nginx -s reloadWatch error rates. If green throws 5xx, drop its weight back to 0. Blue-green deployment AI agents only pays off if you can abort the switch in seconds.
For Envoy, use a weighted cluster:
routes:
- match: { prefix: "/" }
route:
weighted_clusters:
clusters:
- { name: blue, weight: 90 }
- { name: green, weight: 10 }Change weights via the xDS API without restarting the proxy.
Step 5: Verify success under real load
After weights favor green (e.g., 50/50), pull metrics from both stacks. Track tail latency, token throughput, and exception counts. A quick verification loop in Python:
# verify.py
import requests, time
for _ in range(20):
for port in (8080, 8081):
r = requests.post(f"http://localhost:{port}/chat",
json={"prompt":"verify"}, timeout=5)
print(port, r.status_code, len(r.text))
time.sleep(1)Success criteria: green error rate < 0.5%, p95 latency within 20% of blue, and no schema violations in responses. If you use a gateway with per-token metering, compare token usage curves—a broken agent often loops or over-calls the model, spiking tokens on green.
Export Prometheus counters to make this observable:
from prometheus_client import Counter
TOKENS = Counter("agent_tokens_total", "tokens used", ["env"])
TOKENS.labels(os.environ["AGENT_ENV"]).inc(resp.usage.total_tokens)Graph the two env labels side by side; a sudden divergence signals a logic regression.
Step 6: Finalize or roll back
When green is stable at weight 100 for an observation window (15–30 minutes), retire blue:
docker compose -f docker-compose.blue-green.yml stop blueKeep the blue image tagged and deployable for another hour in case late bugs appear. To roll back, reverse the weights and reload NGINX:
sed -i 's/8080 weight=90/8080 weight=100/; s/8081 weight=10/8081 weight=0/' /etc/nginx/conf.d/agent.conf
nginx -s reloadIf you deployed via Kubernetes, flip the selector on the Service instead of weight editing:
kubectl patch service agent -p '{"spec":{"selector":{"version":"green"}}}'That atomic swap is even safer than weights.
Verifying zero downtime
The definitive test: open a long-lived session against the proxy port (80) and run the weight shift while requests flow. If any request returns connection reset, your proxy or agent is not truly stateless. Use a simple loop:
for i in $(seq 1 1000); do
curl -s localhost/chat -d '{"prompt":"continuous"}' >/dev/null || echo "FAIL $i"
doneRun this during the Step 4 shift. Zero FAIL lines means your blue-green deployment AI agents pipeline is solid.
Operational notes
- Automate the weight changes with a small controller (Consul Template, Helm, or a Python script calling your LB API) to avoid manual sed errors.
- Log the
AGENT_ENVvalue on every request so you can attribute post-deploy issues to the correct stack. - If your agent uses streaming, ensure the proxy buffers correctly; broken chunked transfers are the most common silent failure during cutover.
- Keep a runbook: exact commands, expected metrics, and the person to page. Blue-green is mechanical, but humans cause the outages.
Blue-green deployment AI agents is not exotic, but it demands discipline: identical infra, externalized state, and a kill-switch weight. Do that and you ship agent updates as calmly as static web pages.