CI/CD pipelines for shipping AI agents to production

Shipping an LLM agent to production demands the same engineering discipline as any distributed service, but most teams still drag prompts and tool configs around by hand. This guide lays out a concrete CI/CD for AI agent deployment that treats agent logic as versioned code, validates it in CI, and promotes builds through staging to production with automated guardrails.

Step 1: Define the agent as declarative config

Hardcode nothing about the agent’s behavior in Python. Put the model name, system prompt, tool schemas, and sampling parameters in a versioned JSON or YAML file. This makes diffs reviewable and lets you run the same agent against different models by swapping one field.

{
  "model": "gpt-4o-mini",
  "system_prompt": "You are a refund authorization agent. Use tools to verify order status before approving.",
  "tools": [
    {
      "name": "lookup_order",
      "description": "Fetch order by ID",
      "parameters": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"]
      }
    }
  ],
  "max_tokens": 1024,
  "temperature": 0.2
}

Load it at startup and fail fast if the schema is invalid. Treat this file as a first-class artifact; bump its version in Git on every change.

Verify success

python -c "import json; json.load(open('agent_config.json'))" exits 0 and CI shows the file in the diff.

Step 2: Unit-test agent logic with mocked model calls

Your agent code should call a thin LLM wrapper, not the SDK directly. That seam lets you mock completions in unit tests and assert the orchestration logic: tool-call parsing, retry limits, and error handling.

# agent.py
def run_agent(user_msg, llm_complete):
    messages = [{"role": "user", "content": user_msg}]
    resp = llm_complete(messages, model=CONFIG["model"])
    return resp

# test_agent.py
def test_agent_returns_tool_call(monkeypatch):
    import agent
    def fake_complete(messages, **kwargs):
        return {"choices": [{"message": {"tool_calls": [{"name": "lookup_order"}]}}]}
    monkeypatch.setattr(agent, "llm_complete", fake_complete)
    out = agent.run_agent("Refund order 123", agent.llm_complete)
    assert out["choices"][0]["message"]["tool_calls"][0]["name"] == "lookup_order"

Run these in CI before any container build. They execute in milliseconds and catch 80% of logic regressions.

Verify success

pytest -q reports passed tests and the pipeline proceeds to the build stage.

Step 3: Containerize the agent runtime

Package the agent as a stateless HTTP service. Keep the image small and pin the Python version. Inject the config path and API keys at runtime, never bake them in.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY agent.py server.py agent_config.json ./
CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8080"]

Build with a deterministic tag—the Git SHA. That tag is your deployment unit for the rest of the CI/CD for AI agent deployment flow.

Verify success

docker build -t agent:$(git rev-parse --short HEAD) . completes and docker run --rm agent:sha python -c "import agent" exits clean.

Step 4: Build the CI pipeline

A GitHub Actions workflow should run lint, unit tests, then build and push the image. Split jobs so a failing test never wastes a Docker build.

name: ci
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pytest && pytest -q
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t ghcr.io/$GITHUB_REPOSITORY:${GITHUB_SHA::7} .
      - run: docker push ghcr.io/$GITHUB_REPOSITORY:${GITHUB_SHA::7}

Secrets for the container registry live in the CI secret store, not in the repo.

Verify success

The Actions tab shows green checkmarks on both jobs and the image appears in the registry with the SHA tag.

Step 5: Provision staging with infrastructure as code

Don’t click through a console to stand up staging. Use Terraform or Pulumi to declare the task definition, environment variables, and autoscaling. The same module promotes to prod with a different variable file.

resource "aws_ecs_task_definition" "agent" {
  family                = "refund-agent"
  container_definitions = jsonencode([{
    name      = "agent"
    image     = "ghcr.io/yourorg/agent:${var.sha}"
    essential = true
    portMappings = [{ containerPort = 8080 }]
  }])
  cpu       = "256"
  memory    = "512"
}

Apply with terraform apply -var="sha=$(git rev-parse --short HEAD)". Staging gets the same binary prod will run.

Verify success

terraform plan shows no drift and aws ecs describe-tasks lists the new revision running.

Step 6: Run integration tests against a live model gateway

Unit tests mock the model; integration tests must hit a real endpoint to catch prompt regressions and tool-schema mismatches. Point the test suite at an OpenAI-compatible gateway that fronts multiple providers so a single flaky model doesn’t red-light the pipeline. n4n.ai exposes one endpoint covering 240+ models and automatically fails over when a provider is rate-limited, which keeps the integration stage deterministic enough to trust.

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["N4N_KEY"])
resp = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": "Return JSON: {'ok': true}"}],
    response_format={"type": "json_object"},
    extra_headers={"x-cache-control": "ephemeral"}
)
assert "ok" in resp.choices[0].message.content

The gateway forwards cache-control hints and meters per token, so you see exact cost per integration run.

Verify success

The pytest integration marker passes and the CI log prints token usage from the response headers.

Step 7: Promote to production with canary

Ship to prod behind a canary weight. If you run Kubernetes, Argo Rollouts lets you shift 10% traffic, watch metrics, then ramp. The key is that the same image tag from staging moves forward; you never rebuild.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

Automate the pause with a Prometheus query on 5xx rate and p95 latency.

Verify success

kubectl argo rollouts get rollout refund-agent shows healthy pods at full weight and no manual abort.

Step 8: Monitor and roll back

CI/CD for AI agent deployment doesn’t end at rollout. Stream agent logs, token counts, and tool-error rates to your observability stack. Alert on a spike in tool_call_failed or a drop in successful task completion.

If a prompt change causes hallucinations in canary, roll back with one command:

kubectl argo rollouts undo rollout/refund-agent

Keep the previous image tag in the Terraform state so infra and app versions stay in lockstep.

Verify success

Grafana shows stable latency and the rollback completes in under a minute; the stale tag is redeployed.

Closing checklist

• Agent config is JSON in Git.
• Unit tests mock the LLM seam.
• Image tagged with Git SHA.
• CI runs test → build → push.
• Staging and prod use same IaC module.
• Integration tests hit a failover-capable gateway.
• Canary promotes by weight, not by rebuild.
• Rollback is a single declarative command.

Follow these steps and your CI/CD for AI agent deployment will survive real traffic instead of breaking at 2 a.m. because someone edited a prompt in the UI.