Most teams bolt LLM calls onto an existing web service and call it done, then wonder why a prompt change broke production. A proper CI/CD pipeline for LLM apps treats prompts, eval fixtures, and model routing as code, with the same gates you’d apply to any critical path.
Prerequisites
- A GitHub repository with a Python service that calls an LLM.
- Python 3.11+ and
pipavailable locally. - An OpenAI-compatible API key (or a gateway key). If you use n4n.ai, its single OpenAI-compatible endpoint fronts 240+ models with automatic fallback.
ghCLI installed for workflow debugging.
Repository layout
Start with a structure that separates prompts, tests, and eval data:
.
├── prompts/
│ └── support.yaml
├── src/
│ ├── __init__.py
│ └── chain.py
├── tests/
│ ├── test_chain.py
│ └── eval_prompts.py
├── requirements.txt
└── .github/workflows/ci.ymlCommit this skeleton. The pipeline will lint prompts/, run unit tests, execute eval, then deploy.
Step 1: Version prompts as YAML
A prompt file should declare its model, temperature, and body. Lint it in CI to catch missing fields before they hit runtime.
prompts/support.yaml:
name: support_classifier
model: gpt-4o-mini
temperature: 0.0
body: |
Classify the user message into one of: billing, technical, other.
Respond with a single word.Linter script scripts/lint_prompts.py:
import sys, yaml, glob
required = {"name", "model", "temperature", "body"}
for path in glob.glob("prompts/*.yaml"):
with open(path) as f:
data = yaml.safe_load(f)
missing = required - data.keys()
if missing:
print(f"{path}: missing {missing}")
sys.exit(1)
print("prompts OK")Run it locally:
python scripts/lint_prompts.pyExpected output:
prompts OKStep 2: Unit-test the non-LLM logic
Isolate the code that builds the request. This runs fast and needs no network.
src/chain.py:
from openai import OpenAI
def build_messages(prompt_body: str, user_text: str):
return [
{"role": "system", "content": prompt_body},
{"role": "user", "content": user_text},
]
def classify(client: OpenAI, model: str, prompt_body: str, user_text: str):
resp = client.chat.completions.create(
model=model,
messages=build_messages(prompt_body, user_text),
temperature=0.0,
)
return resp.choices[0].message.content.strip().lower()tests/test_chain.py:
from src.chain import build_messages
def test_build_messages():
msgs = build_messages("sys", "hi")
assert msgs[0]["role"] == "system"
assert msgs[1]["content"] == "hi"Run:
pip install -r requirements.txt
pytest tests/test_chain.py -qExpected:
1 passed in 0.02sStep 3: Eval harness against real models
Eval is the part a CI/CD pipeline for LLM apps cannot skip. You need to know if a prompt change degrades accuracy. Use a small fixture set and call the model via an OpenAI-compatible client.
Point the client at your gateway. For example, n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models and falls back automatically when a provider is degraded—useful so a flaky upstream doesn’t red-circle your CI.
tests/eval_prompts.py:
import yaml, os
from openai import OpenAI
from src.chain import classify
client = OpenAI(
base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
api_key=os.environ["LLM_API_KEY"],
)
def load_prompt():
with open("prompts/support.yaml") as f:
return yaml.safe_load(f)
CASES = [
("I was charged twice", "billing"),
("App crashes on login", "technical"),
("What are your hours", "other"),
]
def test_support_accuracy():
p = load_prompt()
correct = 0
for text, expected in CASES:
pred = classify(client, p["model"], p["body"], text)
if pred == expected:
correct += 1
score = correct / len(CASES)
assert score >= 0.99, f"accuracy {score} below gate"Run locally with your key:
export LLM_API_KEY=sk-...
pytest tests/eval_prompts.py -qExpected (if models behave):
1 passed in 2.31sIf a prompt edit breaks the “other” case, the assertion fails and CI goes red.
Step 4: GitHub Actions workflow
Wire the steps into .github/workflows/ci.yml. Keep secrets in GitHub, not in code.
name: ci
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt pytest pyyaml openai
- name: Lint prompts
run: python scripts/lint_prompts.py
- name: Unit tests
run: pytest tests/test_chain.py -q
- name: Eval
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}
run: pytest tests/eval_prompts.py -q
- name: Deploy
if: github.ref == 'refs/heads/main' && success()
run: ./scripts/deploy.shThe LLM_BASE_URL secret can point to your gateway. This CI/CD pipeline for LLM apps fails fast on structural errors, then validates behavior.
Step 5: Gate deployments on eval thresholds
The assert in eval_prompts.py already blocks merge on low accuracy. For stricter control, compute a metric and exit non-zero:
import sys
# after scoring
if score < 0.99:
print(f"Eval gate failed: {score}")
sys.exit(1)In the workflow, the Eval step returning non-zero prevents the Deploy step from running because of success().
Step 6: Deploy script
Keep deploy dumb. It assumes the artifact is built and tests passed.
scripts/deploy.sh:
#!/usr/bin/env bash
set -euo pipefail
echo "Deploying $(git rev-parse HEAD)"
# flyctl deploy or render deploy etc.
./vendor/deploy-tool pushCommit and push. Open a PR that modifies prompts/support.yaml temperature to 0.7. The eval step will likely still pass (classification is deterministic at 0.0, but at 0.7 it may wander). That red build is the pipeline earning its keep.
What you get
A CI/CD pipeline for LLM apps that catches prompt regressions, validates model behavior on every commit, and deploys only when evals hold. The pattern scales: add more eval cases, split prompts per feature, and route via a gateway that meters tokens so you can track CI spend per run.
Treat prompts like code, and your LLM features stop breaking unexpectedly.