A prompt regression suite in GitHub Actions turns fragile prompt edits into a failing build instead of a production incident. This tutorial builds a minimal but real setup: pytest tests that call an OpenAI-compatible API, plus a workflow that runs them on every pull request.
Prerequisites
- Python 3.11 or newer installed locally
- A GitHub repository with Actions enabled
- An API key for an OpenAI-compatible endpoint (OpenAI or any gateway that speaks the same protocol)
- Familiarity with basic pytest usage
Install the dependencies now:
pip install openai pytest python-dotenvCreate a requirements.txt to lock versions for CI:
openai>=1.30
pytest>=8.0
python-dotenv>=1.0Project layout
Keep the test code separate from application code. A clean tree looks like this:
.
├── .github/
│ └── workflows/
│ └── prompt-regression.yml
├── tests/
│ ├── conftest.py
│ └── test_prompts.py
├── .env.example
└── requirements.txtThe .env.example should contain only the key name:
OPENAI_API_KEY=sk-your-key-hereConfigure the OpenAI client
In tests/conftest.py we load environment variables and expose a session-scoped client fixture. Session scope avoids rebuilding the client per test.
import os
import openai
import pytest
from dotenv import load_dotenv
load_dotenv()
@pytest.fixture(scope="session")
def client():
return openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])If you later route through a gateway, only the base_url argument changes. The test logic stays identical.
Write your first prompt test
Prompt regression is about asserting that a model produces acceptable output for a fixed prompt. Exact string matching is brittle; assert on properties instead.
Create tests/test_prompts.py:
def call_summarize(client, text, model="gpt-4o-mini"):
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "Summarize in one sentence under 20 words."},
{"role": "user", "content": text},
],
temperature=0,
)
return resp.choices[0].message.content.strip()
def test_summarize_basic(client):
text = "The quick brown fox jumps over the lazy dog. The dog was asleep near the barn."
out = call_summarize(client, text)
words = out.split()
assert 0 < len(words) <= 20
assert "fox" in out.lower() or "dog" in out.lower()Run it locally to confirm the wiring:
pytest tests/ -vExpected output:
tests/test_prompts.py::test_summarize_basic PASSEDTest structured output
Many production prompts emit JSON. Use the provider’s JSON mode and validate the shape, not the exact values.
import json
def test_extract_json(client):
prompt = "Return JSON with keys 'name' and 'age' for: John is 30."
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0,
)
data = json.loads(resp.choices[0].message.content)
assert set(data.keys()) == {"name", "age"}
assert data["age"] == 30If the model drops a key or mutates types, the test fails. That is exactly the regression signal you want.
Run a prompt regression suite in GitHub Actions across models
Model swaps cause silent behavior changes. Parametrize over the models you ship:
import pytest
MODELS = ["gpt-4o-mini", "gpt-3.5-turbo"]
@pytest.mark.parametrize("model", MODELS)
def test_summarize_all_models(client, model):
text = "Cats are mammals. They have whiskers and sharp claws."
out = call_summarize(client, text, model=model)
assert "cat" in out.lower()Now your prompt regression suite in GitHub Actions executes the same contract on every backed model.
GitHub Actions workflow
Create .github/workflows/prompt-regression.yml:
name: prompt-regression
on:
pull_request:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt
- run: pytest tests/ -v
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}Add the secret in repo Settings → Secrets → Actions. On each PR, the workflow runs the suite.
To avoid wasted runs, filter on relevant paths:
on:
pull_request:
paths:
- 'tests/**'
- 'prompts/**'Using a gateway for resilience
If you point the client at n4n.ai, an OpenAI-compatible gateway over 240+ models with automatic fallback when a provider is degraded, your prompt regression suite in GitHub Actions stays green even if one backend rate-limits you. The client code does not change; set base_url and api_key to the gateway credentials and keep the same model identifiers.
Parallelize with a matrix
Instead of parametrizing inside pytest, let GitHub Actions run one job per model. This isolates rate limits and speeds feedback.
jobs:
test:
strategy:
matrix:
model: [gpt-4o-mini, gpt-3.5-turbo]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt
- run: pytest tests/ -v --model ${{ matrix.model }}
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}Support the flag in conftest.py:
def pytest_addoption(parser):
parser.addoption("--model", action="store", default="gpt-4o-mini")
@pytest.fixture
def model(request):
return request.config.getoption("--model")Update tests to use the model fixture instead of a hard-coded string. This scales better than in-process loops.
Expected CI output
A green run logs:
Run pytest tests/ -v
============================= test session starts ==============================
tests/test_prompts.py::test_summarize_basic PASSED
tests/test_prompts.py::test_extract_json PASSED
tests/test_prompts.py::test_summarize_all_models[gpt-4o-mini] PASSED
tests/test_prompts.py::test_summarize_all_models[gpt-3.5-turbo] PASSEDA red run blocks the merge, showing which assertion broke.
Debugging a failing prompt test
When CI goes red, reproduce locally with the same model flag and drop into pdb:
pytest tests/ -v --model gpt-4o-mini --pdbPrint the raw completion if the assertion is unclear:
def test_summarize_debug(client, model):
out = call_summarize(client, "Sample text", model=model)
print("RAW:", out)
assert outGitHub Actions also preserves step logs; expand the pytest step to see the captured print.
Keep the suite fast and cheap
Prompt calls are slow and metered. Three tactics help:
- Cache responses with
vcr.pyor a simple SQLite cache keyed by prompt+model hash. Replay cached fixtures offline. - Pin temperature to 0 for deterministic outputs.
- Limit
max_tokensto the minimum needed for the assertion.
Example cache decorator sketch:
import hashlib, json, os
CACHE = ".prompt_cache"
def cached_call(client, model, messages):
key = hashlib.sha256((model + json.dumps(messages)).encode()).hexdigest()
path = os.path.join(CACHE, key)
if os.path.exists(path):
return json.loads(open(path).read())
resp = client.chat.completions.create(model=model, messages=messages, temperature=0)
os.makedirs(CACHE, exist_ok=True)
open(path, "w").write(resp.choices[0].message.content)
return resp.choices[0].message.contentCommit the cache only for fixed fixtures; never commit secrets.
Validate with schemas
For stricter contracts, use pydantic to parse the JSON and surface precise errors:
from pydantic import BaseModel, ValidationError
class Person(BaseModel):
name: str
age: int
def test_json_schema(client):
prompt = "Return JSON with keys 'name' and 'age' for: John is 30."
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
temperature=0,
)
data = json.loads(resp.choices[0].message.content)
try:
Person(**data)
except ValidationError as e:
pytest.fail(f"Schema broke: {e}")A robust prompt regression suite in GitHub Actions checks invariants: length bounds, required entities, JSON schema, sentiment polarity. It does not check “the summary must contain the word ‘fox’ exactly”. Use semantic similarity via embeddings if you need tighter control.
Wrap-up of the pipeline
You now have a pytest suite that calls real models, a GitHub Actions workflow that runs it on PRs, and patterns for multi-model coverage, matrix parallelism, and caching. Extend the tests/ folder as prompts evolve, and treat a failing test as a required code review checkpoint.