Prompt regression testing in CI: a GPT-5 walkthrough

Prompt regression testing in CI for GPT-5 is the only way to know your carefully tuned system prompt didn’t quietly break when someone edited a few lines. This tutorial builds a pytest-based harness that runs real GPT-5 completions against fixed assertions on every pull request.

Prerequisites

• Python 3.11 or newer
• An OpenAI account with access to the gpt-5 model
• openai and pytest installed (pip install openai pytest)
• A GitHub repository if you want the CI portion
• Basic familiarity with pytest parametrization

Project layout

Keep the surface area small. You want prompts, test cases, and one test file.

prompt-regression/
├── prompts.py
├── cases.json
├── test_prompts.py
└── .github/
    └── workflows/
        └── prompt-regression.yml

Define the prompt under test

Put the system prompt in code so it is version-controlled. Avoid inline strings in tests; you want a single source of truth.

# prompts.py
def support_classifier_system() -> str:
    return (
        "You are a support ticket classifier. "
        "Respond ONLY with a JSON object of form "
        "{\"queue\": \"billing|tech|general\", \"urgent\": true|false}. "
        "No prose."
    )

Author regression cases

Exact match is brittle. Assert on structural properties: required keys, allowed values, substrings. Store cases as data.

{
  "cases": [
    {
      "system_prompt": "You are a support ticket classifier. Respond ONLY with a JSON object of form {\"queue\": \"billing|tech|general\", \"urgent\": true|false}. No prose.",
      "user_input": "I was charged twice for my subscription, help!",
      "expect_contains": ["billing", "urgent"],
      "expect_json": true
    },
    {
      "system_prompt": "You are a support ticket classifier. Respond ONLY with a JSON object of form {\"queue\": \"billing|tech|general\", \"urgent\": true|false}. No prose.",
      "user_input": "The app crashes on login after update.",
      "expect_contains": ["tech"],
      "expect_json": true
    }
  ]
}

Write the pytest harness

The test loads cases, calls GPT-5 with temperature=0, and validates constraints. Real API calls are slow but that is the point: you are testing the model’s behavior against your prompt.

# test_prompts.py
import os
import json
import pytest
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def load_cases():
    with open("cases.json") as f:
        data = json.load(f)
    return data["cases"]

def _is_valid_json(text: str) -> bool:
    try:
        json.loads(text)
        return True
    except ValueError:
        return False

@pytest.mark.parametrize("case", load_cases())
def test_gpt5_prompt_regression(case):
    resp = client.chat.completions.create(
        model="gpt-5",
        messages=[
            {"role": "system", "content": case["system_prompt"]},
            {"role": "user", "content": case["user_input"]}
        ],
        temperature=0,
        max_tokens=128
    )
    output = resp.choices[0].message.content.strip()
    if case.get("expect_json"):
        assert _is_valid_json(output), f"Output not JSON: {output}"
    for needle in case.get("expect_contains", []):
        assert needle in output, f"Expected '{needle}' in: {output}"

Expected output (local pass)

Run it:

export OPENAI_API_KEY=sk-your-key
pytest -q

..
2 passed in 3.10s

If someone changes the prompt to allow prose, the JSON assertion fails:

>       assert _is_valid_json(output), f"Output not JSON: {output}"
E       AssertionError: Output not JSON: Sure! Here is the classification: {"queue": "billing", "urgent": true}

Wire into GitHub Actions

The CI job runs the same suite on every PR. Secrets stay in GitHub; the key never hits the repo.

# .github/workflows/prompt-regression.yml
name: prompt-regression
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install openai pytest
      - run: pytest -q
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

This workflow makes prompt regression testing in CI for GPT-5 a blocking check. A bad prompt edit cannot merge without breaking the build.

Making it deterministic and cheap

GPT-5 with temperature=0 is mostly stable but not byte-identical across runs. Structure your assertions around properties, not exact strings.

Set max_tokens tight. A classifier response should never need 2k tokens.

If you route through n4n.ai, it honors client routing directives and forwards provider cache-control hints, so you can prefix your system prompt with a cache breakpoint to cut token cost on repeated CI runs. That keeps the suite affordable when it runs on every commit.

For faster feedback, cache the last good responses in a fixture and only hit the API when the prompt hash changes. Pseudo-code:

import hashlib

def prompt_hash(system, user):
    return hashlib.sha256((system+user).encode()).hexdigest()[:8]

# skip API if cached hash matches and golden file exists

What to assert beyond substrings

Substring checks catch drift but miss semantic rot. Add a second layer: embed the output and compare cosine similarity to a stored golden vector. If the prompt changes intended behavior, similarity drops below threshold.

# using openai embeddings
def embed(text):
    r = client.embeddings.create(model="text-embedding-3-small", input=text)
    return r.data[0].embedding

def test_semantic_stability(case, golden):
    out = generate(case)
    sim = cosine(embed(out), golden[case["id"]])
    assert sim > 0.92

Keep thresholds conservative; models update silently.

Handling rate limits in CI

A regression suite that flakes on 429s is worse than none. Retry with backoff at the client level:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def call_gpt5(messages):
    return client.chat.completions.create(model="gpt-5", messages=messages, temperature=0)

If you need cross-provider redundancy, a gateway with automatic fallback removes the 429 entirely. That is the only way prompt regression testing in CI for GPT-5 stays green under heavy PR traffic.

Extending to multiple prompts

As you accumulate prompts, nest them in prompts.py and generate cases per prompt ID. Parametrize by (prompt_id, case). The pattern scales to dozens of prompts without new test code.

@pytest.mark.parametrize("pid,case", [(p, c) for p in PROMPTS for c in CASES[p]])
def test_all_prompts(pid, case):
    sys = PROMPTS[pid]()
    # same call + assert logic

Run the suite daily on main too, not just on PRs, to catch upstream model shifts.

That is the whole loop: version prompts, assert properties, run in CI, fail on drift. Implement it in your next PR.