A prompt edit that drops JSON validity or subtly changes tone is a regression like any other. The discipline of catching prompt regressions before deploy protects users from silent failures that unit tests on application code will never surface. Treat prompts as shipped artifacts, not strings buried in source, and you can block bad changes at the pull request.
Step 1: Version prompts as code
Store every prompt template in its own file with an explicit version field. Keep them out of Python or TypeScript literals so diffs are reviewable and rollbacks are trivial.
# prompts/summarize.yaml
version: 4
model: gpt-4o-mini
template: |
Summarize the following text in {{ max_sentences }} sentences.
Text: {{ input }}
Return strict JSON: {"summary": string}Load it with a tiny helper and assert the version you expect at test time:
import yaml
def load_prompt(name: str) -> dict:
with open(f"prompts/{name}.yaml") as f:
return yaml.safe_load(f)
def test_prompt_version():
p = load_prompt("summarize")
assert p["version"] == 4, "Prompt version drift"Verify success: pytest tests/test_prompts.py passes and the YAML file shows up in git diff when someone changes wording.
Step 2: Build a golden eval set
A regression test needs a fixed reference. Create a JSONL file of inputs paired with either exact expectations or constraint objects. Ten to fifty cases covers most single-purpose prompts.
{"input": "Quantum computing uses superposition.", "constraints": {"must_contain": ["qubit"], "max_words": 20}}
{"input": "The loan was denied due to low credit.", "constraints": {"tone": "neutral", "no_advice": true}}Keep the set in eval/summarize.jsonl. These are not training data; they are assertions you maintain like fixtures.
Step 3: Write deterministic assertion tests
For structured outputs, never trust the model to be consistent. Validate schema and hard constraints before any fuzzy grading. Use Pydantic to parse and fail fast.
from pydantic import BaseModel, ValidationError
class Summary(BaseModel):
summary: str
def check_constraints(output: dict, golden: dict) -> None:
try:
Summary(**output)
except ValidationError as e:
raise AssertionError(f"Schema regression: {e}")
text = output["summary"]
c = golden["constraints"]
if "must_contain" in c:
for term in c["must_contain"]:
assert term.lower() in text.lower(), f"Missing {term}"
if "max_words" in c:
assert len(text.split()) <= c["max_words"], "Exceeded word limit"Run this against recorded baseline outputs first, then against live generations in CI.
Verify success: A test that feeds last week’s saved outputs through check_constraints passes; a hand-broken prompt fails the assertion.
Step 4: Add model-graded regression checks
Deterministic checks miss tone and relevance. Use a separate LLM call as a judge. Keep the judge strict and ask for machine-readable verdicts.
from openai import OpenAI
import os, json
# If you run evals across multiple providers, point the client at a gateway
# such as n4n.ai—one OpenAI-compatible endpoint covering 240+ models with
# automatic fallback when a provider is rate-limited—so a single suite
# exercises diverse backends.
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key=os.environ["N4N_KEY"])
def judge(output: str, golden: dict) -> dict:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a strict grader. Reply only JSON {\"pass\": bool, \"reason\": str}"},
{"role": "user", "content": f"Constraints: {json.dumps(golden['constraints'])}\nOutput: {output}"}
],
response_format={"type": "json_object"}
)
return json.loads(resp.choices[0].message.content)
def test_model_graded():
golden = {"input": "The loan was denied.", "constraints": {"tone": "neutral"}}
out = generate(load_prompt("summarize"), golden["input"])
verdict = judge(out, golden)
assert verdict["pass"], verdict["reason"]Cache judge responses in CI to avoid recompute. The judge model should differ from the generation model when possible.
Verify success: Inject a deliberately unneutral rewrite; the judge returns {"pass": false} and the test fails.
Step 5: Wire into CI/CD
Add a GitHub Actions workflow that runs on every pull request touching prompts/ or eval/.
# .github/workflows/prompt-tests.yml
name: prompt-regression
on:
pull_request:
paths: ["prompts/**", "eval/**", "tests/prompt_regression.py"]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install pyyaml pydantic openai pytest
- run: pytest tests/prompt_regression.py -q
env:
N4N_KEY: ${{ secrets.N4N_KEY }}This blocks merges when a prompt change breaks schema, constraints, or judge checks.
Verify success: Open a PR that deletes must_contain; the workflow turns red and the merge button is disabled.
Step 6: Gate deployments on regression score
A single flaky judge call should not halt shipping, but a cluster of failures must. Compute a pass rate across the eval set and fail the build below a threshold.
def test_regression_suite():
cases = [json.loads(l) for l in open("eval/summarize.jsonl")]
passes = 0
for case in cases:
out = generate(load_prompt("summarize"), case["input"])
check_constraints(out, case)
if judge(out, case)["pass"]:
passes += 1
rate = passes / len(cases)
assert rate >= 0.95, f"Regression rate {rate:.2f} below gate"Run this as the last step in the workflow. Track the rate over time with a simple artifact or dashboard.
Verify success: With the current prompt, the suite reports rate >= 0.95. After a bad edit drops three cases, the gate fails and the deploy is blocked.
Step 7: Keep the eval set alive
Prompts evolve; so should the golden set. When you intentionally change behavior, update the relevant constraints in the same PR as the prompt edit. Reviewers then see both the new wording and the new contract.
Catching prompt regressions before deploy is not exotic. It is version control, fixtures, and a test runner pointed at the same interface your users hit. Do that, and prompt changes become as safe as any other code change.