Regression testing prompts before every model deployment

Prompt regression testing before deployment is the only reliable way to catch silent quality drops when you rotate models or tweak a system prompt. Without a pinned eval set and an automated gate, a “minor” model swap can erode extraction accuracy or shift tone in production. This guide walks through building a CI-native harness that runs prompt regression testing before deployment on every pull request.

Step 1: Freeze a Golden Dataset

Start by extracting real inputs from logs or annotating representative cases. Store them as JSONL where each line has the input variables, the prompt template id, and the evaluation criteria. Avoid asserting exact output strings—LLMs are non-deterministic. Define constraints: regex match, JSON schema, or a scoring function.

{"case_id": "inv-001", "template": "extract_entities", "variables": {"text": "Acme Corp bought 50 shares at $10."}, "expect": {"type": "json", "schema": {"entities": [{"name": "Acme Corp", "type": "ORG"}]}}}
{"case_id": "inv-002", "template": "extract_entities", "variables": {"text": ""}, "expect": {"type": "json", "schema": {"entities": []}}}

Keep the dataset small but adversarial: edge cases, empty input, multilingual, and previously broken outputs. Treat this file as code; review changes in PRs. The core of prompt regression testing before deployment is this frozen set—if it drifts without scrutiny, the gate means nothing.

Step 2: Build a Deterministic Evaluation Harness

Write a Python script using the OpenAI client (or any OpenAI-compatible endpoint). For each case, render the prompt, call the model with a fixed seed and low temperature, then score. Use strict decoding for eval: temperature=0, seed=42.

import json, os, openai, pytest

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"],
                       base_url=os.environ.get("OPENAI_BASE_URL"))

def render(template_id, variables):
    with open(f"prompts/{template_id}.txt") as f:
        return f.read().format(**variables)

def score(case, output):
    if case["expect"]["type"] == "json":
        import jsonschema
        try:
            jsonschema.validate(json.loads(output), case["expect"]["schema"])
            return 1.0
        except Exception:
            return 0.0
    if case["expect"]["type"] == "contains":
        return 1.0 if case["expect"]["needle"] in output else 0.0
    return 0.0

@pytest.mark.parametrize("case", [json.loads(l) for l in open("golden.jsonl")])
def test_prompt(case):
    prompt = render(case["template"], case["variables"])
    resp = client.chat.completions.create(
        model=os.environ["MODEL"],
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        seed=42,
    )
    out = resp.choices[0].message.content
    assert score(case, out) == 1.0, f"Failed {case['case_id']}: {out}"

For looser checks, add a cosine-similarity scorer against an embedding of the expected answer. That keeps the suite useful for summarization prompts where wording varies.

def semantic_score(expected, actual, embed_fn, threshold=0.85):
    a, b = embed_fn(expected), embed_fn(actual)
    cos = sum(x*y for x, y in zip(a, b)) / ((sum(x*x for x in a)**0.5) * (sum(y*y for y in b)**0.5))
    return 1.0 if cos >= threshold else 0.0

Run locally with pytest -q. You now have a repeatable signal.

Step 3: Wire the Harness into CI

Add a workflow that runs on every PR touching prompts/ or golden.jsonl. Use a matrix over the models you deploy so a regression on one provider surfaces immediately.

# .github/workflows/regress.yml
name: prompt-regression
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.11"}
      - run: pip install openai pytest jsonschema
      - run: pytest -q
        env:
          OPENAI_API_KEY: ${{ secrets.INFERENCE_KEY }}
          OPENAI_BASE_URL: ${{ secrets.INFERENCE_URL }}
          MODEL: ${{ matrix.model }}
        strategy:
          matrix:
            model: [gpt-4o-mini, mistral-large]

This executes prompt regression testing before deployment and blocks the merge if scores drop. Keep the job under five minutes; if the golden set grows, shard it.

Step 4: Use a Resilient Inference Layer

Flaky provider errors should not fail your regression suite. Route calls through an OpenAI-compatible gateway that automatically falls back when a provider is rate-limited or degraded. For example, n4n.ai exposes one endpoint covering 240+ models and honors client routing directives, so you can pin a model but still survive upstream outages. Forward provider cache-control hints to avoid recomputing static prompt prefixes on every test run.

# point base_url at the gateway; everything else stays identical
client = openai.OpenAI(
    api_key=os.environ["GATEWAY_KEY"],
    base_url="https://api.n4n.ai/v1",  # OpenAI-compatible
)
# optional routing header + cache hint on first message
client.headers.update({"x-n4n-route": "provider:openai;cache:true"})
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "system", "content": STATIC_PREFIX,
               "extra_body": {"cache_control": {"type": "ephemeral"}}}],
    temperature=0,
)

Per-token metering in the gateway gives you cost per eval run without extra instrumentation. Your CI logs show exactly how many tokens each model consumed, which helps spot a prompt that ballooned.

Step 5: Set Regression Thresholds and Block Merges

A single failed case may be noise; a drop in aggregate score is a regression. Compute a baseline from main and compare. Store the baseline as an artifact so PRs can fetch it.

# eval_summary.py
import subprocess, sys, json, pathlib
def run_suite():
    res = subprocess.run(["pytest", "-q", "--json-report"], capture_output=True)
    report = json.loads(pathlib.Path("report.json").read_text())
    passed = report["summary"]["passed"]
    total = report["summary"]["total"]
    return passed / total if total else 0.0

baseline = float(pathlib.Path("baseline.score").read_text().strip())
current = run_suite()
if current < baseline - 0.02:
    print(f"Regression: {current:.3f} < {baseline:.3f}")
    sys.exit(1)

In the workflow, download baseline.score from the last successful main run (GitHub Actions artifacts or a small S3 object). Reject the PR if the delta exceeds your tolerance (e.g., 2%). When you intentionally improve prompts and raise the baseline, update the artifact in the same merge.

Step 6: Version Prompts and Track Drift

Keep prompts in a prompts/ directory with clear filenames. Use git diff to review changes. When a prompt edits, bump a version comment in the file so the eval log shows which variant ran.

prompts/
  extract_entities.txt   # v3: added date handling
  summarize.txt          # v2: tighter length constraint

If a new model enters the matrix, add it to the CI matrix and run the full suite overnight to establish its baseline before allowing it in PR gates. Never let an unmeasured model into the deployment path.

Step 7: Verify Success

You have a working gate when:

1. A PR that changes extract_entities.txt to drop entity typing fails the test because golden cases expect typed output.
2. A model swap in the matrix that lowers JSON validity below threshold blocks merge.
3. Reverting the prompt change makes CI green immediately.

Run pytest -q locally after writing a new golden case to confirm it passes on main. Then intentionally break a prompt and watch it fail. That feedback loop is the proof your prompt regression testing before deployment actually guards production.

Operating Notes

Schedule a nightly job to expand the golden set with sampled production traffic. Over time, the suite becomes a living spec of model behavior. Keep temperatures at zero for eval runs; if you need creative variance, test that in a separate non-blocking job.

Don’t trust a single model version forever—providers quietly retrain. Your regression set is the canary. Treat a passing suite as a contract: if it goes red, someone changed the deal.