Testing for prompt regressions after a model upgrade

Model snapshots change more often than providers admit, and a silent shift in token distribution can break your extraction pipeline. Testing prompt regressions model upgrade is the only way to know if your carefully tuned system prompt still behaves before you point production traffic at the new version. This walkthrough gives you a repeatable harness to baseline, assert, and diff model outputs in CI.

Step 1: Capture a baseline from the old model

Pick 20–100 representative prompts that exercise the edges of your feature: nested JSON, ambiguous instructions, multilingual input, forced refusals. Fix temperature=0 and a seed so the baseline is reproducible. Store the raw completion, system_fingerprint, and finish_reason alongside the prompt id.

# baseline.py
from openai import OpenAI
import json, os, argparse

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True)
args = parser.parse_args()

with open("prompts.jsonl") as f:
    prompts = [json.loads(line) for line in f]

rows = []
for p in prompts:
    resp = client.chat.completions.create(
        model=args.model,
        messages=p["messages"],
        temperature=0,
        seed=42,
    )
    rows.append({
        "id": p["id"],
        "model": args.model,
        "system_fingerprint": resp.system_fingerprint,
        "output": resp.choices[0].message.content,
        "finish_reason": resp.choices[0].finish_reason,
    })

with open("baseline.jsonl", "w") as f:
    for r in rows:
        f.write(json.dumps(r) + "\n")

Run it against the currently deployed model:

python baseline.py --model gpt-4o-2024-05-13

Keep prompts.jsonl in version control. The baseline file is an artifact; cache it in CI or store it as a release asset.

What makes a good prompt set

Cover the contract your code depends on, not just happy paths. If your parser expects {"status": "ok"}, include a prompt that previously triggered a malformed object. If you rely on the model not mentioning internal tool names, add a prompt that probes for leaks.

Step 2: Define regression assertions

Exact string equality is the wrong tool. Model upgrades change phrasing while preserving structure. Write assertions against properties: valid JSON, required keys, regex patterns, or bounded length. For open-ended text, use a second model as a judge with a tight rubric.

# assertions.py
import json, re

def assert_valid_json(output: str):
    try:
        json.loads(output)
    except Exception as e:
        raise AssertionError(f"output not JSON: {e}")

def assert_has_keys(output: str, keys):
    data = json.loads(output)
    missing = [k for k in keys if k not in data]
    assert not missing, f"missing keys: {missing}"

def assert_no_mention(output: str, forbidden):
    for term in forbidden:
        assert re.search(rf"\b{term}\b", output, re.I) is None, \
            f"leaked forbidden term: {term}"

For subjective quality, call a judge model:

def judge_coherence(client, prompt, output):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Score 1-5 if output follows instruction."},
            {"role": "user", "content": f"P: {prompt}\nO: {output}"},
        ],
        temperature=0,
    )
    score = int(resp.choices[0].message.content.strip()[0])
    assert score >= 4, f"low judge score: {score}"

These functions are your regression suite. They should fail loudly when the new model drifts.

Step 3: Run the candidate model through the same harness

Swap the model id and regenerate. Do not change prompts, temperature, or seed.

python baseline.py --model gpt-4o-2024-08-13 --out candidate.jsonl

If you route through n4n.ai, its OpenAI-compatible endpoint addresses 240+ models, so you change one env var and keep per-token metering instead of juggling multiple provider keys. The client code stays identical.

Cache the baseline run. Only the candidate run needs to execute on every upgrade PR.

Step 4: Diff and score regressions

Load both files by prompt id and run the assertions from Step 2 against the candidate. Count failures and surface the first diff for triage.

# diff_test.py
import json, pytest
from assertions import (assert_valid_json, assert_has_keys,
                        assert_no_mention)

def load(path):
    with open(path) as f:
        return {json.loads(l)["id"]: json.loads(l) for l in f}

BASE = load("baseline.jsonl")
CAND = load("candidate.jsonl")
KEYS = ["status", "data"]
FORBIDDEN = ["internal_tool", "secret"]

@pytest.mark.parametrize("id_", list(BASE.keys()))
def test_prompt_regression(id_):
    cand = CAND[id_]
    out = cand["output"]
    assert_valid_json(out)
    assert_has_keys(out, KEYS)
    assert_no_mention(out, FORBIDDEN)

Run with pytest -q diff_test.py. A green run means the upgrade preserved your contract. A red run prints the exact prompt id and violated property.

Reading the diff

If finish_reason flips from stop to length, your max_tokens is now too small for the loquacious new snapshot. If system_fingerprint changed and JSON validity dropped, the model’s function-call formatting shifted. Treat each failure as a ticket: either tighten the prompt or pin the old model until the provider fixes the snapshot.

Step 5: Automate in CI

Wire the three commands into a workflow that runs on every model-version bump PR. Block merge if pytest fails.

# .github/workflows/model-regression.yml
name: model-regression
on:
  pull_request:
    paths: ["prompts.jsonl", "assertions.py", "baseline.py"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install openai pytest
      - run: python baseline.py --model ${{ vars.OLD_MODEL }} --out baseline.jsonl
      - run: python baseline.py --model ${{ vars.NEW_MODEL }} --out candidate.jsonl
      - run: pytest -q diff_test.py

Cache baseline.jsonl as a workflow artifact from the main branch so PRs don’t regenerate it and introduce noise. The candidate step is the only variable.

Cutting cost

Baseline generation can be expensive at scale. Use provider cache-control hints on the system prompt to avoid re-billing the static prefix on every call. Mark your evaluation prompts with cache_control: {"type": "ephemeral"} where the API supports it, and reuse the cached prefix across baseline and candidate runs.

Step 6: Monitor production fingerprints

Even after testing prompt regressions model upgrade in CI, providers sometimes roll a new snapshot to production without warning. Log system_fingerprint on every request. If it changes mid-flight, trigger the same assertion suite on a shadow sample. Add an alert when finish_reason distribution shifts beyond a threshold you set from baseline stats.

How to verify success

You have a working regression gate when:

1. pytest -q diff_test.py passes on a known-good model swap (e.g., same model id for base and candidate).
2. It fails when you deliberately corrupt candidate.jsonl (remove a required key).
3. The CI job blocks a PR that bumps NEW_MODEL to a snapshot you haven’t vetted.
4. The prompt set lives in git, and the baseline artifact is reproducible from OLD_MODEL.

Run this before every model upgrade, not after. A 10-minute suite beats a 2 a.m. page about malformed JSON in the checkout flow.