If you ship LLM features, you need to automate prompt regression tests with n4n API to catch silent behavior changes before they hit production. This hands-on tutorial builds a pytest harness that runs golden prompts against models, asserts on output shape and content, and runs on every commit.
Prerequisites
- Python 3.11+ and
pip openaiandpytestinstalled- An API key from n4n.ai; the gateway is OpenAI-compatible, fronts 240+ models, and provides automatic fallback when a provider is rate-limited or degraded.
- A directory for test cases and a
.envfile withN4N_API_KEY=...
python -m venv .venv && source .venv/bin/activate
pip install openai pytest python-dotenvProject layout
prompt_regression/
├── cases.json
├── test_prompts.py
└── .envLoad the key via python-dotenv so it is available as os.environ["N4N_API_KEY"].
Step 1: Configure the client
The n4n API speaks the OpenAI chat completions protocol. Point the base URL at the gateway and use your key.
# test_prompts.py
import os
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(
base_url="https://api.n4n.ai/v1",
api_key=os.environ["N4N_API_KEY"],
)Step 2: Define golden cases
Keep expectations as data, not code. Each case pins a model, the messages, and simple string assertions. This makes it easy to review prompt changes in PRs.
// cases.json
[
{
"id": "sql_select_top",
"model": "openai/gpt-4o-mini",
"messages": [
{"role": "user", "content": "Write SQL to get top 5 users by revenue from table users."}
],
"expect": {
"contains": ["SELECT", "ORDER BY", "LIMIT 5"],
"not_contains": ["DROP", "DELETE"]
}
},
{
"id": "json_sentiment",
"model": "openai/gpt-4o-mini",
"messages": [
{"role": "user", "content": "Return JSON with key 'sentiment' and value 'positive' or 'negative' for: 'I love this product'."}
],
"expect": {
"contains": ["sentiment"],
"json": true
}
}
]Step 3: Write the pytest harness
Parametrize over the JSON file. Set temperature=0 to minimize non-determinism. The gateway forwards provider cache-control hints if you pass them, but for tests we keep it simple.
import json
import pytest
def load_cases(path="cases.json"):
with open(path) as f:
return json.load(f)
@pytest.mark.parametrize("case", load_cases())
def test_prompt_regression(case):
resp = client.chat.completions.create(
model=case["model"],
messages=case["messages"],
temperature=0,
)
text = resp.choices[0].message.content
exp = case["expect"]
for token in exp.get("contains", []):
assert token in text, f"{case['id']}: missing {token}"
for forbidden in exp.get("not_contains", []):
assert forbidden not in text, f"{case['id']}: found {forbidden}"
if exp.get("json"):
data = json.loads(text) # raises if invalid
assert "sentiment" in dataStep 4: Run and read output
Execute the suite locally to confirm wiring.
pytest -qExpected output:
..
2 passed in 1.4sIf a model returns LIMIT 10 instead of LIMIT 5, the test fails with sql_select_top: missing LIMIT 5, surfacing the regression immediately.
Step 5: Enforce structured outputs
For stricter contracts, use the response_format parameter. This works through the gateway exactly as with the native API.
def test_structured_sentiment():
resp = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role":"user","content":"Return JSON with keys: name, age for Albert, 42"}],
response_format={"type":"json_object"},
temperature=0,
)
data = json.loads(resp.choices[0].message.content)
assert set(data.keys()) == {"name","age"}
assert data["name"] == "Albert"Run it:
pytest test_prompts.py::test_structured_sentiment -q.
1 passed in 0.9sStep 5b: Reduce non-determinism
Even at temperature 0, some backends vary across deployments. Pass a seed when supported and retry once on parse errors to avoid flaky failures.
def complete(case):
for _ in range(2):
try:
resp = client.chat.completions.create(
model=case["model"],
messages=case["messages"],
temperature=0,
seed=42,
)
return resp.choices[0].message.content
except json.JSONDecodeError:
continue
raise AssertionError("model did not return stable JSON")Step 6: Wire into CI
Commit the harness and add a GitHub Actions workflow. Secrets stay in the runner; the gateway’s per-token usage metering means you can attribute test spend.
# .github/workflows/prompt-regression.yml
name: prompt-regression
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install openai pytest python-dotenv
- run: pytest -q
env:
N4N_API_KEY: ${{ secrets.N4N_API_KEY }}Now every push automatically runs your prompt regression suite.
Step 7: Track cost and avoid flake
Because the gateway returns token usage, log it to watch test cost.
@pytest.mark.parametrize("case", load_cases())
def test_prompt_regression(case):
resp = client.chat.completions.create(
model=case["model"],
messages=case["messages"],
temperature=0,
)
print(f"{case['id']} used {resp.usage.total_tokens} tokens")
# ... assertionsAutomatic fallback keeps tests green when a provider is degraded, but you should still alert on repeated fallbacks. When you automate prompt regression tests with n4n API, treat the golden set as code: review additions in PRs, and bump expectations only after manual verification.
If you need to pin a specific provider path, the gateway honors client routing directives via the model identifier—prefixing with the provider namespace routes the request accordingly. This is useful when a regression only appears on one backend.
For semantic drift, add a cosine-similarity check against an embedded reference using a local model or the gateway’s embedding endpoint. That extends the harness beyond string matching without much overhead.
The full loop—pinned models, data-driven cases, CI execution, and token accounting—gives you a real safety net for prompt changes. To automate prompt regression tests with n4n API at scale, version your cases.json alongside prompts and fail the build on any mismatch.