Every prompt edit risks breaking the fragile assumptions your parser makes about model output. To test prompt changes ci without paying for live inference, you need a deterministic stand-in that mirrors your provider’s API contract and runs fully offline.
Step 1: Pin the request and response contract
Before you can mock anything, you must know exactly what your code sends and what it expects back. Treat the LLM call as a typed function: system prompt, user message, temperature, and the JSON shape you decode.
If you use the OpenAI Python client, the relevant call looks like this:
from openai import OpenAI
def summarize(client: OpenAI, text: str) -> str:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You summarize text in one sentence."},
{"role": "user", "content": text},
],
temperature=0.0,
)
return resp.choices[0].message.content.strip()The contract is the messages array and the choices[0].message.content field. When you test prompt changes ci, you care whether the system string or few-shot examples drift, not whether the model hallucinates.
Capture a real sample
Run the function once against a real endpoint and save the request body. Tools like pytest-recorder or a simple logging middleware can dump the JSON. Keep that fixture as your baseline.
{
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": "You summarize text in one sentence."},
{"role": "user", "content": "Long article..."}
],
"temperature": 0.0
}Step 2: Build a local mock that speaks OpenAI
You don’t need a full provider emulator. A 30-line FastAPI app that echoes a canned completion based on a hash of the prompt is enough to make tests deterministic.
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import hashlib
app = FastAPI()
@app.post("/v1/chat/completions")
async def chat_completions(req: Request):
body = await req.json()
prompt_hash = hashlib.sha256(
str(body["messages"]).encode()
).hexdigest()[:8]
content = f"MOCK SUMMARY ({prompt_hash})"
return JSONResponse({
"id": "mock",
"object": "chat.completion",
"model": body.get("model", "mock"),
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": content},
"finish_reason": "stop"
}]
})Run it with uvicorn mock_llm:app --port 8765. The mock returns instantly and costs zero tokens.
Streaming if you use it
If your client calls stream=True, return text/event-stream chunks. For most prompt regression tests, non-streaming is sufficient; streaming is a transport detail, not a prompt logic detail.
Step 3: Swap the base URL in tests
OpenAI-compatible clients read base_url. Point it at your mock in a pytest fixture so production code never changes.
import pytest
from openai import OpenAI
@pytest.fixture
def client():
return OpenAI(base_url="http://localhost:8765/v1", api_key="test")
def test_summarize_runs(client):
out = summarize(client, "Some input text")
assert out.startswith("MOCK SUMMARY")If your production traffic goes through an OpenAI-compatible gateway such as n4n.ai—which exposes one endpoint covering 240+ models and handles provider fallback—the same base_url override works unchanged because the request and response schemas are identical.
Prevent accidental live calls
Set OPENAI_API_KEY to a dummy and add a pytest hook that fails if a test attempts a socket connection outside localhost. This guarantees your test prompt changes ci job never bills a real account.
# conftest.py
import socket, pytest
def _block_external(sock, *args, **kwargs):
if sock.family == socket.AF_INET:
ip = sock.getpeername()[0]
if not ip.startswith("127."):
raise RuntimeError(f"Blocked external connection to {ip}")
return sock.connect(*args, **kwargs)
@pytest.hookimpl(tryfirst=True)
def pytest_configure():
socket.socket.connect = _block_externalStep 4: Write deterministic prompt snapshot tests
The real value is catching prompt drift. Assert the exact messages your function builds, not just the mock output.
def test_system_prompt_unchanged(client, snapshot):
summarize(client, "input")
assert snapshot == {
"model": "gpt-4o-mini",
"messages": [
{"role": "system", "content": "You summarize text in one sentence."},
{"role": "user", "content": "input"}
],
"temperature": 0.0
}Use pytest-snapshot or syrupy to store the expected request. When a teammate edits the system string, the test fails loudly. That is how you test prompt changes ci without running a single inference.
Test parsing logic too
If your code extracts JSON from the response, feed the mock a fixed JSON string and verify your parser. The mock can branch on prompt hash to return structured data.
if "extract json" in str(body["messages"]).lower():
content = '{"title": "Mock", "score": 0.9}'Step 5: Simulate provider degradation
Live APIs fail. Your CI should prove your retry and fallback paths work.
Return a 429 from the mock:
from fastapi import HTTPException
@app.post("/v1/chat/completions")
async def chat_completions(req: Request):
body = await req.json()
if body.get("model") == "always-rate-limit":
raise HTTPException(status_code=429, detail="rate limited")
# ... normal mockThen test that your wrapper raises or retries appropriately:
def test_rate_limit_handled(client):
with pytest.raises(RateLimitError):
summarize(client, "x") # with model overridden to triggerIf you rely on gateway-level fallback (e.g., automatic reroute on degradation), mock the secondary response too. The point is to exercise your client code’s resilience offline.
Step 6: Run it in CI
Drop a GitHub Actions workflow that starts the mock, runs pytest, and tears down.
name: prompt-tests
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install fastapi uvicorn openai pytest pytest-snapshot
- run: uvicorn mock_llm:app --port 8765 &
- run: pytest -qNo secrets required. The job finishes in seconds. Because there is no network egress, you test prompt changes ci on every commit for free.
Isolation tip
Run the mock in the same job as tests; don’t expose it as a service container unless you need parallel shards. A background process is simpler.
Step 7: Verify the pipeline is actually saving money
A green checkmark means nothing if the tests silently hit production. Verify success with three concrete checks:
- No billed usage. Query your provider’s usage dashboard after a CI run; token count for the CI API key should be zero. If you used the socket block from Step 3, this is enforced.
- Snapshot diffs on prompt edits. Deliberately change the system prompt locally and confirm the snapshot test fails. If it doesn’t, your assertion is too loose.
- Deterministic timing. Mock responses are sub-millisecond. A test suite that takes seconds instead of minutes confirms you’re not waiting on live latency.
When those hold, you have a reliable gate that lets you refactor prompts as casually as you refactor code.
Where to go next
Add mutation testing: randomly perturb the mock’s output to ensure your validation rejects malformed completions. Extend the mock to simulate token overuse or partial JSON. The contract-first approach scales to any OpenAI-compatible surface, so the same harness works whether you call a single model or a routing gateway.
Keep the mock honest: periodically record a fresh real response and diff it against your canned shape. Schemas drift; your mock should track them. That discipline is what makes test prompt changes ci trustworthy over time.