How to snapshot-test LLM API outputs

Snapshot testing LLM output is the fastest way to catch unintended behavior changes when you upgrade a model or tweak a prompt. Unlike unit tests with exact equality, snapshot tests store a known response and diff future runs against it, which fits the probabilistic nature of language models if you constrain the request enough to be repeatable. This guide walks through a concrete pipeline you can drop into CI today.

Step 1: Define a deterministic request contract

Non-determinism is the enemy of snapshot testing llm output. Your first job is to remove every source of variance from the request. Set temperature to 0, top_p to 1, and frequency_penalty and presence_penalty to 0. Pass a seed value when the provider documents support for it—OpenAI and some others honor it, but do not assume all do.

Pin the exact model version. gpt-4o is a moving target; gpt-4o-2024-05-13 is a fixed checkpoint. The same applies to Anthropic, Cohere, or any model behind an OpenRouter-class gateway. If you call claude-3-5-sonnet-latest today and the alias repoints next month, your snapshots will drift for reasons unrelated to your code.

Keep prompts static. Do not embed timestamps, request IDs, or random user data. If you need contextual variation, isolate it in a fixture that is itself snapshot-tested.

from openai import OpenAI

def build_client(base_url: str, api_key: str) -> OpenAI:
    return OpenAI(base_url=base_url, api_key=api_key)

def query_llm(client: OpenAI, user_text: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-2024-05-13",
        messages=[
            {"role": "system", "content": "You output only valid JSON."},
            {"role": "user", "content": user_text},
        ],
        temperature=0,
        top_p=1,
        seed=42,
        max_tokens=200,
        response_format={"type": "json_object"},
    )
    return resp.choices[0].message.content

The response_format flag forces structured output on supporting models, which makes later parsing trivial.

Step 2: Capture the first snapshot with a recording script

You need a baseline artifact. Write a script that sends the deterministic request and persists the response to a version-controlled directory. Use environment variables for credentials; never commit keys.

export LLM_API_KEY=sk-test
export LLM_BASE_URL=https://api.openai.com/v1
python record_snapshot.py

import os
import json
from pathlib import Path
from step1 import build_client, query_llm

SNAP_DIR = Path(__file__).parent / "__snapshots__"
SNAP_DIR.mkdir(exist_ok=True)

def main():
    client = build_client(os.environ["LLM_BASE_URL"], os.environ["LLM_API_KEY"])
    output = query_llm(client, "List three primary colors as JSON array.")
    snap = {
        "model": "gpt-4o-2024-05-13",
        "output": output,
        "system_fingerprint": "fp_test",  # replace with resp.system_fingerprint if available
    }
    (SNAP_DIR / "colors.json").write_text(json.dumps(snap, indent=2))
    print("Wrote snapshot")

if __name__ == "__main__":
    main()

Store one file per test case. For parameterized suites, name them colors__gpt-4o-2024-05-13.json or similar. Review the file manually: open it, confirm the JSON is well-formed and semantically correct. A bad baseline makes every future test meaningless.

Step 3: Wire snapshots into your test runner

Use a mature snapshot library. syrupy for pytest generates readable diffs and supports updating via CLI. Install it:

pip install pytest syrupy

Write a test that compares a fresh call to the stored baseline. Locally you may hit the live API; in CI you will mock (Step 4). The example below shows the syrupy pattern:

import json
from pathlib import Path
from step1 import build_client, query_llm

SNAP = Path(__file__).parent / "__snapshots__" / "colors.json"

def test_colors_snapshot(snapshot):
    client = build_client("https://api.openai.com/v1", "sk-test")
    out = query_llm(client, "List three primary colors as JSON array.")
    data = {"model": "gpt-4o-2024-05-13", "output": out}
    # First run writes; later runs diff.
    snapshot.assert_match(json.dumps(data, indent=2), "colors")

The first execution creates an .ambr binary snapshot. Subsequent runs fail on divergence. To approve a legitimate change, run pytest --snapshot-update and commit the result. Treat that commit as a deliberate act of changing model behavior.

Step 4: Mock the LLM endpoint in CI to avoid cost and flakiness

Running live inferences in CI wastes tokens and fails when a provider has a bad minute. Record the HTTP exchange once, then replay it. respx mocks the transport layer; vcrpy tapes the whole session. Below is a respx test that returns the stored snapshot regardless of network state.

import respx
import httpx
import json
from pathlib import Path

SNAP = Path(__file__).parent / "__snapshots__" / "colors.json"

@respx.mock
def test_colors_mocked():
    payload = json.loads(SNAP.read_text())
    respx.post("https://api.openai.com/v1/chat/completions").mock(
        return_value=httpx.Response(
            200,
            json={
                "id": "chatcmpl-test",
                "choices": [{"message": {"content": payload["output"]}}],
                "model": payload["model"],
            },
        )
    )
    from step1 import build_client, query_llm
    client = build_client("https://api.openai.com/v1", "fake")
    out = query_llm(client, "List three primary colors as JSON array.")
    assert out == payload["output"]

If you route traffic through an OpenAI-compatible gateway like n4n.ai, you can record against its single endpoint—which fronts 240+ models and auto-falls-back when a provider degrades—then replay the captured bytes in CI without touching the network. The gateway’s cache-control hints also let you mark test requests as non-billable in practice.

Step 5: Assert on structured fields, not raw text

Models reformat whitespace, swap key order, or add a trailing newline. Snapshot the parsed object, not the raw string. Use Pydantic to enforce shape and serialize deterministically.

from pydantic import BaseModel

class Color(BaseModel):
    name: str

class ColorList(BaseModel):
    colors: list[Color]

def parse_output(raw: str) -> ColorList:
    import json
    data = json.loads(raw)
    # handle both {"colors": [...]} and raw list
    if isinstance(data, list):
        return ColorList(colors=[Color(name=c) for c in data])
    return ColorList(**data)

def test_parsed_snapshot(snapshot):
    raw = SNAP.read_text()
    parsed = parse_output(json.loads(raw)["output"])
    snapshot.assert_match(parsed.model_dump_json(indent=2), "colors_parsed")

Strip volatile fields before snapshotting. If the API returns created timestamps or system_fingerprint that change per call, exclude them:

clean = {k: v for k, v in resp.items() if k not in ("created", "id")}

This discipline keeps snapshot testing llm output focused on semantics, not noise.

Step 6: Handle model version drift

Even with temperature=0, providers occasionally ship weights on pinned dates. When a snapshot diff appears, read it before approving. A harmless reformulation (“red” vs “Red”) can be normalized in parsing. A missing field means your contract broke.

Maintain a models.lock file. A tiny pytest check can enforce it:

import json
from pathlib import Path

def test_models_locked():
    lock = json.loads((Path(__file__).parent / "models.lock").read_text())
    assert "gpt-4o-2024-05-13" in lock["locked_models"]

When a model is deprecated, update the lock, re-record snapshots, and open a PR that shows the full diff. That PR is your audit trail.

Step 7: Verify success in CI

Add a workflow that installs deps, runs pytest with the mock, and fails on mismatch. The mock ensures the step is fast and free.

name: test
on: [push]
jobs:
  snapshot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install pytest syrupy respx httpx pydantic
      - run: pytest -q

Locally, verify by running pytest (live) then pytest --snapshot-update only after manual review. In CI, success means the suite is green and the snapshot file is unchanged from the last approved commit. If the step fails, the diff is printed; you decide whether to treat it as a regression or a planned change.

That loop—record, mock, assert structured, review diffs—is the only reliable way to practice snapshot testing llm output in a pipeline without burning tokens or accepting flaky tests.