Testing structured outputs against a JSON schema in CI

Flaky LLM responses break pipelines when you assume the model returns clean JSON. Testing structured output schemas in CI catches contract violations before they reach production, and it’s simpler than most teams think. This tutorial builds a minimal but real validation gate using pytest, jsonschema, and GitHub Actions.

Prerequisites

You need Python 3.11+, pytest, jsonschema, and the openai client. For deterministic offline tests, add respx to mock HTTP. A GitHub Actions runner and a repo secret holding an API key round out the setup.

python -m venv .venv
source .venv/bin/activate
pip install pytest jsonschema openai respx

Store your key as OPENAI_API_KEY locally and as a GitHub secret for CI. If you route through a gateway, set OPENAI_BASE_URL instead.

Define the schema contract

We’ll validate a support ticket classifier. The model must return exactly three fields with constrained values. Writing the schema first forces you to think about the real contract.

{
  "type": "object",
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "account", "other"]
    },
    "priority": {
      "type": "string",
      "enum": ["low", "medium", "high", "critical"]
    },
    "confidence": {
      "type": "number",
      "minimum": 0.0,
      "maximum": 1.0
    }
  },
  "required": ["category", "priority", "confidence"],
  "additionalProperties": false
}

Save this as schema/ticket.json. additionalProperties: false is non-negotiable—it blocks silent schema drift from extra fields.

Validate any dict against the schema

A thin wrapper converts jsonschema errors into assertion failures pytest understands.

# validate.py
import json
from jsonschema import validate, ValidationError

def load_schema(path: str) -> dict:
    with open(path) as f:
        return json.load(f)

def assert_valid(instance: dict, schema_path: str = "schema/ticket.json") -> None:
    schema = load_schema(schema_path)
    try:
        validate(instance=instance, schema=schema)
    except ValidationError as e:
        raise AssertionError(f"Schema violation: {e.message}") from e

Run it from a REPL to see the failure mode:

>>> assert_valid({"category": "billing"})
AssertionError: Schema violation: 'priority' is a required property

Call the model with structured output

OpenAI’s json_schema response format instructs the model to conform. Temperature 0 and a fixed seed make CI runs repeatable. If you run this in CI and worry about provider rate limits, point the client at an OpenAI-compatible endpoint like n4n.ai; it automatically falls back when a provider is rate-limited or degraded, so your validation job stays green.

# generate.py
import json, os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    # base_url=os.environ.get("OPENAI_BASE_URL", None)
)

def classify_ticket(text: str, model: str = "gpt-4o-mini") -> dict:
    resp = client.chat.completions.create(
        model=model,
        temperature=0,
        seed=42,
        messages=[
            {"role": "system", "content": "Classify support tickets. Return strict JSON."},
            {"role": "user", "content": text},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "ticket",
                "schema": json.load(open("schema/ticket.json")),
            },
        },
    )
    content = resp.choices[0].message.content
    try:
        return json.loads(content)
    except json.JSONDecodeError as e:
        raise AssertionError(f"Model did not return JSON: {e}") from e

Write the pytest suite

Test the happy path with a mock, and a negative path where the shape is wrong. Mocking keeps CI fast and free.

# test_schema.py
import pytest
import respx
import json
from openai import OpenAI
from generate import classify_ticket
from validate import assert_valid

@respx.mock
def test_valid_output():
    # Mock the chat completion endpoint
    respx.post("https://api.openai.com/v1/chat/completions").respond(
        json={
            "choices": [{
                "message": {
                    "content": json.dumps({
                        "category": "billing",
                        "priority": "high",
                        "confidence": 0.92
                    })
                }
            }]
        }
    )
    out = classify_ticket("I was charged twice.")
    assert_valid(out)
    assert out["category"] == "billing"

def test_schema_rejects_garbage():
    bad = {"foo": "bar"}
    with pytest.raises(AssertionError):
        assert_valid(bad)

For a live call test, skip the mock and guard with an env flag so CI only runs it when a key exists:

import os

@pytest.mark.skipif(not os.environ.get("OPENAI_API_KEY"), reason="no live key")
def test_live_model():
    out = classify_ticket("Server is down, urgent!")
    assert_valid(out)

Run locally and read the output

pytest -q

Expected pass:

..
2 passed

If the model returns {"category": "billing"} only, the failure is explicit:

E       AssertionError: Schema violation: 'priority' is a required property

That red mark is the entire point—it blocks a broken contract from merging.

Wire it into GitHub Actions

Create .github/workflows/ci.yml. Cache pip to keep the job under a minute.

name: ci
on: [push, pull_request]
jobs:
  schema-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
      - run: pip install pytest jsonschema openai respx
      - run: pytest -q
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Commit and open a PR. The checks panel shows the schema test executing on every push.

Catch schema drift across models

Different models honor schemas differently. Parametrize to test each candidate:

MODELS = ["gpt-4o-mini", "mistral-large-latest"]

@pytest.mark.parametrize("model", MODELS)
def test_all_models(model):
    out = classify_ticket("Password reset fails.", model=model)
    assert_valid(out)

Gateways that honor client routing directives let you pin a model per test without code changes. Some, like n4n.ai, forward provider cache-control hints so repeated validation calls with identical prompts hit cache and cut token cost during heavy CI loops.

Keep the schema as the source of truth

Do not inline a copy of the schema in the client call. Load the same file the test uses. If you use TypeScript, compile a Zod schema to JSON and commit both artifacts. A pre-commit hook can run pytest on schema changes. That turns testing structured output schemas in CI from a nice-to-have into a hard gate.

Handle missing fields gracefully

In production, retry on validation failure. In CI, fail fast and print the raw payload. Extend the generator:

def classify_strict(text, model="gpt-4o-mini"):
    out = classify_ticket(text, model)
    try:
        assert_valid(out)
    except AssertionError:
        print("RAW RESPONSE:", out)
        raise
    return out

Final checklist

• Schema lives in schema/ and is loaded by both client and test.
• CI runs pytest on every PR.
• Model calls use temperature=0 and a fixed seed.
• Rate limits mitigated via fallback gateway or retries with backoff.
• Testing structured output schemas in CI runs in under two minutes on a free runner.

That’s the whole loop. No fancy framework required—just jsonschema, pytest, and a YAML file.