Testing function calling schemas for errors

A broken tool definition silently turns a capable agent into a loop of 400 errors. Testing function calling schemas before they ship catches mismatches between your JSON Schema and what the model or provider actually accepts, saving hours of production debugging.

Step 1: Define tools as strict JSON Schema

Start with the tool definition your agent will send. Treat the parameters object as a real JSON Schema, not a loose bag of fields. Enable strict mode by setting additionalProperties: false and declaring every required key. This forces the model to emit conformant arguments and forces you to think about shape up front.

{
  "name": "get_weather",
  "description": "Fetch current weather for a location",
  "parameters": {
    "type": "object",
    "additionalProperties": false,
    "required": ["location"],
    "properties": {
      "location": {"type": "string"},
      "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
    }
  }
}

Keep descriptions concise but specific. Providers differ on max description length and some strip formatting. Write the schema once, load it as a Python dict, and reuse it across tests and client calls.

Step 2: Validate the schema structure locally

Before any model sees the schema, confirm it is a valid JSON Schema. A common failure in testing function calling schemas is shipping a parameters block with unsupported keywords or broken references. Use the jsonschema library to check the schema against its meta-schema.

from jsonschema import Draft202012Validator

tool = {
    "name": "get_weather",
    "description": "Fetch current weather for a location",
    "parameters": {
        "type": "object",
        "additionalProperties": false,
        "required": ["location"],
        "properties": {
            "location": {"type": "string"},
            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
        }
    }
}

# Raises if parameters is not a valid JSON Schema
Draft202012Validator.check_schema(tool["parameters"])
print("Schema is structurally valid")

Run this in a quick script or pytest fixture. If check_schema passes, you know the shape is at least self-consistent. It will not catch semantic mismatches with provider expectations, but it eliminates typos like "type": "str" or missing properties.

Step 3: Mock model output and validate arguments

The model returns arguments as a JSON object string inside a tool call. Your code must parse and validate that object. Write a helper that validates an arbitrary dict against the tool’s parameters, then call it with simulated model output.

import jsonschema

def validate_tool_call(params_schema: dict, args: dict):
    jsonschema.validate(instance=args, schema=params_schema)

# Simulated successful call
fake_args = {"location": "Berlin", "unit": "celsius"}
validate_tool_call(tool["parameters"], fake_args)

# Simulated bad call (should raise)
try:
    validate_tool_call(tool["parameters"], {"unit": "kelvin"})
except jsonschema.ValidationError as e:
    print("Caught expected error:", e.message)

Wrap this in pytest to lock behavior:

def test_valid_args():
    validate_tool_call(tool["parameters"], {"location": "Tokyo"})

def test_invalid_enum():
    import pytest
    with pytest.raises(jsonschema.ValidationError):
        validate_tool_call(tool["parameters"], {"location": "Tokyo", "unit": "kelvin"})

This step isolates your validation logic from network calls and gives fast feedback when you edit the schema.

Step 4: Submit the schema to real providers

Local checks pass, but providers enforce extra rules. Some reject additionalProperties: false unless you also set strict: true in their proprietary field. Others limit nesting depth or disallow certain format values. Testing function calling schemas against live endpoints surfaces these divergences.

If you route through a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint for 240+ models, you can submit the same tool definition to multiple backends in a loop to surface provider-specific rejections without managing separate SDKs.

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
models_to_test = ["gpt-4o-mini", "anthropic/claude-3-haiku", "meta/llama-3-8b"]

for model in models_to_test:
    try:
        resp = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": "Weather in Berlin?"}],
            tools=[{"type": "function", "function": tool}],
            tool_choice="auto"
        )
        print(model, "accepted schema")
    except Exception as e:
        print(model, "rejected schema:", repr(e))

Capture the exceptions. A 400 with a schema error tells you exactly which provider needs a tweaked definition. Repeat this in a staging environment, not production, and cache results so you are not burning tokens on every CI run.

Step 5: Fuzz edge cases and required fields

Real models hallucinate keys or omit required ones under prompt pressure. Your tests should assert that the validator rejects those cases. Build a small matrix of bad inputs.

import pytest

@pytest.mark.parametrize("bad_args", [
    {},                                  # missing required location
    {"location": 123},                   # wrong type
    {"location": "Paris", "unit": "K"},  # enum violation
    {"location": "Rome", "extra": 1},    # additional property
])
def test_rejects_bad_args(bad_args):
    with pytest.raises(jsonschema.ValidationError):
        validate_tool_call(tool["parameters"], bad_args)

Also test nested objects if your tool uses them. Providers sometimes flatten or mangle deep structures. Define a nested schema and verify both local validation and a live call accept it.

{
  "name": "book_flight",
  "parameters": {
    "type": "object",
    "additionalProperties": false,
    "required": ["trip"],
    "properties": {
      "trip": {
        "type": "object",
        "additionalProperties": false,
        "required": ["from", "to"],
        "properties": {
          "from": {"type": "string"},
          "to": {"type": "string"}
        }
      }
    }
  }
}

Run the same validation and provider submission against this. If a provider silently drops the trip wrapper, you will see a validation error on your side or an malformed call on theirs.

Step 6: Automate in CI

Commit the schema files and the pytest suite. Add a scheduled job that runs the live provider check weekly, because provider validation rules change without notice.

pip install jsonschema pytest openai
pytest tests/test_schemas.py -q

A minimal GitHub Actions step:

- name: Test function schemas
  run: |
    pip install jsonschema pytest openai
    pytest tests/test_schemas.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

For the live cross-provider step, gate it behind a manual workflow dispatch or a nightly cron to avoid rate limits and token cost on every push.

Verifying success

You have verified success when:

1. Draft202012Validator.check_schema passes on every tool definition in your repo.
2. Unit tests confirm valid arguments pass and invalid arguments (missing required, bad enum, extra keys) raise.
3. The live submission script reports accepted schema for each target model, or you have documented why a specific provider requires a modified definition.
4. CI runs the local suite on every PR and the provider probe on a schedule.

At that point, testing function calling schemas is part of your pipeline, not a fire drill after agents start returning null tool calls. The schemas are contract-tested like any other API surface, and your agent code can assume validated input.