Production-grade structured outputs tool calling agents fail in boring ways: a missing enum, a loosely typed dict, a model that ignores your instructions. This guide lays out an ordered path to make those agents deterministic enough to ship, from schema to execution loop.
1. Define a strict schema before you write a line of agent code
Treat the JSON schema as the contract, not the prompt. If you let the model emit free-form JSON and pray, you will spend weeks writing validators that should have been enforced at decode time. Write the schema first, commit it, and version it like any other API.
For a weather tool, the schema should forbid extra keys and require every field:
{
"type": "object",
"properties": {
"location": { "type": "string" },
"unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
},
"required": ["location", "unit"],
"additionalProperties": false
}The additionalProperties: false flag is non-negotiable. It forces the decoder to reject stray fields that the model invents under pressure. Version the schema in your repo and bump it when the tool semantics change.
2. Use constrained decoding, not prompt nags
Prompting with “respond only in JSON” is a suggestion, not a constraint. Use the provider’s structured output mode. OpenAI-compatible endpoints expose strict tools or json_schema response formats that guarantee the output conforms to the schema at the token level.
from openai import OpenAI
client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")
resp = client.chat.completions.create(
model="gpt-4o-2024-08-06",
messages=[{"role": "user", "content": "Weather in Berlin?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
},
"required": ["location", "unit"],
"additionalProperties": False
}
}
}],
tool_choice="auto"
)If the model picks the tool, the arguments string parses cleanly into the schema. No regex, no retry loop for malformed JSON.
Pitfall: strict mode rejects nullable by default
Strict structured outputs require all properties listed in required. If a field can be absent, you must model it as an explicit enum with a null value or restructure. Plan for that upfront.
3. Model the tool call as a discriminated action
Multi-tool agents need a clear dispatch path. The simplest robust pattern is one tool per action, letting the model choose. For an agent that can search, calc, or reply, define three tools with strict schemas.
If you need a single structured output that selects among actions, avoid oneOf (poorly supported in strict mode). Instead, use a wrapper with an action enum and a permissive args object that you validate downstream:
{
"type": "object",
"properties": {
"action": {"type": "string", "enum": ["search", "calc", "respond"]},
"args": {"type": "object"}
},
"required": ["action", "args"],
"additionalProperties": false
}This trades some compile-time safety for universal decoder support. You then run a second validation step based on action. It is a reasonable tradeoff for cross-model portability.
4. Handle multi-step loops with explicit state
A single tool call is not an agent. The loop is the agent. Keep the conversation state explicit and append tool results as messages. Do not mutate the original user prompt.
messages = [{"role": "user", "content": "Plan a trip: weather in Paris and convert 100 USD to EUR"}]
done = False
while not done:
resp = client.chat.completions.create(
model="gpt-4o-2024-08-06",
messages=messages,
tools=TOOLS,
tool_choice="auto"
)
msg = resp.choices[0].message
messages.append(msg)
if not msg.tool_calls:
done = True
break
for tc in msg.tool_calls:
result = dispatch(tc.function.name, json.loads(tc.function.arguments))
messages.append({
"role": "tool",
"tool_call_id": tc.id,
"content": json.dumps(result)
})The tool role message must echo the tool_call_id. Miss that and the API rejects the request. Keep the loop bounded with a max iteration counter to avoid runaway costs.
Tradeoff: latency vs. steps
Each loop iteration is a round trip. For latency-sensitive paths, batch independent tool calls in one response (the API supports multiple tool_calls). Only serialize when there is a data dependency.
5. Route and fall back without losing structure
Model providers degrade. Your schema should not care which model served the token. An OpenAI-compatible gateway like n4n.ai forwards provider cache-control hints and automatically falls back when a provider is rate-limited, so your structured schema stays intact across model switches. Write your client against the OpenAI shape and let the gateway handle routing.
client = OpenAI(
base_url="https://gateway.n4n.ai/v1",
api_key=os.environ["N4N_KEY"]
)
# Same strict tools call as before; routing is transparent.If you pin a model that lacks strict mode support, the gateway may route to one that does. Honor the response_format you sent; do not branch on model name in app code.
6. Meter and debug token usage per step
Structured outputs tool calling agents burn tokens on schemas, retries, and tool results. Capture usage on every call and tag it by step type.
usage = resp.usage
print({
"step": "tool_select",
"prompt_tokens": usage.prompt_tokens,
"completion_tokens": usage.completion_tokens,
"cache_read": getattr(usage, "prompt_tokens_details", {}).get("cached_tokens", 0)
})Per-token metering lets you see when a loose schema causes repeated tool calls. If a step consistently uses 3x the tokens of others, tighten the schema or pre-fill a system message with examples.
Pitfall: cached tokens are invisible without hints
Providers cache prompts when you send cache_control markers (or the gateway forwards them). Without that, you pay full price for the static schema on every loop. Set cache breakpoints at the system prompt boundary.
7. Common pitfalls and tradeoffs
Schema drift. You change the tool in code but not the JSON schema. The model still emits the old field. Treat schema and implementation as one PR.
Enum casing. "Celsius" vs "celsius" fails strict validation. Generate enums from a single source of truth, not hand-typed strings.
Over-constraining. Strict mode rejects creative but valid outputs. If the model needs to return a free-text rationale alongside structured fields, add a string field rather than fighting the schema.
Fallback model mismatch. A fallback model may be slower or have smaller context. Your loop must handle timeout independently of structure.
Cost of strict decoding. Constrained decoding adds a small latency overhead. In our experience it pays for itself by eliminating parsing retries.
Building structured outputs tool calling agents is mostly disciplined schema work and a boring loop. The models are capable; the engineering is in making their output something your code can trust without a try-except blanket.