JSON mode vs function calling for structured output

JSON mode vs function calling structured output is the pragmatic choice every team makes once they stop trusting raw model text. Both techniques force the model to emit parseable data, but they diverge on where the schema lives and who enforces it.

Capabilities

JSON mode is a response constraint: the model must return a valid JSON object. It says nothing about keys, types, or nesting beyond what you smuggle into the prompt. Function calling (tool use) binds the generation to a declared function signature. The model emits arguments that match your JSON Schema parameters, and the API wraps them in a tool_calls structure.

The crucial difference is intent. JSON mode is passive—you ask for JSON and hope the model followed your instructions. Function calling is active—the model explicitly decides to invoke a tool, which makes multi-tool routing and conditional extraction natural.

# JSON mode: schema lives in the prompt
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"user","content":"""
Extract {"name": str, "age": int} from: Jane is 42.
Respond ONLY with JSON."""}],
    response_format={"type":"json_object"}
)
data = json.loads(resp.choices[0].message.content)

# Function calling: schema is declared
tools = [{
    "type":"function",
    "function":{
        "name":"extract_person",
        "parameters":{
            "type":"object",
            "properties":{"name":{"type":"string"},"age":{"type":"integer"}},
            "required":["name","age"]
        }
    }
}]
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"user","content":"Jane is 42."}],
    tools=tools,
    tool_choice="auto"
)
args = json.loads(resp.choices[0].message.tool_calls[0].function.arguments)

JSON mode wins for arbitrary shapes—dump a 50-field nested report without defining a function. Function calling wins when the same model call should choose between actions (search, insert, classify).

Cost model

Providers do not price JSON mode and function calling differently; you pay per token for input and output. The hidden cost is schema tokens. In JSON mode you typically repeat the schema in the system prompt, inflating input tokens on every call. In function calling the schema travels in the tools array, which the provider may cache across calls with the same definition.

If you use a gateway with per-token usage metering, both modes show identical token math for the same effective payload. The only savings come from caching tool definitions or trimming prompt-injected JSON schemas.

Latency and throughput

Measured end-to-end, the delta is small. JSON mode may use constrained decoding (e.g., grammar-based sampling) that adds slight overhead on the first token. Function calling requires the model to emit a special token sequence and then arguments, which can delay time-to-first-token by a similar margin.

Throughput scales with the same variables: output length and model size. Neither mode changes batch limits. When a provider is degraded, an OpenAI-compatible endpoint that offers automatic fallback will retry the same request shape against a healthy backend, so mode choice does not affect resilience.

Ergonomics

JSON mode is trivial to drop into an existing prompt pipeline. You validate after the fact:

from pydantic import BaseModel, ValidationError
class Person(BaseModel):
    name: str
    age: int

try:
    p = Person(**data)
except ValidationError as e:
    # handle drift
    pass

Function calling gives you structured args pre-parsed by the SDK, but you must handle the tool_calls list, possibly multiple calls, and the case where the model returns no call. For single-object extraction, JSON mode is less code. For agent loops, function calling removes string parsing entirely.

Ecosystem

OpenAI introduced both; Anthropic and Google support tool use with slightly different schemas. JSON mode (response_format: json_object) is less universally implemented—some open-weight models need a grammar file instead.

An OpenAI-compatible gateway such as n4n.ai that honors client routing directives and forwards provider cache-control hints lets you toggle between these without rewriting calls, but the backend model’s native support still governs behavior. If you target 240+ models through one endpoint, test both modes per model; some only accept tools, not JSON mode.

Limits

JSON mode limits:

• No guarantee of key presence or type—only valid JSON.
• Streaming works, but partial JSON needs a parser.
• Some models ignore response_format if the prompt contradicts it.

Function calling limits:

• Parameter schema must be JSON Schema draft-07 subset.
• Parallel tool calls are not supported on every model.
• The model can still hallucinate values that pass schema but are wrong.

Comparison at a glance

Which to choose

Single-shot extraction from one text block. Use JSON mode. You avoid tool-call boilerplate and can validate with Pydantic. Keep the schema short and explicit in the prompt.

Agentic workflows with branching actions. Use function calling. The model’s native ability to pick search vs create_ticket beats regex on JSON keys.

Strict typing across many calls. Function calling gives you a declared parameter tree that the provider validates structurally. Pair it with application-side validation for semantic correctness.

Cross-model portability. Prefer function calling where possible; it is the more widely adopted standard. If you must use a model without tool support, fall back to JSON mode with a hardened prompt and a retry loop on ValidationError.

High-volume, low-latency classification. JSON mode with a cached system prompt is simplest. Function calling adds request size but may improve routing accuracy; benchmark on your traffic, not someone else’s.

The decision is not permanent. Wrap both behind an output adapter so the rest of your system receives a typed object either way.