Structured outputs with function calling: a comparison

The choice between structured outputs vs function calling shapes how you build extraction, agents, and pipelines. Both coerce model output into typed data, but they differ in what the API contract guarantees and how the model is prompted under the hood.

Capabilities

What each guarantees

Structured outputs (JSON mode, JSON schema, response format) pin the model to a schema you supply. The provider validates the byte stream and either retries or rejects if it diverges. You get a single JSON object back in the message content.

from openai import OpenAI
import json
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "invoice",
            "schema": {
                "type": "object",
                "properties": {
                    "total": {"type": "number"},
                    "currency": {"type": "string"}
                },
                "required": ["total", "currency"]
            }
        }
    },
    messages=[{"role": "user", "content": "Extract: Invoice total $42 USD"}]
)
data = json.loads(resp.choices[0].message.content)

Function calling (tool use) declares one or more functions with parameter schemas. The model returns a tool_calls array referencing a function name and arguments. It is designed for action invocation, not just extraction.

tools = [{
    "type": "function",
    "function": {
        "name": "save_invoice",
        "parameters": {
            "type": "object",
            "properties": {
                "total": {"type": "number"},
                "currency": {"type": "string"}
            },
            "required": ["total", "currency"]
        }
    }
}]

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    tools=tools,
    tool_choice="auto",
    messages=[{"role": "user", "content": "Invoice total $42 USD"}]
)
call = resp.choices[0].message.tool_calls[0]
args = json.loads(call.function.arguments)

The structured outputs vs function calling distinction matters when you evaluate provider guarantees: the former promises parseable content; the latter promises a decision about which registered action to take. Nested objects and enums work identically in both modes because they share JSON Schema semantics. The difference is purely the envelope.

Streaming and partial results

Both modes stream. Structured outputs emit incremental JSON tokens; you can parse with a streaming JSON reader. Function calling streams the arguments string inside tool_calls, which you concatenate before parsing. Neither gives you partial validation for free.

Price/cost model

Most providers bill both by input and output tokens identically regardless of mode. Function calling can inflate output tokens because the model emits a function name plus JSON arguments, and sometimes a surrounding wrapper. Structured outputs emit only the JSON, often slightly fewer tokens for the same data.

Neither mode changes per-token pricing on OpenAI-compatible endpoints. If you route through a gateway that meters per-token usage, the difference shows up as a few percent variance in output token count, not a separate fee. Cache hits apply the same way: a cached system prompt reduces input cost whether you use response_format or tools. If you use provider-specific features like Anthropic’s prompt caching, the cache key includes the tools or response_format block; changing modes invalidates the cache. Plan schema edits accordingly.

Latency/throughput

Function calling requires the model to learn a tool invocation pattern, which adds a small fixed overhead in first-token latency. Structured outputs stream raw JSON, so time-to-first-token is comparable to plain completion, minus validation checks. Speculative decoding and batching are unaffected by mode; the scheduler treats tool tokens like any other.

Throughput at batch scale is similar. The bottleneck is model inference, not the mode. On degraded providers, an OpenAI-compatible endpoint that addresses 240+ models with automatic fallback keeps p95 latency stable for either approach; the mode doesn’t change failover behavior.

Ergonomics

Structured outputs are simpler to wire into a data pipeline. You parse message.content and validate with your own library (pydantic, zod). No need to handle multiple tool calls or a separate arguments string.

// TypeScript: zod parse after structured output
import { z } from "zod";
const Invoice = z.object({ total: z.number(), currency: z.string() });
const parsed = Invoice.parse(JSON.parse(content));

Function calling forces you to handle tool_calls, match function names, and often echo a tool role message back to continue the conversation. That round trip is useful for agents, noisy for ETL.

// Tool call handling adds branches
if (msg.tool_calls) {
  for (const call of msg.tool_calls) {
    if (call.function.name === "save_invoice") {
      const args = JSON.parse(call.function.arguments);
    }
  }
}

Validation and error handling

With structured outputs, validation is a single try/except around your parser. With function calling, you must also guard against unknown function names or empty arguments when tool_choice="auto" returns none. The extra surface area is minor but real. Frameworks like LangChain abstract both, but the abstraction leaks when you need raw token counts or custom retry.

Ecosystem

Every major LLM vendor now supports JSON mode or response schemas: OpenAI, Anthropic, Google, Mistral. Function calling is equally ubiquitous but schema dialects differ (Anthropic uses tools with slightly different fields; some open models need grammar constraints).

For multi-model routing, an OpenAI-compatible request shape works across providers. n4n.ai honors client routing directives and forwards provider cache-control hints, so the same response_format or tools block reaches 240+ models without rewrite. That portability matters more than the mode choice. Client libraries (openai, anthropic-sdk) map their own types; if you hand-roll HTTP, the JSON is identical except for the wrapper key.

Open-source models often implement structured outputs via constrained decoding (grammar sampling), which is stricter than post-hoc validation. Function calling on those models may be emulated by prompt wrapping, raising latency. Check the model card.

Limits

Structured outputs cap you at one object per response (no streaming multiple records unless you pack them in an array property). Recursive schemas are supported but some providers limit depth to a few levels. Schema size is typically capped at a few KB. Output token limits still apply; a huge schema does not grant more completion budget.

Function calling can return parallel calls, letting you extract multiple entities in one turn, but you must dedupe and validate each. Both modes inherit model weaknesses: a schema can’t force factual accuracy. If the prompt lacks info, the model hallucinates compliant garbage.

Comparison table

Which to choose

ETL and labeling pipelines. Use structured outputs. You want a single validated object per document. Function calling adds branches you will strip out anyway.

Single-turn extraction with strict schema. Structured outputs win on ergonomics and token cost. Define the schema once, parse content, move on.

Agent loops and tool-using assistants. Function calling is the right primitive. The model decides among functions, you execute, and return results. Forcing this through JSON content means you reimplement dispatch.

Multi-entity extraction in one call. Function calling with parallel tool calls extracts several records without nesting arrays. Structured outputs can do it via an array property, but the model often truncates long lists.

Cross-model portability requirement. Either works if you stick to OpenAI-compatible shapes. A gateway that forwards cache-control and routes across providers removes the lock-in argument; pick based on use case above.

High-volume cost sensitivity. Structured outputs edge out on token count. At millions of calls, a 5% reduction in output tokens is real money, even if per-token price is flat.

The structured outputs vs function calling decision is mostly about control flow, not data shape. If you need the model to do something, call functions. If you need it to return something, return JSON.