Function calling vs JSON mode: when to use each

The choice between function calling vs json mode shapes how your agent interacts with the outside world, and the trade-offs are concrete rather than aesthetic. Function calling binds a model to a typed interface it can invoke; JSON mode only constrains the shape of the text it returns. Pick wrong and you either drown in parsing glue or ship a model that can’t actually act.

Capabilities

Function calling: typed actions, not just text

Function calling lets the model emit a structured invocation of a declared tool. The API returns a tool_calls array; your code executes the side effect and feeds the result back. This is bidirectional: the model decides whether to call, which function, and with what arguments.

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role":"user","content":"Ship the order #123 to Zurich"}],
    tools=[{
        "type":"function",
        "function":{
            "name":"ship_order",
            "parameters":{
                "type":"object",
                "properties":{
                    "order_id":{"type":"string"},
                    "destination":{"type":"string"}
                },
                "required":["order_id","destination"]
            }
        }
    }],
)
# resp.choices[0].message.tool_calls[0].function.arguments -> '{"order_id":"123","destination":"Zurich"}'

The model never sees the function body. It only sees the schema. That separation is the point.

JSON mode: constrained output, no actions

JSON mode forces the model to emit a valid JSON object according to response_format={"type":"json_object"}. It does not define functions, does not allow the model to request execution, and does not guarantee adherence to an arbitrary schema beyond “valid JSON”. You must validate and route the result yourself.

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role":"system","content":"Return JSON: {\"order_id\":str,\"destination\":str}"},
        {"role":"user","content":"Ship order 123 to Zurich"}
    ],
    response_format={"type":"json_object"},
)
# resp.choices[0].message.content -> '{"order_id":"123","destination":"Zurich"}'

If you need the model to trigger a side effect, JSON mode is silent. It is a serialization format, not a control protocol.

Cost model

Both modes are billed per token on the same meters. Function calling injects the tool schema into the context window, increasing input tokens proportionally to the size of your function definitions. JSON mode shifts that cost to your system prompt if you spell out the schema there. Neither mode changes output token pricing.

The function calling vs json mode cost difference is negligible at the API level. The hidden cost is validation. With function calling, the provider typically validates arguments against the declared JSON schema before returning; with JSON mode you pay compute in your own process to run jsonschema or pydantic. At scale, that CPU time is real but small relative to LLM spend.

Latency and throughput

Empirically, the extra schema tokens in function calling add a small prefill cost. Decoding latency is similar because the model generates structured text either way. JSON mode can be marginally faster on models that implement constrained decoding (grammar masking) to guarantee valid JSON, but many providers implement JSON mode via prompt instruction, which offers no decode-time speedup and may retry on parse failure.

Throughput at the gateway layer is unaffected by mode. A single OpenAI-compatible endpoint like n4n.ai, which fronts 240+ models and automatically falls back when a provider is rate-limited, treats both as ordinary chat requests. Your saturation point is the upstream model’s tokens-per-second, not the mode.

Ergonomics

Function calling wins on developer experience when the action space is known. You declare tools once, the SDK parses arguments into a dict, and you get a clear tool_calls field. Multi-step agent loops become: model calls tool → you execute → you append role:"tool" message → model continues.

The function calling vs json mode ergonomics gap widens with agent complexity. JSON mode pushes the burden to you. You must write a prompt that describes the schema, parse the string, handle missing keys, and map the result to code paths. For a single fixed extraction task, that is fine. For an agent with ten possible actions, the prompt becomes a hand-rolled dispatcher.

import json
data = json.loads(resp.choices[0].message.content)
if "order_id" in data and "destination" in data:
    ship_order(data["order_id"], data["destination"])
else:
    raise ValueError("malformed agent output")

Ecosystem and tooling

Native function calling is supported by OpenAI, Anthropic (via tools), Google Gemini, and most hosted gateways. Open-weight models like Mixtral or Llama-3 can be finetuned or prompted for it, but raw support varies. JSON mode is simpler to emulate: any model that can follow “respond only with JSON” can be coerced, though strictness suffers.

Frameworks (LangChain, LlamaIndex, Semantic Kernel) treat function calling as a first-class primitive. JSON mode is usually a response-format flag they wrap loosely. If you rely on agent orchestration libraries, function calling integrates with their callback and memory layers; JSON mode forces you to bridge manually.

Limits and failure modes

Function calling fails closed when the model hallucinates a function name not in the schema—most providers reject it server-side. But if the model omits required parameters, you get a validation error and must loop. JSON mode fails open: the model may emit valid JSON that semantically misses the mark, and you won’t know until your validator trips.

Parallel calls are a function-calling feature; JSON mode returns one object, so batching requires you to define an array schema and parse it yourself. Streaming with function calling reveals tool_calls deltas; streaming JSON mode gives raw string chunks you must buffer and parse at the end (or use incremental JSON parsers).

Context limits are identical. Both consume space with schemas. Neither mode bypasses provider max-token caps. Smaller models often handle JSON mode prompts more reliably than multi-tool function calling, where the decision space confuses them.

Head-to-head comparison

Which to choose

Use function calling when

• Your system performs side effects (API calls, DB writes, shell commands).
• The action space is discrete and known at request time.
• You want provider-side argument validation and clean multi-turn agent loops.
• You need parallel invocations (e.g., “fetch prices for 5 items”).

Example: an agent that books flights, queries CRMs, or toggles infra. Function calling vs json mode is not a toss-up here—function calling is the only one that actually calls.

Use JSON mode when

• You need extraction or classification with no external action.
• The downstream consumer is a strict schema (e.g., feeding a TypeScript type).
• You target models or providers without native tool support.
• You want minimal request complexity and can own validation.

Example: parsing a support email into {"priority": "high", "category": "billing"}. JSON mode keeps the request lean.

Hybrid patterns

In production agents, the two coexist. Use function calling for the action layer; use JSON mode inside a tool’s return contract when the tool itself needs structured sub-output. For instance, a search function returns raw hits, but you then call the model with JSON mode to summarize them into a fixed report shape.

If you route through a gateway that honors client routing directives and forwards provider cache-control hints, you can keep the same OpenAI-compatible request shape and switch models per call without rewriting your tool or format logic. That portability matters more than micro-optimizing mode choice.

The verdict: function calling vs json mode is a question of control plane vs data plane. Function calling is the control plane for agents; JSON mode is a data-plane constraint for structured text. Build the agent with functions, and reach for JSON mode only when you need guaranteed parseable text without invoking anything.