Function calling vs JSON mode: which one should you use

Function calling and JSON mode are the two primary ways to get structured data out of an LLM, but they operate at different layers of the stack. Function calling is a protocol: the model emits a special tool-call object that your runtime executes, then feeds the result back. JSON mode is a constraint: the model promises its final answer will be valid JSON matching a schema you provide. Confusing them leads to brittle integrations — function calling for simple extraction, JSON mode for multi-step agent loops. This post breaks down the trade-offs so you can pick the right tool without trial and error.

How they work

Function calling

Function calling (also called tool use) extends the chat completion API with a tools parameter. You declare a JSON Schema for each function, and the model decides whether to call one. When it does, the response contains a tool_calls array instead of plain text. Your code executes the function, then sends a tool role message back with the result. The model then produces a final answer.

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "What's the weather in Tokyo?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get current weather for a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {"type": "string"},
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
        },
        "required": ["location"]
      }
    }
  }],
  "tool_choice": "auto"
}

The model responds:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
          "name": "get_weather",
          "arguments": "{\"location\": \"Tokyo\", \"unit\": \"celsius\"}"
        }
      }]
    }
  }]
}

Your runtime parses arguments, calls the real API, then sends:

{
  "role": "tool",
  "tool_call_id": "call_abc123",
  "content": "{\"temp\": 22, \"condition\": \"cloudy\"}"
}

The model then produces a natural-language answer incorporating that data.

JSON mode

JSON mode constrains the model’s final output to valid JSON. You pass response_format: { "type": "json_object" } (or json_schema for strict validation on newer models). The model emits one message — no tool-call round trip — and that message’s content parses as JSON.

{
  "model": "gpt-4o",
  "messages": [
    {"role": "system", "content": "Extract entities as JSON: {\"people\": [], \"orgs\": [], \"locations\": []}"},
    {"role": "user", "content": "Apple announced the iPhone 16 in Cupertino yesterday."}
  ],
  "response_format": { "type": "json_object" }
}

Response:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "{\"people\": [], \"orgs\": [\"Apple\"], \"locations\": [\"Cupertino\"]}"
    }
  }]
}

No execution loop. No second request. Just structured output in one shot.

Capabilities comparison

Function calling shines when the model needs to act — query a database, hit an API, run code, write a file. The model becomes a planner that delegates execution to your runtime. JSON mode shines when you need extraction, classification, or transformation — the model reads input and emits structured data directly.

Price and cost model

Both features consume tokens the same way: input tokens for your prompt and schema, output tokens for the model’s response. Function calling adds overhead:

• Tool definitions in every request (typically 200–800 tokens depending on schema complexity)
• Tool call arguments in the model’s output
• Tool result messages you send back
• A second model turn to synthesize the final answer

A typical function-calling interaction costs 1.5–3× the tokens of an equivalent JSON-mode request. If you’re extracting structured fields from 10,000 documents, JSON mode saves significant spend. If you’re building an agent that makes five API calls per user query, function calling is the only viable architecture — JSON mode cannot express “go fetch this, then decide what to do next.”

Some providers (including n4n.ai) meter per-token usage identically for both modes, so the cost difference is purely token arithmetic. No hidden premiums.

Latency and throughput

JSON mode wins on latency for single-shot tasks. One request, one response, done. Function calling adds at least one network round trip (your runtime → external API → your runtime → model) plus a second model inference. For a weather lookup, that’s 500ms–2s extra. For a complex agent loop with three tool calls, you’re looking at 3–10 seconds.

Throughput behaves differently. Function calling lets you parallelize independent tool calls in a single model turn — the model emits three tool_calls, your runtime executes them concurrently. JSON mode forces sequential requests if you need multiple extractions. For batch workloads, function calling can actually achieve higher effective throughput despite higher per-request latency.

Streaming changes the calculus. JSON mode streams the JSON object token-by-token, so you can start parsing before the response completes (use a streaming JSON parser like json-stream or orjson). Function calling streams tool calls as they’re generated, but you can’t execute until the full call object arrives. For user-facing UIs where progressive rendering matters, JSON mode feels faster.

Ergonomics and developer experience

Function calling ergonomics

Function calling demands more infrastructure:

1. Schema maintenance — Tool definitions live in your codebase, versioned alongside your API contracts.
2. Execution layer — You need a dispatcher that maps name to actual functions, handles auth, retries, timeouts, and error serialization.
3. Conversation management — You must correctly thread tool_call_id through the loop; dropping it breaks the protocol.
4. Testing — Unit test each tool in isolation, then integration-test the full loop with mocked tool responses.

# Minimal dispatcher pattern
async def dispatch_tool_calls(tool_calls: list[ToolCall]) -> list[ToolMessage]:
    results = []
    for call in tool_calls:
        func = TOOL_REGISTRY.get(call.function.name)
        if not func:
            results.append(ToolMessage(
                tool_call_id=call.id,
                content=json.dumps({"error": f"Unknown tool: {call.function.name}"})
            ))
            continue
        try:
            args = json.loads(call.function.arguments)
            result = await func(**args)
            results.append(ToolMessage(
                tool_call_id=call.id,
                content=json.dumps(result)
            ))
        except Exception as e:
            results.append(ToolMessage(
                tool_call_id=call.id,
                content=json.dumps({"error": str(e)})
            ))
    return results

JSON mode ergonomics

JSON mode is simpler to integrate but shifts complexity to prompting:

1. Schema in prompt — You describe the output structure in the system prompt (or use json_schema parameter on supported models).
2. Validation — You still validate the parsed JSON in your code; models occasionally hallucinate keys or violate enums.
3. Repair strategies — On parse failure, you can retry with a stricter prompt or use a repair model.

import json
from pydantic import BaseModel, ValidationError

class Extraction(BaseModel):
    people: list[str]
    orgs: list[str]
    locations: list[str]

def extract_entities(text: str) -> Extraction:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Extract entities as JSON matching this schema: {\"people\": \"string[]\", \"orgs\": \"string[]\", \"locations\": \"string[]\"}"},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    raw = response.choices[0].message.content
    try:
        return Extraction.model_validate_json(raw)
    except ValidationError:
        # Retry with explicit correction prompt
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "Extract entities as JSON matching this schema: {\"people\": \"string[]\", \"orgs\": \"string[]\", \"locations\": \"string[]\"}"},
                {"role": "user", "content": text},
                {"role": "assistant", "content": raw},
                {"role": "user", "content": "The previous response was invalid JSON. Fix it and respond ONLY with valid JSON."}
            ],
            response_format={"type": "json_object"},
            temperature=0
        )
        return Extraction.model_validate_json(response.choices[0].message.content)

JSON mode with json_schema (available on GPT-4o, Claude 3.5 Sonnet, and some open models) moves validation to the provider side — the model is constrained during generation, not just checked after. This reduces but doesn’t eliminate the need for client-side validation.

Ecosystem and model support

Function calling is widely supported across proprietary and open models: OpenAI (all GPT-4/4o variants), Anthropic (Claude 3+ via tool use), Google (Gemini 1.5+), Mistral, Cohere, Llama 3.1+ (via Ollama, vLLM, TGI), and most fine-tunes derived from these. The OpenAI-style tools parameter has become the de facto standard.

JSON mode support is broader but less uniform. OpenAI offers json_object (loose) and json_schema (strict). Anthropic requires prompt-based JSON with no native parameter — you instruct the model and pray. Google Gemini has response_mime_type: "application/json". Open-weight models vary: vLLM and TGI support guided JSON decoding via outlines/grammar, which is stricter than prompting but requires server-side configuration.

If you’re building a multi-provider abstraction layer, function calling is easier to normalize — the request/response shape is consistent. JSON mode requires provider-specific handling for schema enforcement.

Limits and gotchas

Function calling limits

• Context consumption — Tool schemas eat context window. Complex schemas with nested objects can consume 1,000+ tokens. On 128k-context models this rarely matters; on 8k or 16k models it bites.
• Parallel call limits — Most models cap parallel tool calls at 5–10 per turn. Plan accordingly.
• Argument truncation — Models occasionally truncate long JSON arguments mid-string. Validate argument length before dispatching.
• No streaming tool results — You can’t stream a tool’s output back to the model incrementally. The whole result lands in one tool message.

JSON mode limits

• No recursion — You cannot emit JSON that references itself or requires multi-pass reasoning. The model generates linearly.
• Schema drift — With json_object (no strict schema), the model invents keys, changes types, or nests inconsistently across requests. Always use json_schema or guided decoding when available.
• Large output truncation — Models have output token limits (4k–16k typical). A large JSON array can hit this hard. Paginate or stream.
• No partial credit — If the model stops mid-JSON (token limit), you get invalid JSON. Function calling at least gives you valid tool calls up to that point.

Which to choose

Use function calling when:

• The model must act on the world — Query databases, call APIs, run shell commands, write files, send emails. JSON mode cannot do this.
• Multi-step reasoning with external feedback — “Check the user’s subscription, then if premium, fetch their usage data, then generate a report.” Each step depends on the previous result.
• Human-in-the-loop workflows — The model proposes an action (tool call), your UI presents it for approval, then executes on confirmation.
• Parallel independent operations — “Fetch weather for these five cities.” One model turn, five concurrent API calls.
• You need the model to decide whether to use a tool — tool_choice: "auto" lets the model skip tools entirely for simple queries.

Use JSON mode when:

• Pure extraction or classification — Entity extraction, sentiment labeling, document categorization, PII detection. No external calls needed.
• Structured transformation — Convert unstructured text to a known schema: resumes → JSON, invoices → line items, logs → structured events.
• Single-turn, high-volume workloads — Processing millions of records where latency and token cost matter. JSON mode avoids the round trip.
• Streaming progressive UI — Rendering JSON fields as they arrive (autocomplete, live preview, incremental validation).
• Provider-agnostic batch jobs — You want one prompt template that works across OpenAI, Anthropic, and open models without a tool dispatcher.

Hybrid patterns

Real systems often combine both. A classification router (JSON mode) decides which specialist agent (function calling) handles the request. An extraction pipeline (JSON mode) feeds a decision engine (function calling) that queries business logic. The boundary is clean: JSON mode for data shape, function calling for control flow.

# Router (JSON mode) -> Specialist (function calling)
class Route(BaseModel):
    specialist: Literal["billing", "technical", "sales"]
    confidence: float
    reasoning: str

route = extract_route(user_message)  # JSON mode
if route.specialist == "billing":
    result = await billing_agent.handle(user_message)  # function calling
elif route.specialist == "technical":
    result = await tech_agent.handle(user_message)

Summary

Function calling and JSON mode are not interchangeable — they sit at different abstraction layers. Function calling is a runtime protocol for model-driven execution. JSON mode is an output constraint for structured generation. Choose function calling when the model needs to do something. Choose JSON mode when the model needs to emit something. If you find yourself forcing one to act like the other, you’ve picked the wrong primitive.