Function calling explained: how LLMs call external tools

Function calling lets a language model request structured actions from your code by emitting a standardized JSON payload instead of free-form text. The model does not execute anything itself; it only decides what to call and with which arguments, leaving the actual invocation, error handling, and result injection to your application layer. Understanding how function calling works in LLMs means treating the model as a planner that outputs typed RPC requests, not as an agent that reaches out to the world on its own.

The contract: JSON schema as the API surface

Every function you expose to the model is described by a JSON Schema document. That schema becomes part of the prompt context, either via the tools parameter (OpenAI style) or tool_choice/function_call controls. The model reads the schema, matches the user’s intent to a function name, and fills in the arguments.

{
  "name": "get_weather",
  "description": "Return current conditions for a location.",
  "parameters": {
    "type": "object",
    "properties": {
      "location": { "type": "string", "description": "City, e.g. 'San Francisco'" },
      "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "default": "fahrenheit" }
    },
    "required": ["location"]
  }
}

The schema is not decorative. A missing required field, a loose enum, or a vague description all increase hallucination rates on argument values. Treat schema design like you would a public REST API: version it, validate it, and keep it minimal.

The request–response loop

A single turn with function calling follows this sequence:

1. Client sends messages + tools (schemas) to the model.
2. Model replies with a tool_calls array instead of (or alongside) content. Each entry carries id, type: "function", function: { name, arguments }.
3. Your code parses arguments (a JSON string), validates against the schema, executes the real function, and captures the result or error.
4. Client sends a new message with role: "tool", tool_call_id, and content set to the function’s return value (stringified).
5. Model incorporates the result and produces the final user-facing answer.

# Minimal loop using the OpenAI Python SDK
import json
from openai import OpenAI

client = OpenAI()

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
}]

messages = [{"role": "user", "content": "Weather in Tokyo?"}]

# 1. Model decides to call
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

msg = resp.choices[0].message
if msg.tool_calls:
    for tc in msg.tool_calls:
        args = json.loads(tc.function.arguments)
        # 2. Your code executes
        result = {"temp": 18, "condition": "cloudy", "unit": args.get("unit", "celsius")}
        # 3. Feed result back
        messages.append(msg)  # assistant with tool_calls
        messages.append({
            "role": "tool",
            "tool_call_id": tc.id,
            "content": json.dumps(result)
        })
    # 4. Final answer
    final = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        tools=tools
    )
    print(final.choices[0].message.content)

Notice the model never sees your Python function. It only sees the schema and the stringified return value you choose to send back.

Why the indirection matters

The separation between decision and execution is the entire point. It gives you:

• Safety: The model cannot delete a database row, charge a card, or SSH into a box unless your code chooses to do so after validating arguments.
• Observability: Every tool call is a discrete log entry with a correlation ID (tool_call_id). You can trace latency, error rates, and argument distributions per function.
• Portability: The same schema works across OpenAI, Anthropic, Mistral, and any OpenAI-compatible endpoint. Swap providers without rewriting your tool registry.
• Determinism for testing: Replay a recorded tool_call_id → result pair to unit-test the model’s reasoning without hitting external APIs.

Parallel calls and tool_choice control

Models can emit multiple tool_calls in one response. This is not a batch API — it’s the model predicting that several independent lookups are needed before it can answer. Your loop must iterate over the array, execute each, and return all results before the next model turn.

{
  "tool_calls": [
    { "id": "call_1", "function": { "name": "get_weather", "arguments": "{\"location\":\"Tokyo\"}" }},
    { "id": "call_2", "function": { "name": "get_weather", "arguments": "{\"location\":\"Seoul\"}" }}
  ]
}

You control invocation behavior via tool_choice:

Use "required" for workflows where the first step is always a lookup (e.g., “summarize my last 10 orders” → must call list_orders first). Use forced choice for deterministic pipelines where you want to skip the model’s planner entirely.

Streaming function calls

Streaming adds complexity: the tool_calls array arrives incrementally. Arguments stream as partial JSON fragments. You must buffer until the finish_reason for that tool call signals completion, then parse and execute.

# Streaming sketch — buffer arguments per tool_call_id
buffers = {}
for chunk in stream:
    for tc_delta in chunk.choices[0].delta.tool_calls or []:
        idx = tc_delta.index
        if tc_delta.id:
            buffers[idx] = {"id": tc_delta.id, "name": tc_delta.function.name, "args": ""}
        if tc_delta.function.arguments:
            buffers[idx]["args"] += tc_delta.function.arguments
    # When finish_reason == "tool_calls" for this index, parse buffers[idx]["args"]

Do not attempt to execute on partial arguments. Wait for the done signal.

A concrete end-to-end example: SQL query assistant

The user asks: “How many users signed up last week?” Your tool exposes a read-only run_sql function.

Schema

{
  "name": "run_sql",
  "description": "Execute a read-only SELECT query. Returns rows as JSON.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": { "type": "string", "description": "Single SELECT statement, no semicolon" }
    },
    "required": ["query"]
  }
}

Model’s first response

{
  "tool_calls": [{
    "id": "call_abc",
    "function": {
      "name": "run_sql",
      "arguments": "{\"query\": \"SELECT count(*) FROM users WHERE created_at >= now() - interval '7 days'\"}"
    }
  }]
}

Your executor validates: single statement, starts with SELECT, no DDL/DML. Runs it. Returns:

{"rows": [{"count": 142}], "columns": ["count"]}

Model’s final answer: “142 users signed up in the last 7 days.”

The model never sees your database credentials, connection pool, or row-level security policies. It only sees the schema and the result you hand back.

Common misconceptions

“The model calls the API”

No. The model outputs text that looks like a function call. Your code parses that text, validates it, and makes the actual network request. If your parser crashes on malformed JSON, the model has no idea — it already moved on.

“Function calling makes the model an agent”

Function calling is a primitive for building agents. An agent adds: planning loops, memory, error recovery, human-in-the-loop checkpoints, and budget enforcement. The model alone does none of those.

“More functions = better capability”

Each additional function increases the schema token count and the chance the model picks the wrong one. Prefer fewer, composable functions with clear boundaries. A single search_catalog with a filter object beats search_by_color, search_by_size, search_by_brand.

“The model validates arguments”

The model attempts to conform to the schema, but it will hallucinate enum values, omit required fields, and invent properties not in the schema. You must validate server-side. Use a library like pydantic or zod on the parsed arguments before execution.

“Streaming tool calls work like streaming text”

They don’t. Text streams token-by-token. Tool calls stream property-by-property inside a JSON structure. You cannot render partial arguments to the user. Buffer silently, execute on completion.

Error handling patterns

When your function fails, you have two choices for the tool message content:

1. Structured error — return a JSON object the model can reason about:

   {"error": "rate_limited", "retry_after_seconds": 30, "endpoint": "payment_gateway"}

   The model can then apologize, suggest retry, or pick an alternative.

2. Human-readable string — “The payment provider returned 429 Too Many Requests. Please try again in 30 seconds.” Simpler, but the model cannot programmatically react.

Prefer structured errors for recoverable failures (rate limits, transient network, validation errors). Use strings for terminal failures (permission denied, data not found, business rule violation).

Token budget and schema size

Every function schema you pass consumes prompt tokens. A 50-function registry with verbose descriptions can eat 3–5k tokens before the first user message. Strategies:

• Dynamic tool selection: Pass only the 3–5 functions relevant to the current conversation phase. Use a classifier or keyword match to pick the subset.
• Minimal schemas: Strip description from parameters the model infers from the name. Keep top-level description under 160 chars.
• Namespacing: Group related functions under a single dispatch function that takes {action: "list" | "get" | "create", payload: {...}}. Trades schema tokens for argument complexity — evaluate per use case.

Provider differences worth knowing

If you build a provider-agnostic abstraction layer, normalize to OpenAI’s tool_calls/tool format internally and translate at the edges.

Testing strategy

1. Unit test the executor: Feed it known tool_call_id + arguments, assert it returns the expected result shape. No model involved.
2. Golden prompts: Record real model responses (including tool_calls) for a fixed prompt set. Replay them through your executor to catch regressions in argument parsing or result formatting.
3. Schema validation tests: Generate random valid/invalid arguments against each schema; assert your validator accepts the valid and rejects the invalid with clear messages.
4. Latency budgets: Measure p99 of schema → model → executor → model → final token. Function calling adds at least one extra round-trip. Budget accordingly.

When not to use function calling

• Pure generation tasks: Summarization, translation, creative writing — no external data needed.
• High-frequency, low-latency paths: If you need sub-100ms responses, the extra round-trip kills you. Pre-fetch context or use RAG with embedded knowledge.
• Untrusted user input driving destructive actions: Even with validation, a prompt injection that tricks the model into calling delete_user with a crafted ID is a risk. Keep destructive actions behind human confirmation, not direct model invocation.

Closing thought

Function calling is a protocol, not a feature. It standardizes how a model expresses intent to act so your code can decide whether to act. Build your tool registry like an internal API platform: versioned schemas, centralized validation, structured logging, and clear ownership per function. The model is just another client — one that speaks natural language and occasionally hallucinates argument values. Your job is to make the contract tight enough that the hallucinations don’t matter.