What function calling actually costs in tokens

The line item for function calling token cost on your inference bill is rarely what you expect. It is not just the few dozen tokens the model emits as a JSON argument blob; it is the schema you ship on every request, the repeated context across turns, and the amplification from multi-step agent loops. If you treat tool calls as free after the first definition, you will quietly double or triple your spend.

Why the estimate is always low

Engineers budget for the happy path: one call, one response. They open the dashboard, see a 30-token completion, and move on. The invoice at month end tells a different story because the schema tax is invisible in single-call tests.

Every OpenAI-compatible chat completion request that declares tools sends the full JSON schema to the model. The provider bills those schema tokens as input tokens on each call. In a single-turn invocation that is a one-time tax. In an agent that runs twenty turns, the same schema is re-sent nineteen extra times unless you or your gateway use prompt caching.

Schema injection is the silent multiplier

Consider a minimal weather tool:

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Fetch current weather for a location",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {"type": "string", "description": "City and state"},
        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
      },
      "required": ["location"]
    }
  }
}

That definition is roughly 70–120 tokens depending on how verbose your descriptions are. Now wrap it in a 5-turn conversation where the assistant calls the tool each turn. The input token count includes:

• Base system prompt (say 50 tokens)
• Turn 1: user msg + schema + response
• Turn 2: user msg + prior messages + schema + response
• … and so on.

The schema is appended on every turn. With a 100-token schema, that is 500 extra input tokens purely from repetition across five turns. Function calling token cost is therefore a function of schema_size × turns, not a constant.

Scale that to a customer-support agent with 12 tools averaging 150 tokens each. That is 1,800 schema tokens per request. Over a 15-turn conversation, you have injected 27,000 schema tokens that never changed. At common input rates, that alone is a meaningful fraction of the total bill.

Counting it precisely

Use tiktoken to measure your schema before shipping:

import tiktoken, json

schema = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Fetch current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
}

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode(json.dumps(schema))
print(len(tokens))  # typically 80-110 for this shape

Run this in CI. If your schema creeps past 200 tokens per tool and you load 15 tools, you are injecting 3,000 dead tokens per request before the user types anything.

The output side is not free either

When the model decides to call a function, it emits a structured block. That output is billed as completion tokens. For get_weather the arguments might be {"location":"Austin, TX","unit":"fahrenheit"} — maybe 20 tokens. But the model often prefixes the call with reasoning or a role message, especially on larger models. In agent loops, the assistant message that triggered the call is stored and re-sent, so those output tokens become input tokens next turn.

Function calling token cost thus compounds in both directions: output becomes future input, and schema is constant input.

Where it explodes: parallel calls and orchestration

Modern models support multiple tool calls in one response. That is efficient for latency but does not reduce schema cost — the schema for all available tools is still sent once. However, parallel calls can shrink total turns, which is the real win.

A naive agent that does:

1. Call search
2. Wait, call fetch
3. Wait, call summarize

costs three schema injections. If you enable parallel calling and the model batches search + fetch, you cut a turn. Less repetition, lower function calling token cost.

Multi-agent routing

If you run a supervisor that holds 20 tools and delegates to sub-agents that each hold 5 tools, the supervisor still pays for its 20-tool schema on every supervisory turn. Push tool definitions down to the worker only when needed. Better: dynamically trim the tools array based on the current step. The API does not require you to send every tool every time.

A common pattern: use a small classifier model to pick a tool subset, then call the heavy model with only those three tools. The classifier costs fractions of a cent; the schema savings on the heavy model across thousands of calls pay for it.

Tradeoffs: strict schemas vs. loose strings

You can define a tight schema with enums, ranges, and descriptions. This improves model accuracy and reduces post-processing, but each constraint adds tokens. Alternatively, a single run_action(action: string) with a free-form string shifts parsing to your code.

The strict approach raises function calling token cost but lowers error-handling code. The loose approach shrinks input but forces you to validate untrusted output. For high-volume agents, a hybrid works: keep a minimal schema (names and required types only) and validate/prompt-repair on the backend.

I have shipped both. The strict schema wins when the tool is dangerous (e.g., bank transfer) and you need guardrails. The loose schema wins for internal data retrieval where a typo just returns empty and the agent retries.

Caching and gateways

Prompt caching changes the math. Anthropic and OpenAI both support cache control on system blocks or tool schemas. If your gateway forwards cache-control hints, repeated schema tokens are billed at a fraction after the first call.

Some gateways, including n4n.ai, forward provider cache-control hints and expose per-token usage metering across fallbacks, so you can see the actual function calling token cost when a primary provider degrades and a secondary handles the request. That visibility is the only way to confirm your caching strategy works.

Without caching, the only lever is schema size and turn count.

Measuring in production

Turn on usage logging. With an OpenAI-compatible client:

from openai import OpenAI

client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-...")

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Weather in Austin?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Fetch current weather",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"]
            }
        }
    }]
)

print(resp.usage.prompt_tokens, resp.usage.completion_tokens)

Subtract your known conversation tokens to isolate schema overhead. If prompt_tokens jumps by ~100 when you add the tool, that is your per-call tax. Multiply by daily call volume.

If you run automatic fallback, confirm the fallback provider also honors cache. Otherwise a rate-limit event on the primary silently resets your cache and re-bills the full schema. Per-token metering surfaces that; a flat monthly estimate does not.

Concrete optimization playbook

1. Trim descriptions. “Fetch current weather” → “Weather”. The model infers from name + params.
2. Drop enums unless they prevent invalid calls. Validate after.
3. Send fewer tools. Filter the tools list by current intent using a cheap classifier.
4. Cache the schema. Use cache_control on the tools block if your provider supports it.
5. Collapse turns. Use parallel calls; avoid ask-confirm loops.
6. Meter everything. Per-token usage metering shows when a fallback provider re-bills schema without cache.
7. Use small models for routing. A 7B-class model can select tools; the 70B model executes. The schema on the small model is cheaper per token and often accurate enough.

The decisive takeaway

Function calling token cost is dominated by schema repetition, not the call itself. Treat your tool definitions as hot-loop payload: minimize them, cache them, and never send more than the current step requires. Measure the prompt token delta with and without tools on every agent build. Engineers who ignore this pay a 2–5x tax on every multi-turn agent; engineers who trim and cache keep that spend in their pocket.