Function calling latency: GPT-4o vs Claude vs Gemini

Most teams pick a model for tool-using agents based on vague speed impressions. A proper function calling latency benchmark shows the gap between GPT-4o, Claude, and Gemini is narrower than expected once you isolate time-to-first-tool-call from network and parsing noise. The model is only one component in a pipeline that includes request serialization, provider prefill, token streaming, and client-side validation.

What “function calling latency” actually measures

Engineers conflate three different numbers when they complain about slow tools. The first is time-to-first-byte (TTFB) from the API. The second is time-to-structured-call: when the client can confidently hand a function name and arguments to your executor. The third is end-to-end task latency, which includes retries, multi-step orchestration, and your own code.

A function calling latency benchmark must target the second number. TTFB is dominated by provider infrastructure and your geographic distance to it; end-to-end is dominated by your agent loop and error handling. The middle metric reveals model behavior and API design.

Schema overhead is the silent tax

Every call sends a JSON schema describing your tools. GPT-4o and Claude accept OpenAI-style or Anthropic-style tool definitions; Gemini uses function declarations with type strings like "STRING". Larger schemas force the model to attend to more tokens in context and often generate more intermediate tokens before the actual arguments. In our experience, a 10-tool schema adds negligible delay on GPT-4o but visibly increases Claude’s pre-tool text because it restates intent.

{
  "type": "function",
  "function": {
    "name": "search_orders",
    "parameters": {
      "type": "object",
      "properties": {
        "customer_id": {"type": "string"},
        "status": {"type": "string", "enum": ["open","closed"]},
        "page": {"type": "integer", "minimum": 1}
      },
      "required": ["customer_id"]
    }
  }
}

Keep schemas tight. Redundant properties, verbose descriptions, and unnecessary enums are a latency tax you pay on every turn. Strip helper text from production schemas and move it to system prompts if needed.

GPT-4o: early emission, low ceremony

GPT-4o streams tool calls as delta objects inside chat.completions. The model often emits the function.name within the first few hundred milliseconds of generation, then streams arguments as a partial JSON string. Your client can parse incrementally if you implement a state machine, but most SDKs wait for the finish_reason to validate JSON.

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role":"user","content":"Refund order 123"}],
    tools=[{"type":"function","function":{
        "name":"refund_order",
        "parameters":{"type":"object","properties":{"order_id":{"type":"string"}}}
    }}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.tool_calls:
        print(chunk.choices[0].delta.tool_calls[0].function.name)

The decisive advantage in a function calling latency benchmark is GPT-4o’s tight coupling: it does not preface the call with natural language. The token stream goes straight to structure. GPT-4o also supports multiple tool calls in a single assistant message, which lets it parallelize without the envelope overhead Gemini imposes.

One caveat: OpenAI’s tool parser is strict about JSON validity. If the stream is cut at a network boundary, you may need to replay. That is a client concern, not a model latency issue.

Claude: deliberate, sometimes verbose

Claude (tested on 3.5 Sonnet) wraps tool use in a tool_use block, but frequently outputs a sentence of reasoning before the block. That reasoning is useful for debugging but pushes the parsed-call marker later.

resp = anthropic.Anthropic().messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    tools=[{"name":"refund_order","input_schema":{
        "type":"object","properties":{"order_id":{"type":"string"}}
    }}],
    messages=[{"role":"user","content":"Refund order 123"}]
)
# resp.content may contain a text block then a tool_use block

If you measure from response start to the tool_use block’s input field being complete, Claude trails GPT-4o on simple single-call prompts. The gap shrinks when the task requires selecting among many tools, because Claude’s upfront reasoning reduces malformed calls and thus retry latency.

System prompt and tool descriptions

Claude respects long system prompts that describe tools in prose. That context can improve correctness but adds prefill time. GPT-4o is less sensitive. In a function calling latency benchmark, control for system prompt length or you will penalize Claude unfairly. We have observed that 500-token system prompts can add prefill latency of a couple hundred milliseconds on Claude while barely affecting GPT-4o.

Gemini: parallel calls and serialization cost

Gemini (1.5 Pro/Flash) supports multiple function calls in one response. This is powerful for agents that need to fan out. However, the generate_content response wraps calls in candidate.content.parts with function_call objects, and the API often waits to emit the full candidate rather than streaming partial args.

model = genai.GenerativeModel(
    "gemini-1.5-flash",
    tools=[{"function_declarations":[{
        "name":"refund_order",
        "parameters":{"type":"object","properties":{"order_id":{"type":"STRING"}}}
    }]}]
)
out = model.generate_content("Refund order 123")
for part in out.candidates[0].content.parts:
    if part.function_call:
        print(part.function_call.name, part.function_call.args)

Gemini’s latency profile is bimodal: for a single call it may be slightly slower to first parseable structure than GPT-4o because of response envelope overhead; for parallel independent calls it wins decisively versus sequential round-trips on other models. If your agent pattern is “gather N pieces of data then act,” Gemini cuts total latency even if per-call parse is slower.

Building a fair function calling latency benchmark

You cannot trust a single curl test. Write a harness that:

1. Pins model version and region.
2. Uses identical tool schema translated to each provider’s format.
3. Measures from socket write end to client-side parse success.
4. Excludes cold-start by warming up with five discarded calls.
5. Reports p50 and p95, not averages.
6. Validates the parsed arguments against the schema before recording success.

# pseudobench loop for OpenAI-compatible endpoint
for i in $(seq 1 100); do
  curl -s -o /dev/null -w "%{time_starttransfer}\n" https://api.openai.com/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -d @payload.json
done

Replace with signed client calls for Claude and Gemini. The point is to capture the moment your code holds a verified function_name and arguments dict. Run the harness during a stable provider window; avoid measuring during known deployment rolls.

An inference gateway such as n4n.ai can mask provider degradation via automatic fallback when a provider is rate-limited, but that resilience does not alter the intrinsic function calling latency benchmark of each model—it only prevents worst-case tails from killing your agent. A second architectural note: gateways that forward provider cache-control hints (as n4n.ai does) can trim prefill on long tool schemas by reusing cached prefix computations, which narrows Claude’s system-prompt penalty.

Tradeoffs beyond raw milliseconds

Latency is not the only axis. GPT-4o’s speed means more iterative loops per second, but it occasionally emits arguments that violate your schema, requiring a repair prompt. Claude’s slower start buys higher first-call accuracy. Gemini’s parallel calls reduce total turns but complicate your executor’s idempotency and error isolation.

If your agent makes 20 tool calls per task, a 200ms per-call advantage for GPT-4o compounds to 4 seconds saved, but if Claude avoids 3 retries that cost 2 seconds each, Claude wins end-to-end. A function calling latency benchmark that stops at single-call p50 misses this dynamic.

Streaming parsers change the equation

Implement a streaming JSON parser for GPT-4o and you can invoke the local function the moment required fields arrive, cutting effective latency to near-zero after TTFB. Claude’s text-then-tool pattern resists this; you must wait for the block. Gemini’s non-streamed function calls forbid it. Thus client engineering matters as much as model choice. Invest in a tolerant parser before you switch models.

Decisive takeaway

For latency-sensitive single-call agents, GPT-4o is the default: lowest time-to-parsed-call and stream-friendly. Choose Claude when correctness on complex tool selection outweighs raw speed, and accept the preface overhead. Use Gemini when your workload is read-heavy and parallelizable across multiple functions in one turn.

Run your own function calling latency benchmark with the methodology above before committing. The numbers shift with schema size, prompt complexity, and provider load—but the architectural gaps described here are stable. Build your agent loop to parse incrementally, keep schemas lean, and route around provider hiccups with fallback. That discipline beats chasing model leaderboard milliseconds, and it makes your system robust regardless of which frontier model you pin this quarter.