Parallel tool calls: which LLM APIs support them

Parallel tool calls LLM API support has become a baseline requirement for agents that need to fetch multiple independent resources in a single inference turn. OpenAI, Anthropic, and Google Gemini all return multiple invocations in one response, but the wire format, limits, and client ergonomics differ enough to affect your abstraction layer.

What “parallel tool calls” means

The model emits two or more tool invocations in a single assistant message. Your runtime executes them concurrently, collects results, and returns them in one follow-up. This is distinct from sequential tool use, where the model waits for one result before deciding the next step.

The contenders

We compare five providers with publicly documented tool-use APIs:

OpenAI (Chat Completions tools/tool_calls)
Anthropic (Claude Messages tools with tool_use blocks)
Google Gemini (Function Declarations with functionCall parts)
Mistral AI (Chat Completion tool_calls)
Cohere (Command R+ tool_calls in chat)

Capabilities

OpenAI returns an array of tool_calls on the assistant message. Each has an id, function.name, and function.arguments (JSON string). The model can emit any number of these in one turn.

Anthropic returns an array of content blocks; tool_use blocks carry id, name, and input (already-parsed JSON object). Multiple blocks appear in the same assistant message.

Gemini attaches multiple functionCall parts to a single model message. Each part has name and args (object). The API does not assign per-call IDs; you correlate by index or name.

Mistral’s API mirrors OpenAI’s tool_calls array, but in practice the model rarely emits more than a handful; the platform does not document a hard cap.

Cohere’s Command R+ supports multiple tool_calls in a single response, but its agent loop is designed for multi-step retrieval augmented generation, not arbitrary fan-out. The calls are returned as an array with name and parameters.

Despite differences, every parallel tool calls LLM API listed here requires you to return a matching result for each call before continuing the conversation.

Price/cost model

None of the providers charge extra for parallel invocation itself. You pay per input token (including tool definitions) and per output token (including the emitted call arguments). Parallel calls can reduce total latency and thus total turns, which often lowers cost indirectly by avoiding repeated prompt prefixes.

OpenAI, Anthropic, Gemini, Mistral, and Cohere all meter by token. Exact rates vary by model tier; we won’t quote them because they shift frequently.

Latency/throughput

Parallel calls do not speed up the model’s generation—the calls are produced in one token stream. The win is on the client side: you replace N sequential round-trips with one generation plus concurrent execution. Throughput on the provider side is unchanged.

In our experience, OpenAI and Anthropic stream the full set of calls before yielding control. Gemini’s parts arrive together in the non-streaming response; streaming yields parts as generated. Mistral and Cohere behave similarly to OpenAI.

Ergonomics

OpenAI’s format forces you to parse JSON strings from function.arguments. Anthropic gives you a parsed object. Gemini gives objects but no IDs, so you must construct response functionResponse parts in the same order. Mistral copies OpenAI’s string parsing. Cohere returns objects but expects a specific tool_results array shape.

If you build a unified agent loop, normalize these into a common ToolCall type:

from dataclasses import dataclass
import json

@dataclass
class ToolCall:
    id: str | None
    name: str
    args: dict

def from_openai(msg):
    return [ToolCall(tc.id, tc.function.name, json.loads(tc.function.arguments))
            for tc in msg.tool_calls]

def from_anthropic(content_blocks):
    return [ToolCall(b.id, b.name, b.input)
            for b in content_blocks if b.type == "tool_use"]

Streaming accumulation

OpenAI streams tool_calls as deltas with an index. You must buffer partial argument strings per index and parse only when the stream ends. Anthropic streams tool_use blocks with incremental input JSON; its SDK handles assembly. Gemini streams functionCall parts as they complete.

Ecosystem

OpenAI has the largest tooling ecosystem: LangChain, LlamaIndex, and OpenAI’s own Agents SDK assume tool_calls. Anthropic’s SDK is first-class in Claude-specific frameworks. Gemini integrates with Vertex AI tooling. Mistral and Cohere have smaller but functional client libraries.

A gateway like n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and forwards provider cache-control hints, letting you keep the OpenAI-shaped tool_calls loop while routing to any backend that supports parallel tool calls LLM API semantics.

Limits

OpenAI documents no explicit cap on number of tool_calls per message, but the output token limit bounds it. Anthropic similarly bounds by max tokens. Gemini limits by total functionCall parts within context. Mistral and Cohere impose practical limits via output token budgets.

All providers require that you return a matching tool result for each call before continuing the conversation, except Gemini where you return functionResponse parts in parallel.

Comparison table

Provider	Wire format	Per-call ID	Args type	Streaming	Doc’d call cap
OpenAI	`tool_calls[]`	Yes (`id`)	JSON string	Yes	None (token-bound)
Anthropic	`tool_use` blocks	Yes (`id`)	Object	Yes	None (token-bound)
Gemini	`functionCall` parts	No	Object	Yes (parts)	None (context-bound)
Mistral	`tool_calls[]`	Yes (`id`)	JSON string	Yes	None (token-bound)
Cohere	`tool_calls[]`	No (index)	Object	Yes	None (token-bound)

Request shapes

OpenAI request snippet:

{
  "model": "gpt-4o",
  "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object"}}}],
  "messages": [{"role": "user", "content": "Weather in NYC and SF?"}]
}

Response contains:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "tool_calls": [
        {"id": "call_1", "function": {"name": "get_weather", "arguments": "{\"city\":\"NYC\"}"}},
        {"id": "call_2", "function": {"name": "get_weather", "arguments": "{\"city\":\"SF\"}"}}
      ]
    }
  }]
}

Anthropic equivalent:

{
  "model": "claude-3-5-sonnet",
  "tools": [{"name": "get_weather", "input_schema": {"type": "object"}}],
  "messages": [{"role": "user", "content": "Weather in NYC and SF?"}]
}

Returns content blocks:

[
  {"type": "tool_use", "id": "tu_1", "name": "get_weather", "input": {"city": "NYC"}},
  {"type": "tool_use", "id": "tu_2", "name": "get_weather", "input": {"city": "SF"}}
]

Which to choose

Build a general agent framework with one abstraction: Use OpenAI or Anthropic as primary. Their ID-bearing calls simplify result correlation. If you need model portability, put an OpenAI-compatible gateway in front.

Tight Vertex AI integration: Gemini is natural. Write a small normalizer for missing IDs.

European hosting or Mistral-specific models: Mistral works but expect to mirror OpenAI parsing code.

Retrieval-heavy RAG agents: Cohere Command R+ offers built-in multi-step tool loops; parallel calls are supported but the SDK steers you toward its opinionated pattern.

High availability with fallback: Route through a gateway such as n4n.ai that honors client routing directives and provides automatic fallback when a provider is rate-limited or degraded. This keeps your parallel tool calls LLM API logic intact while shifting backends.

Parallel tool calls are no longer experimental. Pick the provider whose wire format matches your client’s tolerance for JSON parsing, and standardize on a single internal representation before the agent loop grows.