Handling parallel tool call results in a single response

Handling parallel tool call results is a routine hurdle when you let a model invoke multiple functions in one turn. When an LLM returns a single assistant message containing several tool_calls, you must run each function, collect the outputs, and feed them back as separate tool messages so the model can synthesize one final answer. This how-to gives a complete, runnable pattern using the OpenAI-compatible request shape.

Step 1: Configure the client and request parallel calls

Point the OpenAI Python client at any OpenAI-compatible endpoint. For example, n4n.ai exposes one endpoint covering 240+ models with automatic fallback when a provider is degraded, but the code below works against any compliant server.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",  # swap for your gateway
    api_key="sk-your-key",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_stock_price",
            "description": "Get stock price for a ticker",
            "parameters": {
                "type": "object",
                "properties": {"ticker": {"type": "string"}},
                "required": ["ticker"],
            },
        },
    },
]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What's the weather in Oslo and the price of AAPL?"}],
    tools=tools,
    parallel_tool_calls=True,  # explicit; default is True but be deliberate
)
assistant_msg = response.choices[0].message

The parallel_tool_calls=True directive tells the model it may emit multiple calls. The returned assistant_msg.tool_calls will be a list if the model complied.

Step 2: Detect and extract the tool calls

Inspect the message. If tool_calls is None, the model answered directly and you are done. Otherwise, you have a list of call objects, each with a unique id, a function.name, and a function.arguments string.

import json

if not assistant_msg.tool_calls:
    print("No tool calls:", assistant_msg.content)
    exit()

calls = []
for tc in assistant_msg.tool_calls:
    calls.append({
        "id": tc.id,
        "name": tc.function.name,
        "args": json.loads(tc.function.arguments),
    })
print(f"Model requested {len(calls)} parallel calls")

Handling parallel tool call results starts with this extraction: you cannot batch the responses into one tool message. Each call needs its own reply.

Step 3: Execute the functions concurrently

Spin up the actual work. Use asyncio if your tools are I/O bound, or a thread pool if they are blocking SDK calls. The key is to launch all of them without waiting for the previous to finish.

import asyncio

async def run_tool(name, args):
    # stub implementations; replace with real HTTP/DB calls
    if name == "get_weather":
        return {"temp_c": 12, "city": args["city"]}
    if name == "get_stock_price":
        return {"price": 227.34, "ticker": args["ticker"]}
    raise ValueError(f"Unknown tool {name}")

async def dispatch(calls):
    tasks = [run_tool(c["name"], c["args"]) for c in calls]
    return await asyncio.gather(*tasks, return_exceptions=True)

results = asyncio.run(dispatch(calls))

Wrapping with return_exceptions=True keeps one failing tool from killing the whole batch. You still need to map each result back to its call by index.

Step 4: Format results as individual tool messages

The API requires a message with role: "tool" for every tool_call_id. The content must be a string, typically JSON.

tool_messages = []
for call, res in zip(calls, results):
    if isinstance(res, Exception):
        content = json.dumps({"error": str(res)})
    else:
        content = json.dumps(res)
    tool_messages.append({
        "role": "tool",
        "tool_call_id": call["id"],
        "content": content,
    })

A common mistake in handling parallel tool call results is reusing a single tool_call_id or omitting one. The server will reject the request if the IDs do not match the assistant message exactly.

Step 5: Send the consolidated context back

Append the original assistant message (with its tool_calls) and all tool messages to your conversation history, then call the model again. Do not set tools unless you want it to call again; usually you omit or keep them but the model should now answer.

follow_up = [{"role": "user", "content": "What's the weather in Oslo and the price of AAPL?"}]
follow_up.append(assistant_msg.model_dump())  # includes tool_calls
follow_up.extend(tool_messages)

final = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=follow_up,
)
print(final.choices[0].message.content)

The model now has all parallel outputs in context and can produce a single natural-language response that references both the weather and the stock price.

Step 6: Verify the round trip

Success means the final message contains no tool_calls and non-empty content. Add a quick assertion in tests:

assert final.choices[0].message.tool_calls is None
assert len(final.choices[0].message.content) > 0
# optional: check usage metering
print(final.usage.model_dump())

If you are using a gateway that meters per-token usage, the usage object reflects both the initial and follow-up calls separately. Run the script end to end; you should see the stubbed weather and stock data merged into one sentence.

Pitfalls when handling parallel tool call results

Streaming

If you stream the first response, tool_calls arrive in fragments. Accumulate delta.tool_calls across chunks before executing. The same ID-matching rule applies.

Partial failures

Returning an error string in a tool message is valid. The model can recover or explain the failure. Do not raise from the dispatch loop; surface it as content.

Order independence

Tool messages do not need to be in the same order as tool_calls, but all must be present. Some SDKs will silently drop a tool message with an unmatched ID.

Cache hints

If your gateway forwards provider cache-control hints, mark static tool schemas with cache_control to cut token cost on the second call. n4n.ai honors client routing directives and forwards provider cache-control hints, which helps when you repeat large tool definitions across multi-step turns.

Closing

The pattern is stable: extract, execute concurrently, map by ID, return all, and let the model merge. Once this loop is abstracted into a small runner, handling parallel tool call results becomes a few lines of orchestration rather than a source of bugs.