Streaming with tool calls in the same response: what breaks

Most LLM gateways treat streaming and tool invocation as orthogonal features, but the moment you ask a model to emit tokens and call a function in one turn, the abstraction leaks. The streaming plus tool calls same response issues are not cosmetic; they expose fundamental mismatches between the OpenAI-compatible schema and how providers actually sequence output.

The protocol mismatch at the core

The OpenAI chat completion streaming schema ships deltas as choices[].delta objects. A delta may contain role, content, or tool_calls. The spec implies a message can hold both content and tool_calls after assembly, but it never defines interleaving rules during the stream.

{
  "choices": [
    {
      "delta": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "index": 0,
            "id": "call_1",
            "type": "function",
            "function": { "name": "get_weather", "arguments": "" }
          }
        ]
      },
      "finish_reason": null
    }
  ]
}

A later chunk carries arguments as a partial JSON string. The client must buffer and concatenate. If a chunk also slipped in content: "Let me check", the client has no guaranteed ordering relative to the tool call start. The streaming plus tool calls same response issues surface first here: you cannot render text and simultaneously commit to a tool invocation without risking a rewrite of the UI when the finish reason resolves to tool_calls.

How providers actually behave

OpenAI native

OpenAI’s models generally emit either content or tool calls, not both. When a tool call is pending, content is null across chunks. The finish_reason switches to tool_calls at the end. This is clean but restrictive: you lose the ability to stream a natural language preface before the call.

Anthropic native

Anthropic’s streaming API uses content blocks. A text block streams deltas, then a tool_use block streams input_json_delta. Interleaving is explicit and ordered. The client gets a content_block_start event for the tool use after text ends. This avoids ambiguity but is not OpenAI-compatible.

Gateways and translation layers

An OpenAI-compatible endpoint such as n4n.ai, which fronts 240+ models with automatic fallback, must translate Anthropic’s block stream into OpenAI chunks while honoring client routing directives and forwarding provider cache-control hints. That translation layer multiplies the streaming plus tool calls same response issues because the gateway cannot invent ordering guarantees the source lacks. If the upstream provider streams text then tool use, the gateway has to emit a content delta chunk, then a tool_calls delta chunk, but the OpenAI client libraries often assume mutual exclusivity and discard the earlier text.

Parsing complexity and failure modes

Naive clients accumulate tool_calls by index. The code works until a provider sends a chunk where arguments is split mid-token inside a JSON string escape.

tool_calls = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.tool_calls:
        for tc in delta.tool_calls:
            idx = tc.index
            tool_calls.setdefault(idx, {"id": "", "name": "", "args": ""})
            if tc.id:
                tool_calls[idx]["id"] = tc.id
            if tc.function and tc.function.name:
                tool_calls[idx]["name"] += tc.function.name
            if tc.function and tc.function.arguments:
                tool_calls[idx]["args"] += tc.function.arguments

# at stream end
for tc in tool_calls.values():
    parsed = json.loads(tc["args"])  # throws if partial

If the connection drops after finish_reason is null, you hold a truncated JSON string. Worse, some gateways emit an empty content delta alongside the first tool_calls delta to satisfy the schema, polluting your text buffer with a ghost newline.

The streaming plus tool calls same response issues also break assumptions about cost metering. Per-token usage in a mixed stream is reported only in the final chunk; if you abort early because you detected a tool call, you may undercount billed tokens.

Tradeoffs of supporting same-response streaming+tool

Pro: lower latency. The model thinks and calls in one round trip. The user sees a spinner or partial text while the function is being selected.

Con: client complexity explodes. You need a state machine that handles four transitions: text-only, text→tool, tool-only, tool→text. Most SDKs ship only the text-only and tool-only paths. If you build a generic agent loop, you will write custom SSE parsing.

Con: error recovery is harder. A malformed tool argument mid-stream forces you to either discard the text preface or replay it. In a strict agent harness, you want deterministic replay; mixed streams defeat that.

A robust pattern: split the phases

The decisive fix is to decouple streaming from tool invocation at the application layer. Phase one: call the model with tools omitted (or tool_choice: "none") and stream the rationale to the user. Phase two: if the policy requires a tool, issue a second non-streamed request with tool_choice forced to the target function.

# Phase 1: stream explanation
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    stream=True,
    tool_choice="none"
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

# Phase 2: force tool call, no streaming
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "get_weather"}}
)
tool_call = resp.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)

This adds one round trip but removes an entire class of parser bugs. For latency-sensitive UIs, you can still stream phase one and show a “preparing action” state the instant phase two starts.

If you must consume a provider that natively interleaves, parse the raw SSE and map blocks explicitly rather than trusting the OpenAI delta union.

// Minimal SSE handler for Anthropic-style blocks via gateway
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value);
  const lines = buffer.split("\n\n");
  buffer = lines.pop() ?? "";
  for (const line of lines) {
    if (line.startsWith("data:")) {
      const evt = JSON.parse(line.slice(5));
      if (evt.type === "content_block_delta" && evt.delta.type === "text_delta") {
        renderText(evt.delta.text);
      } else if (evt.type === "content_block_start" && evt.content_block.type === "tool_use") {
        beginTool(evt.content_block.name);
      }
    }
  }
}

Decisive takeaway

Treat streaming and tool calls as separate response contracts. The streaming plus tool calls same response issues are a symptom of forcing two distinct output modes through one mutable delta schema. If you control the client, split the phases: stream text freely, then resolve tools in a constrained follow-up. If you are building a gateway or a generic agent framework, invest in a block-structured internal representation and translate to OpenAI chunks only at the edge, never the other way around. The slight latency cost buys you correctness, replayability, and one fewer outage at 2 a.m. when a provider changes its chunk ordering.