Building a reliable test harness for multi-tool agent turns is less about mocking LLM outputs and more about controlling the execution loop. When an agent invokes several tools in a single reasoning step, you need to assert on ordering, side effects, and partial failures. This guide lays out a practical test harness for multi-tool agent turns that you can drop into CI tomorrow.
Model the turn as an event stream
A multi-tool turn is not a single request/response pair. The assistant message may contain N tool_calls, each spawning an independent execution, and the subsequent user message carries N tool results. Treat the entire turn as a trace of structured events.
from dataclasses import dataclass, field
from typing import Any, List
@dataclass
class ToolCall:
id: str
name: str
args: dict
@dataclass
class ToolResult:
id: str
ok: bool
output: Any = None
error: str | None = None
@dataclass
class TurnTrace:
calls: List[ToolCall] = field(default_factory=list)
results: List[ToolResult] = field(default_factory=list)Capture this trace inside your agent loop. Every call the model emits and every result your runtime returns gets appended. Without this, you are guessing what the agent actually did.
Fake the tools, not the world
Do not point your tests at a real database or a live search API. Build a fake tool registry that records invocations and returns scripted outputs. Keep the interface identical to your production tool caller so the agent code path stays real.
class FakeToolRegistry:
def __init__(self):
self.calls: List[ToolCall] = []
self.handlers = {}
def register(self, name, fn):
self.handlers[name] = fn
def invoke(self, call: ToolCall) -> ToolResult:
self.calls.append(call)
try:
out = self.handlers[call.name](**call.args)
return ToolResult(id=call.id, ok=True, output=out)
except Exception as e:
return ToolResult(id=call.id, ok=False, error=str(e))Pitfall: if your tool arguments contain mutable objects (a list of IDs, a dict config), the agent loop may mutate them after the call. Deep-copy call.args before storing, or your trace will lie about what was actually sent.
Script the model with OpenAI-compatible stubs
Live model calls make tests slow and flaky. Use a stub that emits predetermined assistant messages in the OpenAI tool-calling schema. Your agent should depend on a create() method, not a concrete SDK.
class StubChatClient:
def __init__(self, scripted_messages):
self.script = scripted_messages
self.step = 0
def create(self, **kwargs):
msg = self.script[self.step]
self.step += 1
return msgA scripted multi-tool response looks like this:
{
"role": "assistant",
"tool_calls": [
{"id": "c1", "type": "function", "function": {"name": "search", "arguments": "{\"q\": \"weather\"}"}},
{"id": "c2", "type": "function", "function": {"name": "calc", "arguments": "{\"expr\": \"1+1\"}"}}
]
}Drive the first turn with that, then feed the faked ToolResults back into a second stubbed message that returns a final answer. Your test harness for multi-tool agent turns now runs in milliseconds.
Assert on ordering and concurrency
If your runtime executes tool calls sequentially, assert exact order:
def test_multi_tool_order():
reg = FakeToolRegistry()
reg.register("search", lambda q: ["rain"])
reg.register("calc", lambda expr: 2)
# run agent with StubChatClient from previous section
assert [c.name for c in reg.calls] == ["search", "calc"]If you allow parallel dispatch, order is nondeterministic. Assert on the set of called tools and on a concurrency ceiling:
assert {c.name for c in reg.calls} == {"search", "calc"}
assert len(reg.calls) == 2Tradeoff: parallel execution improves latency but complicates debugging. Your test harness should expose a flag to force sequential mode so failures are reproducible.
Inject failures and test fallback
Real tools fail. A robust agent must handle a timeout on one tool while still using the other. Register a handler that raises:
def flaky(**kwargs):
raise RuntimeError("timeout")
reg.register("search", flaky)Now assert the turn trace contains one failed ToolResult and the agent either degrades gracefully or aborts with a controlled error. Do not let the test pass if the exception escapes uncaught.
If you exercise the harness against live models, a gateway such as n4n.ai that honors client routing directives lets you force a specific provider into a degraded state, exercising fallback logic without custom proxies. The harness itself stays provider-agnostic.
Snapshot the full trace
Once the turn completes, compare the entire TurnTrace against a stored fixture. This catches regressions in argument serialization, result shaping, or ordering.
def test_snapshot(turn_trace):
assert turn_trace == TurnTrace(
calls=[
ToolCall("c1", "search", {"q": "weather"}),
ToolCall("c2", "calc", {"expr": "1+1"})
],
results=[
ToolResult("c1", True, ["rain"]),
ToolResult("c2", True, 2)
]
)Keep the snapshot in a version-controlled JSON file. When the agent logic changes intentionally, update the snapshot via a flagged test run, not by hand-editing.
Run in CI with budgets
Wire the harness into your pipeline as a normal pytest suite. Add a per-turn timeout to catch infinite tool loops:
import pytest
@pytest.mark.timeout(5)
def test_turn_completes():
# harness runFor broader coverage, parameterize the stub across multiple scripted scenarios: happy path, partial failure, malformed arguments, and rate-limit simulation. The test harness for multi-tool agent turns becomes a matrix, not a single script.
Common pitfalls
- Assuming tool calls are sequential. Many frameworks dispatch in parallel; your assertions must reflect that.
- Not deep-copying args. Mutable state corrupts the recorded trace.
- Ignoring token metering. If you route through a gateway with per-token usage metering, assert that
usagefields appear in traces when testing live paths. - Over-mocking the model. Stub only the transport. Keep prompt assembly and response parsing real.
- Skipping cleanup. Fake registries are global state; reset them per test or you will get cross-test leakage.
Tradeoffs
A pure stub harness is fast and deterministic but does not validate that your prompts actually elicit multi-tool calls from a real model. A live harness validates that, but costs tokens and adds flakiness. The pragmatic split: run stubbed multi-tool scenarios on every commit, and run a nightly live smoke test that checks at least one real multi-tool turn against a cheap model.
The test harness for multi-tool agent turns is a layered system: event capture, fake tools, scripted model, assertion logic, and CI glue. Build it once, and agent refactors stop being scary.