To ship reliable AI agents, you need to simulate multi-turn conversations agent testing before production traffic ever hits them. A single happy-path call doesn’t expose state drift, tool-call loops, or context overflow—only repeated, scripted dialogues do. This guide gives you a concrete harness you can drop into a repo today and extend as your agent grows.
Step 1: Define a deterministic agent harness
Wrap your agent so the test runner owns the message list and can inspect every return value. Do not call the agent’s HTTP server inline; call the core function directly. This keeps tests fast, removes network flakiness, and lets you assert on internal state such as memory or drafted tool calls.
# agent_core.py
from typing import List, Dict, Any, Tuple
def run_agent(messages: List[Dict[str, str]], memory: Dict[str, Any]) -> Tuple[Dict[str, str], Dict[str, Any]]:
"""Pure-ish core. Returns (assistant_message, updated_memory)."""
last_user = messages[-1]["content"] if messages else ""
if "reset" in last_user:
memory.clear()
# Real implementation calls an LLM and maybe tools.
# For tests, inject a fake or mock at this boundary.
reply = f"echo: {last_user}"
memory.setdefault("turns", 0)
memory["turns"] += 1
return {"role": "assistant", "content": reply}, memoryIf your agent calls an LLM, inject the client. In tests, replace it with a stub that returns canned responses keyed by turn index. That makes simulate multi-turn conversations agent testing fully deterministic and milliseconds fast.
Step 2: Build a scripted simulated user
A scripted user is a data file describing what the fake human says and what we expect the agent to do. Keep dialogues as YAML so product managers can add cases without touching Python.
# tests/fixtures/checkout_flow.yaml
user_turns:
- "I want to buy a red shirt"
- "Size medium"
- "Use my saved card"
- "Actually reset my cart"
expected_assistant_substrings:
- "red shirt"
- "size medium"
- "payment"
- "reset"
final_memory_keys:
- "cart"
forbidden_substrings:
- "I don't know"Load and parametrize:
import yaml, glob, os
def load_all_scripts(folder="tests/fixtures"):
for path in glob.glob(f"{folder}/*.yaml"):
with open(path) as f:
yield os.path.basename(path), yaml.safe_load(f)Negative paths matter. Add a script where the user sends gibberish or asks for something impossible; assert the agent declines gracefully instead of hallucinating a tool call.
Step 3: Drive the multi-turn loop with tool calls
Real agents emit tool_calls. Your simulator must echo tool results back as tool role messages, otherwise the next agent turn breaks. Extend the loop:
def simulate_conversation(script: dict, agent_fn=run_agent, fake_tools=None):
messages = []
memory = {}
transcript = []
for turn in script["user_turns"]:
messages.append({"role": "user", "content": turn})
resp, memory = agent_fn(messages, memory)
messages.append(resp)
transcript.append(("user", turn))
transcript.append(("agent", resp["content"]))
# simulate tool round-trip if agent asked for one
if fake_tools and resp.get("tool_calls"):
for call in resp["tool_calls"]:
result = fake_tools(call["function"]["name"], call["function"]["arguments"])
messages.append({"role": "tool", "content": result, "tool_call_id": call["id"]})
return transcript, memory, messagesThis loop is the engine for any simulate multi-turn conversations agent testing setup. It mirrors what the OpenAI chat completions API expects, so you can later swap the fake tools for recorded fixtures.
Step 4: Add assertion and evaluation hooks
A simulation without assertions is a demo. Use pytest to enforce invariants: required substrings, memory shape, forbidden phrases, loop detection, and token budgets.
import pytest
@pytest.mark.parametrize("name,script", load_all_scripts())
def test_scripted_flows(name, script):
_, memory, messages = simulate_conversation(script)
joined = " ".join(m["content"] for m in messages if m["role"] == "assistant")
for exp in script.get("expected_assistant_substrings", []):
assert exp in joined, f"{name}: missing '{exp}'"
for key in script.get("final_memory_keys", []):
assert key in memory, f"{name}: memory missing '{key}'"
for forbid in script.get("forbidden_substrings", []):
assert forbid not in joined, f"{name}: forbidden '{forbid}' appeared"
assistant = [m["content"] for m in messages if m["role"] == "assistant"]
assert max(assistant.count(m) for m in set(assistant)) <= 2, "possible loop"Add a token counter if your agent exposes usage. Fail the test if a 6-turn chat exceeds, say, 4000 tokens—that flags context bloat early.
Step 5: Scale with stochastic simulated users
Scripted tests catch known paths. To find edge cases, generate a simulated user with another LLM. Seed it with a persona and a goal, then let it improvise. Use a gateway that fronts multiple models so you can swap the sim user’s brain without code changes.
If you run these simulations against an OpenRouter-class gateway like n4n.ai, you get automatic fallback when a provider is rate-limited, per-token usage metering, and it forwards provider cache-control hints so long system prompts in repeated turns aren’t re-billed. The OpenAI-compatible endpoint addresses 240+ models, meaning one client works for the agent and the simulator.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def llm_user_actor(goal: str, history: list, seed: int = 42) -> str:
sys = f"You are a user testing an agent. Goal: {goal}. One short message only."
resp = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "system", "content": sys}] + history,
temperature=0.7,
seed=seed,
)
return resp.choices[0].message.contentDrive a bounded loop:
def stochastic_sim(goal: str, max_turns: int = 8):
messages, memory = [], {}
for _ in range(max_turns):
user_msg = llm_user_actor(goal, messages)
messages.append({"role": "user", "content": user_msg})
agent_resp, memory = run_agent(messages, memory)
messages.append(agent_resp)
if "done" in agent_resp["content"].lower():
break
return messages, memoryAlways cap turns. An unbounded agent-vs-agent chat will burn tokens and sometimes never terminate.
Step 6: Run in CI and measure
Put scripted suites on every PR. Stochastic suites run nightly with a fixed seed for reproducibility.
# .github/workflows/agent-tests.yml
name: agent-tests
on: [pull_request]
jobs:
scripted:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.11"}
- run: pip install pytest pyyaml
- run: pytest tests/ -q
nightly-stochastic:
runs-on: ubuntu-latest
if: github.event_name == 'schedule'
steps:
- uses: actions/checkout@v4
- run: python scripts/run_stochastic.py --seed 42 --out artifacts/
- uses: actions/upload-artifact@v4
with: {name: transcripts, path: artifacts/}How to verify success
A green scripted pytest run with all memory and substring assertions passing is the baseline. For stochastic runs, success means: the loop terminated within max_turns, no exception was raised, and an LLM-graded checklist (e.g., “did the agent reach the user’s goal?”) scores above your threshold. Store those scores over time; a dropping trend is an early warning before users complain.
Step 7: Instrument and iterate
Log every simulated transcript to a local file or bucket. When a production incident occurs, write a new scripted YAML from the real conversation and add it to the suite. That feedback loop is what makes simulate multi-turn conversations agent testing pay off—your test corpus grows from real failures, not imagination.
Keep the agent core pure, the user sims swappable, and the assertions strict. Do that and multi-turn regressions stop reaching production.