Multi-turn conversation state bugs are the silent killers of production chatbots: they hide behind seemingly correct single-turn tests and only erupt after the user has sent five messages and a tool call. A multi-turn conversation state bug usually looks like missing context, repeated clarifications, or a suddenly forgetful assistant, and reproducing it requires the exact sequence of prior turns. This article gives you an end-to-end workflow to capture, replay, and pin down those defects without guessing.
Step 1: Capture the full session transcript with metadata
You cannot debug what you did not record. In most frameworks the messages array is assembled at request time from a mix of system prompts, retrieved memory, and tool outputs. If any of those steps silently mutate state, you need the raw inputs and outputs for every turn.
Add a logging sink that writes one JSON line per turn, including the model, tools, and the exact response. Do this at the boundary where you call the LLM, not inside your business logic.
import json, time, uuid
def log_turn(session_id, messages, resp, model, tools=None):
entry = {
"ts": time.time(),
"session_id": session_id,
"messages": messages,
"tools": tools,
"model": model,
"response": resp,
"turn_id": uuid.uuid4().hex[:8]
}
with open(f"sessions/{session_id}.jsonl", "a") as f:
f.write(json.dumps(entry) + "\n")Call log_turn immediately after each completion. Store the file alongside your normal application logs but treat it as forensic data: it contains user inputs.
Verify success: After a buggy session, wc -l sessions/<id>.jsonl equals the number of user+assistant exchanges you expect. If the count is off, your logging hook is not wrapping every path (streaming, fallback, timeouts).
Step 2: Normalize and reconstruct the message history
Raw logs are not enough. You must reconstruct the exact messages array the model saw on the failing turn. A common multi-turn conversation state bug comes from a re-indexing step that drops a system message or merges a tool result into the wrong user turn.
Load the session and sort by timestamp. Assert structural invariants before you trust it.
import json
def load_session(path):
turns = []
with open(path) as f:
for line in f:
turns.append(json.loads(line))
turns.sort(key=lambda t: t["ts"])
ids = [t["turn_id"] for t in turns]
assert len(ids) == len(set(ids)), "duplicate turn ids"
# ensure alternating roles if no tools
return turnsWalk the reconstructed history and print the role sequence. If you see two assistant messages back-to-back, your state machine double-appended a response. If the system prompt vanishes after turn three, your pruning logic is too aggressive.
Verify success: The printed role sequence matches your intended conversation flow, and load_session does not raise.
Step 3: Replay the session against a deterministic harness
With a clean transcript, replay it with temperature=0 against the same model. This removes sampling noise and tells you whether the bug is in your state assembly or in the model itself.
Point your replay script at an OpenAI-compatible endpoint. If you use a gateway such as n4n.ai, its single OpenAI-compatible endpoint spanning 240+ models lets you replay the same transcript against multiple backends to see whether the defect is model-specific or in your own state handling.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
def replay(turns, model):
messages = []
for t in turns:
messages.extend(t["messages"])
messages.append({
"role": "assistant",
"content": t["response"]["choices"][0]["message"]["content"]
})
# drop final assistant message; we want the model to generate it
messages = messages[:-1]
resp = client.chat.completions.create(
model=model,
messages=messages,
temperature=0,
tools=turns[0]["tools"]
)
return resp.choices[0].messageRun the replay and diff the generated final turn against the logged one. For a deterministic model and unchanged state, the content should match exactly (ignoring whitespace normalization if you trimmed tokens).
Verify success: The diff shows no semantic difference in the final assistant message. If it diverges, your replay harness does not faithfully reproduce the production message array—fix the harness before blaming the model.
Step 4: Instrument state mutations explicitly
Most chatbots keep a ConversationState object: user profile, retrieved docs, flag for pending confirmation. A multi-turn conversation state bug often lives in a callback that mutates this object after the snapshot used to build the prompt was taken.
Wrap the state in a tracer that logs every assignment.
class TracedState:
def __init__(self, initial=None):
self._data = dict(initial or {})
def __setattr__(self, name, val):
if name != "_data":
print(f"STATE SET {name} = {val}")
super().__setattr__(name, val)
def update(self, patch):
for k, v in patch.items():
setattr(self, k, v)Replace your plain dict or dataclass with TracedState in a staging environment and replay the buggy session. You will see the exact turn where a tool result overwrites user_name or where context_docs is set to None.
Verify success: The trace shows a state write that occurs after the prompt was assembled for the next turn. That write is your prime suspect.
Step 5: Isolate the turn where state diverges
Once you suspect a specific mutation, bisect the session. Replay only the first k turns and snapshot the state. Compare against the production snapshot taken at the same point.
def replay_until(turns, k, model):
return replay(turns[:k], model)
# manual bisect
for k in [len(turns)//2, len(turns)//4, ...]:
out = replay_until(turns, k, "gpt-4o-mini")
print(f"turn {k}: {out.content[:50]}")If the state is correct at turn k but wrong at k+1, the bug is in the transition. Inspect the tool call or post-processing hook that runs between those turns. In practice, nine out of ten multi-turn conversation state bug reports I have chased were a missing state.update() guard inside an error branch that only triggers when a tool returns a 429.
Verify success: You can name the exact turn index and the function that corrupts state.
Step 6: Fix and add a regression guard
The fix is usually small: deep-copy the state before prompt assembly, or move the mutation before the snapshot. After patching, encode the buggy session as a fixture and assert the replay matches the corrected output.
def test_session_replay():
turns = load_session("fixtures/bug.jsonl")
out = replay(turns, "gpt-4o-mini")
expected = turns[-1]["response"]["choices"][0]["message"]["content"]
assert out.content.strip() == expected.strip()Keep the full transcript in your repo under tests/fixtures. It is the only reliable regression test for session-level logic; unit tests on individual turns will not catch cross-turn leakage.
Verify success: pytest tests/fixtures/bug.jsonl passes, and intentionally reverting the fix makes it fail.
Step 7: Monitor for recurrence in production
A one-off fix is not enough. Add a lightweight runtime check that compares the role sequence and state hash each turn, emitting a metric when they deviate from invariants (e.g., system prompt always present, user_id immutable after turn one).
def assert_invariants(state, messages):
assert messages[0]["role"] == "system"
assert state.user_id is not NoneShip this behind a feature flag with low sampling. When a new multi-turn conversation state bug appears, you will have the violating session id and the exact turn logged before users complain.
Verify success: In staging, force a known corruption and confirm the metric fires and the session is captured to your JSONL store.
Following these steps converts a vague “the bot forgot my name” ticket into a deterministic fixture and a one-line patch. The key is treating conversation state as data you can serialize, replay, and diff—not as something that lives only inside a running process.