A user files a ticket: the assistant hallucinated a discount code mid-conversation. To fix it, you need to reproduce chatbot bug production transcript exactly as it occurred, including the system prompt, model parameters, and tool schemas. Guessing from a screenshot wastes cycles; the transcript is the ground truth.
Step 1: Extract the raw transcript and metadata from your log store
Production chat logs usually land in an object store, data warehouse, or append-only database as a JSON document. A typical schema separates messages from metadata so you can replay the inference call without UI noise.
{
"session_id": "s_8f2c",
"metadata": {
"model": "gpt-4o-mini",
"temperature": 0.2,
"top_p": 1.0,
"max_tokens": 512,
"seed": 12345,
"client_version": "web-2.3.1"
},
"messages": [
{"role": "system", "content": "You are a retail assistant..."},
{"role": "user", "content": "Do you have the blue shirt?"},
{"role": "assistant", "content": "Yes, size M is in stock."},
{"role": "user", "content": "Apply a discount"}
],
"final_output": "Use code BLUE20 for 20% off."
}Load it with a few lines of Python:
import json
with open("s_8f2c.json") as f:
transcript = json.load(f)
messages = transcript["messages"]
meta = transcript["metadata"]If your logs live in Postgres, query by session_id and dump to JSON. Keep the original final_output field; you will need it for diffing in Step 5. Do not truncate the log to “relevant” turns yet—premature trimming hides context that triggered the bug.
What to capture if you aren’t already
If transcripts lack metadata.model or seed, add those fields to your logging layer today. A bug you cannot parameter-match is a bug you cannot reproduce chatbot bug production transcript for. Log the exact request object sent to the model provider, minus secrets.
Step 2: Normalize the conversation into a replayable message list
Raw logs often contain UI artifacts: typing indicators, client-side timestamps, read receipts, or redacted PII placeholders. Strip anything that was not transmitted to the model. The goal is a list compatible with the Chat Completions API.
def normalize(transcript):
allowed = {"system", "user", "assistant", "tool"}
clean = []
for m in transcript["messages"]:
if m["role"] not in allowed:
continue
# Drop client-only fields like 'ts' or 'ui_state'
clean.append({"role": m["role"], "content": m.get("content", "")})
return clean
replay_messages = normalize(transcript)If your bot uses function calls, preserve tool_calls on assistant messages and the subsequent tool role messages exactly. Altering order or dropping a tool result breaks reproduction because the model conditions on that state.
Handling streaming partials
Some logging pipelines capture streaming chunks. Collapse them into the final assistant content before replay. Re-sending partial tokens will skew the context window and change the completion.
# Example: merge streamed deltas for assistant turns
def merge_stream(messages):
out = []
for m in messages:
if m["role"] == "assistant" and isinstance(m.get("content"), list):
m = {**m, "content": "".join(part.get("text","") for part in m["content"])}
out.append(m)
return outStep 3: Recover the exact inference parameters
The bug may be parameter-dependent. Pull model, temperature, top_p, max_tokens, and seed from metadata. If seed is absent, the run is non-deterministic; plan multiple replays.
params = {
"model": meta["model"],
"temperature": meta.get("temperature", 1.0),
"top_p": meta.get("top_p", 1.0),
"max_tokens": meta.get("max_tokens", 1024),
}
if "seed" in meta:
params["seed"] = meta["seed"]When the model id in the log is a routing alias (e.g., retail-default), map it to the concrete provider model using your gateway config. This is where a unified endpoint helps: if production routed through n4n.ai, the same OpenAI-compatible endpoint can replay against the resolved model with automatic fallback when a provider is degraded, so you don’t need to hardcode provider URLs.
If you used response caching in production, note the cache_control hints. Forward the same hints during replay to avoid cache misses that change latency but not output.
Step 4: Replay the session against an OpenAI-compatible endpoint
Use the official openai Python client pointed at your endpoint. The script below sends the full conversation and prints the reconstructed response.
from openai import OpenAI
client = OpenAI(
base_url="https://api.openai.com/v1", # swap for your gateway
api_key="YOUR_KEY"
)
resp = client.chat.completions.create(
messages=replay_messages,
**params
)
print(resp.choices[0].message.content)For a transcript with N turns, replay the entire array up to the faulty turn. To reproduce chatbot bug production transcript at turn K, truncate replay_messages to replay_messages[:K] and inspect the model’s completion.
Run the replay three to five times if no seed was set. LLM outputs drift; a single sample can mislead you into thinking the bug is gone.
Replaying tool-calling sessions
If the transcript includes tool_calls, you must simulate the tool execution. Stub the tool with the exact payload from the production tool message:
if "tool_calls" in replay_messages[-1]:
# append the recorded tool result, do not re-execute
replay_messages.append({
"role": "tool",
"tool_call_id": replay_messages[-1]["tool_calls"][0]["id"],
"content": transcript["tool_results"][0]["content"]
})Step 5: Diff the replay output against production output
With a seed, the output should match byte-for-byte. Without one, use semantic similarity. Compute cosine similarity between embeddings of the production final_output and each replay sample.
import numpy as np
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# Assume embed() calls an embedding model via the same client
prod_emb = embed(transcript["final_output"])
replay_emb = embed(resp.choices[0].message.content)
sim = cosine(prod_emb, replay_emb)
print(f"similarity={sim:.3f}")A similarity above 0.95 means you likely reproduced the behavior. If the replay never emits the bad discount code while production did, your parameters or prompt state are still off—go back to Step 2.
Verifying reproduction
Mark the bug as reproduced only when the replay produces the same failure mode (e.g., hallucinated code) on at least 2 of 5 unseeded runs, or exactly on a seeded run. Document the trigger turn and the similarity score in your ticket.
Step 6: Bisect the conversation to localize the fault
Once you can reproduce chatbot bug production transcript, cut the input to find the minimal trigger. Drop the first user turn, replay, then binary search.
def replay_subset(client, messages, params, end_idx):
return client.chat.completions.create(
messages=messages[:end_idx], **params
)
for i in range(1, len(replay_messages)+1):
out = replay_subset(client, replay_messages, params, i)
if "BLUE20" in out.choices[0].message.content:
print(f"Bug triggered at turn {i}")
breakOften the failure appears only after a specific user correction or after a tool result injects stale data. The bisect tells you whether the system prompt is leaky or a downstream tool response is corrupt. If the bug triggers at turn 3 but not turn 2, inspect the assistant message at turn 2 for a misleading commitment.
Isolate prompt vs. model drift
If you suspect a model version change, pin an older model id in params and re-bisect. Production may have silently shifted from gpt-4o-mini-2024-07 to gpt-4o-mini-2024-09. Your gateway’s per-token metering can confirm which version was billed at incident time.
Step 7: Patch and verify the fix
Suppose the root cause is an underspecified system prompt. Edit only the system message in replay_messages:
replay_messages[0]["content"] += "\nNever invent discount codes. Ask the promotions API."Re-run the replay loop from Step 4. Success criteria: the model no longer outputs a fake code across 5 unseeded runs, and the cosine similarity to a correct reference answer is high.
Verification checklist
- Replay with original params reproduces bug (pre-patch)
- Patched prompt eliminates bug in ≥5 runs
- No regression on earlier turns (re-run full transcript, check prior answers unchanged)
- Tool schemas unchanged unless explicitly fixed
- Seed-based replay (if used) now diverges from buggy output
If you route through a gateway that honors client routing directives, pin the same model version during verification to avoid silent provider swaps.
Step 8: Store the transcript as a regression fixture
Check the anonymized transcript into tests/fixtures/ next to your CI suite. Write a pytest that loads it, replays against a stub or live endpoint, and asserts the bug is absent.
def test_discount_hallucination():
t = json.load(open("tests/fixtures/s_8f2c.json"))
msgs = normalize(t)
msgs[0]["content"] += "\nNever invent discount codes."
out = client.chat.completions.create(messages=msgs, **params)
assert "BLUE20" not in out.choices[0].message.contentNow the next engineer who touches the prompt gets a red build if they reintroduce the defect. This closes the loop: you reproduced the issue, fixed it, and prevented recurrence.
Closing notes on workflow
Reproducing a chatbot defect is forensic work. Treat the production transcript as a test fixture, not a postmortem anecdote. The discipline to reproduce chatbot bug production transcript with the same parameters, same tools, and same message order turns vague complaints into deterministic bugs. Build the replay script once, wire it into CI, and your debugging lead time drops from days to minutes.