A session replay tool chatbot developers can rely on needs to capture exact request and response cycles, not just the final answer. This tutorial builds a lightweight logging and replay harness in Python that works against any OpenAI-compatible API, so you can reproduce production conversations locally and diff model behavior.
Prerequisites
- Python 3.10 or newer
openaiPython package (v1.x):pip install openai- API credentials for an OpenAI-compatible endpoint (OpenAI, Azure, or a gateway)
- Basic comfort with JSON Lines and writing small CLI scripts
If you route through n4n.ai, its OpenAI-compatible endpoint addresses 240+ models and automatically falls back when a provider is rate-limited, which keeps replays from stalling on upstream errors.
Designing the session log format
Start with a flat JSONL file. One line per model call keeps appends cheap and lets you stream-read later. Each record must store the model, the exact input messages, the returned message, and token usage.
{
"ts": "2024-05-12T08:31:02Z",
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Refund policy for orders over 30 days?"}],
"response": {"role": "assistant", "content": "Our policy allows refunds up to 60 days."},
"usage": {"prompt_tokens": 12, "completion_tokens": 9}
}Do not store just the transcript. Store the raw messages array exactly as sent, because system prompts and few-shot examples are where bugs hide.
Instrumenting your chatbot
Wrap the chat completion call. The wrapper writes a record after the response returns. Keep it synchronous for clarity; async works the same with AsyncOpenAI.
import openai, json, datetime, os
client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def chat_with_logging(messages, model="gpt-4o", log_path="sessions.jsonl"):
resp = client.chat.completions.create(model=model, messages=messages)
record = {
"ts": datetime.datetime.utcnow().isoformat() + "Z",
"model": model,
"messages": messages,
"response": {"role": "assistant", "content": resp.choices[0].message.content},
"usage": resp.usage.model_dump() if hasattr(resp.usage, "model_dump") else resp.usage,
}
with open(log_path, "a") as f:
f.write(json.dumps(record) + "\n")
return resp
# Example usage in your bot
if __name__ == "__main__":
msgs = [{"role": "user", "content": "What is your refund window?"}]
r = chat_with_logging(msgs)
print(r.choices[0].message.content)Run it once:
python bot.pyExpected output (content varies):
Our standard refund window is 30 days from delivery.And sessions.jsonl now contains one line. That file is the backbone of your session replay tool chatbot debugging flow.
Writing the replay harness
The replay script reads each line and re-issues the same messages. You can override the model to compare providers or versions.
import json, openai, sys, os
def replay(log_path, model_override=None, base_url=None, api_key=None):
client = openai.OpenAI(api_key=api_key or os.environ["OPENAI_API_KEY"], base_url=base_url)
with open(log_path) as f:
for i, line in enumerate(f):
rec = json.loads(line)
model = model_override or rec["model"]
print(f"[{i}] {rec['ts']} -> replaying with {model}")
try:
resp = client.chat.completions.create(model=model, messages=rec["messages"])
except Exception as e:
print(f" ERROR: {e}")
continue
orig = rec["response"]["content"]
new = resp.choices[0].message.content
print(f" ORIG: {orig}")
print(f" REPL: {new}")
print("---")
if __name__ == "__main__":
path = sys.argv[1] if len(sys.argv) > 1 else "sessions.jsonl"
model = sys.argv[2] if len(sys.argv) > 2 else None
replay(path, model_override=model)Run a replay against a cheaper model:
python replay.py sessions.jsonl gpt-4o-miniExpected output:
[0] 2024-05-12T08:31:02Z -> replaying with gpt-4o-mini
ORIG: Our standard refund window is 30 days from delivery.
REPL: You can request a refund within 30 days of delivery.
---The session replay tool chatbot loop now reproduces the exact context that produced a bad answer.
Diffing outputs for regression checks
Printing side-by-side is fine for one record, but a real replay harness should flag divergences. Use difflib to get a unified diff.
import difflib
def show_diff(a, b):
a_lines = a.splitlines()
b_lines = b.splitlines()
diff = difflib.unified_diff(a_lines, b_lines, lineterm="", fromfile="orig", tofile="replay")
return "\n".join(diff)
# Inside replay loop, after getting new:
if orig != new:
print(show_diff(orig, new))For a changed refund number, you would see:
--- orig
+++ replay
@@ -1 +1 @@
-Our standard refund window is 30 days from delivery.
+You can request a refund within 45 days of delivery.That diff is the signal you care about when a model swap silently changes policy statements.
Handling streaming and tool calls
Production chatbots often stream or call functions. Extend the schema:
{
"ts": "2024-05-12T09:00:00Z",
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Check order #123"}],
"tools": [{"type": "function", "function": {"name": "get_order"}}],
"response": {"role": "assistant", "tool_calls": [{"id": "call_1", "function": {"name": "get_order", "arguments": "{\"id\":123}"}}]},
"usage": {"prompt_tokens": 20, "completion_tokens": 15}
}When replaying, pass tools=rec.get("tools") and check for tool_calls instead of content. Streaming can be captured by accumulating chunks during logging; replay can use stream=False to simplify diffing. Determinism is more important than latency in a session replay tool chatbot debugger.
Running end-to-end with a gateway
If you point the client at a gateway, set base_url. This is where automatic fallback matters: a replay over hundreds of sessions should not abort because one provider returned 429.
replay(
"sessions.jsonl",
model_override="anthropic/claude-3-sonnet",
base_url="https://api.n4n.ai/v1",
api_key=os.environ["N4N_API_KEY"],
)Because the gateway honors client routing directives and forwards provider cache-control hints, cached prompts from the original call may still hit cache on replay, keeping token metering sane.
Limitations and next steps
JSONL is great until you need to query by user ID or session ID. Add a session_id field and load into SQLite or Postgres when volume grows. A web UI that renders the diff inline beats scrolling terminal output.
The session replay tool chatbot approach here is deliberately minimal: it captures the contract between your code and the model. Once that contract is recorded, you can test prompt changes, model upgrades, or provider migrations without asking users to reproduce bugs. That is the whole point of replay debugging—remove the “works on my machine” from LLM behavior.