Most LLM chatbots ship blind. If you are not running session replay chatbot qa, you are guessing about why a conversation went wrong—unable to reproduce the exact model inputs, tool calls, and retries that produced a bad answer. Replay turns intermittent failures into reproducible test cases you can run in CI.
Why logs are not enough
Standard application logs capture latency and maybe the final prompt. They omit the system prompt mutations, the intermediate function outputs, and the specific model version that answered. Session replay chatbot qa records the entire exchange as a first-class artifact.
Without it, debugging a hallucination means manually reconstructing the context window from scattered traces. That reconstruction is itself error-prone and rarely matches production exactly.
Logs also lack the request/response pairing across retries. A provider timeout followed by a fallback changes the model that ultimately answered. Your log shows two calls; your replay shows the sequence and the winner.
Step 1: Capture the full request/response cycle
What to record
At minimum, store:
- The exact messages array sent to the model (including system prompts and any injected context).
- The model identifier and provider routing decision.
- Token usage and finish reason.
- Any tool/function calls and their raw responses.
- Timestamps and a session ID.
If you use a gateway that performs automatic fallback, capture the actual served model. For example, n4n.ai honors client routing directives but may shift to a healthy provider; your replay is worthless if it asserts the wrong backend. Forward provider cache-control hints too, because a cached completion will not regenerate the same way.
Snapshot middleware
Assume an OpenAI-compatible client. Wrap the chat call:
import json, time, uuid
def record_session(session_id, payload, response, model_actual):
snapshot = {
"session_id": session_id,
"ts": time.time(),
"request": payload,
"response": response,
"model_requested": payload.get("model"),
"model_served": model_actual,
"usage": response.get("usage"),
}
# push to durable store; here a local file for illustration
with open(f"sessions/{session_id}.jsonl", "a") as f:
f.write(json.dumps(snapshot) + "\n")
def chat_with_record(client, session_id, **payload):
resp = client.chat.completions.create(**payload)
record_session(session_id, payload, resp.model_dump(), resp.model)
return respThis is minimal. In production, stream to Kafka or S3, not a local file. Add a correlation ID that flows through your tool layer so you can link a tool call to its parent turn.
Step 2: Store sessions with deterministic IDs
Schema tradeoffs
Use a composite key: user_id + conversation_id + turn_number. Avoid relying on timestamps alone; concurrent turns break that. Store each turn as a JSON line to allow append-only writes and easy range scans.
A relational schema helps if you need to join with user metadata, but raw JSONL is faster to implement and sufficient for early QA. Migrate to Parquet after the first week to compress the data substantially.
{
"session_id": "conv_8f2a",
"turn": 3,
"model_served": "gpt-4o-mini",
"request": {"messages": [{"role": "user", "content": "Refund policy?"}]},
"response": {"choices": [{"message": {"content": "..."}}]},
"tool_responses": {"call_1": {"status": 200, "body": {"days": 30}}}
}Keep the original payload verbatim. Normalizing it destroys the ability to reproduce exactly what the model saw. Version your prompt templates separately and reference them by hash in the snapshot.
Step 3: Build a replay harness
Mocking external tools
Most chatbots call APIs (search, DB, CRM). Your replay must freeze those outputs. Record the tool response alongside the model call, then serve it from a stub.
class ToolStub:
def __init__(self, recorded_calls):
self.calls = recorded_calls # dict by call_id
def invoke(self, call_id, *args, **kwargs):
if call_id not in self.calls:
raise AssertionError("Unexpected tool call during replay")
return self.calls[call_id]Replaying a turn in pytest
def test_replay_turn():
snap = load_snapshot("conv_8f2a", turn=3)
stub = ToolStub(snap["tool_responses"])
bot = Bot(tool_client=stub)
out = bot.respond(snap["request"]["messages"])
assert out == snap["response"]["choices"][0]["message"]["content"]If the assertion fails, you have a regression: the same inputs now yield different output. Parametrize across hundreds of stored sessions to build a regression suite.
import pytest
@pytest.mark.parametrize("snap", load_all_snapshots())
def test_all_replays(snap):
stub = ToolStub(snap["tool_responses"])
bot = Bot(tool_client=stub)
assert bot.respond(snap["request"]["messages"]) == snap["response"]["choices"][0]["message"]["content"]Step 4: Diff and assert on behavior
Exact match on generated text is brittle due to sampling. Instead, assert on:
- Tool call sequence (must be identical).
- Final answer satisfies a validator (e.g., contains refund timeframe).
- Latency under threshold (optional).
def validate_refund_answer(text):
return "30 days" in text.lower()
def test_refund_policy_stable():
snap = load_snapshot("conv_8f2a", turn=3)
out = replay(snap, temperature=0)
assert validate_refund_answer(out)Session replay chatbot qa shines here: you catch when a prompt change silently breaks the policy answer. Use semantic diffing when exact text varies: embed both outputs and assert cosine similarity above a threshold.
Step 5: Wire into CI and production sampling
Run replay suites on every PR that touches prompts or tools. Sample 1–5% of production sessions into a replay bucket for nightly regression checks. This catches drift from provider model updates.
Do not replay every production session in CI; that scales poorly and exposes data. Use a redaction pass before storing:
import re
def redact(text):
text = re.sub(r"\b[\w.]+@[\w.]+\b", "[EMAIL]", text)
text = re.sub(r"sk-[a-zA-Z0-9]{20,}", "[KEY]", text)
return textApply redact to every message content before writing the snapshot.
Common pitfalls
PII leakage
Raw messages often contain emails, tokens, secrets. Hash or drop before writing to shared storage. A replay store is a high-value target for attackers. Treat it like a production database.
Non-determinism
Even with temperature=0, some providers return slightly different tokens across versions. Treat replay as a signal, not a hard contract. Pin model versions in your replay metadata and flag mismatches for human review.
Storage cost
A busy bot generates megabytes per hour. Use columnar compression after a week; keep hot JSONL for recent debugging. Delete sessions older than your compliance window.
Over-asserting
Asserting exact text breaks on trivial wording changes. Focus on behavioral invariants: did the bot call the right tool, return a valid JSON, refuse unsafe requests.
Tradeoffs vs live evaluation
Live eval generates synthetic prompts and scores answers. It is great for broad coverage but blind to real user phrasing and edge contexts. Session replay chatbot qa is the inverse: narrow, real, and exact. Use both. Replay anchors you to reality; eval explores the unknown.
Set up replay first. It is cheaper than building an eval pipeline and immediately pays off when your first weird production ticket lands.