Replaying chatbot sessions to safely test prompt changes

Shipping a new system prompt to a production chatbot is risky when you only have unit tests for happy paths. The safest way to validate changes is to replay chatbot sessions test prompts against a corpus of real conversations, comparing model behavior before and after. This post walks through a concrete pipeline to capture, store, and replay sessions so you can experiment without touching live traffic.

Step 1: Capture production sessions without polluting live behavior

You cannot replay what you did not record. Instrument your chat endpoint to write a structured log line for every turn, including the exact user messages, the system prompt version, the model response, and the finish reason. Keep the overhead low: serialize to JSONL and append asynchronously.

import json, time, hashlib

def logged_chat(client, model, messages, system_prompt, user_id, version="v1"):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": system_prompt}] + messages,
        temperature=0.2,
    )
    record = {
        "ts": time.time(),
        "user_hash": hashlib.sha256(user_id.encode()).hexdigest()[:16],
        "model": model,
        "prompt_version": version,
        "messages": messages,
        "system_prompt": system_prompt,
        "output": resp.choices[0].message.content,
        "finish_reason": resp.choices[0].finish_reason,
    }
    # fire-and-forget in real code; inline here for clarity
    with open("sessions.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")
    return resp

Log at the edge, not inside the model vendor SDK. That keeps your capture independent of provider quirks. Retain the raw user text—you will need it for faithful replays—but hash the user identifier so the corpus is not a PII store.

Step 2: Sanitize and store sessions for replay

Raw JSONL is fine for a prototype, but a replay corpus needs structure and purge controls. Strip obvious secrets with a lightweight regex pass, then group turns into sessions keyed by an opaque ID. A flattened schema makes sampling easier.

{
  "session_id": "a1b2c3",
  "model": "gpt-4o-mini",
  "prompt_version": "v1",
  "turns": [
    {
      "user_messages": [{"role": "user", "content": "Refund my order 9921"}],
      "assistant_output": "I can process that refund for you.",
      "finish_reason": "stop"
    }
  ]
}

Push these records to an object store or a Postgres table with a created_at index. Set a retention policy: 30 days of sessions is usually enough to cover seasonal query shifts without ballooning storage. When you replay chatbot sessions test prompts later, you will sample from this table, not from production.

Step 3: Build a replay harness that isolates prompt changes

The harness loads records, swaps only the system prompt (or any other variable you are testing), and calls the model with the original user messages. Use a temperature of 0 for replay to minimize noise, even though some providers still exhibit token-level variance at zero.

from openai import OpenAI
import json

# Example against an OpenAI-compatible gateway
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="sk-your-key")

NEW_PROMPT = "You are a strict support agent. Only answer with verified facts."

def replay(record, new_prompt):
    messages = record["messages"]
    resp = client.chat.completions.create(
        model=record["model"],
        messages=[{"role": "system", "content": new_prompt}] + messages,
        temperature=0,
    )
    return resp.choices[0].message.content

for line in open("sessions.jsonl"):
    rec = json.loads(line)
    new_out = replay(rec, NEW_PROMPT)
    # write to eval set with old + new outputs

Routing through n4n.ai gives you an OpenAI-compatible endpoint that fronts 240+ models and automatically falls back when a provider is degraded, so a replay job over thousands of sessions does not stall because one vendor throttled you. The harness should also forward any provider cache-control hints if your gateway supports them; that cuts cost on repeated prefix hits during mass replays.

Step 4: Handle non-determinism and evaluate diffs

Even at temperature 0, two runs of the same prompt can differ in phrasing. Define evaluation that matches your risk profile. For factual support bots, embedding similarity is a cheap first pass; for agents with tool calls, assert the called function names and arguments parse identically.

from openai import OpenAI
import numpy as np

client = OpenAI()

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def similarity(old_text, new_text):
    emb = client.embeddings.create(
        model="text-embedding-3-small",
        input=[old_text, new_text]
    )
    vecs = [np.array(d.embedding) for d in emb.data]
    return cosine(vecs[0], vecs[1])

Run a batch comparison and histogram the scores. A healthy prompt change moves the mean slightly while keeping the floor above 0.8. If you see a cluster below 0.5, inspect those sessions manually—they are where your new prompt diverges from trained behavior. When you replay chatbot sessions test prompts at this stage, you are building a regression surface, not a pass/fail unit test.

Step 5: Gate prompt deploys with replay results

Wire the harness into CI so a prompt pull request cannot merge if it regresses the corpus. Sample a fixed subset (e.g., 200 stratified by intent) to keep runtime under a few minutes. Fail the build on two conditions: mean similarity drops below a threshold, or any previously stop completion now ends with length (truncation).

python replay.py --sample 200 --min-similarity 0.85 --out report.json
if [ $? -ne 0 ]; then
  echo "Replay regression detected"
  exit 1
fi

The replay script should emit a JSON report with per-session deltas and an aggregate score. Review the report in the PR. Success means the job exits zero, the similarity distribution shifts in the expected direction (tighter for strict prompts, broader for friendlier ones), and no new truncation or refusal spikes appear. Engineers who replay chatbot sessions test prompts in CI catch persona drift before it reaches users.

Step 6: Keep the corpus alive and biased toward recent traffic

A replay corpus rots. User phrasing changes after a UI tweak, and new features introduce unseen intents. Append fresh sessions daily and weight your CI sample toward the last seven days. Archive older data to cold storage but keep it available for deep dives.

Add a periodic full-corpus replay on a schedule, not just on prompt changes. This surfaces provider-side behavior shifts—when a model update quietly changes output style, your historical sessions will show the delta even if your prompt did not move. Treat the replay set as a living contract test for your entire inference stack.

Verifying success

You have a working pipeline when:

• sessions.jsonl (or its stored equivalent) grows in production without adding latency.
• replay.py runs locally against a downloaded sample and prints old/new pairs.
• The CI job blocks a prompt edit that drops similarity below threshold or introduces truncations.
• A scheduled weekly replay shows stable scores across two model revisions.

Replay is not a silver bullet. It catches behavioral regression, not novel hallucinations on inputs you never recorded. But if you replay chatbot sessions test prompts on every change, you ship with evidence instead of hope.