Debugging lost context in multi-turn chatbot sessions

Lost context multi-turn chatbot sessions surface as the model forgetting user preferences stated three turns ago or abruptly changing tone. The root cause is rarely the model itself; it is usually a client that truncates the message array, a proxy that drops headers, or a provider that silently summarizes when the context window fills. Debugging requires reconstructing exactly what each request contained, not guessing from the UI.

Step 1: Capture raw traffic at the client boundary

Instrument the exact point where your code calls the LLM. Log the full request payload and the response usage block before any retry or fallback logic touches it. If you wrap the OpenAI SDK, monkey-patch the create method or use a middleware class.

import openai, json, time, os

client = openai.OpenAI(base_url="https://api.openai.com/v1", api_key=os.environ["KEY"])

def logged_create(**kwargs):
    trace = {
        "ts": time.time(),
        "kwargs": kwargs,
    }
    with open(f"trace_{int(time.time()*1000)}.json", "w") as f:
        json.dump(trace, f, indent=2, default=str)
    resp = client.chat.completions.create(**kwargs)
    with open(f"trace_{int(time.time()*1000)}.resp.json", "w") as f:
        json.dump(resp.model_dump(), f, indent=2)
    return resp

client.chat.completions.create = logged_create

What to capture

• The complete messages array, verbatim.
• Request headers, especially cache-control and any routing directives.
• Response usage.prompt_tokens and usage.completion_tokens.
• The model string actually used (some gateways swap it).

Without this, you are blind to whether the context left your process intact.

Step 2: Store sessions with immutable turn records

A flat log file per request is not enough. Load all turns for a session into a structured store so you can replay them in order. Use a schema that preserves the exact request messages and the assistant reply.

{
  "session_id": "sess_8f2c",
  "turn": 4,
  "model": "gpt-4o-mini",
  "request_messages": [
    {"role": "system", "content": "You are a terse helper."},
    {"role": "user", "content": "My name is Sam and I like Rust."},
    {"role": "assistant", "content": "Got it, Sam."},
    {"role": "user", "content": "What did I say I like?"}
  ],
  "response": {"role": "assistant", "content": "You said you like Rust."},
  "prompt_tokens": 42
}

Write these records atomically. If your service crashes mid-turn, you should still have a consistent prefix.

Step 3: Replay the session against a fixed model

Pick the exact model the production session used and resend the captured request_messages. Use a raw HTTP call so you bypass any client-side mutation that may have caused the bug.

curl https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d @turn_4_request.json

If you route through n4n.ai, the gateway honors client routing directives and forwards provider cache-control hints, so you can pin the exact model and inspect per-token usage metering to confirm prompt size matches your captured prompt_tokens. This eliminates provider fallback as a variable.

Replay from turn 1 upward. At each turn, check whether the model’s answer references facts from earlier turns. If turn 3 answers correctly but turn 4 does not, the break is between those two requests.

Step 4: Compute token counts per turn to locate the drop

Many “lost context” reports are actually silent truncation. Sum the tokens of the messages array you intended to send versus what the trace shows. Use a local tokenizer to avoid round-trips.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def msg_tokens(messages):
    # approximate: role tags add a few tokens, ignore for diffing
    return sum(len(enc.encode(m["content"])) for m in messages)

# expected full history
full = [{"role":"user","content":"My name is Sam and I like Rust."},
        {"role":"assistant","content":"Got it, Sam."},
        {"role":"user","content":"What did I say I like?"}]
print(msg_tokens(full))  # e.g., 24

# actual from trace turn 4
actual = json.load(open("trace_4.json"))["kwargs"]["messages"]
print(msg_tokens(actual))  # e.g., 12 -> last user msg only

A mismatch here proves the client dropped messages before send. If the counts match but the model still forgets, the provider is doing something server-side (summarization, context compression) or your system prompt overwrites state.

Step 5: Fix the history assembly logic

The most common defect is a sliding window that discards system messages or mis-orders turns. Build history from the end, preserving the system prompt and as many recent turns as fit your token budget.

def build_messages(turns, system_prompt, max_prompt_tokens=6000):
    out = [{"role": "system", "content": system_prompt}]
    total = len(enc.encode(system_prompt))
    for turn in reversed(turns):
        req = turn["request_messages"]
        # strip any system msg from turn to avoid duplicates
        user_asst = [m for m in req if m["role"] != "system"]
        t = msg_tokens(user_asst)
        if total + t > max_prompt_tokens:
            break
        out = user_asst + out
        total += t
    return out

This keeps the system message first, respects token limits, and never silently drops the earliest user statement unless the budget forces it. If you must drop, log which turn was evicted.

Client-side cache hints

If your provider supports prompt caching, set cache-control: ephemeral on the stable prefix (system + early turns). Forward that header exactly; do not let a retry library strip it.

Step 6: Guard against provider-side context limits

Some providers summarize when the context window nears full. That is legitimate, but your replay will show prompt_tokens plateau while completion_tokens include a summary marker. Detect this by comparing the token count of your sent messages to the billed prompt_tokens. If the billed amount is far lower than your array size, the provider compressed it.

To prevent surprise compression:

• Send an explicit max_context_tokens if the API supports it.
• Split long sessions into scoped sub-sessions with handed-off summaries you control.
• Monitor the response for meta-tokens like <summary> and fail the turn if found during debugging.

Verify success

After applying the fix, run this verification loop:

1. Replay the original failing session from Step 3 using the patched client.
2. Assert prompt_tokens for each turn equals the sum of your assembled messages (within a small constant for role overhead).
3. Assert the model’s answer at turn 4+ references the first-turn fact (e.g., “Rust”).
4. Check that no turn eviction occurred unless logged and within budget.
5. Run a synthetic 20-turn session with a unique token in turn 1; confirm it appears in turn 20’s response.

If all five hold, the lost context multi-turn chatbot defect is closed. If turn 20 still drops the token but token counts are correct, move the investigation to the provider’s context handling or your system prompt overwriting the user state.

Debugging lost context multi-turn chatbot sessions is mostly forensic work: capture, replay, measure, fix. The code above is the minimum tooling you need to do it on any OpenAI-compatible endpoint without vendor lock-in.