Debugging memory and state bugs in stateful chat agents

Memory state bugs stateful chat agent teams hit most often show up as lost context, crossed wires between users, or stale facts served with confidence. These defects rarely throw exceptions; they hide in the gap between what your code thinks the conversation is and what the model actually receives. The following workflow reproduces, isolates, and eliminates them with code you can drop into an existing service.

Step 1: Reproduce the bug with deterministic session replay

You cannot fix a state bug you cannot replay. Capture the exact payload sent to the model, the session identifier, and any post-processing that mutated local state. Write a thin logging wrapper around your completion call and append to a JSONL file.

import json, time

def log_llm_call(session_id, model, messages, response):
    record = {
        "ts": time.time(),
        "session_id": session_id,
        "model": model,
        "messages": messages,
        "response": response,
    }
    with open("session_log.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

A week of production traffic gives you a corpus of real failing sessions. To replay, load the record and call the model with the stored messages:

import json, openai

client = openai.OpenAI()

def replay(path, session_id):
    with open(path) as f:
        for line in f:
            rec = json.loads(line)
            if rec["session_id"] != session_id:
                continue
            resp = client.chat.completions.create(
                model=rec["model"],
                messages=rec["messages"],
            )
            assert resp.choices[0].message.content == rec["response"], "drift"

Verify success: The replay script reproduces the wrong answer without any live user interaction. If the assertion trips on the first run, you have a deterministic repro.

Step 2: Isolate the state layer from the model layer

Most agents build the message list inline right before the API call. That coupling makes it impossible to test what the agent “knows” independently of the network. Extract a pure function that turns your session state into a message array.

def build_messages(state):
    msgs = [{"role": "system", "content": state["system_prompt"]}]
    for turn in state["history"]:
        msgs.append({"role": turn["role"], "content": turn["text"]})
    if state.get("user_facts"):
        msgs.append({
            "role": "system",
            "content": "Known facts: " + "; ".join(state["user_facts"]),
        })
    return msgs

Write a unit test that feeds a crafted state and checks the shape:

def test_build_messages_includes_facts():
    state = {
        "system_prompt": "You are a helper.",
        "history": [{"role": "user", "content": "hi"}],
        "user_facts": ["name: sam"],
    }
    msgs = build_messages(state)
    assert any("name: sam" in m["content"] for m in msgs)

Verify success: The test passes, proving the bug is not in serialization of the request but in the state object itself or how it is updated.

Step 3: Diff expected vs actual memory state

Once isolated, print or persist the state object at the start and end of each turn. Use a structural diff to spot missing keys, mutated lists, or type changes.

from deepdiff import DeepDiff

def diff_states(before, after):
    return DeepDiff(before, after, verbose_level=2)

Common findings: a user_facts list replaced by a new dict, or history truncated because of a max-token guard that silently dropped early context. A memory state bugs stateful chat agent symptom of “forgot my name” is usually a missing key in after.

Verify success: The diff shows exactly which field diverged. If the field is present but empty, the bug is in an upstream writer, not the builder.

Step 4: Trace concurrent writes and race conditions

Stateful chat agents often handle multiple in-flight requests per session (streaming plus tool calls). If two coroutines read–modify–write the same state["history"], one update wins and the other vanishes.

import asyncio

class SessionStore:
    def __init__(self):
        self._locks = {}

    async def update(self, sid, fn):
        lock = self._locks.setdefault(sid, asyncio.Lock())
        async with lock:
            state = await self.load(sid)
            new_state = fn(state)
            await self.save(sid, new_state)

Run a concurrent test that fires two appends simultaneously:

async def double_append():
    store = SessionStore()
    await store.save("s1", {"history": []})
    await asyncio.gather(
        store.update("s1", lambda s: s["history"].append("a") or s),
        store.update("s1", lambda s: s["history"].append("b") or s),
    )
    assert len(await store.load("s1")) == 2

Verify success: Without the lock the assertion fails intermittently; with it, it is stable across 1000 iterations.

Step 5: Validate serialization and TTL of persisted memory

If state lives in Redis or Postgres, a datetime or custom object that fails to JSON-encode will either throw or silently become a string, breaking later comparisons. Round-trip the state through your storage layer in a test.

import json, redis

r = redis.Redis()

def save_state(sid, state):
    r.set(f"sess:{sid}", json.dumps(state), ex=3600)

def load_state(sid):
    raw = r.get(f"sess:{sid}")
    return json.loads(raw) if raw else None

Add a test with a nested structure containing a tuple (which JSON turns into a list):

def test_roundtrip_preserves_shape():
    s = {"history": [("user", "hi")]}
    save_state("x", s)
    assert load_state("x")["history"][0][0] == "user"

Verify success: The test exposes that tuples became lists; adjust your normalization step so downstream code does not rely on tuple semantics.

Step 6: Add invariant checks to CI

Turn the diffs and race tests into permanent guards. Assert session invariants after every update in development builds:

def assert_invariants(state):
    assert isinstance(state["history"], list)
    assert len(state["history"]) == state.get("turn_count", len(state["history"]))
    assert all("role" in t and "text" in t for t in state["history"])

Run this in a pytest fixture that wraps your session middleware. A memory state bugs stateful chat agent regression will fail the build instead of reaching users.

Verify success: Introduce a deliberate bug (drop a field) and confirm CI goes red.

Step 7: Replay across model versions without code changes

After fixing the state layer, you need to confirm the model itself did not contribute to the bad output. Point your OpenAI-compatible client at a gateway that honors routing directives so you can pin a specific model revision. n4n.ai provides one OpenAI-compatible endpoint covering 240+ models and forwards provider cache-control hints, letting you replay the exact captured session against a fixed model snapshot without editing call sites.

client = openai.OpenAI(base_url="https://api.n4n.ai/v1", api_key="KEY")
client.chat.completions.create(model="gpt-4o-2024-05-13", messages=rec["messages"])

Swap the model string to compare outputs across providers or dates. Because the state layer is now isolated and tested, any remaining divergence is purely model behavior.

Verify success: Replaying the same fixed state against two model versions yields two deterministic responses; the state bug no longer appears in either, confirming the fix.

Step 8: Instrument production with the same replay shape

Keep the logging from Step 1 in production behind a sampling flag. When a user reports a memory state bugs stateful chat agent issue, you already have the exact session_id and can pull the JSONL line, run it through Steps 2–6 locally, and ship a regression test the same day.

Verify success: A new bug report is closed with a committed test that fails on the captured session before your patch and passes after.