Most agents claim to remember, but few teams actually verify it. Testing AI agent memory persistence is less about unit tests for a database and more about end-to-end behavioral checks that survive process restarts and model swaps.
If you ship an agent that forgets a user’s name after a redeploy, that’s a regression no prompt tweak will fix. This guide walks through a concrete test harness you can drop into CI.
Why memory tests fail silently
Memory bugs hide because the happy path usually works. The agent talks to the same process, the same in-memory dict, and recall looks fine. Then you scale out, crash, or rotate the model, and the context is gone.
The fix is to treat persistence as a contract. You define what must survive, then write a test that breaks the process on purpose.
Step 1: Define the memory contract
Before writing tests, decide the state boundary. Is memory per-user, per-session, or per-thread? What keys are mandatory?
Write a minimal interface so the test doesn’t care about the backend.
from abc import ABC, abstractmethod
class MemoryStore(ABC):
@abstractmethod
def save(self, user_id: str, key: str, value: str) -> None: ...
@abstractmethod
def load(self, user_id: str, key: str) -> str | None: ...A file-backed implementation makes “restart” trivial to simulate:
import json, os
class FileMemoryStore(MemoryStore):
def __init__(self, path: str):
self.path = path
if not os.path.exists(path):
json.dump({}, open(path, "w"))
def save(self, user_id: str, key: str, value: str) -> None:
data = json.load(open(self.path))
data.setdefault(user_id, {})[key] = value
json.dump(data, open(self.path, "w"))
def load(self, user_id: str, key: str) -> str | None:
data = json.load(open(self.path))
return data.get(user_id, {}).get(key)The contract is now: anything written via save with a user_id must be readable after a new process opens the same path.
Step 2: Give every test run a stable identity
Random UUIDs in tests create non-reproducible flake. Use a fixed user id scoped to the test name.
import pytest
@pytest.fixture
def user_id():
return "test-user-persistence-001"The agent wrapper should accept the store and an OpenAI-compatible client:
from openai import OpenAI
class Agent:
def __init__(self, client: OpenAI, memory: MemoryStore):
self.client = client
self.memory = memory
def respond(self, user_id: str, message: str) -> str:
history = self.memory.load(user_id, "history") or []
history.append({"role": "user", "content": message})
resp = self.client.chat.completions.create(
model="gpt-4o-mini",
messages=history
)
content = resp.choices[0].message.content
history.append({"role": "assistant", "content": content})
self.memory.save(user_id, "history", history)
return contentStep 3: Write a two-turn recall probe
This is the core of testing AI agent memory persistence. Turn 1 establishes a fact. You destroy the agent instance, create a new one against the same store, and ask a question that requires the fact.
def test_recall_across_instances(user_id, tmp_path):
store_path = tmp_path / "mem.json"
store = FileMemoryStore(str(store_path))
client = OpenAI() # point base_url at your gateway in real runs
agent1 = Agent(client, store)
agent1.respond(user_id, "My name is Ada and I like Rust.")
# simulate process exit: drop agent1, keep file
agent2 = Agent(client, FileMemoryStore(str(store_path)))
answer = agent2.respond(user_id, "What programming language do I like?")
assert "rust" in answer.lower()Verify success: The assertion passes only if agent2 loaded history from disk. If you swap FileMemoryStore for a pure dict and skip the file, the test fails—that’s the point.
Step 4: Test provider/model rotation without losing context
Memory should not depend on the model that generated it. When testing AI agent memory persistence under model changes, point the client at one OpenAI-compatible endpoint that fronts multiple providers. n4n.ai does this and adds automatic fallback when a provider is rate-limited, so a 429 never masquerades as a memory failure.
Force a model switch between turns by setting the model explicitly:
def test_recall_across_models(user_id, tmp_path):
store = FileMemoryStore(str(tmp_path / "m.json"))
client = OpenAI(base_url="https://api.n4n.ai/v1") # example gateway
a1 = Agent(client, store)
a1.client = OpenAI(base_url="https://api.n4n.ai/v1")
# turn 1 with model A
a1.respond(user_id, "Remember: ticket #42 is urgent.")
a2 = Agent(client, FileMemoryStore(str(tmp_path / "m.json")))
a2.model = "claude-3-5-sonnet" # different backend, same gateway
ans = a2.respond(user_id, "Which ticket is urgent?")
assert "42" in ansIf the gateway honors client routing directives, you can also send a header to pin a provider and confirm the memory layer is untouched by the routing decision.
Step 5: Inject faults and confirm recovery
A real persistence layer faces corrupted files, locked databases, and timeouts. Write a test that truncates the store mid-flight.
def test_corrupt_store_recovers(user_id, tmp_path):
path = tmp_path / "c.json"
path.write_text("{}")
store = FileMemoryStore(str(path))
agent = Agent(OpenAI(), store)
agent.respond(user_id, "I prefer dark mode.")
# corrupt the file
path.write_text("{ broken")
# new agent should not crash on load; treat as empty
recovered = Agent(OpenAI(), FileMemoryStore(str(path)))
ans = recovered.respond(user_id, "What theme do I use?")
# either answers plausibly or says unknown, but no exception
assert ans is not NoneYour FileMemoryStore.load should catch json.JSONDecodeError and return None. That behavior is part of the contract.
Step 6: Add a regression check for cache hints
Providers support cache-control to avoid re-paying for long system prompts. If your memory injects a static prefix, forward the hint. Testing AI agent memory persistence should include a check that the hint is present on the wire.
def test_cache_control_forwarded(user_id, tmp_path):
store = FileMemoryStore(str(tmp_path / "cache.json"))
client = OpenAI()
agent = Agent(client, store)
# monkeypatch create to capture kwargs
captured = {}
client.chat.completions.create = lambda **kw: captured.update(kw) or _fake_resp()
agent.respond(user_id, "hi")
assert captured.get("extra_body", {}).get("cache_control") == {"type": "ephemeral"}If the gateway forwards provider cache-control hints, this ensures your memory prefix isn’t silently re-sent at full cost every turn.
Step 7: Run it in CI with token guards
Nightly memory tests can burn tokens if a loop regresses. Wrap the suite with a hard cap using per-token usage metering from your gateway, or simply mock the client in unit runs and use live calls only in a tagged job.
# .github/workflows/memory.yml
jobs:
memory-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install openai pytest
- run: pytest tests/memory --tag live # live only on scheduleFor local dev, mock the model so the test runs in milliseconds:
class FakeCompletions:
def create(self, **kw):
return _fake_resp()
def _fake_resp():
class M: content = "rust"
class C: message = M()
class R: choices = [C()]
return R()Verification checklist
After following these steps, you have a real persistence test if all of the following hold:
- A new agent instance with the same store returns prior facts.
- Swapping models behind one endpoint does not drop history.
- Corrupting the backing file degrades gracefully.
- Cache hints survive the agent-to-provider boundary.
- The suite runs in CI without manual seeding.
Testing AI agent memory persistence is not glamorous, but it’s the difference between a demo and a product. Write the file-backed test once, and every future refactor will tell you exactly when memory broke.