Building long-term memory for Semantic Kernel agents

In this long-term memory semantic kernel agents tutorial we build persistent memory for a Semantic Kernel agent that survives process restarts and scales beyond a single chat. Semantic Kernel gives you the plumbing; you supply the vector store and embedding model. We’ll use a local volatile store for dev and swap in Qdrant for production, with an OpenAI-compatible gateway for inference.

Step 1: Scaffold the project and install dependencies

This long-term memory semantic kernel agents tutorial assumes Python 3.10+ and a virtual environment. Install the core package plus a production-grade vector client:

pip install semantic-kernel qdrant-client

Semantic Kernel’s memory abstractions live in semantic_kernel.memory. The VolatileMemoryStore is in-process and loses data on exit—fine for unit tests, wrong for an agent that should accumulate knowledge. Qdrant is a reasonable default for self-hosted vector storage; the interface is identical, so the swap is a one-line change.

Step 2: Configure the kernel with an OpenAI-compatible endpoint

Semantic Kernel’s OpenAI connectors accept a custom endpoint. Point them at n4n.ai’s OpenAI-compatible endpoint to get automatic fallback across providers and per-token metering without changing your code when a upstream model is rate-limited.

from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, OpenAITextEmbedding

kernel = Kernel()

chat_service = OpenAIChatCompletion(
    ai_model_id="gpt-4o",
    api_key="YOUR_N4N_KEY",
    endpoint="https://api.n4n.ai/v1",
)
embedding_service = OpenAITextEmbedding(
    ai_model_id="text-embedding-3-small",
    api_key="YOUR_N4N_KEY",
    endpoint="https://api.n4n.ai/v1",
)

kernel.add_service(chat_service)
kernel.add_service(embedding_service)

Keep the ai_model_id values aligned with what your gateway exposes. If you later route to a different model family, the embedding dimension must match the vector store’s vector_size or ingestion will throw.

Step 3: Initialize a vector store and bind memory

Create the store, wrap it in SemanticTextMemory, and attach it to the kernel. For local iteration:

from semantic_kernel.memory import VolatileMemoryStore, SemanticTextMemory

store = VolatileMemoryStore()
memory = SemanticTextMemory(storage=store, embeddings=embedding_service)
kernel.memory = memory

For Qdrant (run docker run -p 6333:6333 qdrant/qdrant):

from semantic_kernel.connectors.memory.qdrant import QdrantMemoryStore

store = QdrantMemoryStore(host="localhost", port=6333, vector_size=1536)
memory = SemanticTextMemory(storage=store, embeddings=embedding_service)
kernel.memory = memory

vector_size=1536 matches text-embedding-3-small. If you switch to text-embedding-3-large, use 3072. Mismatches fail silently as zero recalls—verify dimensions before deploying.

Step 4: Write memory helpers with explicit collections

Semantic Kernel organizes memories into named collections. Treat a collection like a table: one per agent persona or knowledge domain.

COLLECTION = "agent_facts"

async def remember(text: str, id: str) -> None:
    # id must be unique; collisions overwrite
    await kernel.memory.save_information(COLLECTION, id=id, text=text)

async def recall(query: str, limit: int = 3) -> list[str]:
    results = await kernel.memory.search(COLLECTION, query, limit=limit)
    return [r.text for r in results]

save_information computes the embedding via the bound service and writes the vector + payload. search returns MemoryQueryResult objects with text, relevance, and metadata. In production, filter by metadata rather than stuffing everything into the prompt.

Step 5: Build the retrieval-augmented agent loop

The retrieval-augmented loop is the core of any long-term memory semantic kernel agents tutorial. The agent pulls relevant facts before responding and writes new facts after. Below is a minimal but complete loop using ChatHistory directly:

from semantic_kernel.contents import ChatHistory

async def agent_respond(user_input: str) -> str:
    relevant = await recall(user_input, limit=3)
    sys_msg = (
        "You are a concise assistant. Use known facts if relevant:\n"
        + "\n".join(f"- {fact}" for fact in relevant)
    )
    history = ChatHistory()
    history.add_system_message(sys_msg)
    history.add_user_message(user_input)

    response = await chat_service.get_chat_message_content(history)
    await remember(user_input, id=str(abs(hash(user_input))))
    return str(response)

This is deliberately naive about what to persist. In a real system, ask the model to extract durable facts (e.g., “summarize any new user preferences”) and store those, not raw transcripts. Storing raw turns bloats the index with low-signal text.

Step 6: Persist across sessions and processes

With VolatileMemoryStore, restarting the Python process loses all vectors. Switch to Qdrant (Step 3) for durability. If you must stay in-process without Qdrant, serialize the store manually:

import json, asyncio
from semantic_kernel.memory import MemoryRecord

async def dump_store(path: str):
    records = await store.get_all(COLLECTION, with_embeddings=True)
    payload = [r.to_dict() for r in records]
    with open(path, "w") as f:
        json.dump(payload, f)

async def load_store(path: str):
    with open(path) as f:
        data = json.load(f)
    for item in data:
        rec = MemoryRecord.from_dict(item)
        await store.upsert(COLLECTION, rec)

Call load_store at boot, dump_store on shutdown. This is a stopgap; Qdrant or Redis handles concurrency and TTLs properly.

Step 7: Verify the end-to-end flow

Write a small driver that simulates two separate sessions:

async def main():
    # Session 1
    await remember("User prefers TypeScript over Python for frontend work.", id="fact-1")
    r1 = await agent_respond("What language should I use for my React app?")
    print("Session 1:", r1)

    # Simulate restart: new kernel, same Qdrant store
    # (skip re-creating store if using Qdrant; volatile would lose fact-1)
    r2 = await agent_respond("Remind me what I like for frontend?")
    print("Session 2:", r2)

if __name__ == "__main__":
    asyncio.run(main())

Success criteria:

• Session 1 response references TypeScript or acknowledges the stored preference.
• Session 2 (with a fresh in-memory kernel but persistent store) recalls fact-1 without it being passed explicitly.
• recall("frontend language preference") returns the fact with relevance > 0.7.

If using volatile store, Session 2 will fail this check—that confirms the persistence boundary is real.

Production notes

• Batch writes. save_information is one vector per call. Collect facts and use store.upsert with a list of MemoryRecord objects to cut embedding round-trips.
• Namespace by tenant. Use a collection per customer, not a metadata field, to avoid cross-tenant recall and to simplify deletion.
• Cache embeddings. Embedding calls are pure functions of text; memoize them in-process to reduce cost when the same fact is written twice.
• Model routing. The patterns from this long-term memory semantic kernel agents tutorial apply unchanged if you swap the gateway: the endpoint parameter is the only coupling.

Long-term memory is not a feature you bolt on; it is the agent’s stateful core. Get the store and embedding dimensions right early, and the rest is prompt design.