Semantic Kernel memory tutorial: recalling conversations

Semantic Kernel saving recalling conversations is a practical requirement for any production LLM application. This tutorial walks through building a persistent conversation memory system using Semantic Kernel’s memory abstractions and a local vector store. You’ll end up with runnable code that stores exchanges, retrieves relevant context, and integrates with chat completion.

Prerequisites

• .NET 8 SDK or later
• An OpenAI-compatible API key (OpenAI, Azure OpenAI, or a gateway like n4n.ai)
• Basic familiarity with Semantic Kernel concepts: kernels, plugins, and prompt functions

Install the required packages:

dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Connectors.OpenAI
dotnet add package Microsoft.SemanticKernel.Connectors.Memory.Qdrant
dotnet add package Microsoft.SemanticKernel.Memory

We’ll use Qdrant as the vector store because it runs locally via Docker and requires no cloud account. If you prefer another store (Redis, Pinecone, Weaviate), swap the connector — the memory API stays the same.

Start Qdrant:

docker run -d -p 6333:6333 qdrant/qdrant

Project structure

Create a console app and organize it like this:

ConversationMemory/
├── Program.cs
├── MemoryService.cs
├── ConversationStore.cs
└── appsettings.json

appsettings.json holds configuration:

{
  "OpenAI": {
    "ApiKey": "YOUR_KEY_HERE",
    "ModelId": "gpt-4o-mini",
    "EmbeddingModelId": "text-embedding-3-small"
  },
  "Qdrant": {
    "Endpoint": "http://localhost:6333",
    "CollectionName": "conversation-memory"
  }
}

Building the memory service

Semantic Kernel’s ISemanticTextMemory interface abstracts the vector store. We’ll wrap it in a service that handles conversation-specific logic: storing turns, retrieving relevant history, and managing collection lifecycle.

MemoryService.cs:

using Microsoft.Extensions.Configuration;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.Memory.Qdrant;
using Microsoft.SemanticKernel.Memory;

namespace ConversationMemory;

public sealed class MemoryService : IDisposable
{
    private readonly ISemanticTextMemory _memory;
    private readonly string _collectionName;
    private bool _disposed;

    public MemoryService(IConfiguration config)
    {
        _collectionName = config["Qdrant:CollectionName"]!;
        
        var qdrantMemoryStore = new QdrantMemoryStore(config["Qdrant:Endpoint"]!);
        _memory = new SemanticTextMemory(qdrantMemoryStore);
    }

    public async Task InitializeAsync(CancellationToken ct = default)
    {
        await _memory.CreateCollectionAsync(_collectionName, ct);
    }

    public async Task SaveConversationTurnAsync(
        string conversationId,
        string userMessage,
        string assistantMessage,
        CancellationToken ct = default)
    {
        var turnId = Guid.NewGuid().ToString();
        var timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();

        // Store user turn
        await _memory.SaveInformationAsync(_collectionName, new SaveInformationRequest
        {
            Id = $"{conversationId}-{turnId}-user",
            Text = userMessage,
            Description = "User message",
            AdditionalMetadata = new Dictionary<string, string>
            {
                ["conversationId"] = conversationId,
                ["role"] = "user",
                ["timestamp"] = timestamp.ToString(),
                ["turnId"] = turnId
            }
        }, ct);

        // Store assistant turn
        await _memory.SaveInformationAsync(_collectionName, new SaveInformationRequest
        {
            Id = $"{conversationId}-{turnId}-assistant",
            Text = assistantMessage,
            Description = "Assistant message",
            AdditionalMetadata = new Dictionary<string, string>
            {
                ["conversationId"] = conversationId,
                ["role"] = "assistant",
                ["timestamp"] = timestamp.ToString(),
                ["turnId"] = turnId
            }
        }, ct);
    }

    public async Task<IReadOnlyList<MemoryQueryResult>> RecallAsync(
        string conversationId,
        string query,
        int limit = 5,
        double minRelevance = 0.7,
        CancellationToken ct = default)
    {
        var filter = MemoryFilter.ByTag("conversationId", conversationId);
        
        var results = new List<MemoryQueryResult>();
        await foreach (var result in _memory.SearchAsync(_collectionName, query, limit, minRelevance, filter, cancellationToken: ct))
        {
            results.Add(result);
        }
        return results.OrderBy(r => r.Metadata?.TryGetValue("timestamp", out var ts) && long.TryParse(ts, out var t) ? t : 0).ToList();
    }

    public async Task<IReadOnlyList<MemoryQueryResult>> GetRecentTurnsAsync(
        string conversationId,
        int limit = 10,
        CancellationToken ct = default)
    {
        // Empty query with filter returns most recent by timestamp metadata
        var filter = MemoryFilter.ByTag("conversationId", conversationId);
        var results = new List<MemoryQueryResult>();
        
        await foreach (var result in _memory.SearchAsync(_collectionName, "", limit, 0.0, filter, cancellationToken: ct))
        {
            results.Add(result);
        }
        
        return results
            .OrderByDescending(r => r.Metadata?.TryGetValue("timestamp", out var ts) && long.TryParse(ts, out var t) ? t : 0)
            .Take(limit)
            .OrderBy(r => r.Metadata?.TryGetValue("timestamp", out var ts2) && long.TryParse(ts2, out var t2) ? t2 : 0)
            .ToList();
    }

    public void Dispose()
    {
        if (!_disposed)
        {
            _disposed = true;
        }
    }
}

Key design decisions:

• Each conversation turn gets two records (user + assistant) linked by turnId and conversationId
• Metadata filters let us scope recall to a single conversation
• GetRecentTurnsAsync uses an empty query with a filter to fetch the latest N turns chronologically
• Relevance threshold (0.7) filters noise; tune based on your embedding model

Conversation store: the orchestration layer

The memory service handles vectors. The conversation store manages the chat loop: building context, calling the model, persisting results.

ConversationStore.cs:

using Microsoft.Extensions.Configuration;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.ChatCompletion;
using Microsoft.SemanticKernel.Connectors.OpenAI;

namespace ConversationMemory;

public sealed class ConversationStore
{
    private readonly Kernel _kernel;
    private readonly IChatCompletionService _chat;
    private readonly MemoryService _memory;
    private readonly string _conversationId;

    public ConversationStore(IConfiguration config, MemoryService memory, string conversationId)
    {
        _memory = memory;
        _conversationId = conversationId;

        var builder = Kernel.CreateBuilder();
        builder.AddOpenAIChatCompletion(
            config["OpenAI:ModelId"]!,
            config["OpenAI:ApiKey"]!);
        builder.AddOpenAITextEmbeddingGeneration(
            config["OpenAI:EmbeddingModelId"]!,
            config["OpenAI:ApiKey"]!);

        _kernel = builder.Build();
        _chat = _kernel.GetRequiredService<IChatCompletionService>();
    }

    public async Task<string> SendAsync(string userMessage, CancellationToken ct = default)
    {
        // 1. Retrieve relevant history for context
        var relevantHistory = await _memory.RecallAsync(_conversationId, userMessage, limit: 3, ct: ct);
        
        // 2. Also get recent turns for continuity
        var recentTurns = await _memory.GetRecentTurnsAsync(_conversationId, limit: 6, ct: ct);

        // 3. Build chat history
        var history = new ChatHistory();
        history.AddSystemMessage("You are a helpful assistant with access to conversation history. Reference prior context naturally when relevant.");

        // Add relevant historical context (deduplicated by turnId)
        var seenTurns = new HashSet<string>();
        foreach (var item in relevantHistory.Concat(recentTurns))
        {
            if (item.Metadata?.TryGetValue("turnId", out var turnId) == true && seenTurns.Add(turnId))
            {
                if (item.Metadata?.TryGetValue("role", out var role) == true)
                {
                    if (role == "user")
                        history.AddUserMessage(item.Metadata.TryGetValue("text", out var t) ? t : item.Text);
                    else
                        history.AddAssistantMessage(item.Metadata.TryGetValue("text", out var t2) ? t2 : item.Text);
                }
            }
        }

        // 4. Add current user message
        history.AddUserMessage(userMessage);

        // 5. Get completion
        var response = await _chat.GetChatMessageContentAsync(history, cancellationToken: ct);

        // 6. Persist the exchange
        await _memory.SaveConversationTurnAsync(_conversationId, userMessage, response.ToString(), ct);

        return response.ToString();
    }
}

Note the deduplication logic: relevant history and recent turns can overlap. We use turnId to avoid duplicating turns in the context window.

Wiring it together

Program.cs demonstrates the full flow:

using Microsoft.Extensions.Configuration;
using ConversationMemory;

var config = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json", optional: false)
    .AddUserSecrets<Program>() // Store API key here in dev
    .Build();

var conversationId = "demo-conversation-" + Guid.NewGuid().ToString()[..8];

await using var memory = new MemoryService(config);
await memory.InitializeAsync();

var store = new ConversationStore(config, memory, conversationId);

Console.WriteLine($"Conversation ID: {conversationId}");
Console.WriteLine("Type 'exit' to quit, 'recall <query>' to test retrieval.\n");

while (true)
{
    Console.Write("You: ");
    var input = Console.ReadLine();
    
    if (string.IsNullOrWhiteSpace(input)) continue;
    if (input.Trim().Equals("exit", StringComparison.OrdinalIgnoreCase)) break;

    if (input.StartsWith("recall ", StringComparison.OrdinalIgnoreCase))
    {
        var query = input["recall ".Length..].Trim();
        var results = await memory.RecallAsync(conversationId, query, limit: 5);
        
        Console.WriteLine($"\n--- Recall results for '{query}' ---");
        foreach (var r in results)
        {
            var role = r.Metadata?.TryGetValue("role", out var ro) == true ? ro : "?";
            var text = r.Text.Length > 100 ? r.Text[..100] + "..." : r.Text;
            Console.WriteLine($"  [{role}] (score: {r.Relevance:0.00}) {text}");
        }
        Console.WriteLine();
        continue;
    }

    try
    {
        var response = await store.SendAsync(input);
        Console.WriteLine($"Assistant: {response}\n");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error: {ex.Message}\n");
    }
}

Running the tutorial

dotnet run

Expected interaction:

Conversation ID: demo-conversation-a1b2c3d4
Type 'exit' to quit, 'recall <query>' to test retrieval.

You: Hi, I'm planning a trip to Japan next month.
Assistant: That sounds exciting! Japan is incredible in spring. Are you thinking cherry blossom season in late March/early April, or a different time? I can help with itinerary ideas, transport, or cultural tips.

You: Yes, cherry blossoms. I want to see Tokyo and Kyoto.
Assistant: Perfect timing. For cherry blossoms, late March to early April hits Tokyo and Kyoto beautifully. A classic 7-10 day split: 3-4 nights Tokyo, 3-4 nights Kyoto, maybe a day trip to Nara or Hakone. Want me to sketch a day-by-day outline?

You: recall cherry blossom timing
--- Recall results for 'cherry blossom timing' ---
  [assistant] (score: 0.87) Perfect timing. For cherry blossoms, late March to early April hits Tokyo and Kyoto beautifully...
  [user] (score: 0.82) Yes, cherry blossoms. I want to see Tokyo and Kyoto.

You: What about food recommendations?
Assistant: Based on your Tokyo/Kyoto plan: In Tokyo, try sushi at Tsukiji outer market, ramen in Shinjuku, tempura in Asakusa. Kyoto: kaiseki in Gion, yudofu near Nanzen-ji, matcha sweets in Uji. Since you're there during sakura season, look for sakura-flavored seasonal treats everywhere.

You: recall food
--- Recall results for 'food' ---
  [assistant] (score: 0.91) Based on your Tokyo/Kyoto plan: In Tokyo, try sushi at Tsukiji outer market...
  [user] (score: 0.78) What about food recommendations?

Production considerations

Context window management

The example naively concatenates retrieved turns. In production, implement a token budget:

private ChatHistory BuildHistoryWithBudget(
    ChatHistory baseHistory,
    IEnumerable<MemoryQueryResult> turns,
    int maxTokens = 3000)
{
    var history = new ChatHistory(baseHistory);
    var encoder = TikToken.EncodingForModel("gpt-4o-mini");
    var currentTokens = encoder.Encode(string.Join("", history.Select(m => m.Content))).Count;

    foreach (var turn in turns)
    {
        var role = turn.Metadata?.GetValueOrDefault("role") ?? "user";
        var content = turn.Text;
        var turnTokens = encoder.Encode(content).Count + 4; // overhead
        
        if (currentTokens + turnTokens > maxTokens) break;
        
        if (role == "user") history.AddUserMessage(content);
        else history.AddAssistantMessage(content);
        
        currentTokens += turnTokens;
    }
    return history;
}

Hybrid retrieval

Vector search alone misses exact matches (names, dates, codes). Combine with keyword search:

public async Task<IReadOnlyList<MemoryQueryResult>> HybridRecallAsync(
    string conversationId,
    string query,
    int vectorLimit = 5,
    int keywordLimit = 3,
    CancellationToken ct = default)
{
    var vectorResults = await RecallAsync(conversationId, query, vectorLimit, ct: ct);
    
    // Keyword filter via metadata (requires store support)
    var keywordFilter = MemoryFilter.ByTag("conversationId", conversationId);
    // Add text search if your vector store supports it (Qdrant does via payload indexes)
    
    return vectorResults; // Simplified — merge and rerank in practice
}

Multi-conversation isolation

The conversationId filter provides logical isolation. For stricter tenancy, use separate collections per user or workspace:

public async Task InitializeUserCollectionAsync(string userId, CancellationToken ct = default)
{
    var collectionName = $"user-{userId}-conversations";
    await _memory.CreateCollectionAsync(collectionName, ct);
}

Embedding model consistency

The embedding model used at write time must match read time. If you migrate models, re-embed the collection or maintain versioned collections.

Testing the memory layer

Unit test the memory service with a fake store:

[Fact]
public async Task SaveAndRecall_ReturnsMatchingTurns()
{
    // Arrange
    var fakeStore = new VolatileMemoryStore();
    var memory = new SemanticTextMemory(fakeStore);
    var collection = "test-collection";
    await memory.CreateCollectionAsync(collection);
    
    var service = new TestableMemoryService(memory, collection);
    var conversationId = "test-conv";
    
    // Act
    await service.SaveConversationTurnAsync(conversationId, "Hello", "Hi there!");
    await service.SaveConversationTurnAsync(conversationId, "How are you?", "I'm well!");
    
    var results = await service.RecallAsync(conversationId, "greeting", limit: 2);
    
    // Assert
    Assert.Equal(2, results.Count);
    Assert.Contains(results, r => r.Text.Contains("Hello"));
    Assert.Contains(results, r => r.Text.Contains("Hi there"));
}

VolatileMemoryStore is in Microsoft.SemanticKernel.Memory and works for fast in-memory tests.

When to use this pattern

Semantic Kernel saving recalling conversations works well when:

• Conversations span multiple sessions or days
• Users reference prior context (“like we discussed last week”)
• You need semantic search over history, not just recent turns
• The conversation corpus fits in a vector store (millions of turns)

Avoid when:

• Conversations are short-lived (single session) — simple ChatHistory suffices
• You need exact replay — store raw transcripts in a database alongside vectors
• Latency is critical — vector search adds 50-200ms per turn

Next steps

• Add a background job to summarize old turns and store summaries as single records (reduces retrieval noise)
• Implement conversation branching: fork a conversation at a turn, share history up to that point
• Add PII detection before embedding (Microsoft.Presidio or custom regex)
• Wire telemetry: track recall latency, relevance scores, token savings from retrieved context

The complete source is available in the n4n.ai examples repository under semantic-kernel/conversation-memory.