Semantic Kernel vector store tutorial: Chroma integration

Semantic Kernel’s vector store abstraction lets you swap backends without rewriting retrieval logic. This tutorial walks through wiring Chroma as the vector store, ingesting documents, and running similarity searches — all with runnable code you can drop into a project today.

Prerequisites

You need Python 3.10+ and a Chroma instance running locally or remotely. Install the Semantic Kernel packages with the Chroma connector:

pip install semantic-kernel[chroma] chromadb

If you prefer a managed Chroma instance, you can point the client at a hosted endpoint — the code stays the same. For local development, start Chroma in a container:

docker run -d -p 8000:8000 chromadb/chroma:latest

Verify it’s up with curl http://localhost:8000/api/v1/heartbeat — you should see a nanosecond timestamp.

Initialize the Chroma vector store

Semantic Kernel uses a VectorStoreRecordDefinition to describe the schema. Define a record that holds an id, the source text, and the embedding vector:

import asyncio
from semantic_kernel.data import VectorStoreRecordDefinition, VectorStoreRecordKeyField, VectorStoreRecordDataField, VectorStoreRecordVectorField
from semantic_kernel.connectors.chroma import ChromaVectorStore

record_definition = VectorStoreRecordDefinition(
    fields={
        "id": VectorStoreRecordKeyField(),
        "content": VectorStoreRecordDataField(),
        "embedding": VectorStoreRecordVectorField(dimensions=1536),
    },
    container_mode=True,
)

vector_store = ChromaVectorStore(
    collection_name="sk_docs",
    record_definition=record_definition,
    connection_string="http://localhost:8000",
)

The dimensions value must match your embedding model. The example above assumes text-embedding-3-small (1536 dimensions). Adjust if you use a different model.

Create an embedding generator

Semantic Kernel separates embedding generation from storage. Wire an OpenAI-compatible embedding client — this works with any OpenAI-compatible endpoint:

from semantic_kernel.connectors.openai import OpenAITextEmbedding
import os

embedding_generator = OpenAITextEmbedding(
    ai_model_id="text-embedding-3-small",
    api_key=os.getenv("OPENAI_API_KEY"),
    # If you route through a gateway that speaks OpenAI format, set base_url:
    # base_url="https://api.n4n.ai/v1",
)

Set OPENAI_API_KEY in your environment. If you’re using a gateway that forwards to multiple providers, point base_url at that gateway — the SDK treats it like any OpenAI endpoint.

Ingest documents

Create a simple ingestion pipeline: chunk text, generate embeddings, upsert into Chroma. Semantic Kernel provides a VectorStoreRecordCollection for typed access:

from semantic_kernel.data import VectorStoreRecordCollection

collection = vector_store.get_collection("sk_docs", record_definition)

documents = [
    {"id": "doc-1", "content": "Semantic Kernel orchestrates plugins, planners, and memories for LLM apps."},
    {"id": "doc-2", "content": "Chroma is an open-source embedding database with a simple HTTP API."},
    {"id": "doc-3", "content": "Vector stores enable retrieval-augmented generation by indexing embeddings."},
]

async def ingest():
    for doc in documents:
        embedding = await embedding_generator.generate_embedding(doc["content"])
        record = {
            "id": doc["id"],
            "content": doc["content"],
            "embedding": embedding,
        }
        await collection.upsert(record)
    print("Ingestion complete")

asyncio.run(ingest())

Expected output:

Ingestion complete

Verify the data landed in Chroma:

curl -X POST http://localhost:8000/api/v1/collections/sk_docs/get \
  -H "Content-Type: application/json" \
  -d '{"ids": ["doc-1", "doc-2", "doc-3"]}'

You should see three records with their embeddings populated.

Run a similarity search

Query the store with a natural-language question. The same embedding generator encodes the query; the collection returns the nearest neighbors:

async def search(query: str, top_k: int = 3):
    query_embedding = await embedding_generator.generate_embedding(query)
    results = await collection.search(
        query_embedding,
        top_k=top_k,
        include_vectors=False,
    )
    for result in results:
        print(f"Score: {result.score:.4f} | ID: {result.record['id']} | Content: {result.record['content']}")

asyncio.run(search("How do I add memory to an LLM app?"))

Expected output (scores will vary slightly):

Score: 0.8234 | ID: doc-1 | Content: Semantic Kernel orchestrates plugins, planners, and memories for LLM apps.
Score: 0.7612 | ID: doc-3 | Content: Vector stores enable retrieval-augmented generation by indexing embeddings.
Score: 0.6541 | ID: doc-2 | Content: Chroma is an open-source embedding database with a simple HTTP API.

The search method returns VectorSearchResult objects with score, record, and optional vector. Set include_vectors=True only if you need the raw embeddings downstream — it increases payload size.

Add metadata filtering

Real workloads need filters — by source, date, tenant, or custom tags. Extend the record definition with filterable fields:

from semantic_kernel.data import VectorStoreRecordKeyField, VectorStoreRecordDataField, VectorStoreRecordVectorField

record_definition_v2 = VectorStoreRecordDefinition(
    fields={
        "id": VectorStoreRecordKeyField(),
        "content": VectorStoreRecordDataField(),
        "source": VectorStoreRecordDataField(is_filterable=True),
        "created_at": VectorStoreRecordDataField(is_filterable=True),
        "embedding": VectorStoreRecordVectorField(dimensions=1536),
    },
    container_mode=True,
)

collection_v2 = vector_store.get_collection("sk_docs_v2", record_definition_v2)

Ingest with metadata:

from datetime import datetime, timezone

async def ingest_with_metadata():
    docs = [
        {"id": "doc-1", "content": "Semantic Kernel orchestrates plugins...", "source": "docs", "created_at": datetime(2024, 1, 15, tzinfo=timezone.utc)},
        {"id": "doc-2", "content": "Chroma is an open-source embedding database...", "source": "blog", "created_at": datetime(2024, 3, 22, tzinfo=timezone.utc)},
        {"id": "doc-3", "content": "Vector stores enable RAG...", "source": "docs", "created_at": datetime(2024, 2, 10, tzinfo=timezone.utc)},
    ]
    for doc in docs:
        embedding = await embedding_generator.generate_embedding(doc["content"])
        record = {**doc, "embedding": embedding}
        await collection_v2.upsert(record)
    print("Metadata ingestion complete")

asyncio.run(ingest_with_metadata())

Search with a filter — only return records where source == "docs":

from semantic_kernel.data import VectorSearchFilter, VectorSearchFilterClause

async def filtered_search(query: str, source: str, top_k: int = 3):
    query_embedding = await embedding_generator.generate_embedding(query)
    filter_ = VectorSearchFilter(
        clauses=[
            VectorSearchFilterClause(field_name="source", operator="eq", value=source),
        ]
    )
    results = await collection_v2.search(
        query_embedding,
        top_k=top_k,
        filter=filter_,
        include_vectors=False,
    )
    for result in results:
        print(f"Score: {result.score:.4f} | Source: {result.record['source']} | Content: {result.record['content']}")

asyncio.run(filtered_search("memory for LLM apps", source="docs"))

Expected output:

Score: 0.8234 | Source: docs | Content: Semantic Kernel orchestrates plugins, planners, and memories for LLM apps.
Score: 0.7612 | Source: docs | Content: Vector stores enable retrieval-augmented generation by indexing embeddings.

The blog post (source: “blog”) is excluded. Chroma translates the filter to its where clause natively — no post-filtering in Python.

Delete and update records

Upsert handles both insert and update. Delete by id:

async def delete_record(record_id: str):
    await collection_v2.delete(record_id)
    print(f"Deleted {record_id}")

asyncio.run(delete_record("doc-2"))

Confirm deletion:

async def verify_deleted(record_id: str):
    result = await collection_v2.get(record_id)
    print(f"Get result: {result}")  # None if deleted

asyncio.run(verify_deleted("doc-2"))

Output:

Get result: None

Use the vector store in a RAG flow

Semantic Kernel’s Kernel ties everything together. Register the collection as a memory skill and invoke it from a prompt:

from semantic_kernel import Kernel
from semantic_kernel.functions import kernel_function
from semantic_kernel.connectors.openai import OpenAIChatCompletion

kernel = Kernel()
kernel.add_service(OpenAIChatCompletion(ai_model_id="gpt-4o-mini", api_key=os.getenv("OPENAI_API_KEY")))

class RagMemory:
    def __init__(self, collection):
        self.collection = collection
        self.embedding_generator = embedding_generator

    @kernel_function(name="search_memory", description="Search the vector store for relevant context")
    async def search_memory(self, query: str, top_k: int = 3) -> str:
        query_embedding = await self.embedding_generator.generate_embedding(query)
        results = await self.collection.search(query_embedding, top_k=top_k, include_vectors=False)
        context = "\n\n".join([r.record["content"] for r in results])
        return context

kernel.add_plugin(RagMemory(collection_v2), plugin_name="memory")

# Invoke from a prompt
prompt = """
Answer the question using only the provided context.

Context:
{{memory.search_memory $question}}

Question: {{$question}}
"""

result = await kernel.invoke_prompt(prompt, question="What is Semantic Kernel?")
print(result)

Expected output (condensed):

Semantic Kernel is a framework that orchestrates plugins, planners, and memories for building LLM applications.

The prompt template syntax {{memory.search_memory $question}} calls the registered function and injects its return value. This pattern scales — swap the collection for Pinecone, Weaviate, or Qdrant by changing only the vector store initialization.

Persist the collection across restarts

Chroma persists to disk by default when running in Docker with a volume. Mount a directory to survive container restarts:

docker run -d -p 8000:8000 -v $(pwd)/chroma_data:/chroma/chroma chromadb/chroma:latest

The Semantic Kernel connector requires no changes — it talks HTTP to the same endpoint. If you switch to Chroma’s embedded mode (chromadb.PersistentClient), update the connection string:

vector_store = ChromaVectorStore(
    collection_name="sk_docs",
    record_definition=record_definition,
    connection_string="./chroma_data",  # local path for embedded client
)

The rest of your code — ingestion, search, filtering — remains identical.

Common pitfalls

Dimension mismatch: If you change embedding models, drop and recreate the collection. Chroma does not support altering vector dimensions in place.

Batch upserts: For large datasets, batch records and call upsert_batch (available on the collection) to reduce round trips:

await collection.upsert_batch(records)  # list of record dicts

Async everywhere: All Semantic Kernel vector store methods are async. Don’t mix blocking calls in the same event loop — use asyncio.run() at the entry point or await throughout.

Filter field types: Only fields marked is_filterable=True in the record definition can appear in VectorSearchFilter. Non-filterable fields are ignored silently.

Next steps

• Add hybrid search by combining vector similarity with BM25 — Chroma supports keyword search via where_document clauses.
• Implement incremental ingestion: track file hashes, only re-embed changed chunks.
• Wire a reranker (Cohere, Jina, or cross-encoder) after the vector search to improve precision.
• Monitor latency and recall with a small eval set — log query, retrieved ids, and human relevance judgments.

The Semantic Kernel vector store abstraction keeps your retrieval logic portable. Chroma gives you a low-friction local backend and a straightforward path to managed hosting. Swap the connector when your scale or compliance requirements change — the plugin, prompt, and planner code stays put.