Customer support bot: LlamaIndex plus a vector database

Building a customer support bot llamaindex vector database stack means wiring together document ingestion, semantic retrieval, and a language model that can ground its answers in your actual knowledge base. This tutorial walks through a complete, runnable implementation using LlamaIndex with Chroma as the vector store and an OpenAI-compatible endpoint for generation. You’ll end up with a service that indexes your support docs, retrieves relevant context per query, and streams citations back to the caller.

Prerequisites

• Python 3.10+
• An OpenAI-compatible API endpoint (OpenAI, Azure, or a gateway like n4n.ai that forwards to 240+ models)
• A directory of support documents in Markdown, PDF, or plain text
• Roughly 15 minutes

Install the dependencies:

pip install llama-index llama-index-vector-stores-chroma llama-index-llms-openai \
    llama-index-embeddings-openai chromadb python-dotenv

Create a .env file with your credentials:

# .env
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.openai.com/v1   # or your gateway endpoint
EMBEDDING_MODEL=text-embedding-3-small
LLM_MODEL=gpt-4o-mini
CHROMA_PERSIST_DIR=./chroma_db

Project structure

support-bot/
├── .env
├── data/                 # drop your support docs here
│   ├── refund-policy.md
│   ├── shipping-faq.md
│   └── troubleshooting.md
├── ingest.py             # one-time (or scheduled) index build
├── query.py              # interactive query loop
└── requirements.txt

Step 1: Ingest documents into the vector store

The ingestion script loads files from data/, splits them into overlapping chunks, embeds each chunk, and persists them to Chroma. Run this whenever your knowledge base changes.

# ingest.py
import os
from pathlib import Path
from dotenv import load_dotenv
from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    Settings,
)
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.embeddings.openai import OpenAIEmbedding
import chromadb

load_dotenv()

PERSIST_DIR = os.getenv("CHROMA_PERSIST_DIR", "./chroma_db")
DATA_DIR = Path(__file__).parent / "data"
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "text-embedding-3-small")

Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)

def build_index() -> VectorStoreIndex:
    # Initialize Chroma client with persistence
    chroma_client = chromadb.PersistentClient(path=PERSIST_DIR)
    chroma_collection = chroma_client.get_or_create_collection("support_docs")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    # Load documents
    documents = SimpleDirectoryReader(str(DATA_DIR)).load_data()
    print(f"Loaded {len(documents)} documents from {DATA_DIR}")

    # Build and persist index
    index = VectorStoreIndex.from_documents(
        documents,
        storage_context=storage_context,
        show_progress=True,
    )
    print(f"Index built and persisted to {PERSIST_DIR}")
    return index

if __name__ == "__main__":
    build_index()

Run it:

python ingest.py

Expected output:

Loaded 3 documents from /path/to/support-bot/data
Index built and persisted to ./chroma_db

Chroma now holds a collection named support_docs with embedded chunks. You can inspect it directly:

python -c "
import chromadb
client = chromadb.PersistentClient(path='./chroma_db')
col = client.get_collection('support_docs')
print(f'Total chunks: {col.count()}')
print(col.peek(3))
"

Step 2: Configure the query engine with citations

The query engine ties together the retriever, the LLM, and a response synthesizer that includes source nodes. We’ll use VectorIndexRetriever with similarity_top_k=4 and enable response_mode="tree_summarize" for multi-document synthesis.

# query.py
import os
from dotenv import load_dotenv
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    Settings,
    get_response_synthesizer,
)
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
import chromadb

load_dotenv()

PERSIST_DIR = os.getenv("CHROMA_PERSIST_DIR", "./chroma_db")
LLM_MODEL = os.getenv("LLM_MODEL", "gpt-4o-mini")
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "text-embedding-3-small")

Settings.llm = OpenAI(model=LLM_MODEL, temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)

def load_index() -> VectorStoreIndex:
    chroma_client = chromadb.PersistentClient(path=PERSIST_DIR)
    chroma_collection = chroma_client.get_collection("support_docs")
    vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    return VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)

def build_query_engine(index: VectorStoreIndex) -> RetrieverQueryEngine:
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=4,
    )

    response_synthesizer = get_response_synthesizer(
        response_mode="tree_summarize",
        streaming=True,
    )

    return RetrieverQueryEngine(
        retriever=retriever,
        response_synthesizer=response_synthesizer,
    )

def format_sources(response) -> str:
    lines = ["\n--- Sources ---"]
    for i, node in enumerate(response.source_nodes, 1):
        meta = node.metadata
        file_name = meta.get("file_name", "unknown")
        page = meta.get("page_label", meta.get("page", "?"))
        score = node.score if node.score is not None else 0.0
        lines.append(f"[{i}] {file_name} (page {page}) — similarity: {score:.3f}")
    return "\n".join(lines)

def main():
    print("Loading index...")
    index = load_index()
    query_engine = build_query_engine(index)

    print("\nCustomer support bot ready. Type 'exit' to quit.\n")
    while True:
        try:
            user_query = input("You: ").strip()
            if user_query.lower() in {"exit", "quit"}:
                break
            if not user_query:
                continue

            print("Bot: ", end="", flush=True)
            streaming_response = query_engine.query(user_query)
            for token in streaming_response.response_gen:
                print(token, end="", flush=True)
            print()  # newline after stream
            print(format_sources(streaming_response))
            print()
        except KeyboardInterrupt:
            break
        except Exception as e:
            print(f"\nError: {e}\n")

if __name__ == "__main__":
    main()

Run the interactive loop:

python query.py

Example session:

Loading index...

Customer support bot ready. Type 'exit' to quit.

You: What's the refund window for digital purchases?
Bot: Our refund policy allows refunds for digital purchases within 14 days of purchase, provided the content has not been downloaded or accessed. For subscription products, you can cancel at any time and receive a prorated refund for the unused portion of the current billing period.

--- Sources ---
[1] refund-policy.md (page 1) — similarity: 0.892
[2] shipping-faq.md (page 1) — similarity: 0.311
[3] troubleshooting.md (page 2) — similarity: 0.287

You: exit

Step 3: Add metadata filtering for product-scoped answers

If your support docs cover multiple products, tag each document at ingestion time and filter at query time. This prevents the bot from mixing policies across products.

Update ingest.py to inject a product field:

# ingest.py (add to build_index, before VectorStoreIndex.from_documents)
for doc in documents:
    # Infer product from directory or filename convention
    # e.g., data/product-a/refund-policy.md -> product-a
    rel_path = Path(doc.metadata.get("file_path", "")).relative_to(DATA_DIR)
    product = rel_path.parts[0] if rel_path.parts else "general"
    doc.metadata["product"] = product

Then modify query.py to accept a product filter:

# query.py (add to build_query_engine)
from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter

def build_query_engine(index: VectorStoreIndex, product: str | None = None) -> RetrieverQueryEngine:
    filters = None
    if product:
        filters = MetadataFilters(filters=[ExactMatchFilter(key="product", value=product)])

    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=4,
        filters=filters,
    )
    # ... rest unchanged

Now call build_query_engine(index, product="product-a") to scope retrieval.

Step 4: Expose as a minimal FastAPI service

Wrap the query engine in a /chat endpoint that streams Server-Sent Events (SSE). This is the pattern you’d deploy behind a load balancer.

# api.py
import os
import json
from dotenv import load_dotenv
from fastapi import FastAPI, Query
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from llama_index.core import VectorStoreIndex, StorageContext, Settings
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers import get_response_synthesizer
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
import chromadb

load_dotenv()

PERSIST_DIR = os.getenv("CHROMA_PERSIST_DIR", "./chroma_db")
LLM_MODEL = os.getenv("LLM_MODEL", "gpt-4o-mini")
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "text-embedding-3-small")

Settings.llm = OpenAI(model=LLM_MODEL, temperature=0.1, streaming=True)
Settings.embed_model = OpenAIEmbedding(model=EMBEDDING_MODEL)

app = FastAPI(title="Support Bot API")

# Load index once at startup
chroma_client = chromadb.PersistentClient(path=PERSIST_DIR)
chroma_collection = chroma_client.get_collection("support_docs")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_vector_store(vector_store, storage_context=storage_context)

class ChatRequest(BaseModel):
    message: str
    product: str | None = None

def create_query_engine(product: str | None):
    from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
    filters = None
    if product:
        filters = MetadataFilters(filters=[ExactMatchFilter(key="product", value=product)])

    retriever = VectorIndexRetriever(index=index, similarity_top_k=4, filters=filters)
    synthesizer = get_response_synthesizer(response_mode="tree_summarize", streaming=True)
    return RetrieverQueryEngine(retriever=retriever, response_synthesizer=synthesizer)

@app.post("/chat")
async def chat(req: ChatRequest):
    query_engine = create_query_engine(req.product)
    streaming_response = query_engine.query(req.message)

    async def event_generator():
        for token in streaming_response.response_gen:
            yield f"data: {json.dumps({'token': token})}\n\n"
        # Send sources at the end
        sources = [
            {
                "file_name": n.metadata.get("file_name", "unknown"),
                "page": n.metadata.get("page_label", n.metadata.get("page", "?")),
                "score": n.score,
            }
            for n in streaming_response.source_nodes
        ]
        yield f"data: {json.dumps({'sources': sources})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

@app.get("/health")
async def health():
    return {"status": "ok", "chunks_indexed": chroma_collection.count()}

Run the API:

pip install fastapi uvicorn
uvicorn api:app --reload --port 8000

Test with curl:

curl -N -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "How do I reset my password?", "product": "product-a"}'

Output (SSE stream):

data: {"token": "To reset your password for Product A, go to the login page and click "}

data: {"token": "\"Forgot Password.\" Enter your registered email address and you'll receive a "}

data: {"token": "reset link within 5 minutes. The link expires after 1 hour."}

data: {"sources": [{"file_name": "troubleshooting.md", "page": "1", "score": 0.912}]}

data: [DONE]

Step 5: Observability — log every query with latency and token counts

Production systems need visibility. Add a lightweight middleware that records request latency, token usage, and the retrieved source count.

# api.py (add near top)
import time
from contextlib import asynccontextmanager
from collections import defaultdict

stats = defaultdict(list)

@asynccontextmanager
async def lifespan(app: FastAPI):
    yield
    # On shutdown, print summary
    if stats["latency"]:
        avg_lat = sum(stats["latency"]) / len(stats["latency"])
        print(f"Processed {len(stats['latency'])} requests, avg latency: {avg_lat:.2f}s")

app = FastAPI(title="Support Bot API", lifespan=lifespan)

# Wrap the chat endpoint
@app.post("/chat")
async def chat(req: ChatRequest):
    start = time.perf_counter()
    query_engine = create_query_engine(req.product)
    streaming_response = query_engine.query(req.message)

    # Capture token usage from the LLM response (OpenAI-compatible)
    prompt_tokens = 0
    completion_tokens = 0
    if hasattr(streaming_response, "metadata") and "token_usage" in streaming_response.metadata:
        usage = streaming_response.metadata["token_usage"]
        prompt_tokens = usage.get("prompt_tokens", 0)
        completion_tokens = usage.get("completion_tokens", 0)

    async def event_generator():
        token_count = 0
        for token in streaming_response.response_gen:
            token_count += 1
            yield f"data: {json.dumps({'token': token})}\n\n"
        sources = [
            {
                "file_name": n.metadata.get("file_name", "unknown"),
                "page": n.metadata.get("page_label", n.metadata.get("page", "?")),
                "score": n.score,
            }
            for n in streaming_response.source_nodes
        ]
        yield f"data: {json.dumps({'sources': sources})}\n\n"
        yield "data: [DONE]\n\n"

    latency = time.perf_counter() - start
    stats["latency"].append(latency)
    stats["prompt_tokens"].append(prompt_tokens)
    stats["completion_tokens"].append(completion_tokens)
    stats["source_count"].append(len(streaming_response.source_nodes))

    return StreamingResponse(event_generator(), media_type="text/event-stream")

Now every request increments counters you can scrape or push to Prometheus.

Step 6: Automate re-indexing on document changes

In production, docs change. Use a file watcher or a scheduled job to rebuild the index incrementally. LlamaIndex supports insert, delete, and update on the index, but for simplicity, a nightly full rebuild is often sufficient for support knowledge bases under 10k documents.

# reindex.py — run via cron or systemd timer
import subprocess
import sys

def main():
    result = subprocess.run([sys.executable, "ingest.py"], capture_output=True, text=True)
    print(result.stdout)
    if result.stderr:
        print(result.stderr, file=sys.stderr)
    sys.exit(result.returncode)

if __name__ == "__main__":
    main()

Add to crontab:

0 3 * * * /path/to/venv/bin/python /path/to/support-bot/reindex.py >> /var/log/support-bot-reindex.log 2>&1

Common pitfalls

Tuning knobs worth adjusting

• Chunk size / overlap: Settings.chunk_size = 512, Settings.chunk_overlap = 50 — smaller chunks improve precision, larger improve recall.
• Embedding model: text-embedding-3-large for higher quality at 3x cost; text-embedding-3-small is the sweet spot for most support bots.
• Reranking: Add a cross-encoder reranker (llama-index-postprocessor-cohere-rerank or sentence-transformers) after retrieval to boost precision before synthesis.
• Hybrid search: Combine dense vectors with BM25 (ChromaVectorStore supports hybrid via query_texts + where filters) for exact keyword matches like error codes.

What’s next

• Add conversation memory with ChatMemoryBuffer so follow-up questions retain context.
• Implement a fallback path: if no source nodes exceed a similarity threshold, respond with “I don’t know” and create a ticket.
• Wire up evaluation: generate a golden set of Q&A pairs and run llama-index-evaluation nightly to catch regressions.
• Deploy the API behind your gateway with per-tenant API keys and rate limits.

You now have a customer support bot llamaindex vector database pipeline that ingests, retrieves, streams, and observes — ready to extend for your product’s specific workflows.