Building a customer support bot with LlamaIndex

This llamaindex customer support chatbot tutorial walks through building a retrieval-augmented support agent that remembers conversation state and answers from your own docs. We’ll use LlamaIndex’s chat engines and memory buffers, with runnable Python you can adapt today.

Prerequisites

• Python 3.10 or newer
• llama-index and llama-index-llms-openai installed
• An OpenAI API key, or any OpenAI-compatible endpoint URL and key
• A support_docs/ directory containing .md or .txt help articles

pip install llama-index llama-index-llms-openai
export OPENAI_API_KEY="sk-..."

If you prefer to avoid a single vendor, you can later point the base URL at a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint spanning 240+ models and handles provider fallback automatically.

Step 1: Load and index your support docs

LlamaIndex treats each file as a document and splits it into nodes. For a support bot, VectorStoreIndex over those nodes is enough to start.

from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    Settings,
)
from llama_index.llms.openai import OpenAI

documents = SimpleDirectoryReader("support_docs").load_data()
index = VectorStoreIndex.from_documents(documents)

This builds an in-memory vector store. For production you’d swap to a persistent store (Chroma, pgvector), but the index API stays identical.

Step 2: Configure the LLM and embeddings

Defaults use OpenAI’s text-embedding-3-small and gpt-3.5-turbo. Set the LLM explicitly so the code is portable.

Settings.llm = OpenAI(model="gpt-3.5-turbo", temperature=0)
# Embeddings default to OpenAI's text-embedding-3-small

To use an OpenAI-compatible gateway instead, change the api_base:

Settings.llm = OpenAI(
    model="anthropic/claude-3-haiku",
    api_base="https://api.n4n.ai/v1",
    api_key="your-gateway-key",
    temperature=0,
)

The rest of the tutorial is unchanged because LlamaIndex only cares about the OpenAI interface.

Step 3: Create a chat engine with memory

A bare index query won’t remember prior turns. ContextChatEngine retrieves relevant nodes each turn and feeds them plus chat history to the LLM. ChatMemoryBuffer caps token usage so context doesn’t grow unbounded.

from llama_index.core.chat_engine import ContextChatEngine
from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=1500)

chat_engine = index.as_chat_engine(
    chat_mode="context",
    memory=memory,
    similarity_top_k=3,
    system_prompt=(
        "You are a customer support agent for Acme Corp. "
        "Answer only from the provided documentation. "
        "If unsure, say you will escalate."
    ),
)

Step 4: Run a conversation loop

The simplest test is a REPL. chat_engine.chat() returns a ChatResponse with the answer and the source nodes.

print("Support bot ready. Type 'exit' to quit.")
while True:
    user_input = input("You: ").strip()
    if user_input.lower() == "exit":
        break
    response = chat_engine.chat(user_input)
    print(f"Bot: {response.response}")

Expected output on first question:

You: How do I reset my password?
Bot: To reset your password, go to Settings > Security > Reset Password. You'll receive an email with a 30-minute link. If the link expires, repeat the step.

The bot pulled that from a support_docs/account.md node. Memory is empty, so it’s pure retrieval.

Step 5: Multi-turn context and citations

Ask a follow-up without repeating context:

You: What if the email doesn't arrive?
Bot: Check spam and ensure the address on file is verified. If still missing after 5 minutes, use the fallback SMS code sent to your registered phone. This is covered in the same account recovery doc.

The ContextChatEngine condenses chat history into the retrieval query, so “the email” resolves to the password reset email. To see what was retrieved:

for node in response.source_nodes:
    print(node.node.get_content()[:200], "...\n")

For stricter grounding, use CondensePlusContextChatEngine which explicitly rewrites the query before retrieval:

from llama_index.core.chat_engine import CondensePlusContextChatEngine

chat_engine = CondensePlusContextChatEngine(
    retriever=index.as_retriever(similarity_top_k=3),
    memory=memory,
    system_prompt="You are Acme support. Cite the doc section in answers.",
)

Step 6: Streaming responses

Support users expect typed-out replies. Enable streaming on the LLM and use stream_chat.

Settings.llm = OpenAI(model="gpt-3.5-turbo", temperature=0, streaming=True)

streaming_engine = index.as_chat_engine(
    chat_mode="context", memory=memory, similarity_top_k=3
)

response = streaming_engine.stream_chat("Explain your refund policy")
for token in response.response_gen:
    print(token, end="")

You’ll see tokens appear incrementally, same as a typical chat UI.

Step 7: Production checks

A few things we’ve shipped and regret skipping:

• Token accounting: ChatMemoryBuffer trims old messages, but log response.metadata to track per-call usage if you meter by token.
• Fallback on provider errors: if you hit rate limits, the gateway or your own retry wrapper should catch APIError and rotate models.
• Stale index: support docs change. Schedule index.refresh() or rebuild nightly; VectorStoreIndex supports insert/delete on persistent stores.
• Guardrails: the system prompt says “escalate if unsure.” Enforce by checking source_nodes length; if zero, route to human.

if len(response.source_nodes) == 0:
    print("Bot: I'll escalate this to a human agent.")

That’s the core of a maintainable support bot. The llamaindex customer support chatbot tutorial above gives you retrieval, memory, and a clean swap path to any OpenAI-compatible model without rewriting your app logic.

Where to extend

Add a RouterQueryEngine to split billing vs technical questions, or plug in SubQuestionQueryEngine for multi-doc reasoning. The chat engine interface stays the same; only the retriever changes.

If you want to run the same code against different providers without code changes, keep the api_base abstraction and let the gateway handle routing directives and cache-control hints. That’s the difference between a demo and a system that survives a provider outage.