LlamaIndex tutorial: a support bot for your help docs

Building a retrieval-augmented assistant over your knowledge base is the highest-leverage use of LLMs for most teams. This llamaindex support bot help docs tutorial takes you from a folder of Markdown articles to a working chat endpoint in under 200 lines of Python. We’ll use LlamaIndex for orchestration, OpenAI-compatible embeddings and completion, and show exactly where to plug in fallbacks.

Prerequisites

Python 3.10 or newer
A directory of help docs (Markdown, HTML, or plain text) — one file per article works best
An API key for an OpenAI-compatible LLM and embedding endpoint
pip and a virtual environment tool

If you don’t have docs handy, create ./help_docs/security/2fa.md with a few sentences about two-factor reset.

Install dependencies

LlamaIndex split into core and integration packages in 2024. Install only what you need.

python -m venv .venv
source .venv/bin/activate
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai python-dotenv fastapi uvicorn

Configure the models

We use the OpenAI client integration but point it at any compatible base URL. If you run into provider rate limits, an inference gateway like n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and automatically falls back when a provider is degraded.

import os
from dotenv import load_dotenv
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI

load_dotenv()

EMBED_BASE = os.getenv("EMBED_BASE_URL", "https://api.openai.com/v1")
LLM_BASE = os.getenv("LLM_BASE_URL", "https://api.openai.com/v1")

embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_base=EMBED_BASE,
    api_key=os.getenv("API_KEY"),
)

llm = OpenAI(
    model="gpt-4o-mini",
    api_base=LLM_BASE,
    api_key=os.getenv("API_KEY"),
    temperature=0.1,
)

Set LLM_BASE_URL and EMBED_BASE_URL in .env to route through your gateway. Keep temperature low; support answers should be deterministic.

Load your help docs

SimpleDirectoryReader handles .md, .html, and .txt. Recursive scan preserves your folder structure, which we’ll later expose as metadata.

from llama_index.core import SimpleDirectoryReader

docs = SimpleDirectoryReader(
    input_dir="./help_docs",
    required_exts=[".md", ".html", ".txt"],
    recursive=True,
).load_data()

print(f"Loaded {len(docs)} documents")

Expected output:

Loaded 42 documents

Tune chunk size before embedding

Default splitting is naive. For help docs, sentence-aware chunks of 512 tokens with 64-token overlap preserve procedural steps better than fixed-size cuts.

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = splitter.get_nodes_from_documents(docs)

for n in nodes:
    n.metadata["file_path"] = n.metadata.get("file_path", "unknown")

Build the vector index

The VectorStoreIndex computes embeddings and holds them in memory. For this llamaindex support bot help docs walkthrough we skip an external vector DB; swap in MilvusVectorStore or PGVector when doc count grows past a few thousand.

from llama_index.core import VectorStoreIndex, Settings

Settings.llm = llm
Settings.embed_model = embed_model

index = VectorStoreIndex(nodes)

Indexing prints batch progress. On 42 small files it finishes in under five seconds.

Query the bot

A query engine retrieves top-k nodes and synthesizes. Use similarity_top_k=4 as a starting point; raise it if answers feel under-informed.

query_engine = index.as_query_engine(similarity_top_k=4)

response = query_engine.query(
    "How do I reset my two-factor authentication?"
)
print(str(response))

Expected output (truncated):

To reset two-factor authentication, go to Settings > Security > 2FA,
click "Reset", and confirm via your recovery code. If you lost the
recovery code, contact support@acme.com with your account email.

Add source citations

A support bot that can’t show its work is a liability. LlamaIndex attaches source nodes with scores. Surface them so agents can verify.

for node in response.source_nodes:
    meta = node.node.metadata
    print(f"{meta.get('file_path')}  score={node.score:.3f}")

Output:

help_docs/security/2fa.md  score=0.87
help_docs/security/login.md  score=0.71
help_docs/account/recovery.md  score=0.64
help_docs/security/overview.md  score=0.58

Make it a chat loop

The ChatEngine keeps state so follow-ups resolve correctly. condense_question mode rewrites the user’s short question into a standalone one using history.

chat_engine = index.as_chat_engine(
    similarity_top_k=4,
    chat_mode="condense_question",
)

while True:
    q = input("You: ")
    if q.lower() in {"exit", "quit"}:
        break
    resp = chat_engine.chat(q)
    print(f"Bot: {resp}")

Run it:

You: How do I invite a teammate?
Bot: Go to Workspace > Members > Invite, enter their email, and pick a role.
You: Can I restrict them to billing only?
Bot: Yes. After inviting, edit their role to "Billing" under Members > Roles.

Filter by metadata

If your docs carry a product field, filter at query time to avoid cross-product confusion.

from llama_index.core.vector_stores import MetadataFilter, MetadataFilters

filters = MetadataFilters(
    filters=[MetadataFilter(key="product", value="enterprise")]
)
filtered_engine = index.as_query_engine(
    similarity_top_k=4,
    filters=filters,
)

Persist the index

Re-embedding on every restart wastes tokens. Persist to disk and reload.

index.storage_context.persist(persist_dir="./storage")

# Later, reload:
from llama_index.core import StorageContext, load_index_from_storage
sc = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(sc, embed_model=embed_model)

Wire up a minimal API

FastAPI turns the engine into an internal endpoint. We skip auth for brevity—add a bearer token before production.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
engine = index.as_query_engine(similarity_top_k=4)

class Ask(BaseModel):
    question: str

@app.post("/ask")
def ask(req: Ask):
    r = engine.query(req.question)
    sources = [
        {"path": n.node.metadata.get("file_path"), "score": n.score}
        for n in r.source_nodes
    ]
    return {"answer": str(r), "sources": sources}

Start with uvicorn main:app --port 8000. Hit it:

curl -X POST localhost:8000/ask \
  -H 'content-type: application/json' \
  -d '{"question":"How do I export invoices?"}'

Response:

{
  "answer": "Go to Billing > Invoices, select a date range, and click Export CSV.",
  "sources": [
    {"path": "help_docs/billing/invoices.md", "score": 0.91}
  ]
}

Evaluate before shipping

Retrieval precision matters more than model size for support bots. Log 50 real questions, check that the cited file_path actually contains the answer, and tune chunk_size or similarity_top_k based on misses. Add a fallback message when all node scores sit below 0.5.

Where to go next

The llamaindex support bot help docs pattern above is a baseline. Add hybrid search (BM25 + vector) for keyword-heavy docs, cache embeddings at the gateway layer, and enforce per-token budgets on the client. If you routed through a single OpenAI-compatible endpoint with fallback, you already get provider redundancy without touching this code.