How to train an AI support agent on your help docs

You don’t fine-tune a model to handle your support queue. To train AI support agent help docs effectively, you build a retrieval pipeline that grounds responses in your existing knowledge base and let a capable LLM synthesize answers. This post walks through a production-grade RAG setup you can ship this week.

Step 1: Inventory and extract your help docs

Most help centers live in multiple places: Markdown in Git, HTML in a CMS, or JSON exported from Zendesk. Pull everything into a common record with slug, title, text, and updated_at.

For HTML, strip tags before chunking. A quick BeautifulSoup pass keeps line breaks intact.

from bs4 import BeautifulSoup
import requests

def load_html(url):
    html = requests.get(url, timeout=10).text
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style"]):
        tag.decompose()
    text = soup.get_text(separator="\n", strip=True)
    return {"slug": url, "text": text, "updated_at": 0}

Keep the metadata. You will filter on updated_at later to avoid serving stale policy details.

Step 2: Chunk with structure preserved

Naive fixed-size splitting destroys tables, code blocks, and heading context. Use a splitter that respects document structure. Aim for 800–1200 token chunks with 100-token overlap so sentences aren’t cut mid-thought.

import tiktoken
from langchain.text_splitter import MarkdownHeaderTextSplitter

enc = tiktoken.get_encoding("cl100k_base")
headers = [("#", "h1"), ("##", "h2"), ("###", "h3")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)

def chunk_doc(doc, max_tokens=1000, overlap=100):
    chunks = splitter.split_text(doc["text"])
    out = []
    for i, c in enumerate(chunks):
        tokens = enc.encode(c.page_content)
        if len(tokens) <= max_tokens:
            out.append(make_record(doc, i, c))
            continue
        # crude overlap split
        start = 0
        j = 0
        while start < len(tokens):
            piece = enc.decode(tokens[start:start+max_tokens])
            out.append(make_record(doc, f"{i}-{j}", piece, c.metadata))
            start += max_tokens - overlap
            j += 1
    return out

def make_record(doc, cid, text, meta=None):
    return {"slug": doc["slug"], "chunk_id": cid, "text": text,
            "meta": meta or {}, "updated_at": doc["updated_at"]}

When you train AI support agent help docs via RAG, chunk quality predicts answer quality more than model choice. Spend time here.

Step 3: Embed and index in a vector store

Use a strong embedding model. text-embedding-3-small is cheap and good enough for most English help desks. Store vectors in Chroma for local dev, or pgvector in Postgres for production.

import openai, chromadb

client = openai.OpenAI()
chroma = chromadb.Client()
col = chroma.create_collection("help_docs")

def embed(texts):
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [d.embedding for d in resp.data]

batch = 100
all_chunks = [c for doc in articles for c in chunk_doc(doc)]
for i in range(0, len(all_chunks), batch):
    slice_ = all_chunks[i:i+batch]
    vecs = embed([c["text"] for c in slice_])
    col.add(
        ids=[f"{c['slug']}#{c['chunk_id']}" for c in slice_],
        embeddings=vecs,
        documents=[c["text"] for c in slice_],
        metadatas=[{"slug": c["slug"], "updated_at": c["updated_at"]} for c in slice_]
    )

In pgvector, use halfvec to cut memory. Create a HNSW index for sub-10ms queries at million-scale.

Step 4: Build the retrieval query path

At inference, embed the user question and pull top-k chunks. Add a metadata prefilter for product area if your docs are large. Hybrid search (BM25 + vector) improves recall on rare error codes.

def retrieve(query, k=5, product=None):
    qvec = embed([query])[0]
    where = {"slug": {"$like": f"{product}%"}} if product else None
    res = col.query(query_embeddings=[qvec], n_results=k, where=where)
    return res["documents"][0], res["metadatas"][0]

To train AI support agent help docs at scale, automate this indexing in a CI job that reruns when the help center changes.

Step 5: Compose the grounded prompt

Never dump raw chunks into the model. Wrap them with explicit instructions and force citations. The system prompt below cuts hallucination in our testing.

SYSTEM = """You are a support agent for Acme Corp.
Answer ONLY from the CONTEXT blocks. If the answer is not present,
say 'I don't have that information' and suggest contacting support.
Cite every claim as [slug#chunk_id]."""

def build_messages(query, ctx_docs, ctx_meta):
    context = "\n\n".join(
        f"[{m['slug']}#{i}] {d}" for i, (d, m) in enumerate(zip(ctx_docs, ctx_meta))
    )
    return [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {query}"}
    ]

Bad prompt: “Answer the question using the docs.” Good prompt: constrains scope and demands provenance.

Step 6: Call the model with fallback

Point your OpenAI client at any OpenAI-compatible endpoint. If you want automatic fallback when a provider is rate-limited, an OpenAI-compatible gateway like n4n.ai fronts 240+ models and handles degradation without code changes.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_KEY"
)

def answer(query, product=None):
    docs, meta = retrieve(query, product=product)
    msgs = build_messages(query, docs, meta)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=msgs,
        temperature=0.1,
        stream=False
    )
    return resp.choices[0].message.content

Set temperature low. Support answers should be deterministic, not creative. Add a 5-second timeout and retry once on connection error.

Step 7: Enforce citations and guardrails

Parse the response for citation tags. If the model answers without a citation, reject and fall back to a human handoff message. Also redact obvious PII patterns before logging.

import re

def validate(response, expected_slugs):
    cites = re.findall(r"\[([^\]]+)#(\d+)\]", response)
    if not cites:
        return False, "no citations"
    for slug, _ in cites:
        if not any(slug.startswith(e) for e in expected_slugs):
            return False, f"bad citation {slug}"
    return True, "ok"

def redact(text):
    return re.sub(r"\b\d{3}-\d{2}-\d{4}\b", "[SSN]", text)

This single check cuts hallucinated policy claims dramatically.

Step 8: Verify success with offline eval

You cannot ship blind. Build a golden set of 50 real tickets with known answers. Run the pipeline and score with citation recall and answer similarity.

def eval_run(golden):
    hits = 0
    for item in golden:
        resp = answer(item["query"], product=item.get("product"))
        ok, _ = validate(resp, [item["slug"]])
        if ok and item["answer_snippet"] in resp:
            hits += 1
    return hits / len(golden)

# accuracy = eval_run(golden)

Define citation recall as: of the chunks the human used to answer, how many were retrieved. Aim for >80% before launch. Track per-token cost via your gateway’s metering to keep margins sane.

Step 9: Automate incremental indexing

Full re-embedding wastes money. Hash each chunk; only embed new or changed hashes. Run the job nightly and on docs merge.

#!/usr/bin/env bash
# cron: 0 3 * * * /opt/support/ingest.sh
python ingest.py --since $(date -d '1 day ago' +%s)

The ingest script should upsert, not append, using chunk id as primary key.

Operational notes

Cache embeddings for unchanged chunks. Forward provider cache-control hints if your gateway supports it to avoid re-embedding stable docs. Log every retrieval score; a low score means the question is out of scope and should route to a human.

The cheapest way to train AI support agent help docs is retrieval, not fine-tuning. You iterate on chunking and prompts in hours, not weeks.