Build a customer support bot with LangChain

If you want to build customer support bot langchain style, you need a retrieval-augmented pipeline that grounds answers in your own documentation and keeps track of the conversation. This tutorial walks through a production-shaped implementation using LangChain, Chroma for vector storage, and an OpenAI-compatible chat model. We’ll ingest a small FAQ, wire up retrieval, add memory, and run end-to-end queries with expected output at each checkpoint.

Prerequisites

• Python 3.10 or newer
• pip install langchain langchain-openai langchain-community chromadb tiktoken python-dotenv
• An API key for an OpenAI-compatible endpoint. If you want a single endpoint that fronts 240+ models with automatic fallback when a provider is degraded, point LangChain at n4n.ai’s OpenAI-compatible URL. Otherwise use OpenAI directly.
• A .env file:

OPENAI_API_KEY=sk-...
# Or for the gateway:
# N4N_API_KEY=...

Create a project folder and a faq.md with realistic support content:

# Billing
## How do I upgrade my plan?
Go to Settings > Billing and click "Upgrade". Prorated charges apply immediately.

## Can I get a refund?
Refunds are issued within 14 days of purchase if no API calls were made.

# Technical
## Why am I getting 429 errors?
Rate limits depend on your tier. Inspect the `x-ratelimit-remaining` header.

Ingest and chunk your docs

LangChain’s text splitter handles markdown reasonably with a separator that respects headers.

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter

loader = TextLoader("faq.md")
docs = loader.load()

splitter = CharacterTextSplitter(
    separator="\n## ",
    chunk_size=400,
    chunk_overlap=40,
    keep_separator=True,
)
chunks = splitter.split_documents(docs)
print(f"Created {len(chunks)} chunks")

Expected output:

Created 4 chunks

Build the retrieval layer

We embed with OpenAI embeddings and store in Chroma. For a support bot, cosine similarity on 1536-dim vectors is sufficient.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

Sanity-check retrieval before involving the LLM:

hits = retriever.invoke("How do I upgrade?")
for h in hits:
    print(h.page_content[:80])

Expected output (truncated):

# Billing
## How do I upgrade my plan?
Go to Settings > Billing and click "Upgrade". Prorated charges apply immediately.

Wire up the LLM and system prompt

Use ChatOpenAI with a tight system prompt that forces grounded answers. If you’re using the gateway, set base_url and api_key accordingly.

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
    # For n4n.ai: base_url="https://api.n4n.ai/v1", api_key=os.getenv("N4N_API_KEY")
    api_key=os.getenv("OPENAI_API_KEY"),
)

system_prompt = """You are a customer support agent. Use ONLY the provided context to answer.
If the answer is not in the context, say "I don't have that information."
Context: {context}
"""

Compose the RAG chain with LCEL

We use create_stuff_documents_chain and pipe the retriever. This is the core pattern when you build customer support bot langchain applications with RAG.

from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains import create_retrieval_chain

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
])

document_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, document_chain)

Run a single turn:

response = rag_chain.invoke({"input": "How do I upgrade my plan?"})
print(response["answer"])

Expected output:

Go to Settings > Billing and click "Upgrade". Prorated charges apply immediately.

Add conversation memory

A support bot that forgets the prior turn is useless. Wrap the chain with RunnableWithMessageHistory using an in-memory store (swap for Redis in production).

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

store = {}

def get_history(session_id: str):
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

conversational_rag = RunnableWithMessageHistory(
    rag_chain,
    get_history,
    input_messages_key="input",
    history_messages_key="chat_history",
)

We need to modify the prompt to accept chat_history. Update the template:

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("placeholder", "{chat_history}"),
    ("human", "{input}"),
])
document_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, document_chain)
conversational_rag = RunnableWithMessageHistory(
    rag_chain,
    get_history,
    input_messages_key="input",
    history_messages_key="chat_history",
)

Now test multi-turn:

cfg = {"configurable": {"session_id": "user-1"}}
print(conversational_rag.invoke({"input": "I'm on free tier."}, config=cfg)["answer"])
print(conversational_rag.invoke({"input": "Can I refund that?"}, config=cfg)["answer"])

Expected output:

I don't have that information.
I don't have that information.

(The FAQ doesn’t mention free tier or refund eligibility for it, so the bot correctly abstains.)

Run the bot end-to-end

Here is the full script consolidated. It is runnable after pip install and env setup.

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain.chains import create_retrieval_chain
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

load_dotenv()

loader = TextLoader("faq.md")
docs = loader.load()
splitter = CharacterTextSplitter(separator="\n## ", chunk_size=400, chunk_overlap=40, keep_separator=True)
chunks = splitter.split_documents(docs)

embeddings = OpenAIEmbeddings()
vs = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")
retriever = vs.as_retriever(search_kwargs={"k": 2})

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0, api_key=os.getenv("OPENAI_API_KEY"))

system_prompt = """You are a customer support agent. Use ONLY the provided context to answer.
If the answer is not in the context, say "I don't have that information."
Context: {context}"""

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("placeholder", "{chat_history}"),
    ("human", "{input}"),
])

doc_chain = create_stuff_documents_chain(llm, prompt)
rag = create_retrieval_chain(retriever, doc_chain)

store = {}
def get_history(sid):
    return store.setdefault(sid, InMemoryChatMessageHistory())

bot = RunnableWithMessageHistory(rag, get_history, input_messages_key="input", history_messages_key="chat_history")

cfg = {"configurable": {"session_id": "u1"}}
print(bot.invoke({"input": "Why 429 errors?"}, config=cfg)["answer"])
print(bot.invoke({"input": "What header shows limit?"}, config=cfg)["answer"])

Expected output:

Rate limits depend on your tier. Inspect the `x-ratelimit-remaining` header.
The `x-ratelimit-remaining` header shows your remaining rate limit.

Production considerations

When you build customer support bot langchain systems for real traffic, hardcode none of the operational details. Externalize the model name and temperature to config. Use a persistent ChatMessageHistory backed by your DB. Add a relevance score threshold on the retriever; if the top hit is below 0.7 cosine similarity, skip the LLM and return a fallback message.

Cache embeddings aggressively—Chroma persists them, but warm the index at startup. For multi-tenant support, namespace the vector collection per account. Finally, instrument the chain with LangSmith or OpenTelemetry so you can see which context chunks drove each answer; that’s how you debug hallucations that slip through the prompt guard.

The pattern to build customer support bot langchain with RAG and memory is now complete: load, chunk, embed, retrieve, ground, remember. Swap the vector store for pgvector if you need SQL joins on metadata, or swap the LLM endpoint to a gateway that handles provider failover without code changes.