This llamaindex condensepluscontext chatbot tutorial walks through building a stateful chat engine that compresses prior turns into a concise context before hitting the model. CondensePlusContext solves the context-window bloat problem that naive chat history approaches hit in production, where you either truncate and lose facts or overflow the prompt.
Prerequisites
- Python 3.10 or newer installed locally with pip
- A working API key for an OpenAI-compatible LLM endpoint (OpenAI, or a gateway like n4n.ai)
- Basic familiarity with Python, environment variables, and async/await
llama-index>= 0.10 (the codebase was modularized; we install only what we need)
If you plan to follow exactly, export two environment variables before running any code:
export OPENAI_API_KEY="sk-..."
# Optional: point at a gateway instead of OpenAI directly
export LLM_BASE_URL="https://api.openai.com/v1"You should also create a fresh virtual environment. LlamaIndex’s split packages will conflict with the old monolithic llama-index install if both are present.
Step 1: Install dependencies
The chat engine lives in llama-index-core, but the OpenAI LLM and embedding adapters are separate distributions. Install the minimal set:
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openaiExpect a few transitive pulls (pydantic, tiktoken, numpy). The install is typically under 50 MB.
Step 2: Configure the model and embeddings
CondensePlusContext needs an LLM for two distinct jobs: rewriting history into a standalone question, and generating the final answer. It also needs an embedding model if you retrieve context from an index, which we do.
import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
llm = OpenAI(
model="gpt-4o-mini",
api_key=os.environ["OPENAI_API_KEY"],
api_base=os.environ.get("LLM_BASE_URL"), # defaults to OpenAI
temperature=0,
)
embed_model = OpenAIEmbedding(
model="text-embedding-3-small",
api_key=os.environ["OPENAI_API_KEY"],
)
Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512Settings is a global singleton in LlamaIndex core. Setting it once avoids passing the LLM into every object. If you route through n4n.ai, set api_base to its OpenAI-compatible endpoint and pick any of the 240+ available models by name—the gateway forwards provider cache-control hints and handles fallback when a backend is degraded.
Step 3: Create a small knowledge base
The chat engine shines when it retrieves relevant nodes per turn. We’ll build an in-memory vector index from three short HR documents so the example runs without external services.
from llama_index.core import Document, VectorStoreIndex
docs = [
Document(text="The company 401k match is 5% of salary, vested immediately."),
Document(text="PTO accrues at 1.5 days per month, capped at 30 days."),
Document(text="Parental leave is 16 weeks paid for all genders, starting after birth or adoption."),
]
index = VectorStoreIndex.from_documents(docs)Building the index triggers embedding of each document. With three tiny docs this is near-instant. In a real system you would persist the index to disk or a vector DB via storage_context.
Step 4: Instantiate CondensePlusContextChatEngine
The engine wraps a retriever (or query engine). It uses a condense_prompt to merge chat history with the latest user message into a standalone query, runs retrieval, then feeds the original history plus retrieved context to the LLM via a context_prompt.
from llama_index.core.chat_engine import CondensePlusContextChatEngine
chat_engine = CondensePlusContextChatEngine.from_defaults(
index=index,
retriever_kwargs={"similarity_top_k": 2},
verbose=True,
)verbose=True prints the condensed question and retrieved nodes, which is invaluable when debugging why an answer ignored earlier context. The engine maintains chat_engine.chat_history as a list of ChatMessage objects.
Step 5: Run a multi-turn conversation
Call chat() synchronously. The first turn establishes context; the second references it without repeating entities.
response = chat_engine.chat("What is the 401k match?")
print(str(response))Expected output (abridged):
Condensed question: What is the 401k match?
Retrieved 2 nodes.
The company offers a 401k match of 5% of salary, vested immediately.Now a follow-up that depends on the prior turn:
response = chat_engine.chat("How does that compare to parental leave weeks?")
print(str(response))Expected output:
Condensed question: What is the 401k match and how does it compare to parental leave weeks?
Retrieved 2 nodes.
The 401k match is 5% of salary, while parental leave is 16 weeks paid. These are different benefits; the match is a percentage of pay, leave is time off.Notice the condensed question fused both intents. That’s the core mechanism: the LLM rewrites the dialogue so retrieval stays relevant even when the user says “that” or “it”.
A third turn shows the history buffer persisting:
response = chat_engine.chat("And the PTO cap?")
print(str(response))Expected output:
Condensed question: What is the PTO cap and how does it relate to the 401k and parental leave?
Retrieved 2 nodes.
PTO is capped at 30 days. This is unrelated to the 401k match (5% of salary) or parental leave (16 weeks paid).Step 6: Inspect the prompt plumbing
If you need to customize behavior—say, force the condenser to preserve numeric values—override the prompts.
from llama_index.core.prompts import PromptTemplate
condense_prompt = PromptTemplate(
"Given the conversation:\n{chat_history}\n"
"Rewrite the user's last message into a standalone question.\n"
"Last message: {question}\nStandalone:"
)
context_prompt = PromptTemplate(
"You are a HR assistant. Use context:\n{context_str}\n"
"Chat history:\n{chat_history}\n"
"Question: {question}\nAnswer:"
)
chat_engine = CondensePlusContextChatEngine.from_defaults(
index=index,
condense_prompt=condense_prompt,
context_prompt=context_prompt,
retriever_kwargs={"similarity_top_k": 2},
)Re-running the same turns now uses your phrasing. The chat_history variable is automatically formatted as alternating Human: and Assistant: lines by LlamaIndex. The context_str variable contains the concatenated retrieved node texts.
Step 7: Streaming and async
Production chat UIs need tokens streamed. The engine supports stream_chat:
stream = chat_engine.stream_chat("Summarize all benefits in one sentence.")
for token in stream.response_gen:
print(token, end="")Async variants (achat, astream_chat) exist for FastAPI or any asyncio loop. They share the same internal state, so mixing sync and async on the same engine instance will corrupt the history buffer—keep one mode per instance.
Step 8: Resetting and persisting state
The engine holds messages in chat_engine.chat_history. To start a new session:
chat_engine.reset()
assert len(chat_engine.chat_history) == 0If you need to persist across restarts, serialize the list of ChatMessage objects:
import json
from llama_index.core.llms import ChatMessage
def dump_history(engine):
return [m.dict() for m in engine.chat_history]
def load_history(engine, data):
engine.chat_history = [ChatMessage.parse_obj(d) for d in data]Store the JSON in Redis or a signed cookie; the condensed query is stateless on the server side because it’s recomputed each turn from the full history.
Debugging common failures
Condensed question loses a constraint. The default condense prompt is short. If a user says “only for part-time staff” in turn one and the condenser drops it, increase the history passed to the condense step or write a stricter prompt as shown in Step 6.
Retriever returns irrelevant nodes. Lower similarity_top_k to 1 or inspect embedding distances. With tiny corpora, cosine similarity can be noisy.
Extra latency. Every turn makes two LLM calls (condense + answer). If your gateway meters per token, this doubles cost on short queries. Cache the condensed question when the chat history hash is unchanged.
Production considerations
CondensePlusContext adds one extra LLM call per turn (the condenser). At scale that doubles your token spend on small queries. Cache the condensed question when the chat history hash hasn’t changed—many gateways already honor cache-control hints on the upstream request.
The retriever top-k should be tuned to your embedding quality. With similarity_top_k=2 we kept prompts small; bump it to 4–6 for sparse documents.
Finally, the default condenser can drop nuanced constraints from long dialogues. If you see answers that ignore a constraint stated three turns ago, either increase the history window passed to the condense prompt or switch to a CondenseQuestionChatEngine plus explicit memory. This llamaindex condensepluscontext chatbot tutorial deliberately uses the context-inclusive variant because it keeps the original history in the final prompt as a safety net.
Where to go next
Wire this engine behind a WebSocket handler, add a tool-calling layer with QueryEngineTool, or swap the vector index for a SQL retriever. The chat engine interface stays identical; only the index argument changes. For a deeper look at memory patterns, read the LlamaIndex docs on ChatMemoryBuffer and the CondensePlusContextChatEngine source.