LlamaIndex chat engines explained: context vs condense

This llamaindex chat engine tutorial puts the two built-in conversation wrappers head to head: ContextChatEngine and CondenseChatEngine. They solve the same problem—multi-turn Q&A over an index—but diverge sharply on how they feed the LLM. Pick wrong and you either burn tokens or lose conversational context.

What each engine does with your chat history

LlamaIndex ships three chat engines in llama_index.core.chat_engine. The two most used are ContextChatEngine and CondenseChatEngine. The third, CondensePlusContextChatEngine, is a hybrid we’ll mention for completeness.

ContextChatEngine

On every chat() call, this engine retrieves nodes from the index using the raw latest user message, then builds a prompt containing:

1. A system prompt
2. The retrieved context nodes
3. The entire chat history from memory

The LLM sees everything that has been said. No summarization occurs.

from llama_index.core import VectorStoreIndex, ChatMemoryBuffer
from llama_index.core.chat_engine import ContextChatEngine

memory = ChatMemoryBuffer.from_defaults(token_limit=4000)
index = VectorStoreIndex.from_documents(docs)

engine = ContextChatEngine.from_defaults(
    index=index,
    memory=memory,
    system_prompt="You are a contract analysis assistant.",
)

# Turn 1
resp1 = engine.chat("What did the contract say about liabilities?")
# Turn 2 — full history + new retrieval sent to LLM
resp2 = engine.chat("And what about indemnification clauses?")

CondenseChatEngine

This engine inserts a rewriting step before retrieval. It calls the LLM once to compress the chat history plus the latest user message into a standalone question, then retrieves against that rewritten query. The main answer call receives only the system prompt, retrieved context, and the condensed query—not the raw history.

from llama_index.core.chat_engine import CondenseChatEngine

engine = CondenseChatEngine.from_defaults(
    index=index,
    memory=memory,
    system_prompt="You are a contract analysis assistant.",
    condense_prompt="Given the conversation and a follow-up question, rephrase as a standalone question.",
)

# Turn 1: history empty, condense step passes query through
resp1 = engine.chat("What did the contract say about liabilities?")
# Turn 2: condense step rewrites "And indemnification?" using history
resp2 = engine.chat("And what about indemnification clauses?")

CondensePlusContextChatEngine

It runs the condense step for retrieval but then sends both the condensed query and the full chat history to the answer LLM. Useful when you want tighter retrieval but can’t afford to drop prior turns from the generation context.

Capabilities

ContextChatEngine preserves exact wording. If a user pastes a clause and later says “summarize that,” the engine still has the clause in the prompt. It handles anaphora (“it”, “that”) natively because the LLM sees the antecedent.

CondenseChatEngine forces the LLM to resolve anaphora during the condense step. If the rewrite is accurate, retrieval improves because the query is self-contained. But any nuance—tone, constraints stated earlier—is lost unless repeated in the condensed string.

For agentic flows that call tools mid-conversation, Context is safer because tool outputs stay in the visible transcript. Condense can drop them if they aren’t captured in the rewrite.

Cost model

Token accounting is the clearest differentiator. Assume a 2k-token context window for retrieved nodes and a 1k-token system prompt.

• Context: Per-turn input tokens ≈ system (1k) + context (2k) + history (grows). After 10 turns of ~200 tokens each, history is 2k, so input is ~5k. After 30 turns, ~9k. You pay for the full accumulation on every call.
• Condense: Per-turn input tokens ≈ system (1k) + context (2k) + condensed_query (~50 tokens) for the answer call, plus a small condense call of history + last message → ~200 tokens. Total per turn stays near 3.3k regardless of session length.

If you point LlamaIndex’s OpenAILike LLM at an OpenAI-compatible inference gateway such as n4n.ai, per-token metering still applies; condensation directly cuts billed input tokens on the primary completion without changing engine code.

Output tokens are similar for both—the answer length is driven by the question, not the history.

Latency and throughput

Context engines suffer from longer prefill. A 9k-token prompt decodes slower and delays time-to-first-token versus a 3k-token prompt. Under concurrent load, that prefill dominates queue time.

Condense adds a sequential dependency: condense call → retrieval → answer call. On a cold start with a small history, Context is faster (one call). Past ~4 turns, the Condense pipeline usually wins because the answer call’s prefill shrinks. Throughput on a shared GPU pool improves because smaller prompts free the batch scheduler.

Ergonomics

Both expose from_defaults() and accept the same memory, llm, and retriever arguments. The difference is operational:

• Context needs a ChatMemoryBuffer with a sane token_limit or it will blow the model context window silently (LlamaIndex truncates from the left).
• Condense requires a working condense_prompt. The default is fine for English QA but fails on code or structured data. Override it:

from llama_index.core.prompts import PromptTemplate

condense_prompt = PromptTemplate(
    "Given the chat log:\n{chat_history}\n"
    "And the follow-up: {question}\n"
    "Output a standalone SQL question, no commentary."
)
engine = CondenseChatEngine.from_defaults(
    index=index, memory=memory, condense_prompt=condense_prompt
)

Context is easier to debug: dump the prompt and you see the whole conversation. Condense is opaque—you must log the condensed query separately to know what was retrieved.

Ecosystem and integration

Both engines sit on BaseChatEngine and work with any BaseIndex (vector, tree, keyword). They accept any LLM implementing the LLM interface, including OpenAI, AzureOpenAI, OpenAILike, and local Ollama.

Memory is pluggable. ChatMemoryBuffer is the default; you can swap in a SimpleComposableMemory if you need per-user isolation. Streaming works identically: engine.stream_chat() returns an async generator for both.

Condense’s extra call is just another llm.predict(); it inherits the same timeout and retry config as the main call. No special middleware required.

Limits

ContextChatEngine

• Hard ceiling at the model’s context window. Truncation drops oldest turns first—bad for “remember the first instruction.”
• Retrieval uses the raw last message. If the user says “compare that to the earlier one,” the retriever sees only “compare that to the earlier one” and may fetch garbage.

CondenseChatEngine

• The condense step can hallucinate or over-compress. “What about the limit in section 4?” might become “What is the liability limit?” dropping the section reference.
• Two LLM calls mean two failure points. If the condense call errors, the engine raises before retrieval.
• Not suitable when the answer must quote exact prior user input (e.g., “repeat my earlier regex”).

Head-to-head comparison

Which to choose

Use ContextChatEngine when:

• Sessions are short (≤8 turns) and you control memory limits.
• The task requires verbatim recall of earlier user messages (legal review, spec authoring).
• You run tool calls or function agents inside the chat loop.
• Retrieval quality on the raw last message is already good.

Use CondenseChatEngine when:

• You expect long-running assistants (support bots, research helpers) with dozens of turns.
• Token budget is tight and you pay per input token.
• Follow-up questions are linguistically simple (“what about X?”, “why?”) and the condense prompt can capture intent.
• You can tolerate occasional rewrite drift and have logging to catch it.

Use CondensePlusContextChatEngine when:

• You need condensed retrieval but cannot drop prior turns from generation (complex multi-step reasoning).
• Your context window is large (32k+) so the hybrid cost is acceptable.

In this llamaindex chat engine tutorial we’ve shown the trade is fundamentally about where the history lives: in the prompt or in a rewrite. Context is the default for a reason—it’s predictable. Condense is the optimization you reach for when the bill or the latency graph tells you to.