Multi-turn RAG conversations in LlamaIndex

Multi-turn RAG conversations require more than stitching a retriever to an LLM. You need conversation memory that fits in context, retrieval that respects dialogue history, and a chat engine that doesn’t hallucinate when the user asks “what about the second one?” three turns in. This guide walks through the LlamaIndex components that make it work, the configuration decisions that matter, and the failure modes you’ll hit in production.

Choose the right chat engine

LlamaIndex ships with several chat engines. For RAG, you almost always want CondensePlusContextChatEngine or ContextChatEngine. The difference: CondensePlusContextChatEngine rewrites the user’s latest message into a standalone query using conversation history, then retrieves. ContextChatEngine stuffs the entire conversation (or a truncated window) into the system prompt alongside retrieved nodes.

from llama_index.core.chat_engine import CondensePlusContextChatEngine
from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=3000)

chat_engine = CondensePlusContextChatEngine.from_defaults(
    retriever=index.as_retriever(similarity_top_k=4),
    memory=memory,
    system_prompt=(
        "You are a support agent for Acme Corp. "
        "Answer only from the provided context. "
        "If the answer isn't in context, say you don't know."
    ),
    verbose=True,
)

CondensePlusContextChatEngine is the safer default. The condensation step prevents the retriever from matching on conversational filler (“yeah,” “the second one,” “that thing you mentioned”). ContextChatEngine works when your corpus is small and you can afford the token budget to pass full history to the model — but it degrades fast as conversations grow.

Configure memory with intention

ChatMemoryBuffer is the simplest memory implementation. It keeps the last N tokens of conversation history. The token_limit parameter is your primary lever. Set it too low and the model forgets early turns. Set it too high and you crowd out retrieved context.

from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core.llms import OpenAI

llm = OpenAI(model="gpt-4o-mini", temperature=0)

# Reserve ~1500 tokens for retrieved nodes + system prompt + response
# Adjust based on your model's context window and typical chunk sizes
memory = ChatMemoryBuffer.from_defaults(
    token_limit=2500,
    llm=llm,  # used for token counting
)

For longer conversations, swap to ChatSummaryMemoryBuffer. It summarizes older turns instead of dropping them. The tradeoff: summarization adds latency and can lose nuance.

from llama_index.core.memory import ChatSummaryMemoryBuffer

memory = ChatSummaryMemoryBuffer.from_defaults(
    token_limit=4000,
    llm=llm,
    summary_prompt="Summarize the key facts and decisions from this conversation.",
)

Pitfall: summarization runs on every turn after the buffer fills. If you have high QPS, the summarization calls become a bottleneck. Consider async summarization or a background job that periodically compresses history.

Make retrieval conversation-aware

The condenser prompt in CondensePlusContextChatEngine controls how the user’s latest message gets rewritten. The default works for simple cases. For domain-specific language (pronouns, acronyms, implicit references), customize it.

from llama_index.core.prompts import PromptTemplate

condense_prompt = PromptTemplate(
    "Given the conversation history and a follow-up question, "
    "rewrite the question as a standalone search query.\n"
    "Preserve entity names, product codes, and technical terms.\n"
    "Do not answer the question.\n\n"
    "Conversation history:\n{chat_history}\n\n"
    "Follow-up question: {question}\n\n"
    "Standalone query:"
)

chat_engine = CondensePlusContextChatEngine.from_defaults(
    retriever=index.as_retriever(similarity_top_k=6),
    memory=memory,
    condense_prompt=condense_prompt,
    system_prompt=system_prompt,
)

Test your condenser with real conversation logs. Common failure: the condenser drops negation (“not the red one” → “the red one”) or merges distinct entities (“compare A and B” → “A B”). Add few-shot examples to the prompt if needed.

Handle the “no relevant context” case

RAG systems hallucinate when retrieval returns nothing useful. Your system prompt must instruct the model to refuse, and your application code should detect low-confidence retrieval.

from llama_index.core.response_synthesizers import CompactAndRefine
from llama_index.core.postprocessor import SimilarityPostprocessor

# Filter nodes below similarity threshold before synthesis
retriever = index.as_retriever(similarity_top_k=6)
retriever = SimilarityPostprocessor(similarity_cutoff=0.72)

chat_engine = CondensePlusContextChatEngine.from_defaults(
    retriever=retriever,
    memory=memory,
    system_prompt=(
        "You are a support agent. Answer only from provided context. "
        "If context is insufficient, respond exactly: "
        "'I don't have enough information to answer that.'"
    ),
    response_synthesizer=CompactAndRefine(),
)

The SimilarityPostprocessor cutoff is empirical. Start at 0.7 for cosine similarity with OpenAI embeddings, then tune against a labeled eval set. Log every query where all nodes fall below cutoff — those are your coverage gaps.

Stream responses for perceived latency

Users tolerate total latency if they see tokens arriving. LlamaIndex supports streaming via the stream_chat method.

# Sync streaming
response = chat_engine.stream_chat("What's the refund policy for enterprise plans?")
for token in response.response_gen:
    print(token, end="", flush=True)

# Async streaming (FastAPI, etc.)
async for token in chat_engine.astream_chat("What's the refund policy?"):
    await websocket.send_text(token)

Streaming works with CondensePlusContextChatEngine, but note: the condensation step runs before streaming starts. You’ll see a pause (condense → retrieve → synthesize) then tokens flow. For lower perceived latency, run condensation and retrieval in parallel with a custom pipeline — but that’s a separate architecture.

Persist memory across sessions

ChatMemoryBuffer is in-memory. For production, you need persistence. LlamaIndex doesn’t include a built-in persistent memory store, but the pattern is straightforward: serialize the message list to your database (Postgres, Redis, DynamoDB) and rehydrate on session resume.

import json
from llama_index.core.base.llms.types import ChatMessage, MessageRole

def save_memory(memory: ChatMemoryBuffer, session_id: str, redis_client):
    messages = [
        {"role": msg.role.value, "content": msg.content}
        for msg in memory.get()
    ]
    redis_client.setex(f"chat:{session_id}", 86400, json.dumps(messages))

def load_memory(session_id: str, redis_client, llm, token_limit=2500):
    data = redis_client.get(f"chat:{session_id}")
    if not data:
        return ChatMemoryBuffer.from_defaults(token_limit=token_limit, llm=llm)
    
    messages = [
        ChatMessage(role=MessageRole(msg["role"]), content=msg["content"])
        for msg in json.loads(data)
    ]
    memory = ChatMemoryBuffer.from_defaults(token_limit=token_limit, llm=llm)
    memory.set(messages)
    return memory

Store the condensed query alongside raw messages if you want to debug retrieval failures later. TTL the session data — 24 hours is typical for support chat, longer for coding assistants.

Evaluate with conversation-level metrics

Single-turn RAG eval (faithfulness, answer relevance) misses multi-turn failures: context drift, entity confusion, contradictory answers across turns. Build a conversation eval harness.

from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator
from llama_index.core.llms import OpenAI

eval_llm = OpenAI(model="gpt-4o", temperature=0)
faithfulness = FaithfulnessEvaluator(llm=eval_llm)
relevancy = RelevancyEvaluator(llm=eval_llm)

def evaluate_conversation(chat_engine, turns: list[str], expected_answers: list[str]):
    results = []
    for i, (user_msg, expected) in enumerate(zip(turns, expected_answers)):
        response = chat_engine.chat(user_msg)
        
        faith = faithfulness.evaluate_response(response=response)
        rel = relevancy.evaluate_response(query=user_msg, response=response)
        
        results.append({
            "turn": i,
            "query": user_msg,
            "response": str(response),
            "expected": expected,
            "faithfulness": faith.passing,
            "relevancy": rel.passing,
            "faithfulness_score": faith.score,
            "relevancy_score": rel.score,
        })
    return results

Run this against a golden set of 20-50 multi-turn conversations. Track faithfulness and relevancy per turn, plus aggregate conversation-level pass rate. A system that scores 0.9 on single-turn but 0.6 on turn 3+ has a memory or condensation problem.

Common pitfalls and fixes

Pitfall: Condenser hallucinates entities. The condenser LLM invents product names or specs not in the user’s message. Fix: lower condenser temperature to 0, add “Do not add information not in the question” to the prompt, or switch to a smaller/faster model just for condensation.

Pitfall: Memory buffer grows unbounded. You set token_limit but forgot that ChatMemoryBuffer counts tokens with the LLM’s tokenizer. If you swap models (e.g., gpt-4o-mini → gpt-4o), token counts shift. Fix: always pass the same llm instance to ChatMemoryBuffer that you use for chat, or use a fixed tokenizer.

Pitfall: Retrieval ignores conversation context. User asks “what about the blue one?” — condenser works, but retriever still matches “blue” chunks from a different product line. Fix: add a reranker (Cohere Rerank, bge-reranker) that sees the condensed query and the previous turn’s retrieved nodes. Or embed the condensed query with a context-aware encoder.

Pitfall: System prompt leaks into memory. If you include the system prompt in ChatMemoryBuffer, it consumes tokens and confuses the condenser. Fix: keep system prompt separate (pass to system_prompt parameter, not memory).

Pitfall: Streaming breaks tool use. If you attach tools to the chat engine (e.g., for API lookups), streaming responses from tool calls requires AgentRunner with streaming=True, not ChatEngine. Different abstraction. Plan for this early if you need tools.

Production checklist

Before shipping:

• Condenser prompt tested on 50+ real conversation logs
• Similarity cutoff tuned on labeled retrieval eval set
• Memory persistence with TTL and encryption at rest
• Conversation eval harness running in CI on every deploy
• Fallback path when condenser fails (catch exception, fall back to raw query)
• Observability: log condensed query, retrieved node IDs, similarity scores, token counts per turn
• Load test: simulate 10-turn conversations at target QPS, measure p99 latency

When to go beyond LlamaIndex defaults

The built-in chat engines cover 80% of use cases. You’ll outgrow them when:

• You need multi-hop reasoning (retrieve → reason → retrieve again within one turn)
• You need dynamic tool selection based on conversation state
• You need human-in-the-loop approval for certain actions
• You need conversation branching (user explores alternative paths)

At that point, build a custom agent loop with AgentRunner or a LangGraph-style state machine. LlamaIndex’s low-level components (retrievers, synthesizers, memory, LLMs) compose cleanly. The chat engines are convenient wrappers, not architectural ceilings.

Start with CondensePlusContextChatEngine, ChatMemoryBuffer, and a tuned similarity cutoff. Add summarization memory only when conversations regularly exceed your token budget. Evaluate at the conversation level, not the turn level. The rest is iteration.