LlamaIndex chat memory buffer: how it works

The ChatMemoryBuffer is LlamaIndex’s default conversation memory implementation, and it’s where most engineers first hit token limit errors in production. It maintains a rolling window of messages, optionally summarizing older turns when the buffer exceeds a token budget. Understanding its behavior — especially around token counting, summarization triggers, and message ordering — saves hours of debugging context window overflows.

How the buffer tracks tokens

ChatMemoryBuffer doesn’t count tokens by calling the model’s tokenizer on every insert. Instead, it uses a tokenizer_fn you provide (defaulting to tiktoken for OpenAI models) to estimate token counts for each message. The buffer maintains a running total and evicts oldest messages when adding a new one would exceed token_limit.

from llama_index.core.memory import ChatMemoryBuffer
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")

memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,
    tokenizer_fn=llm.tokenizer,
)

The token_limit should leave headroom for the system prompt, retrieved context, and the model’s response. A common mistake is setting it to the model’s full context window (e.g., 128k for gpt-4o-mini) and then wondering why the request fails — the buffer only manages chat history, not the entire prompt.

Message structure and roles

Each message stored in the buffer is a ChatMessage with a role (system, user, assistant, tool) and content. System messages are never evicted by the buffer — they’re treated as permanent context. This matters when you inject a large system prompt: it consumes token budget but won’t be summarized or dropped.

from llama_index.core.llms import ChatMessage, MessageRole

memory.put(ChatMessage(role=MessageRole.SYSTEM, content="You are a helpful assistant."))
memory.put(ChatMessage(role=MessageRole.USER, content="What's the capital of France?"))
memory.put(ChatMessage(role=MessageRole.ASSISTANT, content="Paris is the capital of France."))

# Inspect current buffer
for msg in memory.get():
    print(f"{msg.role}: {msg.content[:50]}...")

Tool messages (role=tool) are included in token counting and subject to eviction like any other message. If your agent makes many tool calls, those messages can dominate the buffer quickly.

Summarization: when it triggers and what it costs

When token_limit is exceeded, the buffer doesn’t simply drop oldest messages if you’ve enabled summarization. Instead, it calls an LLM to summarize the oldest N messages into a single summary message, then replaces those messages with the summary. This preserves semantic context at the cost of an extra LLM call and latency.

memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,
    tokenizer_fn=llm.tokenizer,
    llm=llm,  # required for summarization
)

The summarization prompt is hardcoded in LlamaIndex (as of 0.10.x) and roughly says: “Summarize the following conversation concisely.” You cannot customize this prompt through the public API — a limitation if you need domain-specific summaries (e.g., preserving entity names in legal or medical chats).

Tradeoff: Summarization adds 1-2 seconds of latency per trigger and consumes additional tokens. For high-throughput chat, consider whether a simple sliding window (no summarization) with a larger token_limit is more predictable.

Common pitfall: double-counting with chat engines

When you pass a ChatMemoryBuffer to a ChatEngine, the engine internally calls memory.get() to retrieve history and prepends it to the prompt. If you also manually call memory.get() and inject messages into your own prompt template, you’ll double-count history — blowing past token limits.

# Correct: let the chat engine manage memory
from llama_index.core.chat_engine import SimpleChatEngine

chat_engine = SimpleChatEngine.from_defaults(
    llm=llm,
    memory=memory,
    system_prompt="You are a helpful assistant.",
)

response = chat_engine.chat("What did I just ask?")

# Wrong: manual injection duplicates history
messages = memory.get()  # already includes system + history
custom_prompt = "\n".join([f"{m.role}: {m.content}" for m in messages])
custom_prompt += "\nuser: What did I just ask?"
response = llm.complete(custom_prompt)  # history sent twice

Persistence and multi-session handling

ChatMemoryBuffer is in-memory by default. For production, you need persistence across restarts and horizontal scaling. LlamaIndex provides ChatMemoryBuffer.from_defaults(persist_dir=...) which serializes to JSON, but this is file-based and unsuitable for multi-instance deployments.

# File-based persistence (single instance only)
memory = ChatMemoryBuffer.from_defaults(
    token_limit=3000,
    tokenizer_fn=llm.tokenizer,
    persist_dir="./memory_storage",
)

# After chat session
memory.persist()

For multi-instance, extract messages via memory.get() and store in Redis, Postgres, or your session store. Rehydrate on startup:

import json
import redis

r = redis.Redis(decode_responses=True)
SESSION_KEY = "chat_memory:user_123"

# Save
def save_memory(memory: ChatMemoryBuffer, session_id: str):
    messages = [msg.model_dump() for msg in memory.get()]
    r.set(f"chat_memory:{session_id}", json.dumps(messages))

# Load
def load_memory(session_id: str, token_limit: int, tokenizer_fn) -> ChatMemoryBuffer:
    data = r.get(f"chat_memory:{session_id}")
    memory = ChatMemoryBuffer.from_defaults(token_limit=token_limit, tokenizer_fn=tokenizer_fn)
    if data:
        messages = [ChatMessage(**m) for m in json.loads(data)]
        for msg in messages:
            memory.put(msg)
    return memory

Note: ChatMessage.model_dump() requires Pydantic v2 (LlamaIndex 0.10+). On older versions, use msg.dict().

Token limit sizing: a practical heuristic

Start with this formula:

token_limit = model_context_window - (system_prompt_tokens + max_retrieved_context_tokens + max_response_tokens + safety_margin)

For a RAG chat engine with gpt-4o-mini (128k context):

• System prompt: ~500 tokens
• Retrieved chunks (top-5, 512 each): ~2,500 tokens
• Max response: ~1,000 tokens
• Safety margin: ~1,000 tokens

token_limit = 128000 - 5000 = 123,000 — but you rarely need that much history. A practical cap of 8,000-16,000 tokens covers ~20-40 turns while leaving ample room for retrieval and response. Larger buffers increase latency (more context to process) and cost.

When to disable summarization

Disable summarization (omit llm parameter) when:

• Latency is critical and you can tolerate a sliding window
• Conversations are short-lived (support chats, single-task agents)
• You need deterministic token usage — summarization output length varies

Enable summarization when:

• Conversations span hours or days (coaching, tutoring, companionship)
• Long-term context matters (user preferences, prior decisions)
• You can absorb the extra LLM call cost

Debugging buffer state

Add a wrapper to log buffer size and composition on each turn:

class DebugChatMemoryBuffer(ChatMemoryBuffer):
    def put(self, message: ChatMessage) -> None:
        super().put(message)
        total_tokens = sum(self.tokenizer_fn(msg.content or "") for msg in self.get())
        print(f"[Memory] messages={len(self.get())}, tokens={total_tokens}, limit={self.token_limit}")

    def get(self, initial_token_count: int = 0) -> list[ChatMessage]:
        msgs = super().get(initial_token_count)
        print(f"[Memory] retrieving {len(msgs)} messages for prompt")
        return msgs

This surfaces issues like runaway tool messages, system prompt bloat, or summarization not triggering when expected.

Integration with chat engines and agents

ChatMemoryBuffer works with all LlamaIndex chat engines (SimpleChatEngine, CondenseQuestionChatEngine, ContextChatEngine, AgentChatEngine). The engine calls memory.get() before each LLM call and memory.put() for user/assistant messages after.

For AgentChatEngine (ReAct agents), tool call/response pairs are automatically added to memory. If your agent loops tools frequently, the buffer fills fast. Consider a dedicated tool_token_limit or periodic manual compaction:

# Manual compaction: summarize first N messages, keep recent K
def compact_memory(memory: ChatMemoryBuffer, keep_recent: int = 10):
    all_msgs = memory.get()
    if len(all_msgs) <= keep_recent:
        return
    to_summarize = all_msgs[:-keep_recent]
    recent = all_msgs[-keep_recent:]

    # Use LLM to summarize
    summary_prompt = "Summarize this conversation:\n" + "\n".join(
        f"{m.role}: {m.content}" for m in to_summarize
    )
    summary_text = llm.complete(summary_prompt).text

    memory.reset()
    memory.put(ChatMessage(role=MessageRole.SYSTEM, content=all_msgs[0].content))  # preserve system
    memory.put(ChatMessage(role=MessageRole.ASSISTANT, content=f"[Summary of earlier conversation]: {summary_text}"))
    for msg in recent:
        memory.put(msg)

Run this on a background schedule or when len(memory.get()) > threshold.

Summary checklist

• Set token_limit with headroom for system prompt, retrieval, and response — not the full context window
• Provide tokenizer_fn matching your model (tiktoken for OpenAI, HF tokenizer for local models)
• Pass llm to enable summarization; omit for sliding-window behavior
• Never manually inject memory.get() into a prompt if the chat engine already uses that memory
• Persist externally (Redis/DB) for multi-instance deployments
• Monitor buffer size in production — log token count per turn
• Compact manually for agent workflows with heavy tool usage

The ChatMemoryBuffer is a thin, well-scoped abstraction. Its defaults work for simple cases; production workloads demand explicit token budgets, persistence strategy, and observability.