LlamaIndex memory modules for long-running agents

Long-running agents accumulate context faster than most LLM context windows allow, and naive chat history truncation silently drops state your agent needs. A well-designed llamaindex agent memory stack separates transient buffer, compressed summary, and searchable episodic store so the agent stays coherent across sessions. This guide walks through a concrete ordering of memory modules and the tradeoffs you’ll hit when shipping to production.

1. Start with a token-bounded buffer

The first layer is always a ChatMemoryBuffer. It keeps the most recent messages up to a token limit and discards the rest. This is your working set—what the model sees on every call.

from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=4000)

Set token_limit based on your model’s context window minus room for the system prompt, tools, and response. For a 128k model, 4k–8k tokens of buffer is conservative; for an 8k model, 3k is already tight.

Pitfall: developers set token_limit equal to the full context window. The agent then has no space to reason or emit a long response, and the LLM truncates mid-generation. Leave at least 20% headroom.

2. Add summarization before you need it

A buffer alone loses earlier context the moment it overflows. ChatSummaryMemoryBuffer compresses old messages into a running summary using an LLM, then prepends that summary to the recent buffer.

from llama_index.core.memory import ChatSummaryMemoryBuffer
from llama_index.llms.openai import OpenAI

summary_memory = ChatSummaryMemoryBuffer(
    token_limit=4000,
    llm=OpenAI(model="gpt-4o-mini"),
)

The summarizer runs only when the buffer would exceed its limit, not on every turn. Point the summarization LLM at an OpenAI-compatible endpoint with automatic fallback (e.g., n4n.ai) so a provider outage doesn’t stall your agent’s memory compaction.

Tradeoff: summarization is lossy. Complex numeric state or ordered steps can get blurred. If your agent tracks exact counts or IDs, keep those in a separate structured store, not in free-text summary.

3. Offload old facts to a vector store

When the agent needs to recall specific past events—“what did the user say about invoices last week?”—a summary is insufficient. VectorMemory embeds each message and retrieves the top-k most similar ones at query time.

from llama_index.core import VectorStoreIndex
from llama_index.core.memory import VectorMemory

index = VectorStoreIndex.from_documents([])
vector_memory = VectorMemory(
    index=index,
    retriever_kwargs={"similarity_top_k": 3},
)

Attach it as a secondary memory (see next section) or call vector_memory.get() manually inside a tool. Use a low similarity_top_k (2–4) to avoid context pollution.

Pitfall: embedding every chat turn adds latency and cost. Batch writes every N turns instead of per-message if your agent is high-throughput. Also, vector recall is only as good as your embedding model’s grasp of conversational semantics—generic text embeddings often miss pronoun resolution.

4. Compose modules for layered recall

A single memory type forces a compromise. SimpleComposableMemory lets you chain a primary (always-injected) memory with a secondary (retrieved) memory.

from llama_index.core.memory import SimpleComposableMemory

composite = SimpleComposableMemory(
    primary_memory=summary_memory,
    secondary_memory=vector_memory,
)

The primary memory (buffer + summary) gives the model immediate continuity. The secondary memory injects retrieved episodes only when relevant. This pattern is the core of a robust llamaindex agent memory system for agents that run for days.

Wire it into an agent:

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o")
agent = ReActAgent.from_tools(
    tools=[],
    memory=composite,
    llm=llm,
)

5. Persist and isolate concurrent sessions

Memory objects are in-memory by default. For long-running agents you must serialize per session. Use the to_dict / from_dict methods or back the vector index with a persistent store (Postgres, Redis, Pinecone).

# save
state = composite.to_dict()
# later, in another process
restored = SimpleComposableMemory.from_dict(state)

Critical pitfall: never share one memory instance across user IDs. Key your persistence layer by tenant_id:session_id. A common bug is a global agent variable in a FastAPI app that leaks conversation A into conversation B.

If you use async, guard writes with a lock per session. LlamaIndex memory classes are not thread-safe by default.

6. Production tradeoffs: latency, cost, pollution

Every memory layer has a tax:

• Buffer: zero extra latency, but blind to old context.
• Summary: one extra LLM call per overflow event; compresses nuance.
• Vector: retrieval latency (10–100ms) plus embedding cost per message.
• Composite: combines all taxes; needs careful prompt templating to keep the model from confusing summary with live recall.

Context pollution is the silent failure mode. If you inject 3 summarized turns + 3 vector hits + 10 buffer messages, the model may overweight the wrong source. Log the exact memory payload on each step in staging to inspect what the agent actually sees.

7. Pre-flight checklist

Before you ship a long-running agent with llamaindex agent memory:

• token_limit leaves ≥20% headroom in the context window.
• Summarizer LLM has fallback routing; a single provider dependency will break compaction.
• Vector memory uses similarity_top_k ≤4 and batches writes.
• Composite primary/secondary split matches agent’s hot vs cold data needs.
• Persistence keyed by tenant + session; restored on cold start.
• Memory payload logged in a staging run to verify no PII leakage across users.
• Load test with 10x expected session length to confirm summarization frequency stays bounded.

8. When to drop a layer

Not every agent needs all three modules. A short customer-support bot can live on buffer + summary. A personal research assistant that quotes prior sessions needs vector recall. Add layers only when the previous one demonstrably loses required state—each layer is a recurring cost, not a free upgrade.

The right llamaindex agent memory design is the smallest stack that survives your agent’s actual session length without silent context loss. Measure overflow rates in production, then tune.