Engineers building autonomous systems quickly hit the question of short-term vs long-term agent memory: keep everything in the prompt, or persist knowledge across sessions? The trade-offs are not just token counts; they shape latency, cost, and the ergonomics of your whole stack.
Capabilities
Short-term memory
Short-term memory is the working set you hand the model each call: chat history, scratchpad JSON, recently retrieved docs. It lives in the context window and dies when the session ends or you truncate it. The model sees a coherent immediate state with zero external reads.
class ShortTermBuffer:
def __init__(self, max_tokens=8000):
self.msgs = []
self.max_tokens = max_tokens
def add(self, role, content):
self.msgs.append({"role": role, "content": content})
def trim(self):
while estimate_tokens(self.msgs) > self.max_tokens:
self.msgs.pop(0)This covers episodic state (what the user just said) and working memory (intermediate reasoning steps). It cannot recall anything from yesterday unless you replay it.
Long-term memory
Long-term memory persists across sessions and users. It is usually a vector index plus metadata store, sometimes a graph or SQL table. The agent writes facts, then retrieves later via embedding similarity or structured queries.
def recall(mem_db, query_emb, k=5):
return mem_db.execute(
"SELECT text FROM memories ORDER BY embedding <-> $1 LIMIT $2",
(query_emb, k)
)This enables personalization, cumulative learning, and cross-session continuity. You can store semantic facts (“user prefers TypeScript”), episodic logs (“fixed login bug on 2024-05-01”), or procedural templates.
Cost model
Short-term memory bills by tokens every request. A 10-turn conversation with 4k tokens of history costs those 4k input tokens each call. At scale, that dominates spend. If you run 1M calls/month with 4k input tokens, that is 4B input tokens—pure repetition of prior turns.
Long-term memory shifts cost to storage and retrieval. You pay for embedding generation once per write, DB hosting, and small query reads. An inference gateway like n4n.ai forwards provider cache-control hints, so short-term context that repeats across turns can be cached to cut token costs—but only within a single provider’s window.
Long-term writes are cheap per byte but require pipeline maintenance. If you embed on every turn, you still burn tokens for the embedding model. Storage cost is typically cents per GB-month for vector DBs, but query volume can add up.
Latency and throughput
Short-term memory adds zero network hops beyond the LLM call. The bottleneck is prompt processing: a 32k-token context takes longer to first token than a 2k one, especially on GPUs with quadratic attention. Streaming helps perceived latency but not total compute.
Long-term memory inserts a retrieval step. Typical vector search over 1M rows is 5–20ms on pgvector with an IVFFlat index. The agent then builds a smaller prompt. Net effect: lower generation latency, slightly higher orchestration latency.
Throughput scales differently. Short-term blows up your input token volume, capping requests per minute on rate-limited APIs. Long-term keeps prompts small, raising effective throughput per rate limit bucket.
Ergonomics
A rolling buffer is ten lines of code. Debugging is trivial: print the messages. You can snapshot the exact model input for replay.
Long-term memory needs an embedding endpoint, a migration plan, eviction policy, and conflict resolution. You must decide what to store: raw text, summaries, or structured triples.
# writing to long-term
def remember(mem_db, text, emb):
mem_db.execute(
"INSERT INTO memories(text, embedding) VALUES($1,$2)",
(text, emb)
)That looks simple, but production issues—schema drift, duplicate facts, stale embeddings—will eat time. Testing retrieval quality requires golden sets; you cannot eyeball a vector space.
Ecosystem and tooling
Short-term memory is native to every LLM API. OpenAI, Anthropic, and open-weight servers all accept message arrays. No extra dependency. Prompt caching (where supported) is a config flag.
Long-term memory sits in a crowded ecosystem: LangChain memory abstractions, LlamaIndex, Pinecone, Weaviate, pgvector, Redis. Each adds opinionated serialization. You also need an embedding model; text-embedding-3-small is common, but self-hosted alternatives exist. Frameworks help bootstrap, but lock-in is real.
Limits and failure modes
Short-term memory suffers context rot: the model weights early tokens less, and hard truncation loses critical state. Max window is a hard ceiling (200k tokens on some models, far less on others). Concurrent sessions multiply memory footprint in your orchestrator.
Long-term memory suffers retrieval imprecision. Cosine similarity misses negated or temporal facts (“not in EU” vs “in EU”). Without periodic re-indexing, embeddings drift from current language. Writes are eventually consistent; a just-stored fact may not be queryable immediately. Privacy compliance gets harder when data lives forever.
Head-to-head comparison
| Dimension | Short-term memory | Long-term memory |
|---|---|---|
| Capabilities | Immediate state, scratchpad, no external reads | Cross-session persistence, personalization, shared knowledge |
| Cost model | Per-token input on every call | Storage + embedding writes + small reads |
| Latency | Higher prompt processing, no fetch | +5–20ms retrieval, lower generation time |
| Throughput | Limited by input token rate limits | Higher due to smaller prompts |
| Ergonomics | Trivial buffer, easy debug | Pipeline, schema, eviction complexity |
| Ecosystem | Native to all LLM APIs | Vector DBs, embedding models, frameworks |
| Limits | Context window size, truncation loss | Stale data, retrieval miss, write latency |
Which to choose
Single-session chatbots and tool calls. Use short-term memory. You need no persistence; a trimmed buffer with cache control is enough. Keep the last N turns and a scratchpad. Do not build a vector DB for a toy demo.
Long-running personal assistants. Combine both. Short-term holds the active dialog; long-term stores user preferences and past decisions. Retrieve only on relevant turns to control cost. Write summaries nightly, not every message.
Multi-user enterprise knowledge agents. Long-term memory is mandatory. Isolate per-tenant collections in a vector store. Short-term stays per-session for safety and latency. Apply eviction policies to meet data-retention law.
High-throughput batch processing. Prefer long-term for static reference data loaded once, but keep short-term minimal to avoid token blow-up. Cache embeddings aggressively. If the task is stateless, skip long-term entirely.
The split between short-term vs long-term agent memory is not ideological. It is a systems decision driven by session lifetime, cost ceiling, and retrieval accuracy. Build the smallest memory that meets the task, then extend only when state must survive the process exit.