AI agent memory: short-term vs long-term explained

AI agent memory short-term long-term refers to two distinct storage mechanisms: a transient context window that holds recent conversation history and tool outputs for the current reasoning cycle, and a durable external store that persists facts, preferences, and episodic knowledge across sessions. Short-term memory lives inside the model’s context window and disappears when the conversation ends; long-term memory survives across invocations and requires explicit write and retrieval logic. Understanding this distinction is the prerequisite for building agents that don’t forget what they learned five minutes ago.

How short-term memory works

Short-term memory is the model’s context window — the tokens you stuff into the prompt at inference time. Every user message, assistant reply, tool call, and tool result consumes space. When the window fills, older tokens drop off the front. There is no persistence, no indexing, and no semantic retrieval. The model simply attends to whatever tokens remain.

# Typical short-term memory flow
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Alex."},
    {"role": "assistant", "content": "Nice to meet you, Alex!"},
    {"role": "user", "content": "What's my name?"},
]
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,  # entire history fits in context
)

The context window is fast — no network round trip to a vector database, no embedding step. But it has hard limits: 128k tokens for GPT-4o, 200k for Claude 3.5 Sonnet, 1M+ for Gemini 1.5 Pro. Once you exceed the limit, you must summarize, truncate, or offload.

How long-term memory works

Long-term memory is any durable store outside the context window: a PostgreSQL table, a vector database like Pinecone or Qdrant, a key-value store, or even flat files. The agent writes to it explicitly (or via a tool) and reads from it via retrieval — typically semantic search over embeddings, but also keyword lookup, SQL queries, or graph traversal.

# Conceptual long-term memory write
def remember_fact(user_id: str, fact: str, category: str = "general"):
    embedding = embed(fact)
    db.execute("""
        INSERT INTO memories (user_id, content, embedding, category, created_at)
        VALUES (%s, %s, %s, %s, NOW())
    """, (user_id, fact, embedding, category))

# Conceptual long-term memory read
def recall_relevant(user_id: str, query: str, k: int = 5) -> list[str]:
    q_embedding = embed(query)
    rows = db.execute("""
        SELECT content FROM memories
        WHERE user_id = %s
        ORDER BY embedding <=> %s
        LIMIT %s
    """, (user_id, q_embedding, k))
    return [r[0] for r in rows]

The write path is where most agents fail. You need to decide what to store, when to store it, and how to structure it for later retrieval. A naive “store everything” approach floods the index with noise and makes retrieval worse.

Why the distinction matters

Short-term memory handles the immediate reasoning task: “What did the user just ask?” “What did the tool return?” Long-term memory handles continuity: “What does this user prefer?” “What did we decide last week?” “What facts about the domain are stable?”

Mixing them causes two failure modes. First, stuffing long-term knowledge into the context window burns tokens and hits limits fast. Second, relying only on the context window means the agent has amnesia every session. A production agent needs both, with a clear boundary.

Concrete example: a coding agent

Consider an agent that helps developers navigate a codebase. The short-term memory holds the current task, recent file edits, and the last three terminal outputs. The long-term memory stores:

• User preferences (tabs vs spaces, naming conventions)
• Project-specific patterns (where the API layer lives, how errors are handled)
• Learned facts (“this codebase uses FastAPI, not Flask”)
• Episodic memories (“last Tuesday we refactored auth into a middleware”)

# Agent loop with both memory types
async def run_agent(task: str, user_id: str, session_id: str):
    # 1. Load long-term context relevant to the task
    ltm_context = await recall_relevant(user_id, task, k=8)
    
    # 2. Load short-term session history
    stm_messages = await get_session_history(session_id)
    
    # 3. Build prompt: system + long-term facts + short-term history + task
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        *ltm_context,  # injected as system/user messages
        *stm_messages,
        {"role": "user", "content": task},
    ]
    
    # 4. Run the model with tools
    response = await model_with_tools(messages)
    
    # 5. Persist new long-term facts extracted from this turn
    new_facts = extract_facts(response)
    for fact in new_facts:
        await remember_fact(user_id, fact, category="learned")
    
    # 6. Append to short-term history
    await append_session_history(session_id, response)
    
    return response

The agent reads long-term memory before reasoning, writes to it after reasoning, and treats short-term memory as a rolling buffer. This pattern — retrieve, reason, store — is the canonical loop.

Retrieval strategies for long-term memory

Semantic search over embeddings is the default, but it’s not the only tool. Choose based on what you’re storing:

• Facts and preferences: Exact-match or keyword search on structured fields (user_id + category + key). Embeddings add noise here.
• Episodic memories: Hybrid search — embed the query, filter by time range and tags, then rerank.
• Procedural knowledge: Store as structured skills with preconditions and effects; retrieve by matching the current goal state.
• Codebase knowledge: Use a code-aware indexer (tree-sitter + embeddings) rather than raw text chunks.

# Hybrid retrieval example
def retrieve_memories(user_id: str, query: str, time_window_days: int = 30) -> list[Memory]:
    q_emb = embed(query)
    cutoff = datetime.utcnow() - timedelta(days=time_window_days)
    
    # Vector search with metadata filter
    results = vector_db.query(
        vector=q_emb,
        filter={"user_id": user_id, "created_at": {"$gte": cutoff}},
        top_k=20,
    )
    
    # Rerank with cross-encoder for precision
    pairs = [(query, r.payload["content"]) for r in results]
    scores = cross_encoder.predict(pairs)
    
    ranked = sorted(zip(results, scores), key=lambda x: x[1], reverse=True)
    return [Memory.from_payload(r.payload) for r, _ in ranked[:8]]

Common misconceptions

“Long-term memory is just a bigger context window”

A 1M-token context window is not long-term memory. It doesn’t persist across sessions, it doesn’t support semantic retrieval, and it charges you for every token on every request. Long-term memory is a database with a write path, a read path, and a schema.

“RAG solves long-term memory”

RAG (retrieval-augmented generation) is a retrieval pattern, not a memory architecture. You still need to decide what gets indexed, when it gets updated, how to handle conflicts, and how to garbage-collect stale entries. RAG over a static corpus is search; RAG over an agent’s lived experience is memory.

“Store everything, let the model sort it out”

Dumping every tool output and intermediate thought into long-term memory creates a retrieval nightmare. The signal-to-noise ratio tanks. Be selective: store distilled facts, decisions, and user preferences. Discard verbose logs unless you have a specific forensic use case.

“Short-term and long-term memory are separate systems”

They interact constantly. Long-term memory seeds the short-term context at the start of each turn. Short-term memory produces the candidates for long-term storage at the end. The boundary is porous by design — that’s where the engineering lives.

“Vector databases are required”

PostgreSQL with pgvector, SQLite with sqlite-vec, or even a well-indexed JSON column can handle millions of memories. Don’t reach for a managed vector database until you have a scale or latency requirement that justifies it. The embedding model and retrieval logic matter more than the storage backend.

Memory consolidation: the missing piece

Human memory consolidates — short-term experiences become long-term memories during sleep. Agents need an analogous process. You can run a background job that:

1. Scans recent short-term conversations
2. Extracts salient facts, decisions, and preferences
3. Deduplicates against existing long-term memories
4. Writes consolidated entries with confidence scores

# Nightly consolidation job (simplified)
async def consolidate_user_memories(user_id: str, lookback_days: int = 7):
    recent_sessions = await get_sessions(user_id, since=days_ago(lookback_days))
    
    for session in recent_sessions:
        messages = await get_session_history(session.id)
        
        # Use a smaller, cheaper model for extraction
        facts = await extract_facts_llm(messages, model="gpt-4o-mini")
        
        for fact in facts:
            # Deduplicate: check if similar fact exists
            existing = await find_similar_fact(user_id, fact.content, threshold=0.85)
            if existing:
                # Merge: update confidence, last_seen
                await update_fact(existing.id, confidence=min(1.0, existing.confidence + 0.1))
            else:
                await remember_fact(user_id, fact.content, category="consolidated")

This prevents the “first session writes it, fifty sessions later it’s still there unchanged” problem. Memories should decay, strengthen, or merge over time.

Putting it together in production

A minimal viable memory stack for an agent:

1. Short-term: Redis list or Postgres table keyed by session_id, TTL 24-72 hours. Append-only, trim to last N messages or M tokens.
2. Long-term: Postgres + pgvector with tables for facts, preferences, episodes, skills. Each row: user_id, content, embedding, metadata, confidence, created_at, last_accessed.
3. Retrieval: Hybrid — exact match for structured fields, vector search for semantic, cross-encoder rerank for top-k.
4. Write path: Explicit tool calls (remember_fact, update_preference) + background consolidation job.
5. Observability: Log every read and write with latency, token count, and retrieval scores. You cannot tune what you don’t measure.

The n4n.ai gateway can help here by metering per-token usage across the multiple model calls this pattern requires — retrieval, extraction, consolidation, and the main reasoning loop all hit different models with different cost profiles.

Summary

Short-term memory is the context window: fast, limited, ephemeral. Long-term memory is a database you design: durable, queryable, your responsibility. The engineering is in the boundary — what crosses from short-term to long-term, when, and how it’s retrieved next time. Build the write path first. The read path is useless without it.