AI agent memory selection determines which observations, tool outputs, and user statements survive beyond the immediate context window. Without a deliberate policy, agents either drown in irrelevant vectors or lose critical state, and the difference shows up as broken reasoning and inflated token bills. The thesis here is that memory selection is a ranking problem under hard constraints, and the only scalable pattern is tiered storage with explicit salience scoring rather than indiscriminate writes to a single vector index.
The selection problem
An agent in production emits far more text than any context window can hold across a long session. A typical customer-support agent might process 200 tool calls, each returning JSON or HTML fragments of 1–5k tokens. If every fragment is stored and later retrieved by cosine similarity alone, the top-k results will be dominated by the most recent or most verbose items, not the most useful ones.
The core constraint is not storage capacity—vector databases scale cheaply. The constraint is injection budget: you can only place a few thousand tokens of retrieved memory into the next prompt before latency and cost explode, and before the model’s attention dilutes. AI agent memory selection is therefore about maximizing the expected utility of the small subset you surface.
Memory tiers and what gets kept
A useful distinction is three tiers:
- Episodic: raw interaction logs, tool outputs, errors. High volume, low per-item value.
- Semantic: extracted facts, user preferences, entity attributes. Compact, high value.
- Procedural: durable instructions or self-modified plans. Rarely changed, always loaded.
Selection is the process of promoting episodic items into semantic memory through compaction. A raw Slack message “I hate the new dashboard” is episodic; the extracted fact user:adam dislikes dashboard_v2 is semantic. The promotion decision should be explicit, not accidental.
A memory record should carry metadata that later filters can use:
{
"id": "mem_8f2a",
"type": "semantic",
"content": "adam dislikes dashboard_v2",
"importance": 0.9,
"access_count": 4,
"ts": "2024-05-12T10:22:00Z",
"embedding": [0.12, -0.03, ...]
}Without type, ts, and importance, retrieval becomes a blind similarity search.
Scoring salience: code and criteria
Storage-time selection asks: should this item be promoted or retained at all? A lightweight scoring function beats a blanket “store everything” policy. Consider a composite score:
import math
def cosine(a, b):
dot = sum(x*y for x, y in zip(a, b))
na = math.sqrt(sum(x*x for x in a))
nb = math.sqrt(sum(x*x for x in b))
return dot / (na*nb + 1e-9)
def score_memory(mem, query_emb, now, weights):
age_hours = (now - mem["ts"]).total_seconds() / 3600
recency = 1.0 / (1.0 + age_hours / 24.0) # half-life ~1 day
freq = min(mem["access_count"] / 10.0, 1.0) # saturates at 10 hits
imp = float(mem.get("importance", 0.0))
sim = cosine(query_emb, mem["embedding"]) if query_emb else 0.0
return (weights["r"]*recency +
weights["f"]*freq +
weights["i"]*imp +
weights["s"]*sim)The importance field can be assigned by a small classifier or by the LLM at write time: “Rate the long-term value of this observation from 0 to 1.” When batching these curation calls, an OpenAI-compatible gateway such as n4n.ai that honors cache-control hints can reuse the system prompt prefix across many items, cutting embedding and generation cost without changing your application code.
The weights are task-specific. For a debugging agent, imp and sim dominate; for a daily journal agent, recency matters more.
Retrieval-time selection matters as much as storage
Even perfectly curated memory is useless if you retrieve the wrong slice. AI agent memory selection at read time means applying structural filters before similarity ranking:
{
"filter": {
"type": "semantic",
"updated_after": "2024-05-01T00:00:00Z",
"min_importance": 0.6
},
"top_k": 5,
"similarity_threshold": 0.75
}This prevents the retriever from surfacing a high-similarity but stale episodic log from three months ago. The two-stage approach—metadata prefilter, then vector sort—is non-negotiable at scale. Pure vector search has no notion of “only facts about this user.”
A concrete failure mode: an agent retrieves a previous error trace that looks similar to the current one but belonged to a different tenant. Metadata scoping by tenant_id is the fix, not a better embedding model.
Tradeoffs: latency, cost, fidelity
Every selection strategy trades something:
Full history compaction. Summarize the entire episode into semantic memory hourly. Pro: bounded storage. Con: details lost; summarization is a model call that can hallucinate.
Write-time LLM scoring. Score each item as it appears. Pro: precise promotion. Con: adds latency to the agent loop; if the scorer is a frontier model, cost multiplies per step.
No selection (dump to vector DB). Pro: zero logic. Con: retrieval noise forces larger top-k, bloating context and degrading answers.
Latency is the silent killer. A retriever that takes 300ms is fine; a retriever that triggers a synchronous LLM re-rank adding 800ms per step will blow your p99. Prefer precomputed scores and simple math at request time.
Cost is dominated by embedding and storage writes. Embedding every episodic tool output is wasteful if 80% are never accessed. A cheap heuristic—“only embed items longer than 200 tokens or containing error keywords”—cuts write volume dramatically with minimal fidelity loss.
A reference implementation sketch
Below is a minimal agent step that separates write and read selection. It is intentionally synchronous for clarity.
async def agent_step(agent, observation):
# 1. Retrieve candidate memories using metadata filter + vector
cands = await agent.memory.query(
embedding=observation.embedding,
filter={"type": "semantic", "tenant_id": agent.tenant},
top_k=20
)
# 2. Rank by composite score with current query
ranked = sorted(
cands,
key=lambda m: score_memory(m, observation.embedding, now(), agent.weights),
reverse=True
)
context = [m["content"] for m in ranked[:5]]
# 3. Run the model
resp = await agent.llm.chat(
messages=[{"role": "system", "content": agent.system_prompt},
*[{"role": "memory", "content": c} for c in context],
{"role": "user", "content": observation.text}]
)
# 4. Write-time selection: should we promote anything from this step?
if observation.is_tool_result and observation.size_tokens > 200:
fact = await agent.extract_fact(resp) # small model call
if fact.importance > 0.5:
await agent.memory.add({
"type": "semantic",
"content": fact.text,
"importance": fact.importance,
"ts": now(),
"embedding": await agent.embed(fact.text)
})This loop never stores raw tool output unless it passes a size and importance gate, and it never retrieves more than five semantic memories per step.
Decisive takeaway
Treat AI agent memory selection as a constrained ranking system, not a dumping ground. Implement tiered memory with explicit promotion from episodic to semantic, assign importance at write time with a cheap model or heuristic, and always apply metadata filters before vector similarity at read time. Skip the temptation to “just use RAG” on raw logs—without selection, retrieval precision collapses and your agent pays for tokens it doesn’t need. The engineers who ship reliable agents are the ones who treat memory as a curated index with a budget, not an append-only lake.