Long-running agents accumulate context faster than any context window allows. Effective agent memory summarization is the difference between a coherent assistant and a rambling process that loses the plot by step 500. This guide gives an ordered, implementable path for engineers who need summarization that survives production workloads.
1. Model the memory store before writing summarization code
Start with an explicit schema. Treat memory as an append-only log of typed records, not just chat messages. Without typing, your summarizer cannot distinguish a user preference from a transient tool response.
from dataclasses import dataclass, field
from enum import Enum
import time
class RecordType(Enum):
OBSERVATION = "observation"
ACTION = "action"
FACT = "fact"
SUMMARY = "summary"
@dataclass
class MemoryRecord:
type: RecordType
content: str
ts: float = field(default_factory=time.time)
token_estimate: int = 0Agent memory summarization fails when you summarize untyped blobs. Keep facts separate from chatter from day one.
Pitfall: Storing entire HTTP responses verbatim. Summarize at ingest, not later. A 5KB API dump will blow your token budget before the agent finishes its first task.
2. Trigger on token pressure, not message count
Message count is a lie. One message can be a single word or a 10K-token document. Count tokens approximately, then verify with a real tokenizer if you cross a threshold.
def estimate_tokens(text: str) -> int:
return len(text) // 4
def should_summarize(records, max_ctx=8000, threshold=0.7):
used = sum(r.token_estimate or estimate_tokens(r.content) for r in records)
return used > max_ctx * thresholdTradeoff: A threshold too high risks mid-step truncation; too low wastes compute on frequent summarization calls. For most agents, 0.7 of the model’s context limit is a sane starting point.
3. Pick a granularity strategy
Effective agent memory summarization requires choosing how much history to compress and how often. Three patterns dominate:
Sliding window
Keep the last N tokens raw, summarize everything older.
def sliding_summarize(client, old_records, model="gpt-4o-mini"):
old_text = "\n".join(f"[{r.type.value}] {r.content}" for r in old_records)
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "Compress the following agent log into a terse summary preserving decisions and open tasks."},
{"role": "user", "content": old_text}
]
)
return resp.choices[0].message.contentHierarchical
Summarize summaries on a schedule (e.g., hourly or per-session). Good for agents that run for days.
Full compaction
One summary of entire history at each trigger. Simplest, but recent detail gets blurred into older narrative.
For most production agents, a sliding window of recent records plus a hierarchical daily merge balances recency and coherence.
4. Run an extractive pass before abstractive summary
Abstractive summarization alone drifts. The model paraphrases and silently drops constraints. Extract structured facts first, then summarize around them.
EXTRACT_PROMPT = """Extract commitments, entities, and unresolved questions from the log as JSON.
Log:
{log}
Output only JSON."""
def extract_facts(client, text):
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role":"user","content": EXTRACT_PROMPT.format(log=text)}],
response_format={"type":"json_object"}
)
return resp.choices[0].message.contentStore the JSON as RecordType.FACT. A later summary can reference: “Open tasks: X, Y (see facts).” This reduces hallucinated omissions.
Pitfall: Ignoring negative constraints (“do not delete the prod DB”). Your extractive prompt must explicitly capture negations and prohibitions, or the summarizer will treat them as noise.
5. Cache summarization prompts and reuse across steps
Summarization system prompts rarely change. If you call a model through n4n.ai, forward cache-control hints on the static system prompt so providers bill only once for repeated instructions. The gateway honors client routing directives and forwards those hints automatically.
client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[
{"role":"system","content":"You compress agent logs.","cache_control":{"type":"ephemeral"}},
{"role":"user","content": old_text}
]
)Without caching, summarization overhead can exceed 30% of total token cost on long sessions. Cache the invariant parts: role definition, output format, examples.
6. Handle degradation and provider errors
Summarization is a hard dependency. When the model is rate-limited or degraded, fall back to naive truncation rather than blocking the agent.
def safe_summarize(client, old_records, max_ctx=2000):
try:
return sliding_summarize(client, old_records)
except Exception:
kept = []
used = 0
for r in reversed(old_records):
t = estimate_tokens(r.content)
if used + t > max_ctx: break
kept.insert(0, r)
used += t
return "TRUNCATED:" + "\n".join(r.content for r in kept)Tradeoff: Truncated memory loses early context, but the agent keeps running. Log the fallback so you can replay and repair later.
7. Test with replay, not unit tests
Record a real session of 1000+ steps. Replay it through your summarizer at intervals to see if critical facts survive.
def replay(session_log, summarize_fn, window=50):
mem = []
for entry in session_log:
mem.append(entry)
if should_summarize(mem):
summary = summarize_fn(mem[:-window])
mem = mem[-window:] + [MemoryRecord(RecordType.SUMMARY, summary)]
return memMeasure how often a fact from step 10 appears in final memory. If recall drops below acceptable, adjust your extractive schema or window size.
Pitfall: Evaluating on synthetic 10-message chats. Real agents diverge after hundreds of steps because small drifts compound. Only replay exposes this.
8. Meter and bound summarization cost
Track tokens spent on summarization separately from agent reasoning. If you route through n4n.ai, its per-token usage metering lets you tag summarization calls with a task header to attribute cost precisely.
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[...],
extra_headers={"x-task": "summarize"}
)
metrics.record("summarization_tokens", resp.usage.total_tokens)Tradeoff: More frequent summarization improves coherence but linearly increases spend. Set a budget alert at 15% of total token throughput; beyond that, tune thresholds.
9. Practical default configuration
For a production agent, start with:
- Trigger at 70% of context limit.
- Sliding window of 50 recent records raw.
- Extractive JSON facts every trigger.
- Hierarchical daily merge.
- Cache system prompts via provider cache-control.
- Fallback truncation on error.
- Replay tests on sessions > 1000 steps.
Adjust thresholds based on replay metrics, not intuition.
Agent memory summarization is infrastructure, not a feature. Build it with the same rigor as a database layer, and your long-running agents will stay coherent when it matters.