Sliding window vs summarization: managing agent memory

Every agent that survives past a handful of turns hits a memory wall. The choice between sliding window vs summarization determines whether you silently drop old context or spend tokens compressing it, and that trade-off shows up in your bill and your p99 latency. This is a practical comparison, not a survey.

How the two strategies actually work

Sliding window

A sliding window keeps a fixed-size tail of the conversation. You drop the oldest messages once the token budget is exceeded. Implementation is a trim function that runs locally before each model call:

def trim_messages(messages, max_tokens=4000, token_fn=len):
    # naive char proxy; use tiktoken or llama-tokenizer in prod
    out = []
    total = 0
    for m in reversed(messages):
        cost = token_fn(m["content"]) + 4
        if total + cost > max_tokens:
            break
        out.insert(0, m)
        total += cost
    return out

No external calls. The model never sees what fell off the edge. If the user said “my name is Sam” in turn one and the window is 6 turns, Sam is gone by turn eight.

Summarization

Summarization periodically condenses the backlog into a running summary. You store the summary as a system note and feed only recent raw messages plus that note to the main model.

def summarize_history(messages, client):
    transcript = "\n".join(f'{m["role"]}: {m["content"]}' for m in messages)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Compress the transcript to key facts and user preferences. Output only the summary."},
            {"role": "user", "content": transcript}
        ]
    )
    return resp.choices[0].message.content

# usage
summary = summarize_history(old_messages, client)
messages = [{"role": "system", "content": f"Prior context summary: {summary}"}] + recent_messages

You pay for the summarize call, but the main context stays small. The summary is lossy: exact phrasing and edge cases bleed out.

Dimensions that matter in production

Capabilities

Sliding window preserves exact recent text; anything beyond the window is gone irretrievably. Summarization keeps a lossy global view. For a coding agent that needs the exact stack trace from turn one, sliding window fails unless the window is huge. For a travel planner that needs to recall “user is vegetarian and prefers trains,” summarization captures it even after 50 turns.

The real capability gap is recall vs precision. Sliding window gives precision on recent state. Summarization gives recall on gist.

Price/cost model

Sliding window has zero auxiliary inference cost. You pay only for the tokens you send to the main model, which shrinks over time as old turns drop. Summarization adds a recurring cost: each compaction is a generation priced per output token. Over a 100-turn session with compaction every 10 turns, you issue ~10 extra calls. If you route through a gateway that meters per-token usage and forwards provider cache-control hints, you can mark the summary prompt as cacheable to cut repeat cost. n4n.ai does this when you set cache_control on the summary block, so the cached prefix isn’t re-billed on every subsequent turn.

That said, summarization can reduce main-call cost on long sessions because the live context stays bounded. With a 128k context model, sliding window of 32k tokens still bills 32k every turn; a summary of 2k tokens plus 8k recent is far cheaper.

Latency/throughput

Sliding window adds only local list operations—sub-millisecond. Summarization injects a network round-trip and generation time. If you run it synchronously before each response, users feel it: a 500ms summarize call becomes part of every turn. Async compaction on a background timer hides most of it, but you risk race conditions where the main call reads a stale buffer.

Ergonomics

Sliding window is a few lines and a unit test. Summarization needs: a summary store (KV or DB), a merge policy (when to compact), a prompt that doesn’t hallucinate, and a fallback when the summarizer returns garbage. You also must decide whether to keep a rolling raw buffer alongside the summary. Debugging a poisoned summary is harder than debugging a dropped message because the failure is silent and propagates.

Ecosystem

Both are supported by LangChain (ConversationBufferWindowMemory vs ConversationSummaryMemory), LlamaIndex, and raw API code. Sliding window is universal; summarization requires a callable model. If you use an OpenAI-compatible endpoint that addresses 240+ models, you can swap the summarizer to a cheap model without code changes. Most frameworks treat summarization as a special memory class; you still own the prompt.

Limits

Sliding window’s limit is the window size—pick wrong and the agent amnesia hits at the worst time. Summarization’s limit is drift: summaries lose edge cases, and a bad summary poisons all future turns. You must cap summary length or it becomes its own context hog. Neither strategy handles structured long-term memory (e.g., a database of user facts) without extra scaffolding.

Comparison table

Implementation patterns that work

Hybrid window+summary

Keep a sliding window of the last 8 messages, plus a summary of everything older. This bounds latency and retains gist.

def build_context(full_history, summary, window=8):
    recent = full_history[-window:]
    sys = [{"role": "system", "content": f"Summary of earlier: {summary}"}]
    return sys + recent

This is the production default for assistants that need both exact recent instructions and long-term user attributes.

Cache the summary

If your provider supports prompt caching, tag the summary block. On a gateway that honors client routing directives, set:

{
  "messages": [
    {"role": "system", "content": "Summary of prior turns...", "cache_control": {"type": "ephemeral"}}
  ]
}

That avoids re-billing the summary tokens on every call. Without caching, summarization’s cost advantage shrinks.

Threshold-based compaction

Don’t summarize on a fixed turn count; summarize when raw buffer exceeds a token threshold:

if estimate_tokens(raw_buffer) > 6000:
    summary = summarize_history(raw_buffer, client)
    raw_buffer = []  # or keep last 2 turns

This adapts to verbose vs terse users.

Which to choose

Use sliding window when:

• Sessions are short (<20 turns) or recovery from old context is unnecessary.
• You need predictable p99 latency and zero extra calls.
• Throughput matters more than recall (e.g., a query classifier or a single-shot tool caller).
• Your model context is cheap enough that a 16k tail per turn is acceptable.

Use summarization when:

• The agent runs long and must remember user constraints, decisions, or facts from early turns.
• You can tolerate occasional async compaction latency.
• Cost of a large context every turn exceeds the cost of periodic summarize calls (true for many 100k+ token sessions).
• You have a cheaper, reliable model to act as summarizer.

Use hybrid when:

• You want both exact recent text and long-term gist. This is the default for production assistants.
• You can implement a background worker that summarizes only when the raw buffer crosses a threshold.
• You can monitor summary quality with a simple diff or LLM-judge in CI.

Avoid summarization if:

• Your summarizer model is weaker than the main model and may distort instructions.
• You cannot afford the extra call on a degraded provider—though automatic fallback to another provider mitigates this if your gateway supports it.
• Regulatory constraints require exact transcript retention; summaries are not audit-grade.

Sliding window vs summarization is not a religious choice. It is a budget and latency decision. Measure your session length distribution, then pick the cheapest strategy that meets your recall bar. Most teams end up at hybrid because pure window amnesia bites first, and pure summarization drift bites second.