Strategies for summarizing history to save context tokens

Long-running agent sessions bleed tokens. Every retained user message, tool result, and system note eats context that could go to reasoning. To summarize history save context tokens, you need a deterministic compression loop that runs before each model call, not a vague “make it shorter” hope. This guide walks through a concrete implementation you can drop into an OpenAI-compatible chat pipeline.

Step 1: Define a token budget and trigger threshold

Pick a hard context limit based on your model (e.g., 8k for smaller models, 32k for large). Set a trigger below that limit so compression runs before you error out. The goal is to summarize history save context tokens without losing critical state at the boundary.

Use a local tokenizer to count accurately. tiktoken with cl100k_base covers most modern models.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def count_tokens(messages):
    n = 0
    for m in messages:
        n += 4  # per-message framing overhead
        n += len(enc.encode(m.get("content", "")))
    return n

BUDGET = 8000
TRIGGER = 6000  # compress when projected usage crosses this

Run count_tokens on your full message list after each turn. If it exceeds TRIGGER, compress before the next inference call.

Step 2: Select which messages to compress

Never compress the system prompt or the most recent exchanges. Recent context is hot; old context is cold. Keep the first system message and the last N user/assistant pairs verbatim.

def split_history(messages, keep_recent=4):
    sys = messages[0] if messages and messages[0]["role"] == "system" else None
    rest = messages[1:] if sys else messages
    if len(rest) <= keep_recent:
        return messages, []
    to_compress = rest[:-keep_recent]
    retained = rest[-keep_recent:]
    if sys:
        retained = [sys] + retained
    return retained, to_compress

This gives you a clean separation: retained goes to the model unchanged, to_compress becomes summary fodder.

Step 3: Build a strict summarization prompt

A summarizer that free-forms will drop IDs and dates. Force structure. Ask for JSON with a summary string and an entities array. Use a cheap model for this pass.

from openai import OpenAI

client = OpenAI(base_url="https://api.openai.com/v1", api_key="sk-...")

def summarize(messages_to_compress):
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in messages_to_compress)
    prompt = f"""Compress the following conversation into a terse factual summary.
Retain: user intents, tool call IDs, dates, unresolved questions.
Drop: pleasantries, repetition, verbose reasoning.
Output JSON: {{"summary": str, "entities": [str]}}

CONVERSATION:
{convo}
"""
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"user","content":prompt}],
        response_format={"type":"json_object"}
    )
    return resp.choices[0].message.content

The response_format guard keeps parsing simple. If your client library differs, validate the JSON manually.

Step 4: Merge summary into a single message

Replace the compressed block with one system message that sits right after the original system prompt. This is the core operation to summarize history save context tokens at scale.

import json

def compress_history(messages):
    retained, to_compress = split_history(messages)
    if not to_compress:
        return messages
    out = summarize(to_compress)
    data = json.loads(out)
    summary_msg = {
        "role": "system",
        "content": f"Summary of earlier turns: {data['summary']} | Entities: {', '.join(data['entities'])}"
    }
    if retained and retained[0]["role"] == "system":
        retained.insert(1, summary_msg)
    else:
        retained.insert(0, summary_msg)
    return retained

One message instead of twenty. The model sees a dense recap, not a scroll of stale text.

Step 5: Preserve structured data explicitly

Naive summarization loses primary keys. The entities field from Step 3 is your insurance. Push order IDs, timestamps, and customer numbers into that list and echo them in the summary message. Later steps that need to call tools can still reference order_4921 because it survived the cut.

If your domain is strict, extend the schema:

{
  "summary": "User asked to cancel order_4921 placed 2024-05-01.",
  "entities": ["order_4921", "2024-05-01"],
  "open_tasks": ["confirm refund method"]
}

Parse and store open_tasks as a separate system note. That turns compression into state management, not just text trimming.

Step 6: Implement the rolling loop in your handler

Wire compression into the session object so it fires automatically. This keeps the main call site clean.

class ChatSession:
    def __init__(self):
        self.messages = [{"role":"system","content":"You are a helpful agent."}]
    def send(self, user_text):
        self.messages.append({"role":"user","content":user_text})
        if count_tokens(self.messages) > TRIGGER:
            self.messages = compress_history(self.messages)
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=self.messages
        )
        assistant_msg = resp.choices[0].message.content
        self.messages.append({"role":"assistant","content":assistant_msg})
        return assistant_msg

Every turn self-polices. If the session grows, it shrinks itself before the model sees the bloat.

Step 7: Verify token savings and quality

Success is not “it ran.” Measure two things: token reduction and fact retention.

before = count_tokens(original_messages)
after = count_tokens(compressed_messages)
assert after < before, "compression increased size"

Then run a small eval: ask the compressed session a question that only the old context can answer (e.g., “What was the order date?”). If the model answers correctly, the summary held.

If your inference gateway provides per-token usage metering, reconcile the billed tokens against your local counts to confirm you actually summarize history save context tokens in production rather than just shifting cost to the summarizer. Gateways such as n4n.ai surface per-token metering and automatic fallback, so a summarization call that hits a provider rate limit won’t break the loop.

Step 8: Cache summaries to avoid recomputation

The same cold prefix re-appears across turns. Cache it locally keyed by content hash, and set provider cache hints where supported.

import hashlib

cache = {}

def cached_summarize(messages_to_compress):
    key = hashlib.md5(
        "\n".join(m["content"] for m in messages_to_compress).encode()
    ).hexdigest()
    if key in cache:
        return cache[key]
    # ... build prompt as in Step 3 ...
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"user","content":prompt}]
    )
    cache[key] = resp.choices[0].message.content
    return cache[key]

When you call the summarizer repeatedly with similar prefixes, set the appropriate cache-control hint for your provider (e.g., Anthropic’s cache_control block). n4n.ai honors client routing directives and forwards provider cache-control hints, so the summarization prefix is cached at the provider edge and you stop paying to re-encode the same conversation slice.

Verification checklist

• count_tokens drops by ≥40% after compression on long sessions.
• Recent keep_recent messages are byte-identical before and after.
• Entities from Step 5 appear in the merged system message.
• A factual question about compressed turns still resolves correctly.
• Summarizer token cost (from metering) is less than the tokens saved on the main call.

Follow these steps and the context window stops being a wall. You turn history into a rolling, queryable state that costs a fraction of the raw transcript.