Managing sliding window context in long chat sessions

A sliding window context chat session breaks silently when your chat history grows past the model’s context limit, dropping earlier turns and degrading answers. The fix is not just truncating old messages—it’s a deliberate policy for what stays, what gets summarized, and what gets discarded. Engineers building production LLM apps need a reproducible method to keep relevant context without blowing up token cost.

1. Count tokens before you trim

Guesswork about message size will burn you. Count tokens with the same tokenizer the model uses, or at minimum measure usage.prompt_tokens from the API response.

import tiktoken

def count_messages_tokens(messages, model="gpt-4o"):
    enc = tiktoken.encoding_for_model(model)
    total = 0
    for m in messages:
        total += len(enc.encode(m.get("role", "")))
        total += len(enc.encode(m.get("content", "")))
    # OpenAI adds ~4 tokens per message plus 2 for the reply prefix
    return total + 4 * len(messages) + 2

Pitfall: system prompts, tool schemas, and developer messages all count. A 500-token system prompt eats into a 8k context fast. Measure the full request object, not just the conversational turns.

2. Define a window policy from a token budget

Pick a hard ceiling below the model’s max context to leave room for the completion and overhead. For a 128k model, a 100k prompt budget is sane; for a 8k model, stay under 6k.

POLICY = {
    "max_prompt_tokens": 30000,
    "reserved_system_tokens": 800,
    "keep_recent_pairs": 10,
}

A sliding window context chat session should retain the system prompt verbatim, keep the most recent N user/assistant pairs, and compress everything older into a rolling summary. Do not evict the system message—ever.

What to keep vs. what to summarize

• System/developer instructions: always.
• Last K turns: always verbatim (K depends on token budget).
• Older turns: merged into a summary message injected as a leading “context” block.

Tradeoff: verbatim recent history gives highest fidelity; summary loses nuance but saves tokens. Tune K by inspecting failure modes in your eval set.

3. Summarize evicted turns incrementally

When a message falls outside the window, fold it into the existing summary with a cheap model call. Batch evictions to avoid per-message summarization spam.

def merge_summary(old_summary, evicted, client, model="gpt-4o-mini"):
    evicted_text = "\n".join(f"{m['role']}: {m['content']}" for m in evicted)
    resp = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": f"Current summary:\n{old_summary}\n\n"
                       f"Integrate these messages without losing key facts:\n{evicted_text}\n\n"
                       f"Return updated summary only."
        }]
    )
    return resp.choices[0].message.content.strip()

Call this only when the window overflows, not on every turn. If you evict 5 messages at once, send them together.

Pitfall: summarization drift. Over many turns the summary becomes vague. Periodically regenerate the summary from the raw evicted store if you keep it, or accept that very old context is approximate.

4. Implement the sliding buffer

A minimal session class enforces the policy:

class SlidingWindowSession:
    def __init__(self, client, max_prompt_tokens=30000, summary_model="gpt-4o-mini"):
        self.client = client
        self.max_prompt_tokens = max_prompt_tokens
        self.summary_model = summary_model
        self.system = None
        self.summary = ""
        self.recent = []  # list of {role, content}

    def set_system(self, text):
        self.system = text

    def add(self, role, content):
        self.recent.append({"role": role, "content": content})
        self._enforce_window()

    def _window_tokens(self):
        msgs = []
        if self.system:
            msgs.append({"role": "system", "content": self.system})
        if self.summary:
            msgs.append({"role": "system", "content": "Prior context: " + self.summary})
        msgs.extend(self.recent)
        return count_messages_tokens(msgs)

    def _enforce_window(self):
        while self._window_tokens() > self.max_prompt_tokens and len(self.recent) > 2:
            evicted = [self.recent.pop(0)]
            # pop assistant/user pair if possible
            if self.recent and self.recent[0]["role"] != evicted[0]["role"]:
                evicted.append(self.recent.pop(0))
            self.summary = merge_summary(self.summary, evicted, self.client, self.summary_model)

    def get_messages(self):
        out = []
        if self.system:
            out.append({"role": "system", "content": self.system})
        if self.summary:
            out.append({"role": "system", "content": "Prior context: " + self.summary})
        out.extend(self.recent)
        return out

This keeps the most recent dialogue intact and pushes old context into a compressed prefix.

5. Protect tool calls and structured state

Function-calling workflows break if you drop a tool message or the assistant message that invoked it. The model expects a strict pairing.

def safe_evict(recent):
    # never evict a message with tool_calls or role == 'tool' unless its pair is also evicted
    if recent[0].get("tool_calls") or recent[0]["role"] == "tool":
        return False
    return True

In _enforce_window, skip eviction when safe_evict is false, or evict a full block: assistant tool_calls plus all subsequent tool responses together.

Tradeoff: tool payloads (JSON, SQL results) are often large. Truncate or compress their content before summarization—e.g., keep only the returned IDs or row counts—to stay within budget.

6. Adapt to provider limits and routing

A sliding window context chat session must know the max context of whatever model actually serves the request. If you route across providers, each has different limits (e.g., 32k vs 200k). n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models and applies automatic fallback when a provider is rate-limited or degraded, but your trimming code still needs the target model’s token ceiling passed in at request time. Honor client routing directives and forward provider cache-control hints so summarized prefixes can be cached instead of recomputed.

# pseudo-route hint
headers = {"x-n4n-model": "anthropic/claude-3.5-sonnet", "x-cache-control": "summary"}

If fallback switches to a smaller model mid-session, recompute the window with the new max_prompt_tokens or you’ll overflow the smaller context.

7. Log, meter, and tune

Emit structured logs per turn:

logger.info({
    "event": "window_trim",
    "prompt_tokens": self._window_tokens(),
    "summary_tokens": len(self.summary.split()),
    "recent_pairs": len(self.recent) // 2,
    "eviction_count": eviction_total,
})

Use per-token usage metering to spot summarization cost creep. If summary tokens grow past 10% of your budget, raise eviction aggressiveness or compress the summary format.

Test with replay of long real sessions, not synthetic loops. The failure mode is always specific: a forgotten variable from turn 40, a dropped tool result, or a summary that paraphrases a user constraint into mush.

8. Common pitfalls checklist

• Counting only content: ignore role overhead and you’ll overflow.
• Summarizing too early: if K is small, the model loses thread before it can act.
• Dropping system mid-session: some frameworks rebuild messages and omit it.
• Assuming all models same limit: a session tuned for 128k dies on an 8k fallback.
• Not caching summaries: recomputing the same summary on every request wastes tokens.

A sliding window context chat session is a stateful component, not a one-liner. Build it as a class, test it against your longest conversations, and meter it like any other paid dependency.