Context window management is the practice of explicitly controlling which tokens an LLM sees on each call, how they are truncated, summarized, or retrieved, and how state persists across turns. Without it, agent loops silently degrade as prompts overflow the model’s finite context and earlier instructions get evicted.
How context window management works
Every LLM request is a flat sequence of tokens bounded by a hard limit (128k for many current models, 200k for some). The sequence is typically assembled from several sources:
- System prompt and static instructions
- Tool or function schemas
- Conversation history (user turns, assistant turns, tool results)
- Retrieved context (RAG documents, memory blobs)
- The current step’s input
Context window management assigns each source a priority and a budget. When the sum exceeds the limit, you drop or compress lower-priority items before sending.
Token budgeting in code
A minimal budget allocator looks like this:
from dataclasses import dataclass
@dataclass
class ContextSlot:
name: str
tokens: int
priority: int # higher = keep longer
def trim_to_budget(slots: list[ContextSlot], max_tokens: int) -> list[str]:
slots.sort(key=lambda s: s.priority, reverse=True)
kept = []
used = 0
for s in slots:
if used + s.tokens <= max_tokens:
kept.append(s.name)
used += s.tokens
else:
# partial keep or drop; here we drop
pass
return keptThis is naive but shows the shape: you measure, you prioritize, you cut.
Counting tokens correctly
Never estimate token counts with len(text.split()). Use the model’s tokenizer. For OpenAI models, tiktoken is exact:
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
def count_tokens(text: str) -> int:
return len(enc.encode(text))If you call a gateway that fronts multiple providers, each model may tokenize differently. n4n.ai exposes an OpenAI-compatible endpoint across 240+ models; the token counts returned in usage reflect the upstream provider’s accounting, so trust the API response over local guesses. Effective context window management requires knowing the real count per model, not a generic average.
Output tokens count too
The assistant’s reply on turn N becomes part of the input on turn N+1. If you budget only for the current input, you will overflow after a long generation. Reserve headroom equal to max_completion_tokens when sizing the prompt.
Why agents need it
An agent is a loop: observe, think, act, repeat. Each iteration appends new tokens—tool outputs are notorious for being large. A single curl response or database dump can be 10k tokens. After ten steps you have 100k tokens of history, and your system prompt is now a distant memory.
Instruction drift
LLMs attend to the whole context, but in practice saliency drops for tokens early in a long sequence. If your core directive (“never execute destructive SQL”) sits at position 0 and gets pushed out by tool logs, the agent will violate it. Context window management keeps invariant instructions in a reserved high-priority slot.
Cost and latency
Input tokens are billed. Repeatedly resending 50k tokens of unchanged history on every step wastes money and adds latency. Sliding window or summarization cuts both. When a provider is rate-limited and a gateway like n4n.ai performs automatic fallback to a different model, the context limit may shrink; your trimmer must adapt to the target model’s max tokens, not assume a fixed constant.
Multi-agent sprawl
If you run parallel sub-agents, each maintains its own context. Aggregate token consumption across the system can explode. Centralized budgeting per agent is mandatory.
A concrete agent example
Consider a support agent that queries a ticket system and replies to users. Here is a stripped-down loop with explicit context window management:
from collections import deque
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
MAX_CTX = 120_000
SYS = "You are a support agent. Never leak internal IDs."
history = deque(maxlen=20) # keep last 20 turns only
def build_messages(user_msg, tool_results):
messages = [{"role": "system", "content": SYS}]
for turn in history:
messages.append(turn)
for res in tool_results:
messages.append({"role": "tool", "content": res[:2000]}) # hard truncate tool output
messages.append({"role": "user", "content": user_msg})
# budget check
total = sum(len(enc.encode(m["content"])) for m in messages)
while total > MAX_CTX and len(history) > 0:
dropped = history.popleft()
total -= len(enc.encode(dropped["content"]))
return messages
# inside loop:
# history.append({"role": "user", "content": user_msg})
# resp = client.chat.completions.create(model="gpt-4o", messages=build_messages(...))
# history.append({"role": "assistant", "content": resp.choices[0].message.content})The deque caps history; tool results are hard-truncated; system prompt is always first. This is context window management doing its job.
Summarization instead of dropping
For longer tasks, dropping history loses context. A better pattern: every N steps, summarize the oldest M turns into a compact memory blob.
def summarize(old_turns, client):
text = "\n".join(f'{t["role"]}: {t["content"]}' for t in old_turns)
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role":"system","content":"Compress to 200 tokens."},
{"role":"user","content":text}]
)
return resp.choices[0].message.contentStore the summary in a memory slot with medium priority. The raw turns can be evicted.
Common misconceptions
“Long context means I don’t need management”
A 200k window is not infinite. Real agent traces with verbose tools, full HTML, or base64 payloads blow past that in minutes. Moreover, longer contexts increase the chance of the model fixating on irrelevant middle content (lost-in-the-middle effect). You still need budgets.
“The model remembers everything from earlier calls”
Unless you persist and re-inject, it doesn’t. Each API call is stateless. The model’s memory is the current prompt. Context window management is the mechanism that decides what gets re-injected.
“Vector search replaces context management”
RAG retrieves relevant docs, but the retrieved chunks still consume tokens in the live prompt. If you retrieve 20 chunks of 1k tokens each, that’s 20k tokens you must place and trim. Retrieval is a source; management is the controller.
“Summarization is lossless”
It is not. A summary of a stack trace loses the exact line numbers. Use summarization for narrative memory, not for data the agent must act on precisely. Keep structured data in external stores and inject only pointers.
Practical patterns engineers actually use
Reserved system slot
Always reserve a fixed token budget (e.g., 2k) for system instructions. Never let dynamic content evict it.
Tiered memory
- Ephemeral: current turn, always full.
- Working: last K turns, sliding window.
- Long-term: summarized or retrieved, lower priority.
Provider cache hints
Some providers support cache_control to pin prefixes. If you send a stable system prompt plus large knowledge base, mark the prefix as cached. Gateways that honor client routing directives forward those hints—n4n.ai does this while providing automatic fallback when a provider is degraded. That reduces recomputation cost, but you still must trim the uncached tail yourself.
Monitor usage
Log usage.prompt_tokens on every call. A sudden spike means a tool returned unexpectedly large payload. Alert on it.
{
"model": "gpt-4o",
"usage": {
"prompt_tokens": 84211,
"completion_tokens": 312,
"total_tokens": 84523
}
}If prompt_tokens approaches your MAX_CTX, your trimmer failed.
Closing thoughts
Context window management is not a feature you buy; it’s a discipline you implement in the agent’s input assembly layer. Measure tokens, assign priority, trim ruthlessly, and persist what matters outside the prompt. Agents that skip this work fail in production not with errors, but with quiet amnesia.