The distinction between working memory vs context window matters the moment your agent needs to remember something beyond a single turn. The context window is the fixed token buffer the model attends to on each forward pass; working memory is any state your agent persists and retrieves through tool calls or external stores. Treat them as complementary budgets, not competitors.
What the context window is
The context window is a literal slice of the transformer’s attention matrix. Everything you put in the prompt—system instructions, chat history, retrieved docs, tool schemas—occupies tokens that are processed on every generation step. Exceed the limit and the request fails or gets silently truncated by the client.
Under the hood, the model builds a KV cache from your prompt. Repeated prefixes (stable system prompts, few-shot examples) can be cached by the provider to cut prefill compute, but you still pay per token on the wire unless the provider applies a cache discount. The window is ephemeral: it dies when the request returns.
A minimal call that leans entirely on the context window:
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="...")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a support bot."},
{"role": "user", "content": long_conversation_history},
],
)No external state required. The model “remembers” only what fits, and it attends to all of it with equal (though not always effective) weight.
What working memory is
Working memory is agent-controlled persistence. It lives outside the model: a key-value store, SQL table, vector index, or even a flat file. The agent writes to it via tool calls and reads selectively. This decouples what the model knows now from what it has seen before.
There are three practical flavors:
- Volatile working memory: in-process dict or Redis ephemeral keys, reset on restart.
- Persistent working memory: Postgres, S3, or a vector DB that survives across sessions.
- Structured working memory: JSON schemas or graph edges the agent mutates explicitly.
A toy example with explicit tools:
def read_memory(key: str) -> str:
return redis.get(f"agent:{key}") or ""
def write_memory(key: str, value: str) -> None:
redis.set(f"agent:{key}", value)
# Tool schema the model sees
tools = [{
"type": "function",
"function": {
"name": "write_memory",
"parameters": {
"type": "object",
"properties": {"key": {"type": "string"}, "value": {"type": "string"}}
}
}
}]The model never sees the full history unless you explicitly load a slice into the context window.
Head-to-head comparison
Capabilities
The context window gives the model immediate, zero-latency access to all contained tokens with full attention. It excels at syntactic consistency, short-range reasoning, and following instructions that reference earlier lines in the same prompt. Working memory holds arbitrarily large data and supports queries, updates, and sharing across sessions or agents. But the model only sees what you inject, and it can’t introspect the store without a tool round-trip that consumes its own reasoning tokens.
Cost model
Context window cost is linear in tokens per request. A 32k-token prompt billed at $0.01/1k input costs $0.32 every call, regardless of how many tokens actually matter. Working memory shifts cost to infrastructure (database, embeddings) and adds token overhead for read/write tool messages. A gateway like n4n.ai meters per-token usage across providers, so context-window spend stays transparent even when you switch models to shrink the buffer.
Working memory can cut total token cost by keeping the live context small, but you pay for storage and retrieval latency. Prompt caching can soften context cost if your prefix is stable; external memory softens it by never sending old data twice.
Latency and throughput
Prefill time scales with context length. A 100k-token prompt can add seconds before the first token on smaller GPUs. Working memory keeps the prompt short; the penalty is a tool-call round trip (tens of milliseconds for Redis, hundreds for vector search with rerank). For high-throughput batch jobs, stuffing context is simpler and avoids extra network hops. For interactive agents, external memory keeps TTFT low and predictable.
Ergonomics
Context window is dumb but easy: concatenate and send. Debugging is straightforward—the prompt is the state. Working memory demands schema design, cache invalidation, and serialization. You must handle stale writes, retrieval misses, and concurrent updates. Frameworks help, but you own the bugs. The working memory vs context window tradeoff shows up hardest here: one is a string, the other is a system.
Ecosystem
Every LLM API speaks the context window natively. Working memory has no standard: LangChain Memory, LlamaIndex, custom Redis, or raw SQL. n4n.ai’s OpenAI-compatible endpoint fronts 240+ models, so you can trade context size by swapping model names without rewriting memory code. The model-agnostic nature of the context window is its biggest advantage; the fragmentation of working memory is its biggest tax.
Limits
Context windows hit hard caps (8k–200k tokens depending on model) and suffer attention dilution on long inputs—models routinely forget middle content (“lost in the middle”). Working memory limits are your infra’s: Redis memory, vector recall quality, and the agent’s discipline. There is no free lunch; both degrade if mismanaged.
Comparison table
| Dimension | Context window | Working memory |
|---|---|---|
| Capability | Full attention over fixed tokens | Arbitrary external state via tools |
| Cost | Per-token per request, scales with size | Infra + token overhead for reads/writes |
| Latency | Prefill grows with length | Tool round-trip, usually lower TTFT |
| Ergonomics | Trivial concat, easy debug | Schema, retrieval, invalidation |
| Ecosystem | Native to all LLMs | Fragmented, framework-dependent |
| Limits | Hard token cap, mid-context loss | Store capacity, recall quality |
Hybrid patterns that survive contact
In production, pure context stuffing breaks at session two. Pure external memory breaks when the model needs immediate coherence. Use a hybrid: keep the last N turns in context, archive older turns to working memory with embeddings, and inject a retrieved summary when relevant.
recent = history[-8:]
summary = vector_store.search(query=current_topic, top_k=3)
messages = [system, *summary, *recent]This keeps the working memory vs context window split explicit: context for immediacy, memory for persistence. You can also write a rolling compaction job that summarizes the oldest context into a memory key every K turns.
Which to choose
Single-shot tasks and low-latency APIs
If the job fits in a few thousand tokens—classification, extraction, short chat—put everything in the context window. Adding working memory is pure overhead and another failure mode.
Long-running or multi-session agents
Once a conversation spans days or thousands of turns, working memory is mandatory. Store facts, user preferences, and prior summaries externally; load only what the current step needs. The context window becomes a scratchpad, not an archive.
Knowledge-intensive retrieval
For QA over document corpora, working memory (vector store) is the only scalable option. The context window is a cache for the top-k chunks, not the source of truth. Inject chunks per query; never preload the corpus.
Strict token budgets
When cost per call must stay flat, cap context at a small size and lean on working memory reads. You trade a little latency for predictable spend. Monitor cache hit rates to confirm the prefix stays stable.
Prototyping vs production
During prototyping, max out the context window to validate logic. In production, move stale data to working memory before the bill arrives. The working memory vs context window decision is iterative, not one-time.
Pick based on data lifetime and attention needs, not ideology. The split is a design lever, not a religion.