A context window is the maximum number of tokens a large language model can process in a single forward pass, including both the input prompt and the generated output. It functions as the model’s working memory — everything the model “knows” during inference must fit within this fixed token budget. When a conversation or document exceeds the window, the oldest tokens are dropped unless you implement a strategy to manage them.
How the context window works
At the architecture level, the context window is determined by the maximum sequence length the model’s positional embeddings support. Most transformer-based LLMs use either absolute positional embeddings (fixed sinusoidal or learned vectors per position) or relative positional encodings like RoPE (Rotary Positional Embeddings). In both cases, the model has a hard maximum position index it can attend to.
During inference, the KV cache stores key and value projections for every token in the context. This cache grows linearly with sequence length and is the primary memory consumer during generation. A 7B parameter model with a 4,096 token context window requires roughly 1.5 GB of VRAM just for the KV cache at FP16 precision (2 bytes per parameter × 2 [key + value] × 32 layers × 4,096 tokens × 4,096 head dimension / 32 heads ≈ 1.5 GB). At 128k context, that same model needs ~48 GB for the cache alone.
The attention mechanism computes scores between every token and every other token in the context. This is why context length affects both memory (KV cache) and compute (quadratic attention complexity, though flash attention and other optimizations reduce the constant factor). You cannot simply feed a model more tokens than its trained context window — the positional embeddings don’t exist for those positions, and the attention patterns degrade.
Why context window size matters for engineers
Context window size directly constrains what applications you can build without additional engineering. Here are the practical boundaries:
Single-document QA: A 4k window handles most technical documentation, API references, and short papers. A 32k window covers entire codebases, legal contracts, or financial reports. A 128k+ window enables whole-repository analysis or book-length inputs.
Multi-turn conversations: Each turn consumes tokens from the window. A 4k window supports roughly 10-15 substantive exchanges before you must summarize or truncate. A 128k window supports hundreds of turns.
In-context learning: Few-shot prompting consumes window space. With 4k tokens, you might fit 5-10 examples. With 128k, you can fit hundreds, enabling stronger in-context learning for niche tasks.
RAG systems: Retrieval-augmented generation stuffs retrieved chunks into the context. Larger windows mean more chunks, higher recall, and less aggressive chunking — but also higher latency and cost per request.
Agent workflows: Tool calls, observations, and reasoning traces accumulate quickly. A coding agent that reads files, runs tests, and iterates can burn 50k+ tokens in a single session.
The trade-off is linear: larger context means higher per-request latency, higher memory requirements, and higher cost (most providers charge per input token). You should choose the smallest window that reliably fits your use case.
Concrete example: fitting a codebase into context
Suppose you’re building a code review assistant. Your target repository has 200 Python files averaging 300 lines each (~1,500 tokens per file). That’s 300,000 tokens total — far exceeding any current context window.
# Naive approach: fails silently or truncates
def review_codebase(repo_path: str, model: str) -> str:
all_code = ""
for file in Path(repo_path).rglob("*.py"):
all_code += file.read_text() + "\n\n"
prompt = f"Review this codebase:\n{all_code}"
return llm_complete(prompt, model=model) # Truncates at context limitYou need a strategy. Common approaches:
1. Retrieval + reranking — Embed the codebase, retrieve top-k relevant files for the specific review task, fit those into context.
def review_with_rag(task: str, repo_path: str, model: str, k: int = 10) -> str:
# Embed all files once (offline)
embeddings = embed_codebase(repo_path)
# Retrieve relevant files for this task
query_embedding = embed(task)
top_files = retrieve_top_k(embeddings, query_embedding, k)
context = "\n\n".join(f"# {f.path}\n{f.content}" for f in top_files)
prompt = f"Task: {task}\n\nCode:\n{context}"
return llm_complete(prompt, model=model)2. Hierarchical summarization — Summarize each file, then summarize summaries, feed the hierarchy.
def hierarchical_summary(repo_path: str, model: str) -> str:
file_summaries = {}
for file in Path(repo_path).rglob("*.py"):
file_summaries[file] = llm_complete(
f"Summarize this file in 100 words:\n{file.read_text()}",
model=model
)
# Second-level summary fits in context
all_summaries = "\n".join(f"{p.name}: {s}" for p, s in file_summaries.items())
return llm_complete(
f"High-level architecture summary:\n{all_summaries}",
model=model
)3. Sliding window with overlap — Process the codebase in overlapping chunks, aggregate findings.
def sliding_window_review(repo_path: str, model: str, window: int = 8000, overlap: int = 1000) -> list:
all_code = "\n\n".join(f.read_text() for f in Path(repo_path).rglob("*.py"))
tokens = tokenize(all_code)
findings = []
for i in range(0, len(tokens), window - overlap):
chunk = detokenize(tokens[i:i + window])
finding = llm_complete(f"Find issues in this code:\n{chunk}", model=model)
findings.append(finding)
# Final aggregation pass
return llm_complete(f"Aggregate these findings:\n{findings}", model=model)Each strategy trades off completeness, latency, and implementation complexity. The context window size determines which strategies are viable.
Common misconceptions
Misconception: “The model remembers everything from training.” The context window is the only memory the model has at inference time. Training data influences weights, but the model cannot “recall” specific training documents. If you need the model to know something, it must be in the context window (or in its weights via fine-tuning, which is a different mechanism).
Misconception: “Larger context is always better.” Larger contexts increase latency quadratically in attention compute (though flash attention helps) and linearly in KV cache memory. They also increase the chance of “lost in the middle” degradation — models attend less reliably to information in the middle of long contexts. A 2023 study by Liu et al. showed performance on needle-in-haystack tasks peaks around 20-30% of the nominal context length for many models. Don’t use 128k context when 8k suffices.
Misconception: “Context window equals output length.” The context window is shared between input and output. If your model has a 4,096 token window and you send a 3,500 token prompt, you have at most 596 tokens for the response (minus overhead). Plan accordingly.
Misconception: “All models with the same nominal context perform equally.” Nominal context length (e.g., “32k”) is a training hyperparameter. Effective context — the length at which the model actually maintains coherence and retrieval accuracy — varies significantly. Some models trained with 32k context degrade noticeably after 8k. Others use extrapolation techniques (YaRN, PI, LongRoPE) to extend beyond their training length with varying success. Test your specific model on your specific task.
Misconception: “You can just truncate the beginning.” Naive truncation drops the system prompt, early conversation turns, or critical instructions. This changes model behavior unpredictably. Use structured truncation strategies: preserve system prompts, summarize older turns, or use sliding windows with overlap.
Misconception: “Context window limits are purely technical.” Provider APIs often impose lower limits than the model supports. A model may support 128k context, but the API caps requests at 32k. Some providers charge different rates for different context tiers. Check the API documentation, not just the model card.
Managing context in production
If you’re building a production system, you need explicit context management. Here’s a pattern that works:
class ContextManager:
def __init__(
self,
max_tokens: int,
system_prompt: str,
tokenizer,
reserve_output: int = 1000
):
self.max_tokens = max_tokens
self.system_prompt = system_prompt
self.tokenizer = tokenizer
self.reserve_output = reserve_output
self.messages = [{"role": "system", "content": system_prompt}]
def token_count(self, text: str) -> int:
return len(self.tokenizer.encode(text))
def available_tokens(self) -> int:
used = sum(self.token_count(m["content"]) for m in self.messages)
return self.max_tokens - used - self.reserve_output
def add_user_message(self, content: str) -> bool:
"""Returns False if message doesn't fit even after truncation."""
needed = self.token_count(content)
if needed > self.available_tokens():
self._truncate_history(needed)
if needed > self.available_tokens():
return False
self.messages.append({"role": "user", "content": content})
return True
def add_assistant_message(self, content: str):
self.messages.append({"role": "assistant", "content": content})
def _truncate_history(self, needed: int):
"""Remove oldest non-system messages until space available."""
while self.available_tokens() < needed and len(self.messages) > 1:
# Never remove system prompt (index 0)
removed = self.messages.pop(1)
# Optionally: summarize removed message and prepend to next
# This preserves information at token cost
def get_messages(self) -> list:
return self.messagesThis pattern preserves the system prompt, reserves space for output, and truncates oldest history first. For more sophisticated needs, replace _truncate_history with summarization or retrieval.
Context window evolution
Context windows have grown rapidly: GPT-3 (2k) → GPT-3.5 (4k/16k) → GPT-4 (8k/32k/128k) → GPT-4o (128k). Open models followed: LLaMA 2 (4k) → LLaMA 3 (8k/128k) → various fine-tunes pushing to 1M+ via extrapolation.
This growth enables new application categories but doesn’t eliminate the need for context management. Even with 1M token windows, you face latency, cost, and attention quality trade-offs. The engineering discipline of fitting the right information into the available window remains the same — the window is just larger.
When evaluating models for a task, test effective context length on your data. Nominal context is a starting point, not a guarantee.