GPT-5's context window for long-running agents

Building gpt-5 context window agents that run for hours or days tempts you to treat the model’s context as a limitless scratchpad. The larger window reduces but does not remove the need for disciplined state management, because attention degradation, latency, and cost still scale with token count. This guide gives an ordered path to ship agents that survive long horizons without silently losing track of goals.

1. Quantify the token budget before writing loops

The context window is a hard ceiling measured in tokens, not messages. A system prompt with tool schemas, a few retrieved docs, and 50 round-trips can eat 30–40% of a large window before your agent does real work. Compute the steady-state size of one step and multiply by expected steps.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
system = "You are a long-running agent with tools: search, exec, store."
tool_defs = '[{"name":"search","params":{...}}]'  # approx 800 tokens
step_overhead = len(enc.encode(system)) + len(enc.encode(tool_defs)) + 1200
window = 200_000  # example gpt-5 context window size in tokens
max_steps = (window * 0.7) // step_overhead  # keep 30% headroom
print(f"Max safe steps: {max_steps}")

If max_steps is under your expected run length, you cannot naively append. Plan externalization now. Treat the window as a cache, not a database.

2. Never append every observation to the message list

The common failure is a loop that pushes every tool result into messages:

# anti-pattern
for step in range(1000):
    resp = client.chat.completions.create(model="gpt-5", messages=messages)
    messages.append(resp.choices[0].message)
    messages.append({"role": "tool", "content": observe()})

This grows linearly and guarantees context overflow or truncated attention. Instead, keep only the rolling recent window in context and move everything else to a store. A correct loop holds the system prompt, the current objective, and the last 5–10 interactions.

# pattern
recent = []
for step in range(run_length):
    resp = client.chat.completions.create(model="gpt-5", messages=[system] + recent)
    recent.append(resp.choices[0].message)
    obs = observe()
    save_to_store(run_id, step, obs)   # external
    recent.append({"role": "tool", "content": obs[:500]})  # truncated inline
    if len(recent) > 10:
        recent = recent[-10:]

3. Externalize state to a typed store

Define a minimal schema for agent state. Use a fast key-value or document store. The context should hold only: system prompt, current objective, last 5–10 steps, and pointers to archived state.

import redis

r = redis.Redis()

def save_checkpoint(run_id, step, data: dict):
    r.hset(f"agent:{run_id}:step:{step}", mapping=data)

def load_recent(run_id, n=8):
    keys = sorted(r.keys(f"agent:{run_id}:step:*"))[-n:]
    return [r.hgetall(k) for k in keys]

Store full outputs, errors, and decisions outside the model context. Retrieve on demand via embedding search (section 5). Never put raw HTTP responses or stack traces inline; summarize them to 200 tokens and archive the rest.

4. Implement checkpoint compaction

Every N steps, force a summary. Call the model with the accumulated steps and instruct it to produce a tight state digest. Replace the old messages with the digest.

def compact(run_id, client, n=10):
    recent = load_recent(run_id, n)
    digest_prompt = [
        {"role": "system", "content": "Summarize agent state in 200 words: goals, done, pending, blockers."},
        {"role": "user", "content": str(recent)}
    ]
    sum_resp = client.chat.completions.create(model="gpt-5", messages=digest_prompt)
    digest = sum_resp.choices[0].message.content
    save_checkpoint(run_id, "digest", {"text": digest})
    # reset live context to digest + current objective
    return [
        {"role": "system", "content": "Resumed agent. State: " + digest},
        {"role": "user", "content": "Continue with next actionable step."}
    ]

Run this on a step count or time interval. The digest becomes the new baseline. Without it, gpt-5 context window agents drift as early goals fade into the middle of a huge prompt.

5. Retrieve, don’t replay

For long horizons, embed each checkpoint and tool result. At each step, fetch the top-k most relevant past entries and inject as a constrained context block.

from openai import OpenAI
client = OpenAI()

def embed(text):
    return client.embeddings.create(model="text-embedding-3-small", input=text).data[0].embedding

# store embedding with checkpoint in a vector db (e.g., redis search, pgvector)
def relevant(run_id, query_vec, k=3):
    # pseudo: cosine scan
    return vector_db.query(run_id, query_vec, top_k=k)

This keeps the live context small while preserving access to full history. It also mitigates lost-in-the-middle failures where the model ignores early instructions because they sit at token 2,000 of 180,000.

6. Route across providers with fallback

A single provider outage kills a multi-hour run. Front your agent with an OpenAI-compatible endpoint that supports fallback. For example, n4n.ai exposes one endpoint for 240+ models, applies automatic fallback when a provider is rate-limited or degraded, and forwards cache-control hints so provider-side prompt caching survives model swaps. Your code stays identical:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.n4n.ai/v1",
    api_key="YOUR_KEY"
)
# same chat.completions.create call; routing directives via headers if needed
resp = client.chat.completions.create(
    model="gpt-5",
    messages=messages,
    extra_headers={"x-routing": "prefer:openai,fallback:azure"}
)

Per-token usage metering lets you attribute cost per agent run without building your own middleware. When a primary provider returns 429, the gateway shifts traffic without your loop crashing.

7. Pitfalls: latency, cache misses, and silent truncation

Large contexts are slow. A 150K-token prompt can add seconds of TTFT even on fast GPUs. Use cache-control on static prefixes (system prompt, tool defs) to avoid recomputation:

{
  "messages": [{"role": "system", "content": "..."}],
  "cache_control": {"type": "ephemeral", "prefix": true}
}

If the provider ignores or drops the hint, your bill spikes. Monitor cache hit rates via usage metrics. Lost-in-the-middle is real: models attend unevenly across long contexts. Put immutable goals at both ends of the prompt, not just the top.

Another trap: compaction that silently drops errors. If a step failed, the digest must record the failure signature, or the agent will repeat it. Log raw failures outside the context but surface a one-line error marker inline.

8. Cost and tradeoffs of compaction

Compaction loses nuance. A 200-word digest cannot capture every edge case. Trade frequency against fidelity: compact every 10 steps for chatty agents, every 50 for stable ones. Keep raw logs in the store for post-hoc debugging.

Per-token pricing means a 200K context refilled 100 times costs 20M tokens in input alone. External retrieval at 2K tokens per step cuts that 100x. The tradeoff is retrieval quality—bad embeddings return irrelevant history and the agent stalls. Evaluate recall on a held-out set of past runs before shipping.

9. Launch checklist for gpt-5 context window agents

1. Compute token budget; set max_steps headroom to 30%.
2. Separate messages (live) from Redis/DB (archive).
3. Add compact() every N steps with digest checkpoint.
4. Embed checkpoints; add relevant() retrieval to system prompt.
5. Use a gateway with fallback and cache-control forwarding.
6. Pin goals at top and bottom of context.
7. Alert on cache miss rate > 20% or step latency > 2x baseline.
8. Keep raw traces; never trust digest as source of truth.
9. Load-test with simulated 24h runs using replayed inputs.
10. Document the compaction schema so another engineer can read the store.

Follow this order and your gpt-5 context window agents will run for days without context overflow or goal amnesia. The window is large; your discipline determines whether you use it or drown in it.