Debugging context length exceeded errors

A context length exceeded error is the blunt-force response you get when your assembled prompt plus planned completion overshoots the model’s maximum context window. It shows up in logs as a 400 from the API, often after you’ve stacked a system prompt, chat history, and retrieved documents without accounting for token growth. The following steps take you from reproduction to a hardened request pipeline that stays inside limits.

Step 1: Reproduce the error with explicit token counts

You cannot fix what you cannot measure. Send a request that deliberately violates the limit, capture the exact API response, and separately count tokens with a tokenizer so you trust the numbers.

from openai import OpenAI, APIStatusError

client = OpenAI()  # defaults to OpenAI-compatible /v1/chat/completions

oversized = "repeat this please " * 50000
try:
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": oversized}],
        max_tokens=100,
    )
except APIStatusError as e:
    if e.status_code == 400:
        print("CAUGHT:", e.response.json()["error"]["message"])

The message typically reads This model's maximum context length is 128000 tokens. However, you requested .... That confirms a context length exceeded error rather than a rate limit or auth failure.

Now count tokens locally. For OpenAI models use tiktoken; for other families load the matching tokenizer.

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding("cl100k_base")  # safe approximation
    return len(enc.encode(text))

print(count_tokens(oversized))  # ~250000 for the snippet above

If you are on a Llama or Mistral model, use transformers.AutoTokenizer from the model repo. The count will differ from cl100k by 10–30%; never assume parity.

Step 2: Identify the actual limit for your model

Context windows are per-model, not per-API. A gateway may expose many models behind one endpoint, but each has its own cap. Common values: gpt-4o is 128k, claude-3.5-sonnet is 200k, gemini-1.5-pro is 1M+ (with caveats). The limit is inclusive of both input and output tokens.

Query the model list to avoid hardcoding:

curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  | jq '.data[] | select(.id=="gpt-4o") | {id, context_window: .context_window}'

If the field is absent, fall back to the provider’s documentation. Store the limit in configuration, not inline, because you will swap models during debugging.

Remember: max_tokens in your request is reserved from the same budget. A 128k window with max_tokens=4096 leaves 123k for input. The context length exceeded error fires when input_tokens + max_tokens > window.

Step 3: Break down your prompt components

Most overflow is silent accumulation across four sources. Separate them and print a ledger.

components = {
    "system": system_prompt,
    "history": format_history(messages),
    "retrieval": "\n".join(chunks),
    "current": user_query,
}

ledger = {k: count_tokens(v) for k, v in components.items()}
ledger["reserved_output"] = 1024
total = sum(ledger.values())
print(ledger, "total:", total, "limit:", LIMIT)

System prompt bloat

Engineers paste style guides, few-shot examples, and safety boilerplate into the system prompt. Audit it. Remove duplicated instructions; collapse examples into one canonical pattern.

History growth

Appending every turn without bound is the classic leak. A 20-turn conversation at 500 tokens/turn is 10k tokens—fine until retrieval adds 30k.

Retrieval context

Top-k search with chunk size 512 and k=50 injects 25k tokens. Tune k down or compress chunks before sending.

Output reservation

Always reserve the maximum you will actually use. If you never generate more than 512 tokens, don’t reserve 4096.

Step 4: Apply truncation or summarization

Once you know the overflow source, cut it. Naive truncation drops oldest history first:

def trim_to_budget(messages, budget_tokens: int, counter=count_tokens):
    # messages: list of {"role","content"}
    while sum(counter(m["content"]) for m in messages) > budget_tokens and len(messages) > 1:
        messages.pop(0)  # drop oldest
    return messages

Better: summarize evicted turns with a cheap model and prepend the summary.

def summarize_old(messages, keep_last: int = 4):
    if len(messages) <= keep_last:
        return messages
    old, recent = messages[:-keep_last], messages[-keep_last:]
    summary = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role":"system","content":"Compress the chat to 3 bullets."},
                  {"role":"user","content":format_history(old)}],
        max_tokens=200,
    ).choices[0].message.content
    return [{"role":"system","content":f"Prior summary: {summary}"}] + recent

For retrieval, use a sliding window with overlap only if the task needs local context; otherwise rank and keep only the top 3–5 chunks.

Step 5: Use provider-native features

Prompt caching reduces repeated cost of static prefixes but does not raise the limit. If your gateway forwards cache-control hints, mark the stable system prompt:

{
  "messages": [
    {"role": "system", "content": "You are a terse SQL expert.",
     "cache_control": {"type": "ephemeral"}}
  ]
}

For documents larger than the window, implement map-reduce: embed, split, answer per chunk, then fold. Do not hope the model “reads the whole PDF”—it physically cannot.

Step 6: Handle the error at runtime with fallback and routing

Trimming logic will have edge cases. Catch the context length exceeded error explicitly and apply a降级 path. If you use a gateway that honors client routing directives, you can switch to a larger-context model without rewriting endpoints. For example, n4n.ai exposes one OpenAI-compatible endpoint for 240+ models and respects routing headers, so a header can push a request to a 200k model when your default 128k fills up.

def safe_complete(messages, model="gpt-4o"):
    try:
        return client.chat.completions.create(
            model=model, messages=messages, max_tokens=1024)
    except APIStatusError as e:
        msg = e.response.json()["error"]["message"]
        if "context length" in msg:
            trimmed = trim_to_budget(messages, LIMIT - 2048)
            # route to larger window via model swap
            return client.chat.completions.create(
                model="anthropic/claude-3.5-sonnet",
                messages=trimmed, max_tokens=1024)
        raise

Automatic provider fallback covers rate limits and degradation, not client-side token overflow. You must resize the payload or pick a bigger window yourself.

Step 7: Verify success

Verification is two-layered: a unit check on token math and an integration smoke test.

def test_fits_budget():
    msgs = build_messages()
    used = sum(count_tokens(m["content"]) for m in msgs) + 1024
    assert used < LIMIT, f"over by {used - LIMIT}"

def test_live_no_error():
    resp = client.chat.completions.create(
        model="gpt-4o", messages=build_messages(), max_tokens=16)
    assert resp.choices[0].message.content

Run the unit test in CI on every prompt change. Run the live test nightly against a small model to catch regression without burning spend. If both pass, the context length exceeded error is gone from that code path.

Operational note

Emit the ledger from Step 3 to your logs with the request ID. When a user reports truncation, you can see exactly which component ate the budget. Per-token metering (if your stack provides it) lets you correlate spikes in context usage with cost anomalies before they become outages.

Debugging context limits is not glamorous, but it is the difference between a demo and a system that survives real conversation lengths. Measure, trim by priority, cache what is stable, and keep a fallback that degrades gracefully instead of throwing a 400 at your users.