The context length exceeded error openai returns (HTTP 400 with error code context_length_exceeded) is the most common hard failure when you put LLM features in front of real users. It means the total token count of your request—system prompt, conversation history, retrieved context, and the reserved completion space—surpasses the model’s fixed context window, and the API rejects the call before generating anything.
You cannot fix this by retrying the same payload. You have to detect the error precisely, reduce or redistribute tokens, and sometimes change models. Below is an end-to-end procedure that works in production Python services.
Step 1: Detect the error shape accurately
OpenAI’s REST API returns a 400 with a JSON body. The Python SDK (v1.x) raises openai.BadRequestError. The code attribute is the string "context_length_exceeded". Do not string-match on the message—it varies by model and locale.
from openai import OpenAI, BadRequestError
client = OpenAI()
def complete(messages, model="gpt-3.5-turbo"):
try:
return client.chat.completions.create(model=model, messages=messages)
except BadRequestError as e:
if e.code == "context_length_exceeded":
# handle specifically
raise ContextLengthExceeded(e.message) from e
raiseWrap this in your own exception so upstream code doesn’t depend on SDK internals.
Step 2: Count tokens client-side before sending
Round-tripping a failure wastes latency and quota. Count tokens locally with tiktoken. For chat models, count per-message overhead: every message adds 4 tokens, plus 2 for the reply prefix.
import tiktoken
def count_chat_tokens(messages, model="gpt-3.5-turbo"):
enc = tiktoken.encoding_for_model(model)
# per-message overhead is 3, plus 2 for the assistant reply priming
overhead = 3 * len(messages) + 2
text_tokens = sum(len(enc.encode(m["content"])) for m in messages)
return overhead + text_tokens
def max_context_for(model):
# known limits; verify against model list endpoint
limits = {"gpt-3.5-turbo": 16385, "gpt-4-turbo": 128000}
return limits.get(model, 8192)Subtract your planned max_tokens for the completion. If count_chat_tokens(...) + max_tokens > max_context_for(model), you are guaranteed to hit the context length exceeded error openai would reject.
Step 3: Truncate or compress the prompt deterministically
Once you know you are over budget, decide what to drop. For chat logs, drop the oldest user/assistant pairs first; they are usually least relevant to the immediate turn. For RAG pipelines, rank retrieved chunks by score and keep the top N that fit.
def truncate_messages(messages, model, max_tokens):
enc = tiktoken.encoding_for_model(model)
budget = max_context_for(model) - max_tokens - 2
kept = []
# iterate from most recent backward
for m in reversed(messages):
t = len(enc.encode(m["content"])) + 3
if t <= budget:
budget -= t
kept.append(m)
else:
break
return list(reversed(kept))If you need the dropped content, summarize it asynchronously with a smaller model and inject the summary. That is a separate call but avoids the hard error.
Step 4: Switch to a model with a larger context window
When truncation loses required information, change the model. gpt-3.5-turbo caps at 16K; gpt-4-turbo and gpt-4o support 128K. The tradeoff is cost per token and latency, not just capability.
FALLBACK_CHAIN = [
("gpt-3.5-turbo", 16385),
("gpt-4-turbo", 128000),
("gpt-4o", 128000),
]
def pick_model_for(tokens_needed):
for name, ctx in FALLBACK_CHAIN:
if tokens_needed <= ctx:
return name
raise ValueError("no model with sufficient context")If you route through a gateway such as n4n.ai, which exposes one OpenAI-compatible endpoint across 240+ models, you can send a routing directive preferring larger-context models and let the gateway handle provider selection without changing your client code.
Step 5: Implement a retry layer with fallback
Combine Steps 1–4 into a single resilient call. On context_length_exceeded, truncate once; if still over, upgrade model. Cap the loop to avoid infinite escalation.
def resilient_complete(messages, max_tokens=512, model="gpt-3.5-turbo"):
attempts = 0
while attempts < 3:
attempts += 1
if count_chat_tokens(messages, model) + max_tokens > max_context_for(model):
needed = count_chat_tokens(messages, model) + max_tokens
try:
model = pick_model_for(needed)
except ValueError:
messages = truncate_messages(messages, model, max_tokens)
try:
return complete(messages, model)
except ContextLengthExceeded:
messages = truncate_messages(messages, model, max_tokens)
raise RuntimeError("exhausted context fallback attempts")This pattern turns a hard 400 into a degraded but successful response.
Step 6: Verify success with a regression test
Write a test that constructs a payload explicitly over the limit and asserts the wrapper returns a completion rather than raising.
def test_context_fallback():
big = [{"role": "user", "content": "word " * 20000}]
# gpt-3.5-turbo limit is 16385; this must fallback or truncate
resp = resilient_complete(big, max_tokens=100, model="gpt-3.5-turbo")
assert resp.choices[0].message.content is not NoneRun it in CI with a mocked OpenAI client so you don’t burn API keys. The test proves your handling of the context length exceeded error openai surfaces actually triggers.
Operational notes
Log the original token count, the truncated count, and the model finally used. Those three fields tell you whether you are silently losing context in production. If you see constant truncation on a core path, that is a signal to move that workload to a 128K model permanently, not to keep patching.
Provider cache-control hints matter here: if you prefix static system prompts, mark them cacheable so repeated calls over large contexts don’t re-bill the same tokens. Gateways that forward those hints (or honor client routing) reduce the pain without code changes.
The error is not a bug in your integration; it is a hard constraint of the transformer architecture. Engineer around it with measurement, not hope.