A truncated response max_tokens limit is the most common silent failure when shipping LLM features. You ask for a long completion, the model hits the cap, and your downstream parser receives a half-written JSON object or cut-off sentence. This guide gives you an end-to-end workflow to reproduce, diagnose, and eliminate that class of bug.
Step 1: Reproduce the truncation deterministically
Set max_tokens artificially low so the model cannot finish. Use any OpenAI-compatible client. Point the OpenAI client at an OpenAI-compatible gateway such as n4n.ai’s endpoint to keep per-token metering consistent while you test.
from openai import OpenAI
client = OpenAI(base_url="https://api.n4n.ai/v1", api_key="YOUR_KEY")
resp = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role": "user", "content": "Write a 500-word essay on HTTP caching."}],
max_tokens=50,
)
print(resp.choices[0].message.content)
print("finish_reason:", resp.choices[0].finish_reason)If you see finish_reason: length and the text stops mid-word, you have a truncated response max_tokens limit exactly as it appears in production. The API returns HTTP 200, so naive error handling will miss it.
Step 2: Read usage and finish_reason
The finish_reason field is your primary signal. Possible values:
stop: model emitted a stop token.length: hitmax_tokenscap.content_filter: provider blocked output.tool_calls: model returned a function call instead of text.
Log the full response metadata:
print(resp.usage)
# Example output:
# {
# "prompt_tokens": 12,
# "completion_tokens": 50,
# "total_tokens": 62
# }A truncated response max_tokens limit always pairs finish_reason == "length" with completion_tokens == max_tokens. The SDK does not raise an exception, which is why this slips into logs as “valid” responses.
Step 3: Distinguish from prompt truncation and hidden token drains
Some providers silently truncate the prompt when it exceeds the context window. That is not a response max_tokens issue. Verify by checking prompt_tokens against the model’s documented context window. If you sent 10k tokens to an 8k model, the gateway may drop earlier messages. Use the usage fields to confirm.
Also watch for hidden reasoning tokens. Models like o1 consume completion budget for internal thinking before emitting visible text. Your max_tokens caps the total, not just the visible characters. A truncated response max_tokens limit on those models may show fewer visible tokens than the cap implies.
When the model returns length, the prompt fit; only the generation was capped.
Step 4: Compute a safe max_tokens value
You need max_tokens = context_window - prompt_tokens - safety_margin. Fetch prompt token count from the previous call’s usage.prompt_tokens, or estimate locally with tiktoken for OpenAI models.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
prompt_text = "Write a 500-word essay on HTTP caching."
prompt_tokens = len(enc.encode(prompt_text))
CONTEXT_WINDOW = 128_000
SAFETY_MARGIN = 500
max_tokens = CONTEXT_WINDOW - prompt_tokens - SAFETY_MARGIN
print(max_tokens) # 127_488Never hardcode max_tokens to the model’s absolute limit. Leave headroom for dynamic few-shot examples, middleware injections, or reasoning tokens. For non-OpenAI models, call the models endpoint to retrieve the real context size instead of guessing.
Step 5: Implement continuation logic
If the task legitimately needs more tokens than fit in one response, chain completions. Detect length and resume. This pattern turns a truncated response max_tokens limit into a transparent loop.
def complete_long(client, model, messages, max_tokens=4096, max_iter=5):
collected = ""
for _ in range(max_iter):
resp = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=max_tokens,
)
piece = resp.choices[0].message.content or ""
collected += piece
if resp.choices[0].finish_reason != "length":
break
messages = messages + [
{"role": "assistant", "content": piece},
{"role": "user", "content": "Continue exactly where you left off, no repetition."},
]
return collectedTrade latency for completeness: each iteration is a new round trip. For user-facing chat, stream the pieces as they arrive. For batch jobs, larger max_tokens with a single call is cheaper if it fits.
Step 6: Stream to fail fast and recover
Streaming does not change the token math, but it lets you abort early and log partial state. Request usage in the stream tail.
stream = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role": "user", "content": "Generate a large SQL schema."}],
max_tokens=200,
stream=True,
stream_options={"include_usage": True},
)
text = ""
finish = None
for event in stream:
if event.choices and event.choices[0].delta.content:
text += event.choices[0].delta.content
if event.choices and event.choices[0].finish_reason:
finish = event.choices[0].finish_reason
print("finish_reason:", finish, "chars:", len(text))If finish is length, you know the cap was hit before the connection closed. You can then persist the partial text and trigger the continuation loop from Step 5.
Step 7: Add gateway-level resilience and caching
If you route through n4n.ai, it honors client routing directives and forwards provider cache-control hints, but the finish_reason contract stays identical. Automatic fallback covers provider degradation, not token limits. Your code must still branch on length.
Set explicit cache-control on long prompts to avoid re-paying prompt tokens on each continuation:
{
"messages": [{"role": "system", "content": "You are a SQL expert."}],
"max_tokens": 4096,
"route": {"prefer": ["openai", "anthropic"]},
"cache_control": {"type": "ephemeral"}
}That JSON is forwarded as-is to providers that support caching, reducing cost when you retry after a truncated response max_tokens limit.
Step 8: Write tests that assert finish_reason
Write a test that asserts the success condition. For JSON extraction, parse the result; for free text, assert finish_reason == "stop".
def test_no_truncation():
resp = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role": "user", "content": "Return a JSON object with keys a,b,c."}],
max_tokens=2000,
)
assert resp.choices[0].finish_reason == "stop"
import json
json.loads(resp.choices[0].message.content) # raises if truncated
def test_detects_truncation():
resp = client.chat.completions.create(
model="openai/gpt-4o-mini",
messages=[{"role": "user", "content": "Write 1000 words about DNS."}],
max_tokens=20,
)
assert resp.choices[0].finish_reason == "length"Run them in CI:
pytest tests/test_truncation.py -k truncationIf both pass, you have closed the loop on the truncated response max_tokens limit.
Step 9: Monitor truncation rates in production
Instrument your gateway logs to emit finish_reason as a metric. A sudden spike in length means a prompt template grew or a model swap reduced the context window. Alert when length exceeds 1% of completions for a given route.
# Pseudo-logging snippet
logger.info("completion", extra={
"model": resp.model,
"finish_reason": resp.choices[0].finish_reason,
"completion_tokens": resp.usage.completion_tokens,
})Per-token metering (available on OpenAI-compatible gateways) lets you attribute the extra cost of continuation loops to the specific endpoint that triggered them.
Operational checklist
- Log
finish_reasonon every completion. - Compute
max_tokensfrom liveprompt_tokens, not static guesses. - Treat
lengthas a retry/continue signal, not an error. - Keep a safety margin for middleware overhead and reasoning tokens.
- Stream in user-facing paths to surface partial output gracefully.
- Cache long prompts to cut cost on continuation retries.
- Test both the happy path (
stop) and the capped path (length) in CI.
Following these steps removes a whole category of silent LLM bugs from your service.