Setting max_tokens for long-form content generation requires understanding both the model’s hard limits and the practical constraints of your use case. Most engineers set this parameter once and forget it, then wonder why outputs truncate mid-sentence or why costs spike unexpectedly. This guide walks through calculating the right value, handling outputs that exceed a single call, and verifying the configuration works in production.
Step 1: Determine the model’s actual output limit
Every model publishes a maximum output token count, but the usable limit is often lower. Check the provider’s documentation for the exact number — GPT-4o allows 16,384 output tokens, Claude 3.5 Sonnet allows 8,192, and Gemini 1.5 Pro allows 8,192. These are hard ceilings; the API will return an error if you request more.
# Quick reference for common models (verify against current docs)
MODEL_OUTPUT_LIMITS = {
"gpt-4o": 16384,
"gpt-4o-mini": 16384,
"claude-3-5-sonnet-20241022": 8192,
"claude-3-5-haiku-20241022": 8192,
"gemini-1.5-pro": 8192,
"gemini-1.5-flash": 8192,
"llama-3.1-405b": 4096,
"llama-3.1-70b": 4096,
}Reserve 10-15% headroom for the model’s internal formatting and potential reasoning tokens. If you need 8,000 tokens of clean output, request 9,000-9,200.
Step 2: Calculate your target output size
Estimate tokens for your desired output length. A rough heuristic: 1 token ≈ 0.75 words for English prose, but code and structured data run closer to 1 token per 3-4 characters. For a 2,000-word article, budget ~2,700 tokens. For a 500-line Python file, budget ~4,000-5,000 tokens.
def estimate_output_tokens(word_count: int, content_type: str = "prose") -> int:
"""Estimate output tokens needed for a given word count."""
if content_type == "prose":
return int(word_count / 0.75 * 1.15) # 15% buffer
elif content_type == "code":
return int(word_count * 1.3 * 1.15) # ~1.3 tokens/word for code
elif content_type == "json":
return int(word_count * 1.5 * 1.15) # verbose structure
return int(word_count * 1.2) # default conservativeFactor in the prompt’s token count too. The context window is shared: prompt_tokens + max_tokens ≤ context_window. If your prompt uses 10,000 tokens on a 128k model, you have 118k theoretical space — but the output limit still caps you at the model’s output maximum.
Step 3: Set max_tokens in your request
Pass the calculated value directly in the API call. Use the OpenAI-compatible parameter name max_tokens (most providers accept this; Anthropic uses max_tokens_to_sample in legacy endpoints but max_tokens in Messages API).
import os
from openai import OpenAI
client = OpenAI(
api_key=os.getenv("N4N_API_KEY"),
base_url="https://api.n4n.ai/v1" # single endpoint, 240+ models
)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Write a comprehensive technical guide."},
{"role": "user", "content": "Explain distributed consensus algorithms in depth."}
],
max_tokens=12000, # below 16384 limit with headroom
temperature=0.3,
)
print(response.choices[0].message.content)
print(f"Usage: {response.usage}")If you’re using a gateway that supports routing directives, you can specify fallback models with different output limits:
# Example: prefer high-output model, fall back gracefully
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
max_tokens=12000,
extra_headers={
"X-Route-Preference": "max_output_tokens",
"X-Fallback-Models": "claude-3-5-sonnet-20241022,gemini-1.5-pro"
}
)Step 4: Handle outputs that exceed the limit
When your content needs more tokens than a single call allows, you have three patterns: continuation, chunking, or recursive refinement.
Continuation (simplest for streaming)
Request the first chunk, then feed the partial output back as a continuation prompt.
def generate_long_form(client, model, messages, total_tokens, chunk_size=4000):
"""Generate content in chunks via continuation."""
full_content = []
remaining = total_tokens
while remaining > 0:
current_chunk = min(chunk_size, remaining)
if full_content:
# Continuation prompt
messages = [
{"role": "assistant", "content": "".join(full_content)},
{"role": "user", "content": f"Continue the response. Write approximately {int(current_chunk * 0.75)} more words."}
]
response = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=current_chunk,
temperature=0.3,
stream=True # stream to detect natural stopping points
)
chunk_text = ""
for chunk in response:
if chunk.choices[0].delta.content:
chunk_text += chunk.choices[0].delta.content
full_content.append(chunk_text)
remaining -= current_chunk
# Stop if model naturally concludes
if any(stop in chunk_text.lower() for stop in ["conclusion", "in summary", "finally,"]):
break
return "".join(full_content)Chunking by section (better for structured content)
Break the outline into sections, generate each independently, then stitch.
def generate_by_sections(client, model, outline, tokens_per_section=2000):
"""Generate each outline section separately."""
sections = []
for i, section_title in enumerate(outline):
prompt = f"""Write section {i+1}: "{section_title}".
Target length: ~{int(tokens_per_section * 0.75)} words.
Maintain technical depth and continuity with previous sections."""
messages = [
{"role": "system", "content": "You are writing a technical guide. Be precise."},
{"role": "user", "content": prompt}
]
if sections:
# Provide context from previous sections
context = "\n\n".join(sections[-2:]) # last 2 sections
messages.insert(1, {"role": "assistant", "content": f"Previous sections:\n{context}"})
response = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=tokens_per_section,
temperature=0.3,
)
sections.append(response.choices[0].message.content)
return "\n\n".join(sections)Recursive refinement (highest quality for complex topics)
Generate a skeleton, then expand each node recursively.
def recursive_expand(client, model, node, depth=0, max_depth=3, tokens_per_level=1500):
"""Recursively expand outline nodes into full content."""
if depth >= max_depth:
return node.get("content", "")
# Generate subsections for this node
subsections = node.get("subsections", [])
if not subsections:
return node.get("content", "")
expanded = []
for sub in subsections:
prompt = f"Expand this subsection: {sub['title']}\nGuidance: {sub.get('guidance', '')}"
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=tokens_per_level,
temperature=0.3,
)
sub["content"] = response.choices[0].message.content
expanded.append(recursive_expand(client, model, sub, depth + 1, max_depth, tokens_per_level))
return "\n\n".join(expanded)Step 5: Implement stop sequences for cleaner boundaries
Stop sequences prevent the model from rambling past a natural endpoint, saving tokens and improving structure. Define sequences that match your output format.
# Stop sequences for different content types
STOP_SEQUENCES = {
"markdown_sections": ["## ", "### ", "\n## ", "\n### "],
"json_objects": ["}\n{", "}\n {", "\n}"],
"code_blocks": ["```\n```", "```\n\n```"],
"numbered_lists": ["\n1. ", "\n2. ", "\n3. "],
"conclusion_markers": ["\n## Conclusion", "\n## Summary", "\n## Final"],
}
# Use in request
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
max_tokens=12000,
stop=STOP_SEQUENCES["markdown_sections"], # stops at next header
temperature=0.3,
)Combine stop sequences with max_tokens as a safety net. The model stops at whichever condition triggers first.
Step 6: Monitor actual usage vs. configured limits
Log the usage object from every response. Compare completion_tokens against your max_tokens setting to detect truncation.
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def generate_with_monitoring(client, model, messages, max_tokens):
response = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=max_tokens,
temperature=0.3,
)
usage = response.usage
completion = usage.completion_tokens
prompt = usage.prompt_tokens
total = usage.total_tokens
# Detect truncation
truncation_ratio = completion / max_tokens
if truncation_ratio > 0.95:
logger.warning(
f"Near truncation: {completion}/{max_tokens} tokens used "
f"({truncation_ratio:.1%}). Consider increasing max_tokens or chunking."
)
elif response.choices[0].finish_reason == "length":
logger.error(
f"Hard truncation at {max_tokens} tokens. Output incomplete."
)
logger.info(f"Tokens - prompt: {prompt}, completion: {completion}, total: {total}")
return response.choices[0].message.content, usageTrack these metrics in your observability stack. A dashboard showing completion_tokens / max_tokens distribution across requests reveals whether your sizing is consistently too low, too high, or well-calibrated.
Step 7: Verify success with automated tests
Write integration tests that validate output completeness for your target lengths.
import pytest
@pytest.mark.integration
class TestLongFormGeneration:
"""Verify max_tokens configuration produces complete outputs."""
def test_article_generation_completes(self, client):
"""A 2000-word article should finish without length truncation."""
messages = [
{"role": "user", "content": "Write a 2000-word technical article on Redis internals."}
]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
max_tokens=3500, # ~2600 words with headroom
temperature=0.3,
)
assert response.choices[0].finish_reason != "length", \
"Output truncated by max_tokens limit"
word_count = len(response.choices[0].message.content.split())
assert 1800 <= word_count <= 2500, \
f"Word count {word_count} outside expected range"
def test_code_generation_completes(self, client):
"""A 300-line Python module should generate fully."""
messages = [
{"role": "user", "content": "Write a complete async HTTP client with retry logic, "
"connection pooling, and request/response middleware. ~300 lines."}
]
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
max_tokens=5000,
temperature=0.2,
)
assert response.choices[0].finish_reason != "length"
lines = response.choices[0].message.content.count("\n")
assert lines >= 250, f"Only {lines} lines generated"
# Verify syntactic validity
import ast
try:
ast.parse(response.choices[0].message.content)
except SyntaxError as e:
pytest.fail(f"Generated code has syntax error: {e}")
def test_continuation_preserves_context(self, client):
"""Multi-chunk generation should maintain coherence."""
# First chunk
response1 = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Write a detailed guide to Kubernetes operators. Start with introduction."}],
max_tokens=2000,
)
# Continuation
response2 = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "assistant", "content": response1.choices[0].message.content},
{"role": "user", "content": "Continue with the implementation section."}
],
max_tokens=2000,
)
combined = response1.choices[0].message.content + response2.choices[0].message.content
assert "implementation" in combined.lower()
assert response2.choices[0].finish_reason != "length"Run these in CI against a staging environment. They catch regressions when model behavior shifts or when you switch providers.
Step 8: Adjust for provider-specific quirks
Different providers handle max_tokens differently at the boundary.
OpenAI: Returns finish_reason: "length" when hitting the limit. The partial output is valid.
Anthropic (Messages API): Also returns "stop_reason": "max_tokens". The content array may contain a partial text block.
Google (Gemini): Returns finishReason: "MAX_TOKENS". Output may cut mid-token, producing invalid UTF-8 at the boundary.
Open-weight models (via vLLM, TGI, etc.): Behavior varies by server configuration. Some enforce hard cutoffs; others allow slight overrun.
Normalize handling in your client wrapper:
def normalize_finish_reason(response, provider: str) -> str:
"""Map provider-specific finish reasons to canonical values."""
mapping = {
"openai": {"length": "max_tokens", "stop": "stop_sequence", "content_filter": "safety"},
"anthropic": {"max_tokens": "max_tokens", "stop_sequence": "stop_sequence"},
"google": {"MAX_TOKENS": "max_tokens", "STOP": "stop_sequence", "SAFETY": "safety"},
"vllm": {"length": "max_tokens", "stop": "stop_sequence"},
}
provider_map = mapping.get(provider, {})
raw = getattr(response.choices[0], "finish_reason", None) or \
getattr(response.choices[0], "stop_reason", None) or \
response.get("finishReason", "")
return provider_map.get(raw, raw)
def is_truncated(response, provider: str) -> bool:
return normalize_finish_reason(response, provider) == "max_tokens"Step 9: Optimize cost by right-sizing max_tokens
Setting max_tokens higher than needed wastes money on some providers (you pay for the reservation) and increases latency. Set it to your actual expected maximum plus buffer, not the model’s ceiling.
def calculate_optimal_max_tokens(
target_words: int,
content_type: str = "prose",
buffer_pct: float = 0.15,
hard_cap: int = 16384
) -> int:
"""Calculate right-sized max_tokens for a request."""
base = estimate_output_tokens(target_words, content_type)
with_buffer = int(base * (1 + buffer_pct))
return min(with_buffer, hard_cap)
# Usage
max_tokens = calculate_optimal_max_tokens(
target_words=2500,
content_type="prose",
buffer_pct=0.15,
hard_cap=MODEL_OUTPUT_LIMITS["gpt-4o"]
)
# Returns ~3800 instead of 16384For variable-length outputs, consider dynamic sizing based on prompt analysis:
def dynamic_max_tokens(prompt: str, model_limits: dict) -> int:
"""Estimate output need from prompt keywords."""
prompt_lower = prompt.lower()
# Heuristic multipliers based on prompt intent
if any(kw in prompt_lower for kw in ["comprehensive", "detailed", "in-depth", "complete guide"]):
multiplier = 3.0
elif any(kw in prompt_lower for kw in ["summary", "brief", "overview", "tl;dr"]):
multiplier = 0.5
elif any(kw in prompt_lower for kw in ["code", "implement", "function", "class"]):
multiplier = 2.0
else:
multiplier = 1.5
# Estimate from prompt length (rough proxy)
prompt_words = len(prompt.split())
estimated_output = int(prompt_words * multiplier)
return min(estimated_output, model_limits.get("output", 4096))Step 10: Document the configuration for your team
Create a shared reference that maps use cases to max_tokens values. This prevents every engineer from rediscovering the same limits.
# max_tokens Reference Guide
## By Use Case
| Use Case | Target Words | Content Type | max_tokens | Model |
|----------|-------------|--------------|------------|-------|
| Blog post | 2,000 | prose | 3,500 | gpt-4o |
| Technical guide | 4,000 | prose | 6,500 | gpt-4o |
| Code module | 300 lines | code | 5,000 | gpt-4o |
| API spec (OpenAPI) | 500 lines | yaml | 4,000 | claude-3-5-sonnet |
| Legal summary | 1,500 | prose | 2,500 | gemini-1.5-pro |
## By Model (Output Limits)
| Model | Hard Limit | Recommended Max | Notes |
|-------|-----------|-----------------|-------|
| gpt-4o | 16,384 | 14,000 | Best for very long single-call |
| claude-3-5-sonnet | 8,192 | 7,000 | Strong reasoning, lower ceiling |
| gemini-1.5-pro | 8,192 | 7,000 | Large context, same output cap |
| llama-3.1-405b | 4,096 | 3,500 | Requires chunking for long-form |
## Chunking Thresholds
- **Prose**: Chunk at >6,000 tokens (use continuation)
- **Code**: Chunk at >4,000 tokens (use section-based)
- **Structured**: Chunk at >3,000 tokens (use recursive)Store this in your repo’s docs/llm-params.md and link it from onboarding.
Verification checklist
Before deploying a new long-form generation pipeline:
-
max_tokens≤ model’s documented output limit minus 10% headroom - Prompt tokens +
max_tokens≤ context window - Stop sequences defined for structured outputs
- Logging captures
completion_tokensandfinish_reasonon every request - Alert fires when
completion_tokens / max_tokens > 0.95on >5% of requests - Integration tests validate completion for each target length
- Fallback model configured with compatible output limits
- Cost projection matches expected
max_tokens× volume × price
Set it once, measure continuously, adjust when the data demands it.